-----============= acceptance-small: sanity-flr ============----- Wed Aug 19 01:58:09 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg336-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 01:58:28 (1787119108) === oleg336-client.virtnet: executing check_config_client /mnt/lustre oleg336-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg336-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9f19c7d6f000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f19c7d6f000.idle_timeout=debug disable quota as required oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 01:59:04 (1787119144) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7639] == sanity-flr test 0a: lfs mirror create with -N option == 01:59:06 (1787119146) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:59:24 (1787119164) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:59:27 (1787119167) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:59:31 (1787119171) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:59:55 (1787119195) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:59:58 (1787119198) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 02:00:02 (1787119202) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9f19c7d6f000.stats=clear osc.lustre-OST0001-osc-ffff9f19c7d6f000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 02:00:09 (1787119209) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 02:00:20 (1787119220) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 02:00:27 (1787119227) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 02:00:30 (1787119230) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 02:00:39 (1787119239) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 02:00:48 (1787119248) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 02:00:55 (1787119255) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 02:00:58 (1787119258) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 02:01:06 (1787119266) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 02:01:17 (1787119277) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 02:01:23 (1787119283) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.977781 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134334 s, 7.8 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 02:01:33 (1787119293) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.186637 s, 33.7 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 02:01:41 (1787119301) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 02:01:51 (1787119311) Stopping /mnt/lustre-ost1 (opts:) on oleg336-server oleg336-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f19c7d6f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f19c7d6f000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg336-server: oleg336-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg336-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg336-server oleg336-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f19c7d6f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f19c7d6f000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg336-server: oleg336-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg336-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (72s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 02:03:03 (1787119383) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.296312 s, 21.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg336-server oleg336-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f19c7d6f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f19c7d6f000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg336-server: oleg336-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg336-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (39s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 02:03:42 (1787119422) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg336-server oleg336-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f19c7d6f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f19c7d6f000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg336-server: oleg336-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg336-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg336-server oleg336-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f19c7d6f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f19c7d6f000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg336-server: oleg336-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg336-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (76s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 02:04:58 (1787119498) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.33155 s, 9.8 MB/s -rw-r--r-- 1 root root 52428800 Aug 19 02:05 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.970s user 0m0.000s sys 0m0.182s Stopping /mnt/lustre-ost1 (opts:) on oleg336-server oleg336-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f19c7d6f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f19c7d6f000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.103s user 0m0.000s sys 0m0.190s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg336-server: oleg336-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg336-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg336-server oleg336-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f19c7d6f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f19c7d6f000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.467s user 0m0.000s sys 0m0.182s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg336-server: oleg336-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg336-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg336-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f19c7d6f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f19c7d6f000.ost_server_uuid in FULL state after 0 sec PASS 33b (93s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 02:06:31 (1787119591) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 02:06:35 (1787119595) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 02:06:39 (1787119599) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 02:06:42 (1787119602) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 02:06:49 (1787119609) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.768887 s, 16.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 02:07:01 (1787119621) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302036 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327796 s, 32.0 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 02:07:08 (1787119628) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 02:07:15 (1787119635) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000803705 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00110256 s, 929 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00142935 s, 716 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00146219 s, 700 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 02:07:25 (1787119645) create test files with size 15739273 .. 0f4be89e42e18475ff63679c13f4e756 - 3dd3c5ef3ec04fa6a71aa43c7b54f0bc - 6b63d5fcb51ec611fbe0db1b9c8f12e5 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg336-client.virtnet /mnt/lustre (opts:) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 02:07:48 (1787119668) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 02:08:06 (1787119686) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 02:08:12 (1787119692) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934481 s, 11.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200045 s, 52.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 02:08:19 (1787119699) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.117321 s, 17.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.313641 s, 13.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.165098 s, 12.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (13s) == sanity-flr test 42: lfs mirror verify ================= 02:08:32 (1787119712) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 02:08:35 (1787119715) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 02:08:38 (1787119718) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313349 s, 33.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 02:08:45 (1787119725) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 02:08:48 (1787119728) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 02:08:56 (1787119736) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.228571 s, 45.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg336-client.virtnet /mnt/lustre (opts:) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 02:09:05 (1787119745) llite.lustre-ffff9f19d1879000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.317949 s, 33.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg336-client.virtnet /mnt/lustre (opts:) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9f19d8e7b000.xattr_cache=1 PASS 44e (12s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 02:09:18 (1787119758) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 02:09:56 (1787119796) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (31s) == sanity-flr test 47: Verify mirror obj alloc =========== 02:10:27 (1787119827) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 02:10:30 (1787119830) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.205589 s, 15.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.139509 s, 22.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x161ec037 (nosync,stale) Mirror 2: 0x161ec037 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 64703aedc8a2097e29a26505d40023ea - mirror 1 checksum: 8a3ce1fa7d3acebb210dae9a126c1cdb - mirror 2 checksum: 64703aedc8a2097e29a26505d40023ea - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-19T02:10:33-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-19T02:10:33-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 64703aedc8a2097e29a26505d40023ea - mirror 1 checksum: 8a3ce1fa7d3acebb210dae9a126c1cdb - mirror 2 checksum: 8a3ce1fa7d3acebb210dae9a126c1cdb - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-19T02:10:33-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-19T02:10:33-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } PASS 48 (9s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 02:10:41 (1787119841) Failing mds1 on oleg336-server Stopping /mnt/lustre-mds1 (opts:) on oleg336-server 02:10:47 (1787119847) shut down facet: mds1 facet_host: oleg336-server facet_failover_host: oleg336-server Failover mds1 to oleg336-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-MDT0000 02:11:09 (1787119869) targets are mounted 02:11:09 (1787119869) facet_failover done oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481133 s, 21.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } PASS 50A (44s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 02:11:26 (1787119886) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 02:11:29 (1787119889) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 02:11:32 (1787119892) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00634009 s, 2.6 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 02:11:39 (1787119899) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 02:11:43 (1787119903) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0142644 s, 287 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0833762 s, 49.1 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 02:11:50 (1787119910) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 02:12:11 (1787119931) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 02:12:40 (1787119960) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 02:13:03 (1787119983) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 02:13:27 (1787120007) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 02:13:35 (1787120015) mirror create pid 54332 mirror split pid 54334 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54332 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54334 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 02:14:44 (1787120084) Seed set to 86 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x26c69 000001 1787120086.543906 trunc from 00000000 to 0x026c68 (0x26c69 bytes) 000002 1787120086.573668 read 0x016f70 thru 0x01fca0 (0x8d31 bytes) 000003 1787120086.590502 MIRROR_EXTEND to 2 mirrors 000007 1787120087.579244 mapwrite 0x00cd36 thru 0x0152cc (0x8597 bytes) 000008 1787120087.769377 mapread 0x004ca5 thru 0x012ca8 (0xe004 bytes) 000009 1787120087.780425 read 0x010cb1 thru 0x018c30 (0x7f80 bytes) 000010 1787120087.786481 read 0x011bbc thru 0x01d441 (0xb886 bytes) 000011 1787120087.800293 trunc from 0x026c69 to 0x021161 (0x5b07 bytes) 000012 1787120087.826213 read 0x001e0e thru 0x010c71 (0xee64 bytes) 000013 1787120087.828730 read 0x00b563 thru 0x016c42 (0xb6e0 bytes) 000014 1787120087.830778 read 0x005ed9 thru 0x00bd3f (0x5e67 bytes) 000018 1787120087.833385 mapwrite 0x02eae4 thru 0x032bc0 (0x40dd bytes) 000019 1787120087.961101 write 0x0365f8 thru 0x03c105 (0x5b0e bytes) 000020 1787120087.962450 trunc from 0x03c106 to 0x004cd8 (0x3742d bytes) 000021 1787120087.989520 mapread 0x0035ce thru 0x004cd8 (0x170b bytes) 000022 1787120087.992259 trunc from 0x004cd9 to 0x016aea (0x11e12 bytes) 000025 1787120088.014657 trunc from 0x016aeb to 0x0074f8 (0xf5f2 bytes) truncating to largest ever: 0x3e9f6 000026 1787120088.035502 trunc from 0x0074f9 to 0x03e9f5 (0x374fd bytes) 000029 1787120088.061551 trunc from 0x03e9f6 to 0x018a01 (0x25ff4 bytes) 000032 1787120088.089333 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) 000034 1787120088.217893 mapwrite 0x001729 thru 0x00903f (0x7917 bytes) 000036 1787120088.276085 trunc from 0x018a02 to 0x0036bd (0x15344 bytes) 000040 1787120088.314528 read 0x000f4a thru 0x0036bd (0x2774 bytes) 000042 1787120088.326710 mapwrite 0x01d7e9 thru 0x02d73f (0xff57 bytes) 000043 1787120088.460217 mapread 0x015aad thru 0x024d39 (0xf28d bytes) 000044 1787120088.475470 trunc from 0x02d740 to 0x035b4f (0x8410 bytes) 000045 1787120088.491136 mapread 0x00a2e6 thru 0x01839e (0xe0b9 bytes) 000049 1787120088.506188 trunc from 0x035b50 to 0x018894 (0x1d2bb bytes) 000051 1787120088.521242 write 0x006b2d thru 0x010efb (0xa3cf bytes) 000054 1787120088.525606 write 0x02696b thru 0x0329da (0xc070 bytes) 000055 1787120088.527782 write 0x03b4d6 thru 0x03ffff (0x4b2a bytes) 000056 1787120088.529392 mapwrite 0x016109 thru 0x01f19e (0x9096 bytes) 000057 1787120088.593189 mapread 0x020ec7 thru 0x027dda (0x6f14 bytes) 000059 1787120088.599780 mapwrite 0x00d70b thru 0x01075c (0x3052 bytes) 000060 1787120088.649331 write 0x013b22 thru 0x014752 (0xc31 bytes) 000063 1787120088.656932 trunc from 0x040000 to 0x011b44 (0x2e4bb bytes) 000064 1787120088.682865 read 0x00c3df thru 0x011b44 (0x5766 bytes) 000065 1787120088.683649 trunc from 0x011b45 to 0x02ed34 (0x1d1f0 bytes) 000066 1787120088.698826 mapwrite 0x02f49a thru 0x03a1c6 (0xad2d bytes) 000068 1787120088.788875 trunc from 0x03a1c7 to 0x029228 (0x10f9e bytes) 000069 1787120088.805405 read 0x002c8e thru 0x00cbda (0x9f4d bytes) 000070 1787120088.806888 write 0x018301 thru 0x026325 (0xe025 bytes) 000073 1787120088.811872 mapwrite 0x00f631 thru 0x01aab9 (0xb489 bytes) 000075 1787120088.861527 read 0x01a849 thru 0x01eb99 (0x4351 bytes) 000079 1787120088.866282 mapread 0x025e96 thru 0x029228 (0x3393 bytes) 000080 1787120088.868373 mapread 0x021734 thru 0x022df0 (0x16bd bytes) 000082 1787120088.871118 mapwrite 0x003999 thru 0x00dba0 (0xa208 bytes) 000087 1787120088.913413 trunc from 0x029229 to 0x009e88 (0x1f3a0 bytes) 000089 1787120088.931036 mapread 0x005b06 thru 0x008452 (0x294d bytes) 000090 1787120088.932469 trunc from 0x009e89 to 0x02c2e8 (0x22460 bytes) 000091 1787120088.949849 trunc from 0x02c2e9 to 0x02ae1b (0x14cd bytes) 000092 1787120088.964377 mapwrite 0x016dac thru 0x02400c (0xd261 bytes) 000094 1787120089.021379 mapread 0x0279e0 thru 0x02ae1b (0x343c bytes) 000096 1787120089.024100 mapread 0x006d8a thru 0x0108bb (0x9b32 bytes) 000097 1787120089.026039 write 0x03b07f thru 0x03f5b0 (0x4532 bytes) 000098 1787120089.028298 trunc from 0x03f5b1 to 0x017911 (0x27c9f bytes) 000100 1787120089.052792 read 0x0153cc thru 0x017911 (0x2546 bytes) 000101 1787120089.054290 mapread 0x00ba77 thru 0x00da65 (0x1fef bytes) 000102 1787120089.056404 mapwrite 0x0154c9 thru 0x020cbf (0xb7f7 bytes) 000104 1787120089.142440 mapread 0x00d511 thru 0x016bbb (0x96ab bytes) 000108 1787120089.146819 write 0x00822a thru 0x00d090 (0x4e67 bytes) 000109 1787120089.148364 trunc from 0x020cc0 to 0x02f9e6 (0xed27 bytes) 000112 1787120089.164494 mapread 0x00e7e3 thru 0x01c636 (0xde54 bytes) 000114 1787120089.168877 write 0x0385f9 thru 0x03ffff (0x7a07 bytes) 000116 1787120089.171670 mapread 0x032b8c thru 0x03f6fe (0xcb73 bytes) 000121 1787120089.186838 write 0x0069e7 thru 0x00c01c (0x5636 bytes) 000122 1787120089.189041 write 0x0329fe thru 0x03dd00 (0xb303 bytes) 000123 1787120089.192346 trunc from 0x040000 to 0x01d844 (0x227bb bytes) 000124 1787120089.208635 write 0x01cedc thru 0x029034 (0xc159 bytes) 000125 1787120089.212236 mapread 0x012ad7 thru 0x014a27 (0x1f51 bytes) 000126 1787120089.214506 mapread 0x028050 thru 0x029034 (0xfe5 bytes) 000127 1787120089.216263 trunc from 0x029035 to 0x03995f (0x1092b bytes) 000130 1787120089.233046 write 0x01a7f5 thru 0x01aad7 (0x2e3 bytes) 000133 1787120089.242367 mapread 0x020398 thru 0x0212de (0xf47 bytes) 000134 1787120089.245241 read 0x014f21 thru 0x0190b1 (0x4191 bytes) 000135 1787120089.248650 read 0x01eb8b thru 0x020380 (0x17f6 bytes) 000138 1787120089.271068 mapwrite 0x001217 thru 0x0039d6 (0x27c0 bytes) 000139 1787120089.366297 read 0x0095d2 thru 0x00aafd (0x152c bytes) 000140 1787120089.367003 write 0x02f27c thru 0x03dc74 (0xe9f9 bytes) 000141 1787120089.384348 mapwrite 0x002335 thru 0x00897c (0x6648 bytes) 000142 1787120089.450410 mapread 0x0130cc thru 0x01807e (0x4fb3 bytes) 000144 1787120089.456081 read 0x01e917 thru 0x026c4d (0x8337 bytes) 000148 1787120089.459577 read 0x0058d4 thru 0x00b44f (0x5b7c bytes) 000149 1787120089.460936 read 0x007370 thru 0x00e295 (0x6f26 bytes) 000151 1787120089.463501 trunc from 0x03dc75 to 0x0009ee (0x3d286 bytes) 000152 1787120089.479861 write 0x01af4e thru 0x026c24 (0xbcd7 bytes) 000153 1787120089.487484 trunc from 0x026c25 to 0x00d7ec (0x19438 bytes) 000154 1787120089.519559 read 0x004de2 thru 0x00d7ec (0x8a0b bytes) 000155 1787120089.532599 write 0x01982c thru 0x01dac4 (0x4299 bytes) 000156 1787120089.540237 MIRROR_EXTEND to 2 mirrors 000159 1787120090.553796 read 0x01384e thru 0x01dac4 (0xa277 bytes) 000160 1787120090.581501 read 0x00afde thru 0x015f14 (0xaf37 bytes) 000162 1787120090.594036 mapwrite 0x035e3e thru 0x03baa0 (0x5c63 bytes) 000164 1787120090.821787 mapwrite 0x02818e thru 0x02f6f7 (0x756a bytes) 000165 1787120090.891942 mapread 0x0314c3 thru 0x033840 (0x237e bytes) 000166 1787120090.894208 mapread 0x0118ed thru 0x01a8b6 (0x8fca bytes) 000167 1787120090.952213 MIRROR_RESYNC 2 mirrors 000169 1787120091.630535 mapwrite 0x025e19 thru 0x02da2a (0x7c12 bytes) 000174 1787120091.916406 read 0x02fbaf thru 0x038742 (0x8b94 bytes) 000176 1787120091.925923 trunc from 0x03baa1 to 0x01770d (0x24393 bytes) 000181 1787120091.954860 trunc from 0x01770e to 0x001cfa (0x15a13 bytes) 000182 1787120091.970369 read 0x00182a thru 0x001cfa (0x4d1 bytes) 000183 1787120091.984326 write 0x0103eb thru 0x0117f6 (0x140c bytes) 000185 1787120091.986902 trunc from 0x0117f7 to 0x00bf79 (0x587d bytes) 000187 1787120092.001256 write 0x01ed4f thru 0x023153 (0x4405 bytes) 000188 1787120092.006452 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) 000189 1787120092.168962 MIRROR_EXTEND to 2 mirrors 000191 1787120093.131223 mapread 0x005f37 thru 0x012024 (0xc0ee bytes) 000192 1787120093.170048 trunc from 0x023154 to 0x0081b8 (0x1af9b bytes) 000195 1787120093.295166 read 0x007890 thru 0x0081b8 (0x929 bytes) 000197 1787120093.309821 trunc from 0x0081b9 to 0x03d35a (0x351a2 bytes) 000199 1787120093.356447 write 0x0330c3 thru 0x0380ee (0x502c bytes) 000201 1787120093.404360 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d35b): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d35b): Mirror 2: 0x1e977a91 (stale) CRC-32 checksum value for chunk [0, 0x3d35b): Mirror 1: 0xffdf92a4 Mirror 3: 0xffdf92a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d35b: skipped 000202 1787120094.652616 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3d35b): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d35b): Mirror 2: 0x1e977a91 (stale) CRC-32 checksum value for chunk [0, 0x3d35b): Mirror 1: 0xffdf92a4 Mirror 3: 0xffdf92a4 Mirror 4: 0xffdf92a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d35b: skipped 000204 1787120095.868500 mapread 0x032f7c thru 0x0364f1 (0x3576 bytes) 000205 1787120095.914307 MIRROR_RESYNC 4 mirrors 000207 1787120097.183342 mapwrite 0x02b865 thru 0x03659a (0xad36 bytes) 000208 1787120097.456620 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d35b): Mirror 3: 0xffdf92a4 (stale) Mirror 4: 0xffdf92a4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d35b): Mirror 3: 0xffdf92a4 (stale) Mirror 4: 0xffdf92a4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d35b: skipped 000212 1787120097.983355 write 0x00e26d thru 0x01d37c (0xf110 bytes) 000213 1787120098.033983 read 0x016871 thru 0x01dcce (0x745e bytes) 000214 1787120098.034833 write 0x03a9a3 thru 0x03bb67 (0x11c5 bytes) 000215 1787120098.055069 write 0x007b9b thru 0x010dd5 (0x923b bytes) 000216 1787120098.076081 read 0x00c5d0 thru 0x01138d (0x4dbe bytes) 000219 1787120098.098156 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3d35b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d35b): Mirror 3: 0xffdf92a4 (stale) Mirror 4: 0xffdf92a4 (stale) CRC-32 checksum value for chunk [0, 0x3d35b): Mirror 1: 0x2ab6382d Mirror 5: 0x2ab6382d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d35b: skipped 000220 1787120099.414063 trunc from 0x03d35b to 0x006cbe (0x3669c bytes) 000221 1787120099.520295 read 0x003b83 thru 0x006cbe (0x313c bytes) 000222 1787120099.527360 read 0x006774 thru 0x006cbe (0x54b bytes) 000223 1787120099.528044 mapread 0x004cac thru 0x006cbe (0x2013 bytes) 000227 1787120099.533453 trunc from 0x006cbf to 0x02a4f2 (0x23834 bytes) 000228 1787120099.554420 mapread 0x00285b thru 0x0080ff (0x58a5 bytes) 000229 1787120099.567074 mapwrite 0x0293cb thru 0x02c77a (0x33b0 bytes) 000231 1787120099.642239 mapread 0x00ca3a thru 0x011e79 (0x5440 bytes) 000232 1787120099.643415 read 0x004658 thru 0x006fa2 (0x294b bytes) 000233 1787120099.651547 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c77b): Mirror 4: 0x49af785 (stale) Mirror 5: 0x1fdc00f3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c77b): Mirror 4: 0x49af785 (stale) Mirror 5: 0x1fdc00f3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c77b: skipped 000235 1787120100.088453 trunc from 0x02c77b to 0x03782e (0xb0b4 bytes) 000237 1787120100.105178 write 0x0362bd thru 0x036a14 (0x758 bytes) 000245 1787120100.115952 trunc from 0x03782f to 0x01a7d6 (0x1d058 bytes) 000246 1787120100.128106 mapread 0x006165 thru 0x00970e (0x35aa bytes) 000248 1787120100.160321 trunc from 0x01a7d7 to 0x03c13b (0x21965 bytes) 000249 1787120100.195125 read 0x023d79 thru 0x02d77f (0x9a07 bytes) 000252 1787120100.219354 read 0x002b31 thru 0x00b9ec (0x8ebc bytes) 000254 1787120100.229309 trunc from 0x03c13c to 0x02cdca (0xf371 bytes) 000255 1787120100.271368 write 0x01a938 thru 0x02934e (0xea17 bytes) 000256 1787120100.283200 trunc from 0x02cdcb to 0x00dd57 (0x1f073 bytes) 000260 1787120100.313088 mapwrite 0x032b74 thru 0x033dc3 (0x1250 bytes) 000263 1787120100.486009 mapwrite 0x00039b thru 0x0051ca (0x4e30 bytes) 000264 1787120100.667132 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33dc4): Mirror 4: 0xc70349ca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33dc4): Mirror 4: 0xc70349ca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33dc4: skipped 000265 1787120101.109974 read 0x023947 thru 0x02bb5f (0x8219 bytes) 000268 1787120101.140672 read 0x00786c thru 0x00e410 (0x6ba5 bytes) 000269 1787120101.165353 MIRROR_RESYNC 2 mirrors 000270 1787120102.389808 read 0x0213f0 thru 0x02c093 (0xaca4 bytes) 000272 1787120102.407980 mapread 0x02fb0e thru 0x033dc3 (0x42b6 bytes) 000273 1787120102.423614 mapwrite 0x01c0bd thru 0x0220bb (0x5fff bytes) 000278 1787120102.567114 read 0x00c044 thru 0x01ac8c (0xec49 bytes) 000281 1787120102.577374 mapwrite 0x000367 thru 0x00bd86 (0xba20 bytes) 000285 1787120102.626005 mapread 0x01242f thru 0x01a1e3 (0x7db5 bytes) 000291 1787120102.630707 mapwrite 0x01e6d1 thru 0x02af53 (0xc883 bytes) 000292 1787120102.677763 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x33dc4): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33dc4): Mirror 4: 0x48147ae5 (stale) CRC-32 checksum value for chunk [0, 0x33dc4): Mirror 1: 0x5f8c66b5 Mirror 5: 0x5f8c66b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33dc4: skipped 000293 1787120103.703001 write 0x0203c8 thru 0x02732d (0x6f66 bytes) 000294 1787120103.842137 trunc from 0x033dc4 to 0x030ae6 (0x32dd bytes) 000296 1787120103.862249 read 0x0032bf thru 0x003a32 (0x774 bytes) 000298 1787120103.870664 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x30ae7): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30ae7): Mirror 4: 0xfeaf0d38 (stale) Mirror 5: 0xd9a3eaa0 (stale) CRC-32 checksum value for chunk [0, 0x30ae7): Mirror 1: 0x318f1dcb Mirror 6: 0x318f1dcb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30ae7: skipped 000299 1787120104.859725 mapwrite 0x032a46 thru 0x03ffff (0xd5ba bytes) 000300 1787120105.092758 write 0x031d9f thru 0x0387a5 (0x6a07 bytes) 000302 1787120105.097047 read 0x023b86 thru 0x02e9d5 (0xae50 bytes) 000305 1787120105.112784 mapwrite 0x01b484 thru 0x02abeb (0xf768 bytes) 000307 1787120105.190490 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xbea34e81 (stale) Mirror 5: 0x9819d322 (stale) Mirror 6: 0x8682fb1b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5075d45f Mirror 7: 0x5075d45f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000309 1787120106.181258 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xbea34e81 (stale) Mirror 5: 0x9819d322 (stale) Mirror 6: 0x8682fb1b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5075d45f Mirror 7: 0x5075d45f Mirror 8: 0x5075d45f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000312 1787120107.155750 write 0x033434 thru 0x0343d8 (0xfa5 bytes) 000313 1787120107.362922 trunc from 0x040000 to 0x00fc4d (0x303b2 bytes) 000314 1787120107.417062 trunc from 0x00fc4e to 0x0204aa (0x1085d bytes) 000317 1787120107.439496 mapwrite 0x00d33a thru 0x015f68 (0x8c2f bytes) 000319 1787120107.537769 read 0x009da5 thru 0x00d493 (0x36ef bytes) 000320 1787120107.539571 write 0x033f81 thru 0x038d00 (0x4d80 bytes) 000322 1787120107.545468 read 0x031eab thru 0x038d00 (0x6e56 bytes) 000323 1787120107.566796 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38d01): Mirror 4: 0xa0482adc (stale) Mirror 5: 0x6054e7e0 (stale) Mirror 6: 0xc2ebb670 (stale) Mirror 8: 0x29297c33 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38d01): Mirror 4: 0xa0482adc (stale) Mirror 5: 0x6054e7e0 (stale) Mirror 6: 0xc2ebb670 (stale) Mirror 8: 0x29297c33 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38d01: skipped 000324 1787120108.106660 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38d01): Mirror 4: 0xa0482adc (stale) Mirror 6: 0xc2ebb670 (stale) Mirror 8: 0x29297c33 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38d01): Mirror 4: 0xa0482adc (stale) Mirror 6: 0xc2ebb670 (stale) Mirror 8: 0x29297c33 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38d01: skipped 000325 1787120108.433389 mapwrite 0x031dea thru 0x039a15 (0x7c2c bytes) 000327 1787120108.514944 mapwrite 0x005cc7 thru 0x00669f (0x9d9 bytes) 000328 1787120108.580119 trunc from 0x039a16 to 0x004ddc (0x34c39 bytes) 000329 1787120108.605909 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x4ddd): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4ddd): Mirror 4: 0xad05e76 (stale) Mirror 6: 0x438c33e (stale) Mirror 8: 0x438c33e (stale) CRC-32 checksum value for chunk [0, 0x4ddd): Mirror 1: 0x438c33e Mirror 9: 0x438c33e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4ddd: skipped 000331 1787120109.478225 trunc from 0x004ddd to 0x0362bd (0x314e1 bytes) 000332 1787120109.554131 read 0x015d76 thru 0x023439 (0xd6c4 bytes) 000334 1787120109.563738 trunc from 0x0362be to 0x02fedd (0x63e0 bytes) 000335 1787120109.580984 trunc from 0x02fede to 0x017d32 (0x181ab bytes) 000337 1787120109.593338 mapread 0x00431d thru 0x009a2d (0x5711 bytes) 000339 1787120109.616967 mapread 0x004968 thru 0x0082ea (0x3983 bytes) 000341 1787120109.629417 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17d33): Mirror 4: 0xbbe02dff (stale) Mirror 8: 0x5c1346c9 (stale) Mirror 9: 0x2e92dd9f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17d33): Mirror 4: 0xbbe02dff (stale) Mirror 8: 0x5c1346c9 (stale) Mirror 9: 0x2e92dd9f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17d33: skipped 000343 1787120110.066190 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x17d33): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17d33): Mirror 4: 0xbbe02dff (stale) Mirror 8: 0x5c1346c9 (stale) Mirror 9: 0x2e92dd9f (stale) CRC-32 checksum value for chunk [0, 0x17d33): Mirror 1: 0x2e92dd9f Mirror 10: 0x2e92dd9f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17d33: skipped 000345 1787120111.243649 mapwrite 0x007a07 thru 0x01256c (0xab66 bytes) 000347 1787120111.498506 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x17d33): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17d33): Mirror 4: 0xbbe02dff (stale) Mirror 8: 0x5c1346c9 (stale) Mirror 9: 0x2e92dd9f (stale) Mirror 10: 0x2e92dd9f (stale) CRC-32 checksum value for chunk [0, 0x17d33): Mirror 1: 0xe2bcc0b3 Mirror 11: 0xe2bcc0b3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17d33: skipped 000349 1787120112.761958 mapwrite 0x037962 thru 0x03ffff (0x869e bytes) 000350 1787120112.958871 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xbea34e81 (stale) Mirror 8: 0x5075d45f (stale) Mirror 9: 0x63ecab66 (stale) Mirror 10: 0x63ecab66 (stale) Mirror 11: 0xe5ae1d7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xeb4447b5 Mirror 12: 0xeb4447b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000353 1787120114.594113 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xbea34e81 (stale) Mirror 8: 0x5075d45f (stale) Mirror 9: 0x63ecab66 (stale) Mirror 10: 0x63ecab66 (stale) Mirror 11: 0xe5ae1d7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xeb4447b5 Mirror 12: 0xeb4447b5 Mirror 13: 0xeb4447b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000354 1787120115.642932 trunc from 0x040000 to 0x01d16a (0x22e95 bytes) 000359 1787120115.732656 mapread 0x011f6e thru 0x016e36 (0x4ec9 bytes) 000360 1787120115.763327 write 0x009975 thru 0x017473 (0xdaff bytes) 000364 1787120115.775369 read 0x014244 thru 0x01d16a (0x8f27 bytes) 000365 1787120115.782624 trunc from 0x01d16b to 0x02287d (0x5713 bytes) 000368 1787120115.807514 read 0x00a606 thru 0x0169f5 (0xc3f0 bytes) 000370 1787120115.809381 read 0x00ce2f thru 0x0106f4 (0x38c6 bytes) 000371 1787120115.810711 trunc from 0x02287e to 0x00140c (0x21471 bytes) 000372 1787120115.823634 read 0x0001bb thru 0x00115b (0xfa1 bytes) 000373 1787120115.829306 read 0x000d06 thru 0x00140c (0x707 bytes) 000374 1787120115.829983 mapread 0x000aee thru 0x00140c (0x91f bytes) 000375 1787120115.830989 mapwrite 0x03b17b thru 0x03ffff (0x4e85 bytes) 000376 1787120115.922018 mapwrite 0x021d9d thru 0x02ac25 (0x8e89 bytes) 000377 1787120116.020407 trunc from 0x040000 to 0x006638 (0x399c7 bytes) 000379 1787120116.051444 mapread 0x002f16 thru 0x006638 (0x3723 bytes) 000380 1787120116.055353 write 0x028d22 thru 0x02946b (0x74a bytes) 000382 1787120116.062988 mapread 0x02599b thru 0x02946b (0x3ad1 bytes) 000385 1787120116.101532 MIRROR_RESYNC 8 mirrors 000387 1787120117.987876 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) 000390 1787120118.369235 mapwrite 0x02afcd thru 0x039502 (0xe536 bytes) 000392 1787120118.704190 write 0x004219 thru 0x00bc0a (0x79f2 bytes) 000393 1787120118.724385 mapwrite 0x01296b thru 0x0217a3 (0xee39 bytes) 000395 1787120118.840956 write 0x0235d3 thru 0x023672 (0xa0 bytes) 000396 1787120118.842605 trunc from 0x039503 to 0x006e4c (0x326b6 bytes) 000398 1787120118.885259 write 0x029ae3 thru 0x03206f (0x858d bytes) 000399 1787120118.889902 mapread 0x02e777 thru 0x02fb72 (0x13fc bytes) 000401 1787120118.892563 write 0x0137b2 thru 0x0162c5 (0x2b14 bytes) 000403 1787120118.907537 write 0x01299c thru 0x019ad1 (0x7136 bytes) 000404 1787120118.923020 mapwrite 0x0172d7 thru 0x0198bc (0x25e6 bytes) 000405 1787120119.006155 trunc from 0x032070 to 0x029447 (0x8c28 bytes) 000407 1787120119.033648 mapwrite 0x001b4f thru 0x00dd27 (0xc1d9 bytes) 000409 1787120119.186598 mapread 0x022595 thru 0x0228f3 (0x35f bytes) 000411 1787120119.212432 read 0x01a451 thru 0x026c1f (0xc7cf bytes) 000413 1787120119.244747 write 0x03f7ea thru 0x03ffff (0x816 bytes) 000414 1787120119.261219 MIRROR_RESYNC 7 mirrors 000418 1787120121.046642 MIRROR_EXTEND to 8 mirrors 000419 1787120122.044612 mapwrite 0x03f552 thru 0x03ffff (0xaae bytes) 000422 1787120122.250252 MIRROR_RESYNC 8 mirrors 000423 1787120124.044658 write 0x007d8e thru 0x014905 (0xcb78 bytes) 000424 1787120124.181193 write 0x012c43 thru 0x012dc0 (0x17e bytes) 000425 1787120124.183575 trunc from 0x040000 to 0x00b790 (0x3486f bytes) 000427 1787120124.215599 read 0x0078e5 thru 0x00872e (0xe4a bytes) 000428 1787120124.217567 mapwrite 0x0023c5 thru 0x00359d (0x11d9 bytes) 000429 1787120124.295857 read 0x009742 thru 0x00b790 (0x204f bytes) 000430 1787120124.296512 write 0x02fd4f thru 0x03fb28 (0xfdda bytes) 000431 1787120124.298194 trunc from 0x03fb29 to 0x035fac (0x9b7c bytes) 000432 1787120124.334648 write 0x02c8e2 thru 0x036a57 (0xa176 bytes) 000433 1787120124.354124 mapwrite 0x0311a1 thru 0x0320c6 (0xf26 bytes) 000435 1787120124.425627 MIRROR_RESYNC 8 mirrors 000437 1787120126.679182 mapwrite 0x01ca82 thru 0x01dde6 (0x1365 bytes) 000439 1787120126.907639 read 0x0302c3 thru 0x036a57 (0x6795 bytes) 000440 1787120126.924053 read 0x0052a2 thru 0x013528 (0xe287 bytes) 000442 1787120126.944539 mapread 0x0118aa thru 0x0186bc (0x6e13 bytes) 000444 1787120126.954465 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36a58): Mirror 4: 0x1c0b4e32 (stale) Mirror 8: 0x1c0b4e32 (stale) Mirror 9: 0x1c0b4e32 (stale) Mirror 11: 0x1c0b4e32 (stale) Mirror 12: 0x1c0b4e32 (stale) Mirror 13: 0x1c0b4e32 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36a58): Mirror 4: 0x1c0b4e32 (stale) Mirror 8: 0x1c0b4e32 (stale) Mirror 9: 0x1c0b4e32 (stale) Mirror 11: 0x1c0b4e32 (stale) Mirror 12: 0x1c0b4e32 (stale) Mirror 13: 0x1c0b4e32 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36a58: skipped 000445 1787120127.599182 read 0x036679 thru 0x036a57 (0x3df bytes) 000446 1787120127.618211 mapwrite 0x0285e6 thru 0x02cca6 (0x46c1 bytes) 000449 1787120127.751660 read 0x012499 thru 0x0194a3 (0x700b bytes) 000450 1787120127.765030 read 0x00e0ce thru 0x010629 (0x255c bytes) 000452 1787120127.766879 trunc from 0x036a58 to 0x037c21 (0x11ca bytes) 000453 1787120127.788255 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x37c22): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37c22): Mirror 4: 0x92afc1f2 (stale) Mirror 8: 0x92afc1f2 (stale) Mirror 9: 0x92afc1f2 (stale) Mirror 11: 0x92afc1f2 (stale) Mirror 12: 0x92afc1f2 (stale) Mirror 13: 0x92afc1f2 (stale) CRC-32 checksum value for chunk [0, 0x37c22): Mirror 1: 0x30f97dfe Mirror 14: 0x30f97dfe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37c22: skipped 000455 1787120128.895204 read 0x000171 thru 0x00041f (0x2af bytes) 000456 1787120128.905857 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37c22): Mirror 4: 0x92afc1f2 (stale) Mirror 8: 0x92afc1f2 (stale) Mirror 9: 0x92afc1f2 (stale) Mirror 11: 0x92afc1f2 (stale) Mirror 13: 0x92afc1f2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37c22): Mirror 4: 0x92afc1f2 (stale) Mirror 8: 0x92afc1f2 (stale) Mirror 9: 0x92afc1f2 (stale) Mirror 11: 0x92afc1f2 (stale) Mirror 13: 0x92afc1f2 (stale) CRC-32 checksum value for chunk [0, 0x37c22): Mirror 1: 0x30f97dfe Mirror 14: 0x30f97dfe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37c22: skipped 000457 1787120130.187587 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37c22): Mirror 8: 0x92afc1f2 (stale) Mirror 9: 0x92afc1f2 (stale) Mirror 11: 0x92afc1f2 (stale) Mirror 13: 0x92afc1f2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37c22): Mirror 8: 0x92afc1f2 (stale) Mirror 9: 0x92afc1f2 (stale) Mirror 11: 0x92afc1f2 (stale) Mirror 13: 0x92afc1f2 (stale) CRC-32 checksum value for chunk [0, 0x37c22): Mirror 1: 0x30f97dfe Mirror 14: 0x30f97dfe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37c22: skipped 000459 1787120131.392824 read 0x0270b6 thru 0x029f72 (0x2ebd bytes) 000460 1787120131.437991 trunc from 0x037c22 to 0x027bc4 (0x1005d bytes) 000461 1787120131.640925 trunc from 0x027bc5 to 0x02d0bc (0x54f8 bytes) 000464 1787120131.687180 mapwrite 0x01b153 thru 0x0287ec (0xd69a bytes) 000465 1787120131.840768 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d0bd): Mirror 8: 0x1048fa0a (stale) Mirror 11: 0x1048fa0a (stale) Mirror 13: 0x1048fa0a (stale) Mirror 14: 0xd2cc780c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d0bd): Mirror 8: 0x1048fa0a (stale) Mirror 11: 0x1048fa0a (stale) Mirror 13: 0x1048fa0a (stale) Mirror 14: 0xd2cc780c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d0bd: skipped 000466 1787120132.421161 write 0x038078 thru 0x0388e6 (0x86f bytes) 000468 1787120132.423343 mapwrite 0x01ec08 thru 0x023cd4 (0x50cd bytes) 000469 1787120132.563210 MIRROR_RESYNC 5 mirrors 000470 1787120133.878131 read 0x0245bf thru 0x025607 (0x1049 bytes) 000471 1787120133.883371 trunc from 0x0388e7 to 0x01cd00 (0x1bbe6 bytes) 000474 1787120133.987810 mapwrite 0x026b6a thru 0x030403 (0x989a bytes) 000478 1787120134.121105 mapwrite 0x03c8e7 thru 0x03ffff (0x3719 bytes) 000479 1787120134.320723 mapwrite 0x03b31d thru 0x03ffff (0x4ce3 bytes) 000481 1787120134.477095 MIRROR_RESYNC 5 mirrors 000482 1787120135.540477 mapread 0x005571 thru 0x00c962 (0x73f2 bytes) 000483 1787120135.564452 mapwrite 0x02e50e thru 0x030dd2 (0x28c5 bytes) 000484 1787120135.816323 mapread 0x009f7f thru 0x014041 (0xa0c3 bytes) 000485 1787120135.856256 write 0x02d906 thru 0x0339e7 (0x60e2 bytes) 000486 1787120135.857590 mapread 0x03b434 thru 0x03c36a (0xf37 bytes) 000487 1787120135.870180 trunc from 0x040000 to 0x00f4bf (0x30b40 bytes) 000488 1787120135.893792 trunc from 0x00f4c0 to 0x0373b3 (0x27ef4 bytes) 000489 1787120135.919130 MIRROR_RESYNC 5 mirrors 000491 1787120137.089322 mapread 0x02aa95 thru 0x033919 (0x8e85 bytes) 000492 1787120137.110523 write 0x00d2c6 thru 0x00dece (0xc09 bytes) 000493 1787120137.170000 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x373b4): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x373b4): Mirror 8: 0x1a7372f5 (stale) Mirror 11: 0x1a7372f5 (stale) Mirror 13: 0x1a7372f5 (stale) Mirror 14: 0x1a7372f5 (stale) CRC-32 checksum value for chunk [0, 0x373b4): Mirror 1: 0xcffbd541 Mirror 15: 0xcffbd541 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x373b4: skipped 000494 1787120138.502283 mapread 0x033718 thru 0x0373b3 (0x3c9c bytes) 000495 1787120138.574213 mapwrite 0x037a1d thru 0x03bc53 (0x4237 bytes) 000496 1787120138.704917 mapread 0x0049d3 thru 0x011607 (0xcc35 bytes) 000497 1787120138.722712 mapread 0x0294d7 thru 0x02da98 (0x45c2 bytes) 000498 1787120138.734097 trunc from 0x03bc54 to 0x017a4c (0x24207 bytes) 000500 1787120138.755496 trunc from 0x017a4d to 0x002f01 (0x14b4b bytes) 000502 1787120138.772669 trunc from 0x002f02 to 0x033a9c (0x30b9b bytes) 000503 1787120138.793424 read 0x000feb thru 0x00dc4f (0xcc65 bytes) 000504 1787120138.806788 write 0x012d5a thru 0x013f26 (0x11cd bytes) 000505 1787120138.816411 read 0x011969 thru 0x0130f9 (0x1791 bytes) 000506 1787120138.845876 mapwrite 0x01c254 thru 0x022be1 (0x698e bytes) 000507 1787120138.941534 MIRROR_RESYNC 6 mirrors 000512 1787120140.094523 MIRROR_RESYNC 6 mirrors 000515 1787120140.543736 read 0x030a45 thru 0x033a9c (0x3058 bytes) 000516 1787120140.556238 read 0x000856 thru 0x00988c (0x9037 bytes) 000521 1787120140.574977 write 0x00eb09 thru 0x01a9e8 (0xbee0 bytes) 000523 1787120140.698456 mapwrite 0x036660 thru 0x03ffff (0x99a0 bytes) 000524 1787120140.849831 read 0x02c8f6 thru 0x034626 (0x7d31 bytes) 000530 1787120140.868083 trunc from 0x040000 to 0x028ec1 (0x1713e bytes) 000533 1787120140.910775 mapwrite 0x03321d thru 0x03d5c0 (0xa3a4 bytes) 000538 1787120141.110385 read 0x013d2d thru 0x01ecbe (0xaf92 bytes) 000539 1787120141.123000 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d5c1): Mirror 11: 0xc2584791 (stale) Mirror 13: 0xc2584791 (stale) Mirror 14: 0xc2584791 (stale) Mirror 15: 0xc2584791 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d5c1): Mirror 11: 0xc2584791 (stale) Mirror 13: 0xc2584791 (stale) Mirror 14: 0xc2584791 (stale) Mirror 15: 0xc2584791 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d5c1: skipped 000541 1787120141.633857 mapwrite 0x0103ca thru 0x01eab7 (0xe6ee bytes) 000544 1787120141.752670 mapread 0x0049a3 thru 0x009691 (0x4cef bytes) 000547 1787120141.780018 MIRROR_RESYNC 5 mirrors 000548 1787120143.037368 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) 000550 1787120143.499333 write 0x004f54 thru 0x013930 (0xe9dd bytes) 000552 1787120143.639152 mapwrite 0x02430a thru 0x029c13 (0x590a bytes) 000554 1787120143.733035 trunc from 0x03d5c1 to 0x01c387 (0x21239 bytes) 000555 1787120143.762548 trunc from 0x01c388 to 0x034a3a (0x186b3 bytes) 000556 1787120143.796316 mapwrite 0x02585a thru 0x027648 (0x1def bytes) 000558 1787120143.864268 trunc from 0x034a3b to 0x00338f (0x316ab bytes) 000559 1787120143.889969 read 0x000c74 thru 0x00338f (0x271c bytes) 000560 1787120143.904049 trunc from 0x003390 to 0x00340b (0x7c bytes) truncating to largest ever: 0x3f36e 000561 1787120143.929306 trunc from 0x00340c to 0x03f36d (0x3bf62 bytes) 000562 1787120143.951783 read 0x03343a thru 0x0395ee (0x61b5 bytes) 000563 1787120143.992307 read 0x03da5a thru 0x03f36d (0x1914 bytes) 000564 1787120143.993987 trunc from 0x03f36e to 0x023243 (0x1c12a bytes) 000565 1787120144.021393 mapwrite 0x031b52 thru 0x039275 (0x7724 bytes) 000568 1787120144.197058 write 0x01c732 thru 0x01fbc8 (0x3497 bytes) 000571 1787120144.202131 read 0x00794e thru 0x00f01c (0x76cf bytes) 000572 1787120144.202966 write 0x03db76 thru 0x03ffff (0x248a bytes) 000573 1787120144.211336 MIRROR_RESYNC 4 mirrors 000574 1787120145.041900 write 0x01f762 thru 0x021375 (0x1c14 bytes) 000576 1787120145.150224 read 0x03b28c thru 0x03ffff (0x4d74 bytes) 000579 1787120145.169032 read 0x027b5f thru 0x02d498 (0x593a bytes) 000580 1787120145.180537 trunc from 0x040000 to 0x03c381 (0x3c7e bytes) 000583 1787120145.249267 trunc from 0x03c382 to 0x03d389 (0x1008 bytes) 000586 1787120145.275614 mapwrite 0x0265f6 thru 0x02cfdf (0x69ea bytes) 000589 1787120145.329994 trunc from 0x03d38a to 0x02d005 (0x10384 bytes) 000591 1787120145.347041 read 0x002b4c thru 0x00e23a (0xb6ef bytes) 000592 1787120145.348415 mapread 0x000062 thru 0x005aa7 (0x5a46 bytes) 000594 1787120145.356464 MIRROR_RESYNC 4 mirrors 000595 1787120146.467431 mapread 0x0251ea thru 0x02d005 (0x7e1c bytes) 000596 1787120146.488238 write 0x00d003 thru 0x014ab4 (0x7ab2 bytes) 000597 1787120146.569085 mapread 0x0074e4 thru 0x01481b (0xd338 bytes) 000598 1787120146.587479 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2d006): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d006): Mirror 13: 0xf0d9e633 (stale) Mirror 14: 0xf0d9e633 (stale) Mirror 15: 0xf0d9e633 (stale) CRC-32 checksum value for chunk [0, 0x2d006): Mirror 1: 0x5038f35c Mirror 16: 0x5038f35c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d006: skipped 000599 1787120147.598974 mapwrite 0x0349e4 thru 0x03be15 (0x7432 bytes) 000600 1787120147.785897 mapwrite 0x01d283 thru 0x01d470 (0x1ee bytes) 000602 1787120147.842616 mapread 0x00beba thru 0x013a4f (0x7b96 bytes) 000604 1787120147.858872 trunc from 0x03be16 to 0x01446c (0x279a9 bytes) 000607 1787120147.883067 mapwrite 0x03fae3 thru 0x03ffff (0x51d bytes) 000609 1787120147.952212 trunc from 0x040000 to 0x015e19 (0x2a1e6 bytes) 000610 1787120147.977637 read 0x00e8d6 thru 0x015e19 (0x7544 bytes) 000611 1787120147.984316 mapwrite 0x018891 thru 0x01f9ac (0x711c bytes) 000612 1787120148.066969 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1f9ad): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f9ad): Mirror 13: 0xc1e8ba94 (stale) Mirror 14: 0xc1e8ba94 (stale) Mirror 15: 0xc1e8ba94 (stale) Mirror 16: 0x7cff6a75 (stale) CRC-32 checksum value for chunk [0, 0x1f9ad): Mirror 1: 0xa35e2e30 Mirror 17: 0xa35e2e30 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f9ad: skipped 000614 1787120149.244563 mapread 0x0019ad thru 0x00719d (0x57f1 bytes) 000615 1787120149.309712 write 0x024250 thru 0x02d6f6 (0x94a7 bytes) 000616 1787120149.446359 write 0x016bbf thru 0x01de37 (0x7279 bytes) 000618 1787120149.478764 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2d6f7): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d6f7): Mirror 13: 0x9f93f562 (stale) Mirror 14: 0x9f93f562 (stale) Mirror 15: 0x9f93f562 (stale) Mirror 16: 0x6f657ce8 (stale) Mirror 17: 0xf071a4cc (stale) CRC-32 checksum value for chunk [0, 0x2d6f7): Mirror 1: 0x2587c86 Mirror 18: 0x2587c86 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d6f7: skipped 000620 1787120150.737813 read 0x019938 thru 0x01e693 (0x4d5c bytes) 000621 1787120150.756179 write 0x006fa3 thru 0x00d40c (0x646a bytes) 000623 1787120150.905806 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2d6f7): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d6f7): Mirror 13: 0x9f93f562 (stale) Mirror 14: 0x9f93f562 (stale) Mirror 15: 0x9f93f562 (stale) Mirror 16: 0x6f657ce8 (stale) Mirror 17: 0xf071a4cc (stale) Mirror 18: 0x2587c86 (stale) CRC-32 checksum value for chunk [0, 0x2d6f7): Mirror 1: 0x21534b9b Mirror 19: 0x21534b9b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d6f7: skipped 000625 1787120151.864743 trunc from 0x02d6f7 to 0x02eea7 (0x17b1 bytes) 000626 1787120152.037115 mapread 0x00bf0f thru 0x01b880 (0xf972 bytes) 000630 1787120152.078294 mapread 0x01bd61 thru 0x02b32e (0xf5ce bytes) 000631 1787120152.099061 mapwrite 0x02816a thru 0x02b3e1 (0x3278 bytes) 000632 1787120152.144818 trunc from 0x02eea8 to 0x003809 (0x2b69e bytes) 000633 1787120152.169570 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x380a): Mirror 13: 0x2b081a20 (stale) Mirror 14: 0x2b081a20 (stale) Mirror 16: 0x2b081a20 (stale) Mirror 17: 0x2b081a20 (stale) Mirror 18: 0x2b081a20 (stale) Mirror 19: 0x2b081a20 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x380a): Mirror 13: 0x2b081a20 (stale) Mirror 14: 0x2b081a20 (stale) Mirror 16: 0x2b081a20 (stale) Mirror 17: 0x2b081a20 (stale) Mirror 18: 0x2b081a20 (stale) Mirror 19: 0x2b081a20 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x380a: skipped 000637 1787120152.605264 mapread 0x0018e2 thru 0x003809 (0x1f28 bytes) 000639 1787120152.629292 trunc from 0x00380a to 0x033acd (0x302c4 bytes) 000640 1787120152.647418 mapwrite 0x01506c thru 0x023cf8 (0xec8d bytes) 000641 1787120152.760151 trunc from 0x033ace to 0x0130e6 (0x209e7 bytes) 000642 1787120152.794142 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x130e7): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x130e7): Mirror 13: 0x2aa3d50f (stale) Mirror 14: 0x2aa3d50f (stale) Mirror 16: 0x21f638d6 (stale) Mirror 17: 0x21f638d6 (stale) Mirror 18: 0x21f638d6 (stale) Mirror 19: 0x32d19782 (stale) CRC-32 checksum value for chunk [0, 0x130e7): Mirror 1: 0x2aa3d50f Mirror 20: 0x2aa3d50f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x130e7: skipped 000643 1787120153.796368 read 0x0057ff thru 0x00f7b0 (0x9fb2 bytes) 000644 1787120153.815442 write 0x02478f thru 0x025274 (0xae6 bytes) 000653 1787120153.916932 mapwrite 0x016498 thru 0x01c063 (0x5bcc bytes) 000654 1787120154.004662 read 0x0170a6 thru 0x023de2 (0xcd3d bytes) 000655 1787120154.018390 mapread 0x01a054 thru 0x01a2e0 (0x28d bytes) 000656 1787120154.019415 trunc from 0x025275 to 0x01e9ad (0x68c7 bytes) 000659 1787120154.047178 trunc from 0x01e9ae to 0x0220aa (0x36fd bytes) 000660 1787120154.064498 trunc from 0x0220ab to 0x038dd7 (0x16d2d bytes) 000661 1787120154.081217 read 0x0283ac thru 0x02c46f (0x40c4 bytes) 000663 1787120154.099730 read 0x00eb37 thru 0x01bc15 (0xd0df bytes) 000665 1787120154.101954 trunc from 0x038dd8 to 0x021d9b (0x1703c bytes) 000666 1787120154.128896 read 0x02042c thru 0x021d9b (0x1970 bytes) 000667 1787120154.131127 trunc from 0x021d9c to 0x03867a (0x168df bytes) 000668 1787120154.155334 write 0x022def thru 0x02b484 (0x8696 bytes) 000670 1787120154.167075 mapwrite 0x029584 thru 0x034085 (0xab02 bytes) 000674 1787120154.254453 mapread 0x01cb6e thru 0x022811 (0x5ca4 bytes) 000675 1787120154.256200 mapwrite 0x00db91 thru 0x010eef (0x335f bytes) 000676 1787120154.367149 trunc from 0x03867b to 0x01ba49 (0x1cc31 bytes) 000677 1787120154.388370 mapwrite 0x001a30 thru 0x010649 (0xec1a bytes) 000678 1787120154.461823 MIRROR_RESYNC 8 mirrors 000681 1787120156.006851 mapread 0x008597 thru 0x0138a4 (0xb30e bytes) 000683 1787120156.038828 mapwrite 0x03110c thru 0x03264d (0x1542 bytes) 000684 1787120156.301377 read 0x00f246 thru 0x015c80 (0x6a3b bytes) 000685 1787120156.315852 mapwrite 0x0069af thru 0x00840b (0x1a5d bytes) 000686 1787120156.376804 mapwrite 0x014941 thru 0x02469e (0xfd5e bytes) 000687 1787120156.472308 trunc from 0x03264e to 0x01039c (0x222b1 bytes) 000688 1787120156.494235 trunc from 0x01039d to 0x004d8c (0xb610 bytes) 000690 1787120156.527767 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x4d8d): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4d8d): Mirror 13: 0xdd453203 (stale) Mirror 14: 0xdd453203 (stale) Mirror 16: 0xdd453203 (stale) Mirror 17: 0xdd453203 (stale) Mirror 18: 0xdd453203 (stale) Mirror 19: 0xdd453203 (stale) Mirror 20: 0xdd453203 (stale) CRC-32 checksum value for chunk [0, 0x4d8d): Mirror 1: 0xdd453203 Mirror 21: 0xdd453203 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4d8d: skipped 000692 1787120157.665277 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x4d8d): Mirror 13: 0xdd453203 (stale) Mirror 14: 0xdd453203 (stale) Mirror 17: 0xdd453203 (stale) Mirror 18: 0xdd453203 (stale) Mirror 19: 0xdd453203 (stale) Mirror 20: 0xdd453203 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4d8d): Mirror 13: 0xdd453203 (stale) Mirror 14: 0xdd453203 (stale) Mirror 17: 0xdd453203 (stale) Mirror 18: 0xdd453203 (stale) Mirror 19: 0xdd453203 (stale) Mirror 20: 0xdd453203 (stale) CRC-32 checksum value for chunk [0, 0x4d8d): Mirror 1: 0xdd453203 Mirror 21: 0xdd453203 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4d8d: skipped 000694 1787120158.445143 trunc from 0x004d8d to 0x01630f (0x11583 bytes) 000696 1787120158.558804 trunc from 0x016310 to 0x017326 (0x1017 bytes) 000697 1787120158.580186 trunc from 0x017327 to 0x00724d (0x100d9 bytes) 000698 1787120158.592386 read 0x0059db thru 0x005fda (0x600 bytes) 000700 1787120158.598351 mapwrite 0x02b0ac thru 0x031a2d (0x6982 bytes) 000701 1787120158.670031 mapwrite 0x03199e thru 0x03db6f (0xc1d2 bytes) 000702 1787120158.763243 mapread 0x01d912 thru 0x021d5e (0x444d bytes) 000706 1787120158.778957 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3db70): Mirror 13: 0x4ff90218 (stale) Mirror 17: 0x4ff90218 (stale) Mirror 18: 0x4ff90218 (stale) Mirror 19: 0x4ff90218 (stale) Mirror 20: 0x4ff90218 (stale) Mirror 21: 0x55b7e71b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3db70): Mirror 13: 0x4ff90218 (stale) Mirror 17: 0x4ff90218 (stale) Mirror 18: 0x4ff90218 (stale) Mirror 19: 0x4ff90218 (stale) Mirror 20: 0x4ff90218 (stale) Mirror 21: 0x55b7e71b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3db70: skipped 000707 1787120159.278262 write 0x009052 thru 0x011671 (0x8620 bytes) 000708 1787120159.298367 MIRROR_RESYNC 7 mirrors 000709 1787120160.682025 write 0x02c15a thru 0x03b632 (0xf4d9 bytes) 000712 1787120160.742967 write 0x0063f3 thru 0x01353e (0xd14c bytes) 000714 1787120160.753336 mapwrite 0x00f687 thru 0x014238 (0x4bb2 bytes) 000715 1787120160.817757 mapread 0x000aa0 thru 0x002535 (0x1a96 bytes) 000716 1787120160.835934 mapread 0x01b3f1 thru 0x01e89e (0x34ae bytes) 000718 1787120160.851265 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3db70): Mirror 13: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3db70): Mirror 13: 0x91221ba3 (stale) Mirror 17: 0x91221ba3 (stale) Mirror 18: 0x91221ba3 (stale) Mirror 19: 0x91221ba3 (stale) Mirror 20: 0x91221ba3 (stale) Mirror 21: 0x91221ba3 (stale) CRC-32 checksum value for chunk [0, 0x3db70): Mirror 1: 0x68a17314 Mirror 22: 0x68a17314 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3db70: skipped 000722 1787120161.904334 mapread 0x01d4ac thru 0x025024 (0x7b79 bytes) 000724 1787120161.942235 mapwrite 0x03af27 thru 0x03ffff (0x50d9 bytes) 000727 1787120162.176822 trunc from 0x040000 to 0x0383ca (0x7c35 bytes) 000733 1787120162.218373 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x383cb): Mirror 13: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x383cb): Mirror 13: 0x4e6afa2f (stale) Mirror 17: 0x4e6afa2f (stale) Mirror 18: 0x4e6afa2f (stale) Mirror 19: 0x4e6afa2f (stale) Mirror 20: 0x4e6afa2f (stale) Mirror 21: 0x4e6afa2f (stale) Mirror 22: 0x99869dd2 (stale) CRC-32 checksum value for chunk [0, 0x383cb): Mirror 1: 0x99869dd2 Mirror 23: 0x99869dd2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x383cb: skipped 000735 1787120163.491357 write 0x00b07d thru 0x015180 (0xa104 bytes) 000736 1787120163.589066 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x383cb): Mirror 13: 0x4e6afa2f (stale) Mirror 17: 0x4e6afa2f (stale) Mirror 18: 0x4e6afa2f (stale) Mirror 19: 0x4e6afa2f (stale) Mirror 20: 0x4e6afa2f (stale) Mirror 22: 0x99869dd2 (stale) Mirror 23: 0x99869dd2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x383cb): Mirror 13: 0x4e6afa2f (stale) Mirror 17: 0x4e6afa2f (stale) Mirror 18: 0x4e6afa2f (stale) Mirror 19: 0x4e6afa2f (stale) Mirror 20: 0x4e6afa2f (stale) Mirror 22: 0x99869dd2 (stale) Mirror 23: 0x99869dd2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x383cb: skipped 000737 1787120164.102345 write 0x03fd84 thru 0x03ffff (0x27c bytes) 000738 1787120164.104303 mapread 0x039501 thru 0x03ffff (0x6aff bytes) 000739 1787120164.122287 trunc from 0x040000 to 0x01c604 (0x239fb bytes) 000746 1787120164.145473 write 0x0019eb thru 0x006aa5 (0x50bb bytes) 000747 1787120164.154637 mapread 0x019c50 thru 0x01c604 (0x29b5 bytes) 000749 1787120164.164545 mapwrite 0x0309e8 thru 0x03df8b (0xd5a4 bytes) 000750 1787120164.256341 write 0x027e4f thru 0x03038d (0x853f bytes) 000752 1787120164.262436 mapread 0x01beca thru 0x02a0cf (0xe206 bytes) 000753 1787120164.281767 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3df8c): Mirror 13: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3df8c): Mirror 13: 0x41a6314b (stale) Mirror 17: 0x41a6314b (stale) Mirror 18: 0x41a6314b (stale) Mirror 19: 0x41a6314b (stale) Mirror 20: 0x41a6314b (stale) Mirror 22: 0x713d577d (stale) Mirror 23: 0x75b1c950 (stale) CRC-32 checksum value for chunk [0, 0x3df8c): Mirror 1: 0x1780816f Mirror 24: 0x1780816f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3df8c: skipped 000754 1787120165.235285 mapread 0x033dbc thru 0x03c365 (0x85aa bytes) 000757 1787120165.301032 read 0x038b13 thru 0x03df8b (0x5479 bytes) 000760 1787120165.318264 read 0x000b2d thru 0x00a9d0 (0x9ea4 bytes) 000762 1787120165.345272 read 0x0240bb thru 0x0248fa (0x840 bytes) 000763 1787120165.361453 MIRROR_RESYNC 9 mirrors 000764 1787120166.746644 write 0x01a175 thru 0x022e4d (0x8cd9 bytes) 000765 1787120166.821649 write 0x02c3e5 thru 0x037866 (0xb482 bytes) 000766 1787120166.830446 mapwrite 0x0245f7 thru 0x03413f (0xfb49 bytes) 000771 1787120166.943503 read 0x02d472 thru 0x02e533 (0x10c2 bytes) 000772 1787120166.944263 trunc from 0x03df8c to 0x008902 (0x35689 bytes) 000775 1787120166.990240 read 0x00749c thru 0x00806c (0xbd1 bytes) 000776 1787120166.999660 read 0x006368 thru 0x008902 (0x259b bytes) 000777 1787120167.004930 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x8903): Mirror 13: 0x3b879e05 (stale) Mirror 17: 0x3b879e05 (stale) Mirror 18: 0x3b879e05 (stale) Mirror 20: 0x3b879e05 (stale) Mirror 22: 0x3b879e05 (stale) Mirror 23: 0x3b879e05 (stale) Mirror 24: 0x3b879e05 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8903): Mirror 13: 0x3b879e05 (stale) Mirror 17: 0x3b879e05 (stale) Mirror 18: 0x3b879e05 (stale) Mirror 20: 0x3b879e05 (stale) Mirror 22: 0x3b879e05 (stale) Mirror 23: 0x3b879e05 (stale) Mirror 24: 0x3b879e05 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8903: skipped 000781 1787120167.508866 MIRROR_RESYNC 8 mirrors 000783 1787120168.896591 write 0x01e360 thru 0x02b881 (0xd522 bytes) 000784 1787120169.002102 read 0x00e240 thru 0x00ed0c (0xacd bytes) 000785 1787120169.016789 write 0x03d7a5 thru 0x03ffff (0x285b bytes) 000787 1787120169.023164 mapwrite 0x01ad0f thru 0x022524 (0x7816 bytes) 000789 1787120169.224874 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x7f10f2c2 (stale) Mirror 17: 0x7f10f2c2 (stale) Mirror 20: 0x7f10f2c2 (stale) Mirror 22: 0x7f10f2c2 (stale) Mirror 23: 0x7f10f2c2 (stale) Mirror 24: 0x7f10f2c2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x7f10f2c2 (stale) Mirror 17: 0x7f10f2c2 (stale) Mirror 20: 0x7f10f2c2 (stale) Mirror 22: 0x7f10f2c2 (stale) Mirror 23: 0x7f10f2c2 (stale) Mirror 24: 0x7f10f2c2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000792 1787120169.683251 MIRROR_RESYNC 7 mirrors 000793 1787120170.984343 trunc from 0x040000 to 0x00b0b9 (0x34f46 bytes) 000794 1787120171.085003 trunc from 0x00b0ba to 0x03962d (0x2e574 bytes) 000796 1787120171.113217 read 0x02f752 thru 0x035fd7 (0x6886 bytes) 000797 1787120171.122552 mapwrite 0x0247c0 thru 0x026cbd (0x24fe bytes) 000798 1787120171.198083 mapread 0x018359 thru 0x01e898 (0x6540 bytes) 000804 1787120171.239072 MIRROR_RESYNC 7 mirrors 000805 1787120173.009768 write 0x03a09b thru 0x03ffff (0x5f65 bytes) 000806 1787120173.071648 mapread 0x00102a thru 0x008035 (0x700c bytes) 000807 1787120173.100331 write 0x03a069 thru 0x03e0f6 (0x408e bytes) 000808 1787120173.106623 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xd9e2b576 (stale) Mirror 17: 0xd9e2b576 (stale) Mirror 20: 0xd9e2b576 (stale) Mirror 22: 0xd9e2b576 (stale) Mirror 23: 0xd9e2b576 (stale) Mirror 24: 0xd9e2b576 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4d4342f5 Mirror 25: 0x4d4342f5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000811 1787120174.203199 MIRROR_RESYNC 8 mirrors 000812 1787120175.888681 MIRROR_EXTEND to 9 mirrors 000813 1787120176.845974 trunc from 0x040000 to 0x015af1 (0x2a50e bytes) 000814 1787120176.980655 read 0x00ca2d thru 0x00f954 (0x2f28 bytes) 000815 1787120176.994647 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15af2): Mirror 13: 0x735296e4 (stale) Mirror 17: 0x735296e4 (stale) Mirror 20: 0x735296e4 (stale) Mirror 22: 0x735296e4 (stale) Mirror 23: 0x735296e4 (stale) Mirror 24: 0x735296e4 (stale) Mirror 25: 0x735296e4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15af2): Mirror 13: 0x735296e4 (stale) Mirror 17: 0x735296e4 (stale) Mirror 20: 0x735296e4 (stale) Mirror 22: 0x735296e4 (stale) Mirror 23: 0x735296e4 (stale) Mirror 24: 0x735296e4 (stale) Mirror 25: 0x735296e4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15af2: skipped 000816 1787120177.739135 mapwrite 0x03fa80 thru 0x03ffff (0x580 bytes) 000817 1787120177.893803 read 0x03e383 thru 0x03ffff (0x1c7d bytes) 000818 1787120177.915878 read 0x012c9d thru 0x0163ba (0x371e bytes) 000820 1787120177.938059 trunc from 0x040000 to 0x012607 (0x2d9f8 bytes) 000823 1787120177.979371 mapread 0x00189c thru 0x003a68 (0x21cd bytes) 000824 1787120177.981278 write 0x020d66 thru 0x0262d1 (0x556c bytes) 000826 1787120177.985982 mapwrite 0x0057ff thru 0x006a83 (0x1285 bytes) 000827 1787120178.115164 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x262d2): Mirror 13: 0x5d1b7e10 (stale) Mirror 17: 0x5d1b7e10 (stale) Mirror 20: 0x5d1b7e10 (stale) Mirror 22: 0x5d1b7e10 (stale) Mirror 24: 0x5d1b7e10 (stale) Mirror 25: 0x5d1b7e10 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x262d2): Mirror 13: 0x5d1b7e10 (stale) Mirror 17: 0x5d1b7e10 (stale) Mirror 20: 0x5d1b7e10 (stale) Mirror 22: 0x5d1b7e10 (stale) Mirror 24: 0x5d1b7e10 (stale) Mirror 25: 0x5d1b7e10 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x262d2: skipped 000828 1787120178.803238 trunc from 0x0262d2 to 0x013e1e (0x124b3 bytes) 000829 1787120178.827379 read 0x0101db thru 0x013e1e (0x3c44 bytes) 000830 1787120178.836654 mapwrite 0x013340 thru 0x01f8e2 (0xc5a3 bytes) 000831 1787120178.927194 mapwrite 0x03663d thru 0x03ffff (0x99c3 bytes) 000832 1787120179.075877 read 0x0322fe thru 0x0374b5 (0x51b8 bytes) 000833 1787120179.076762 read 0x011685 thru 0x020d75 (0xf6f1 bytes) 000834 1787120179.086115 mapwrite 0x00fd3c thru 0x01b9d7 (0xbc9c bytes) 000837 1787120179.169532 trunc from 0x040000 to 0x00018a (0x3fe75 bytes) 000838 1787120179.192291 read 0x000145 thru 0x00018a (0x46 bytes) 000839 1787120179.193116 trunc from 0x00018b to 0x037134 (0x36faa bytes) 000840 1787120179.224411 write 0x0168f3 thru 0x019938 (0x3046 bytes) 000842 1787120179.254592 mapread 0x0044b0 thru 0x00ceb8 (0x8a09 bytes) 000843 1787120179.256759 write 0x026c94 thru 0x02bf1d (0x528a bytes) 000846 1787120179.265601 trunc from 0x037135 to 0x006aff (0x30635 bytes) 000849 1787120179.288993 mapwrite 0x03c6bd thru 0x03ffff (0x3943 bytes) 000852 1787120179.370493 read 0x030f5c thru 0x0322f0 (0x1395 bytes) 000854 1787120179.385287 trunc from 0x040000 to 0x02140c (0x1ebf3 bytes) 000855 1787120179.406785 MIRROR_RESYNC 7 mirrors 000856 1787120181.217106 write 0x034f3d thru 0x039aea (0x4bae bytes) 000860 1787120181.301374 mapwrite 0x02282d thru 0x025755 (0x2f29 bytes) 000864 1787120181.398540 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x39aeb): Mirror 13: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39aeb): Mirror 13: 0x9bde25bf (stale) Mirror 17: 0x9bde25bf (stale) Mirror 20: 0x9bde25bf (stale) Mirror 22: 0x9bde25bf (stale) Mirror 24: 0x9bde25bf (stale) Mirror 25: 0x9bde25bf (stale) CRC-32 checksum value for chunk [0, 0x39aeb): Mirror 1: 0x4d27c594 Mirror 26: 0x4d27c594 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39aeb: skipped 000867 1787120182.337434 mapwrite 0x008cba thru 0x00c49a (0x37e1 bytes) 000868 1787120182.480396 mapwrite 0x039723 thru 0x03ffff (0x68dd bytes) 000869 1787120182.645214 MIRROR_RESYNC 8 mirrors 000870 1787120184.972187 read 0x03f0d6 thru 0x03ffff (0xf2a bytes) 000873 1787120184.980787 mapread 0x0135f9 thru 0x01b101 (0x7b09 bytes) 000874 1787120185.016319 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) 000876 1787120185.438907 mapwrite 0x02b3f4 thru 0x039e4b (0xea58 bytes) 000878 1787120185.694164 read 0x028b75 thru 0x02a081 (0x150d bytes) 000879 1787120185.697150 trunc from 0x040000 to 0x02f4e8 (0x10b17 bytes) 000880 1787120185.740537 read 0x029a8a thru 0x02b4e1 (0x1a58 bytes) 000881 1787120185.747410 read 0x012959 thru 0x014b25 (0x21cd bytes) 000882 1787120185.764596 MIRROR_RESYNC 7 mirrors 000885 1787120187.565947 mapwrite 0x01d353 thru 0x01eb01 (0x17af bytes) 000886 1787120187.754611 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f4e9): Mirror 17: 0xc86f8bad (stale) Mirror 20: 0xc86f8bad (stale) Mirror 24: 0xc86f8bad (stale) Mirror 25: 0xc86f8bad (stale) Mirror 26: 0xc86f8bad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f4e9): Mirror 17: 0xc86f8bad (stale) Mirror 20: 0xc86f8bad (stale) Mirror 24: 0xc86f8bad (stale) Mirror 25: 0xc86f8bad (stale) Mirror 26: 0xc86f8bad (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f4e9: skipped 000888 1787120188.152577 read 0x0052a2 thru 0x0151af (0xff0e bytes) 000889 1787120188.170855 read 0x027f6d thru 0x02f4e8 (0x757c bytes) 000890 1787120188.180618 write 0x023abb thru 0x030dc5 (0xd30b bytes) 000891 1787120188.182186 read 0x020d73 thru 0x02921c (0x84aa bytes) 000892 1787120188.183060 mapwrite 0x008e5e thru 0x00eccb (0x5e6e bytes) 000895 1787120188.288133 mapread 0x001ce2 thru 0x00e1e3 (0xc502 bytes) 000896 1787120188.299597 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30dc6): Mirror 17: 0xbe24070e (stale) Mirror 24: 0xbe24070e (stale) Mirror 25: 0xbe24070e (stale) Mirror 26: 0xbe24070e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30dc6): Mirror 17: 0xbe24070e (stale) Mirror 24: 0xbe24070e (stale) Mirror 25: 0xbe24070e (stale) Mirror 26: 0xbe24070e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30dc6: skipped 000897 1787120188.722604 mapread 0x00a67d thru 0x00fadd (0x5461 bytes) 000901 1787120188.740913 mapread 0x006cf9 thru 0x00d9f0 (0x6cf8 bytes) 000902 1787120188.743270 mapread 0x00adda thru 0x00bdf6 (0x101d bytes) 000904 1787120188.749950 read 0x00f1b0 thru 0x01354e (0x439f bytes) 000907 1787120188.755201 mapread 0x00cdb8 thru 0x014e5b (0x80a4 bytes) 000908 1787120188.759252 read 0x0035fb thru 0x01244d (0xee53 bytes) 000909 1787120188.782445 mapread 0x02ecba thru 0x02ff88 (0x12cf bytes) 000910 1787120188.795319 MIRROR_RESYNC 5 mirrors 000911 1787120189.944336 read 0x025c99 thru 0x02ad13 (0x507b bytes) 000912 1787120189.965948 trunc from 0x030dc6 to 0x01452a (0x1c89b bytes) 000913 1787120190.040966 mapwrite 0x001632 thru 0x0028da (0x12a9 bytes) 000915 1787120190.096286 write 0x039b30 thru 0x03f802 (0x5cd3 bytes) 000916 1787120190.100059 write 0x03912b thru 0x03c487 (0x335d bytes) 000917 1787120190.109886 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3f803): Mirror 17: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f803): Mirror 17: 0x6ef49744 (stale) Mirror 24: 0x6ef49744 (stale) Mirror 25: 0x6ef49744 (stale) Mirror 26: 0x6ef49744 (stale) CRC-32 checksum value for chunk [0, 0x3f803): Mirror 1: 0xc9c2c29d Mirror 27: 0xc9c2c29d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f803: skipped 000918 1787120191.103335 mapwrite 0x038117 thru 0x03f680 (0x756a bytes) 000919 1787120191.287579 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3f803): Mirror 17: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f803): Mirror 17: 0x6ef49744 (stale) Mirror 24: 0x6ef49744 (stale) Mirror 25: 0x6ef49744 (stale) Mirror 26: 0x6ef49744 (stale) Mirror 27: 0xc9c2c29d (stale) CRC-32 checksum value for chunk [0, 0x3f803): Mirror 1: 0x5c705e16 Mirror 28: 0x5c705e16 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f803: skipped 000921 1787120192.420655 read 0x0008eb thru 0x00140e (0xb24 bytes) 000925 1787120192.465278 mapwrite 0x03a90b thru 0x03ffff (0x56f5 bytes) 000927 1787120192.655847 trunc from 0x040000 to 0x01592f (0x2a6d0 bytes) 000929 1787120192.675737 read 0x015370 thru 0x01592f (0x5c0 bytes) 000930 1787120192.687525 read 0x00c3fe thru 0x01592f (0x9532 bytes) 000931 1787120192.689120 read 0x00ea95 thru 0x01592f (0x6e9b bytes) 000932 1787120192.697981 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x15930): Mirror 17: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15930): Mirror 17: 0xe651e8f7 (stale) Mirror 24: 0xe651e8f7 (stale) Mirror 25: 0xe651e8f7 (stale) Mirror 26: 0xe651e8f7 (stale) Mirror 27: 0x1168aefa (stale) Mirror 28: 0x1168aefa (stale) CRC-32 checksum value for chunk [0, 0x15930): Mirror 1: 0x1168aefa Mirror 29: 0x1168aefa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15930: skipped 000934 1787120193.692267 mapread 0x00a633 thru 0x015692 (0xb060 bytes) 000935 1787120193.728672 trunc from 0x015930 to 0x02d0e2 (0x177b3 bytes) 000936 1787120193.911764 read 0x010b80 thru 0x020537 (0xf9b8 bytes) 000939 1787120193.935142 read 0x012b4c thru 0x0205fc (0xdab1 bytes) 000940 1787120193.937463 mapread 0x029e00 thru 0x02d0e2 (0x32e3 bytes) 000943 1787120193.983472 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d0e3): Mirror 17: 0xd881e9e2 (stale) Mirror 25: 0xd881e9e2 (stale) Mirror 26: 0xd881e9e2 (stale) Mirror 27: 0x8bd969e4 (stale) Mirror 28: 0x8bd969e4 (stale) Mirror 29: 0x8bd969e4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d0e3): Mirror 17: 0xd881e9e2 (stale) Mirror 25: 0xd881e9e2 (stale) Mirror 26: 0xd881e9e2 (stale) Mirror 27: 0x8bd969e4 (stale) Mirror 28: 0x8bd969e4 (stale) Mirror 29: 0x8bd969e4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d0e3: skipped 000945 1787120194.521036 write 0x00658d thru 0x010d69 (0xa7dd bytes) 000946 1787120194.532040 mapwrite 0x008f1d thru 0x00955d (0x641 bytes) 000947 1787120194.584329 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2d0e3): Mirror 17: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d0e3): Mirror 17: 0xd881e9e2 (stale) Mirror 25: 0xd881e9e2 (stale) Mirror 26: 0xd881e9e2 (stale) Mirror 27: 0x8bd969e4 (stale) Mirror 28: 0x8bd969e4 (stale) Mirror 29: 0x8bd969e4 (stale) CRC-32 checksum value for chunk [0, 0x2d0e3): Mirror 1: 0x2a94022b Mirror 30: 0x2a94022b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d0e3: skipped 000948 1787120195.754299 trunc from 0x02d0e3 to 0x03e3d8 (0x112f6 bytes) 000950 1787120195.898108 write 0x037ef9 thru 0x03f49f (0x75a7 bytes) 000951 1787120195.908958 mapread 0x0326e3 thru 0x03f49f (0xcdbd bytes) 000952 1787120195.943553 read 0x0321cd thru 0x03f49f (0xd2d3 bytes) 000954 1787120195.944860 mapwrite 0x0331ff thru 0x035e35 (0x2c37 bytes) 000956 1787120196.034068 trunc from 0x03f4a0 to 0x0071cb (0x382d4 bytes) 000957 1787120196.067492 read 0x0024bf thru 0x0071cb (0x4d0d bytes) 000965 1787120196.098535 read 0x00175a thru 0x0071cb (0x5a72 bytes) 000967 1787120196.107380 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x71cc): Mirror 17: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x71cc): Mirror 17: 0x27764c1c (stale) Mirror 25: 0x27764c1c (stale) Mirror 26: 0x27764c1c (stale) Mirror 27: 0xe2aac04f (stale) Mirror 28: 0xe2aac04f (stale) Mirror 29: 0xe2aac04f (stale) Mirror 30: 0x46dbfb76 (stale) CRC-32 checksum value for chunk [0, 0x71cc): Mirror 1: 0x46dbfb76 Mirror 31: 0x46dbfb76 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x71cc: skipped 000970 1787120197.090798 write 0x01bc98 thru 0x01ef63 (0x32cc bytes) 000971 1787120197.172612 mapread 0x011750 thru 0x016fa3 (0x5854 bytes) 000972 1787120197.188889 read 0x00067f thru 0x00f96f (0xf2f1 bytes) 000974 1787120197.200572 trunc from 0x01ef64 to 0x032d97 (0x13e34 bytes) 000976 1787120197.245984 read 0x02ed6b thru 0x032d97 (0x402d bytes) 000978 1787120197.259565 read 0x0180d7 thru 0x027291 (0xf1bb bytes) 000979 1787120197.260678 read 0x029f98 thru 0x032d97 (0x8e00 bytes) 000980 1787120197.262578 read 0x00010a thru 0x00d792 (0xd689 bytes) 000981 1787120197.268761 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32d98): Mirror 17: 0xcb06c172 (stale) Mirror 25: 0xcb06c172 (stale) Mirror 26: 0xcb06c172 (stale) Mirror 27: 0x7c9b06b2 (stale) Mirror 29: 0x7c9b06b2 (stale) Mirror 30: 0x3edb477f (stale) Mirror 31: 0xce3729cd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32d98): Mirror 17: 0xcb06c172 (stale) Mirror 25: 0xcb06c172 (stale) Mirror 26: 0xcb06c172 (stale) Mirror 27: 0x7c9b06b2 (stale) Mirror 29: 0x7c9b06b2 (stale) Mirror 30: 0x3edb477f (stale) Mirror 31: 0xce3729cd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32d98: skipped 000982 1787120197.764103 read 0x01b3b6 thru 0x024c84 (0x98cf bytes) 000983 1787120197.776227 trunc from 0x032d98 to 0x02f9f3 (0x33a4 bytes) 000984 1787120197.791913 mapread 0x0232c0 thru 0x0254fd (0x223e bytes) 000992 1787120197.810913 MIRROR_RESYNC 8 mirrors 000996 1787120199.212869 mapwrite 0x00123a thru 0x00571d (0x44e4 bytes) 000997 1787120199.471387 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2f9f4): Mirror 17: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f9f4): Mirror 17: 0x4ed1c87a (stale) Mirror 25: 0x4ed1c87a (stale) Mirror 26: 0x4ed1c87a (stale) Mirror 27: 0x4ed1c87a (stale) Mirror 29: 0x4ed1c87a (stale) Mirror 30: 0x4ed1c87a (stale) Mirror 31: 0x4ed1c87a (stale) CRC-32 checksum value for chunk [0, 0x2f9f4): Mirror 1: 0x6e5f92a9 Mirror 32: 0x6e5f92a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f9f4: skipped 000998 1787120200.721615 trunc from 0x02f9f4 to 0x00232b (0x2d6c8 bytes) 000999 1787120200.864197 read 0x00202c thru 0x00232b (0x300 bytes) 001000 1787120200.869046 read 0x001e72 thru 0x00232b (0x4ba bytes) All operations completed A-OK! PASS 70a (121s) == sanity-flr test 71: check mirror extend parent fid ==== 02:16:45 (1787120205) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 02:16:48 (1787120208) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.22524 s, 9.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185807 s, 5.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 02:16:56 (1787120216) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre2 Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 7285760 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7036000 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8237792 .. failed Extending file size to 8625248 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8871072 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9243168 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57475 57476 57478 57480 57481 done 192.168.203.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre2 (opts:) 192.168.203.136@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 02:18:15 (1787120295) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre2 Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3640256 .. Extending file size to 8832480 .. done Extending file size to 5814016 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7533664 .. Extending file size to 8910080 .. Extending file size to 9351552 .. Extending file size to 7842816 .. Extending file size to 8923680 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 59063 59064 59066 59068 59069 failed 192.168.203.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre2 (opts:) 192.168.203.136@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 02:19:33 (1787120373) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0721225 s, 29.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.360658 s, 5.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x5a68f060 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 02:19:51 (1787120391) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0977124 s, 21.5 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 02:19:57 (1787120397) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f8:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ed:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ed:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ee:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 02:20:04 (1787120404) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.267735 s, 7.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 02:20:12 (1787120412) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.205567 s, 10.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 02:20:19 (1787120419) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.362074 s, 5.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 02:20:52 (1787120452) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg336-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123175 s, 8.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f3:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f3:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f4:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 02:21:48 (1787120508) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg336-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.196021 s, 10.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (53s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 02:22:41 (1787120561) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg336-server mds-ost sync done. Creating new pool test_204f oleg336-server: Pool lustre.test_204f created Adding targets to pool oleg336-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg336-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.251497 s, 8.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg336-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg336-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg336-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 02:24:02 (1787120642) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 02:24:10 (1787120650) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 02:24:18 (1787120658) oleg336-server: Pool lustre.test_206 created Adding targets to pool oleg336-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg336-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg336-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg336-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg336-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 02:24:44 (1787120684) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588691 s, 17.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631561 s, 16.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xaf69fad9 Mirror 3: 0xaf69fad9 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 02:24:55 (1787120695) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 02:24:58 (1787120698) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 02:25:01 (1787120701) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0588629 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xc00000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xbf0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xbf0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xbf0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1700000 Read this (4 bytes): BBBB 192.168.203.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 02:25:40 (1787120740) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0190304 s, 430 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 02:25:47 (1787120747) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405317 s, 25.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x21d lmm_fid: [0x200000404:0x21d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x21d lmm_fid: [0x200000404:0x21d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x300:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834461 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-19 02:25:49.000000000 -0400 Modify: 2026-08-19 02:25:49.000000000 -0400 Change: 2026-08-19 02:25:50.000000000 -0400 Birth: 2026-08-19 02:25:49.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 02:25:55 (1787120755) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 02:25:58 (1787120758) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.554383 s, 73.9 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0721254 s, 56.8 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 19 02:26 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21f:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 19 02:26 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 02:26:07 (1787120767) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 02:26:16 (1787120776) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1695 sec =========== 02:26:26 (1787120786) === sanity-flr: start cleanup 02:26:28 (1787120788) === === sanity-flr: finish cleanup 02:26:34 (1787120794) ===