-----============= acceptance-small: sanity-flr ============----- Tue Apr 28 20:01:38 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg438-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 20:01:44 (1777420904) === oleg438-client.virtnet: executing check_config_client /mnt/lustre oleg438-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg438-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9ac2903e9000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9ac2903e9000.idle_timeout=debug disable quota as required oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 20:01:54 (1777420914) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7555] == sanity-flr test 0a: lfs mirror create with -N option == 20:01:55 (1777420915) PASS 0a (4s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:01:59 (1777420919) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:02:00 (1777420920) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 20:02:01 (1777420921) PASS 0d (8s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:02:09 (1777420929) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:02:10 (1777420930) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 20:02:12 (1777420932) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9ac2903e9000.stats=clear osc.lustre-OST0001-osc-ffff9ac2903e9000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:02:21 (1777420941) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:02:30 (1777420950) PASS 0j (12s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:02:42 (1777420962) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 20:02:46 (1777420966) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 20:02:55 (1777420975) PASS 2 (5s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:02:59 (1777420979) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:03:01 (1777420981) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:03:05 (1777420985) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (4s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 20:03:11 (1777420991) PASS 7 (4s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:03:15 (1777420995) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111706 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111415 s, 9.4 MB/s PASS 21 (5s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:03:20 (1777421000) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.0749212 s, 70.0 MB/s PASS 22 (5s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:03:25 (1777421005) Stopping /mnt/lustre-ost1 (opts:) on oleg438-server oleg438-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ac2903e9000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ac2903e9000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg438-server: oleg438-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg438-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg438-server oleg438-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ac2903e9000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ac2903e9000.ost_server_uuid in DISCONN state after 1 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg438-server: oleg438-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg438-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (38s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:04:03 (1777421043) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.250621 s, 41.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg438-server oleg438-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ac2903e9000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ac2903e9000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg438-server: oleg438-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg438-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (19s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:04:22 (1777421062) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg438-server oleg438-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ac2903e9000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ac2903e9000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg438-server: oleg438-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg438-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg438-server oleg438-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ac2903e9000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ac2903e9000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg438-server: oleg438-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg438-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (35s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:04:57 (1777421097) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.91067 s, 27.4 MB/s -rw-r--r-- 1 root root 52428800 Apr 28 20:05 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.376s user 0m0.000s sys 0m0.161s Stopping /mnt/lustre-ost1 (opts:) on oleg438-server oleg438-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ac2903e9000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ac2903e9000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.561s user 0m0.001s sys 0m0.176s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg438-server: oleg438-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg438-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg438-server oleg438-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ac2903e9000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ac2903e9000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.568s user 0m0.001s sys 0m0.181s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg438-server: oleg438-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg438-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg438-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ac2903e9000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ac2903e9000.ost_server_uuid in FULL state after 0 sec PASS 33b (41s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:05:38 (1777421138) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:05:39 (1777421139) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:05:41 (1777421141) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 20:05:43 (1777421143) PASS 35 (5s) == sanity-flr test 36a: write to mirrored files ========== 20:05:48 (1777421148) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.353933 s, 35.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (8s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:05:56 (1777421156) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185363 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256779 s, 40.8 MB/s PASS 36b (4s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:06:00 (1777421160) PASS 36c (5s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:06:05 (1777421165) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00135719 s, 754 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00113502 s, 902 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00082128 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000814681 s, 1.3 MB/s flv=10 olv=10 PASS 36d (7s) == sanity-flr test 37: mirror I/O API verification ======= 20:06:12 (1777421172) create test files with size 15728981 .. 27cf65d5452e9bc8abe1c0c22842e8bb - d34ab8e0d1016c711d12ff9154085c72 - ab77e08613621526fd08f3987a8d7a46 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg438-client.virtnet /mnt/lustre (opts:) Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre PASS 37 (13s) == sanity-flr test 38: resync ============================ 20:06:25 (1777421185) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (9s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:06:34 (1777421194) PASS 39 (4s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:06:38 (1777421198) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800233 s, 13.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168608 s, 62.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (5s) == sanity-flr test 41: lfs mirror resync check =========== 20:06:43 (1777421203) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0933774 s, 22.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.209715 s, 20.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.111617 s, 18.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (9s) == sanity-flr test 42: lfs mirror verify ================= 20:06:52 (1777421212) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 20:06:54 (1777421214) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:06:56 (1777421216) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217946 s, 48.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 20:07:00 (1777421220) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:07:02 (1777421222) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:07:07 (1777421227) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.121538 s, 86.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg438-client.virtnet /mnt/lustre (opts:) Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:07:19 (1777421239) llite.lustre-ffff9ac289702800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.124002 s, 84.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg438-client.virtnet /mnt/lustre (opts:) Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9ac282cbc800.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:07:24 (1777421244) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (18s) == sanity-flr test 46: Verify setstripe --copy option ==== 20:07:42 (1777421262) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (18s) == sanity-flr test 47: Verify mirror obj alloc =========== 20:08:00 (1777421280) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 20:08:01 (1777421281) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.131108 s, 24.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0853667 s, 36.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x6c026e8 (nosync,stale) Mirror 2: 0x6c026e8 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 2b80b54c7215a258e060e8f12a4b441c - mirror 1 checksum: 873159c298ad201601ee2ee51c6453df - mirror 2 checksum: 2b80b54c7215a258e060e8f12a4b441c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Apr 28 20:08:03 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Apr 28 20:08:03 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 2b80b54c7215a258e060e8f12a4b441c - mirror 1 checksum: 873159c298ad201601ee2ee51c6453df - mirror 2 checksum: 873159c298ad201601ee2ee51c6453df - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Apr 28 20:08:03 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Apr 28 20:08:03 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } PASS 48 (5s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:08:07 (1777421287) Failing mds1 on oleg438-server Stopping /mnt/lustre-mds1 (opts:) on oleg438-server 20:08:11 (1777421291) shut down facet: mds1 facet_host: oleg438-server facet_failover_host: oleg438-server Failover mds1 to oleg438-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-MDT0000 20:08:27 (1777421307) targets are mounted 20:08:28 (1777421308) facet_failover done oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255187 s, 41.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 50A (29s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:08:36 (1777421316) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:08:38 (1777421318) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:08:40 (1777421320) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00789508 s, 2.1 MB/s PASS 50c (4s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:08:44 (1777421324) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:08:46 (1777421326) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:08:48 (1777421328) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (12s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:09:00 (1777421340) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (22s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:09:23 (1777421363) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (20s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:09:42 (1777421382) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (21s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:10:03 (1777421403) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (5s) == sanity-flr test 70: mirror create and split race ====== 20:10:08 (1777421408) mirror create pid 52238 mirror split pid 52239 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4989: 52238 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4989: 52239 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 20:11:11 (1777421471) Seed set to 1472 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1777421472.570771 write 0x022d16 thru 0x031687 (0xe972 bytes) 000002 1777421472.580991 write 0x004770 thru 0x008fcf (0x4860 bytes) 000007 1777421472.594988 mapread 0x01d814 thru 0x02cc40 (0xf42d bytes) 000010 1777421472.607743 read 0x02f350 thru 0x031687 (0x2338 bytes) 000011 1777421472.608773 write 0x031e69 thru 0x03ee4c (0xcfe4 bytes) 000014 1777421472.613147 read 0x031292 thru 0x038a64 (0x77d3 bytes) 000016 1777421472.615154 mapwrite 0x0196c9 thru 0x021aff (0x8437 bytes) 000017 1777421472.669372 read 0x01a4c8 thru 0x01d190 (0x2cc9 bytes) truncating to largest ever: 0x36b8b 000018 1777421472.670042 trunc from 0x03ee4d to 0x036b8a (0x82c2 bytes) 000019 1777421472.686997 mapread 0x035a0a thru 0x036b8a (0x1181 bytes) 000020 1777421472.688258 write 0x003b4f thru 0x00ec5b (0xb10d bytes) 000022 1777421472.702533 MIRROR_EXTEND to 2 mirrors 000023 1777421473.050463 mapwrite 0x005a49 thru 0x005f34 (0x4ec bytes) 000024 1777421473.162600 write 0x012cac thru 0x018982 (0x5cd7 bytes) 000025 1777421473.168978 mapread 0x00f451 thru 0x012341 (0x2ef1 bytes) 000028 1777421473.171861 trunc from 0x036b8b to 0x027759 (0xf431 bytes) 000029 1777421473.196452 trunc from 0x02775a to 0x0296c9 (0x1f70 bytes) 000030 1777421473.206850 write 0x00c086 thru 0x0182d5 (0xc250 bytes) 000031 1777421473.208437 mapread 0x002c83 thru 0x009c9f (0x701d bytes) 000032 1777421473.217386 MIRROR_RESYNC 2 mirrors 000033 1777421473.538655 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) 000036 1777421473.632889 read 0x00625d thru 0x0104b6 (0xa25a bytes) 000037 1777421473.638699 write 0x03e565 thru 0x03ffff (0x1a9b bytes) 000039 1777421473.647823 trunc from 0x040000 to 0x0255a6 (0x1aa59 bytes) 000040 1777421473.669258 mapread 0x010b14 thru 0x01ab6a (0xa057 bytes) 000041 1777421473.684811 mapwrite 0x010995 thru 0x016b7a (0x61e6 bytes) 000043 1777421473.713936 write 0x01a1ec thru 0x0215e4 (0x73f9 bytes) 000049 1777421473.723019 mapread 0x01aaab thru 0x01d4dd (0x2a33 bytes) 000051 1777421473.724144 read 0x0154a8 thru 0x018bd2 (0x372b bytes) 000054 1777421473.725455 mapwrite 0x026ffd thru 0x02c268 (0x526c bytes) 000060 1777421473.792827 write 0x02dde7 thru 0x039c6d (0xbe87 bytes) 000061 1777421473.794167 trunc from 0x039c6e to 0x02e874 (0xb3f9 bytes) 000062 1777421473.810857 mapwrite 0x00e361 thru 0x016455 (0x80f5 bytes) 000064 1777421473.847992 trunc from 0x02e875 to 0x00b492 (0x233e2 bytes) 000065 1777421473.863052 read 0x005b67 thru 0x00b492 (0x592c bytes) truncating to largest ever: 0x3dd3c 000066 1777421473.863970 trunc from 0x00b493 to 0x03dd3b (0x328a9 bytes) 000068 1777421473.877054 mapread 0x00ebf8 thru 0x00fb8d (0xf96 bytes) 000070 1777421473.887996 mapwrite 0x03b8a4 thru 0x03ffff (0x475c bytes) 000071 1777421473.940040 MIRROR_EXTEND to 2 mirrors 000077 1777421474.340219 write 0x0000bc thru 0x004abf (0x4a04 bytes) 000080 1777421474.416821 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) 000081 1777421474.521363 read 0x014c45 thru 0x0249b5 (0xfd71 bytes) 000084 1777421474.530171 trunc from 0x040000 to 0x013ffe (0x2c001 bytes) 000086 1777421474.547953 mapwrite 0x019599 thru 0x01a4e8 (0xf50 bytes) 000087 1777421474.627792 read 0x004084 thru 0x00c1b5 (0x8132 bytes) 000088 1777421474.637158 trunc from 0x01a4e9 to 0x00c787 (0xdd61 bytes) 000090 1777421474.654199 trunc from 0x00c788 to 0x018c0b (0xc484 bytes) 000091 1777421474.671154 trunc from 0x018c0c to 0x03a938 (0x21d2d bytes) 000095 1777421474.683786 mapwrite 0x02b3f1 thru 0x02f3c8 (0x3fd8 bytes) 000096 1777421474.715253 write 0x010d6d thru 0x016b80 (0x5e14 bytes) 000097 1777421474.726686 read 0x02d9e4 thru 0x033da7 (0x63c4 bytes) 000098 1777421474.728740 write 0x033aaa thru 0x036ca8 (0x31ff bytes) 000099 1777421474.730079 write 0x000056 thru 0x00b334 (0xb2df bytes) 000103 1777421474.733189 mapread 0x026ab0 thru 0x03330e (0xc85f bytes) 000106 1777421474.736078 read 0x01c1d7 thru 0x01d88a (0x16b4 bytes) 000107 1777421474.740635 MIRROR_EXTEND to 2 mirrors 000109 1777421475.061223 mapwrite 0x02cdc7 thru 0x0376f1 (0xa92b bytes) 000110 1777421475.161713 write 0x027167 thru 0x033a08 (0xc8a2 bytes) 000111 1777421475.167491 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) 000112 1777421475.261570 write 0x00ad4f thru 0x00f3a2 (0x4654 bytes) 000113 1777421475.268652 trunc from 0x03a939 to 0x01ead5 (0x1be63 bytes) 000114 1777421475.288426 read 0x0028a9 thru 0x0058f0 (0x3048 bytes) 000116 1777421475.293486 write 0x014a5b thru 0x0182e9 (0x388f bytes) 000117 1777421475.299813 mapread 0x009c06 thru 0x014845 (0xac40 bytes) 000119 1777421475.304529 write 0x038041 thru 0x03b061 (0x3021 bytes) 000121 1777421475.309442 MIRROR_EXTEND to 2 mirrors 000125 1777421475.686043 mapread 0x036063 thru 0x03b061 (0x4fff bytes) 000128 1777421475.703942 write 0x006e74 thru 0x00d05b (0x61e8 bytes) 000130 1777421475.778596 mapread 0x007b69 thru 0x010f20 (0x93b8 bytes) 000131 1777421475.779767 write 0x039902 thru 0x03ffff (0x66fe bytes) 000132 1777421475.786979 read 0x020f0b thru 0x02b957 (0xaa4d bytes) 000133 1777421475.801581 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3c2372a9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x79a550b1 Mirror 3: 0x79a550b1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000135 1777421476.289077 mapread 0x0009f9 thru 0x007c33 (0x723b bytes) 000137 1777421476.323640 trunc from 0x040000 to 0x0349f7 (0xb608 bytes) 000138 1777421476.377041 mapread 0x003845 thru 0x0094fd (0x5cb9 bytes) 000143 1777421476.393560 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x349f8): Mirror 3: 0x58145b51 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x349f8): Mirror 3: 0x58145b51 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x349f8: skipped 000144 1777421476.578841 mapread 0x02cac2 thru 0x03229c (0x57db bytes) 000147 1777421476.595681 read 0x034842 thru 0x0349f7 (0x1b6 bytes) 000148 1777421476.596376 read 0x02995a thru 0x031ecf (0x8576 bytes) 000149 1777421476.597032 write 0x03011e thru 0x03ca73 (0xc956 bytes) 000151 1777421476.604058 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ca74): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ca74): Mirror 3: 0xac252fff (stale) CRC-32 checksum value for chunk [0, 0x3ca74): Mirror 1: 0x53c60930 Mirror 4: 0x53c60930 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ca74: skipped 000152 1777421477.048364 write 0x0016bc thru 0x009f7a (0x88bf bytes) 000156 1777421477.110187 write 0x01c75e thru 0x025b95 (0x9438 bytes) 000158 1777421477.123056 mapread 0x0091a2 thru 0x0178fc (0xe75b bytes) 000160 1777421477.133557 write 0x02d40c thru 0x034a63 (0x7658 bytes) 000161 1777421477.139967 mapwrite 0x03a9e1 thru 0x03e2b9 (0x38d9 bytes) 000163 1777421477.226758 trunc from 0x03e2ba to 0x019d06 (0x245b3 bytes) 000165 1777421477.237690 write 0x008871 thru 0x00f464 (0x6bf4 bytes) 000166 1777421477.238843 read 0x0115d8 thru 0x019d06 (0x872f bytes) 000168 1777421477.240015 write 0x02da45 thru 0x038661 (0xac1d bytes) 000169 1777421477.241412 mapread 0x02d9c7 thru 0x032799 (0x4dd3 bytes) 000170 1777421477.242211 read 0x03038e thru 0x038661 (0x82d4 bytes) 000171 1777421477.247741 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x38662): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38662): Mirror 3: 0xff2809d7 (stale) Mirror 4: 0x6774c837 (stale) CRC-32 checksum value for chunk [0, 0x38662): Mirror 1: 0x6bc95033 Mirror 5: 0x6bc95033 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38662: skipped 000174 1777421477.691973 trunc from 0x038662 to 0x03b60d (0x2fac bytes) 000175 1777421477.769686 read 0x02fa61 thru 0x03b60d (0xbbad bytes) 000178 1777421477.781653 write 0x00c25b thru 0x01832c (0xc0d2 bytes) 000180 1777421477.793940 write 0x00bc5d thru 0x014d01 (0x90a5 bytes) 000181 1777421477.801976 write 0x00f7e5 thru 0x013f35 (0x4751 bytes) 000184 1777421477.804246 mapread 0x0390fb thru 0x03b60d (0x2513 bytes) 000185 1777421477.805405 write 0x000a54 thru 0x0050e0 (0x468d bytes) 000188 1777421477.814143 trunc from 0x03b60e to 0x02c82e (0xeddf bytes) 000190 1777421477.838027 trunc from 0x02c82f to 0x033a20 (0x71f2 bytes) 000191 1777421477.852710 MIRROR_RESYNC 4 mirrors 000192 1777421478.273208 mapread 0x006bef thru 0x00ab4f (0x3f61 bytes) 000193 1777421478.286581 mapwrite 0x0035d1 thru 0x0057a3 (0x21d3 bytes) 000195 1777421478.379248 write 0x03e089 thru 0x03ffff (0x1f77 bytes) 000196 1777421478.381257 mapread 0x011367 thru 0x018f82 (0x7c1c bytes) 000197 1777421478.392621 write 0x0051ce thru 0x00e490 (0x92c3 bytes) 000198 1777421478.394192 mapread 0x019d74 thru 0x021953 (0x7be0 bytes) 000199 1777421478.407249 trunc from 0x040000 to 0x01e7ce (0x21831 bytes) 000200 1777421478.421267 mapread 0x01288d thru 0x01a209 (0x797d bytes) 000203 1777421478.423225 read 0x00c8d6 thru 0x0151e8 (0x8913 bytes) 000204 1777421478.423833 read 0x0049ad thru 0x0065fb (0x1c4f bytes) 000205 1777421478.424304 mapread 0x0163c8 thru 0x01be40 (0x5a79 bytes) 000207 1777421478.425530 read 0x008ce1 thru 0x014910 (0xbc30 bytes) 000208 1777421478.426256 mapwrite 0x025d2d thru 0x0281f4 (0x24c8 bytes) 000210 1777421478.474455 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x281f5): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x281f5): Mirror 3: 0x1c290013 (stale) Mirror 4: 0x1c290013 (stale) Mirror 5: 0x1c290013 (stale) CRC-32 checksum value for chunk [0, 0x281f5): Mirror 1: 0x818a7f72 Mirror 6: 0x818a7f72 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x281f5: skipped 000211 1777421478.870280 trunc from 0x0281f5 to 0x0020bb (0x26139 bytes) 000213 1777421478.939390 read 0x001a85 thru 0x0020bb (0x637 bytes) 000215 1777421478.944614 trunc from 0x0020bc to 0x02bfe7 (0x29f2c bytes) 000219 1777421478.959538 trunc from 0x02bfe8 to 0x01dfb1 (0xe036 bytes) 000220 1777421478.968895 mapread 0x0177b8 thru 0x01bdfe (0x4647 bytes) 000221 1777421478.984274 MIRROR_RESYNC 5 mirrors 000224 1777421479.431424 mapread 0x01232a thru 0x018af3 (0x67ca bytes) 000225 1777421479.442725 mapread 0x005bfb thru 0x009c20 (0x4026 bytes) 000226 1777421479.448622 mapread 0x012d6a thru 0x01dc0d (0xaea4 bytes) 000227 1777421479.452582 read 0x0091f9 thru 0x00f681 (0x6489 bytes) 000230 1777421479.453837 mapread 0x01a755 thru 0x01dfb1 (0x385d bytes) 000232 1777421479.456647 MIRROR_EXTEND to 6 mirrors 000234 1777421479.796227 MIRROR_RESYNC 6 mirrors 000239 1777421480.040548 mapread 0x009834 thru 0x00de97 (0x4664 bytes) 000243 1777421480.053136 trunc from 0x01dfb2 to 0x031003 (0x13052 bytes) 000244 1777421480.095906 read 0x017782 thru 0x01d633 (0x5eb2 bytes) 000245 1777421480.104905 mapwrite 0x0170fb thru 0x025b82 (0xea88 bytes) 000248 1777421480.154898 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x31004): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31004): Mirror 3: 0x3bb810be (stale) Mirror 4: 0x3bb810be (stale) Mirror 5: 0x3bb810be (stale) Mirror 6: 0x3bb810be (stale) Mirror 7: 0x3bb810be (stale) CRC-32 checksum value for chunk [0, 0x31004): Mirror 1: 0x4ac498f2 Mirror 8: 0x4ac498f2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31004: skipped 000249 1777421480.572018 write 0x012f92 thru 0x019a27 (0x6a96 bytes) 000250 1777421480.613878 trunc from 0x031004 to 0x00e4f0 (0x22b13 bytes) 000252 1777421480.640193 mapread 0x007b2c thru 0x00e4f0 (0x69c5 bytes) 000253 1777421480.646262 mapread 0x00b562 thru 0x00e4f0 (0x2f8f bytes) 000255 1777421480.651221 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xe4f1): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe4f1): Mirror 3: 0x16d1873 (stale) Mirror 4: 0x16d1873 (stale) Mirror 5: 0x16d1873 (stale) Mirror 6: 0x16d1873 (stale) Mirror 7: 0x16d1873 (stale) Mirror 8: 0x16d1873 (stale) CRC-32 checksum value for chunk [0, 0xe4f1): Mirror 1: 0x16d1873 Mirror 9: 0x16d1873 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe4f1: skipped 000256 1777421481.142193 mapread 0x00d6bc thru 0x00e4f0 (0xe35 bytes) 000257 1777421481.149614 read 0x00ca9f thru 0x00e4f0 (0x1a52 bytes) 000258 1777421481.155312 MIRROR_RESYNC 8 mirrors 000259 1777421481.792514 mapwrite 0x03e8f4 thru 0x03ffff (0x170c bytes) 000262 1777421481.879189 MIRROR_RESYNC 8 mirrors 000263 1777421482.613411 read 0x017e3e thru 0x027013 (0xf1d6 bytes) 000264 1777421482.626934 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) 000265 1777421482.831790 MIRROR_EXTEND to 8 mirrors 000266 1777421483.278766 mapread 0x03c041 thru 0x03ffff (0x3fbf bytes) 000267 1777421483.296707 read 0x03469d thru 0x03df2b (0x988f bytes) 000268 1777421483.304648 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) 000269 1777421483.524789 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) 000270 1777421483.726323 trunc from 0x040000 to 0x023146 (0x1ceb9 bytes) 000271 1777421483.771596 mapread 0x008705 thru 0x00d4ce (0x4dca bytes) 000273 1777421483.787948 mapwrite 0x03e2aa thru 0x03ffff (0x1d56 bytes) 000274 1777421483.833624 write 0x03c2bc thru 0x03ffff (0x3d44 bytes) 000276 1777421483.835053 read 0x01f541 thru 0x029cc7 (0xa787 bytes) 000277 1777421483.842351 read 0x021a1e thru 0x023d06 (0x22e9 bytes) 000279 1777421483.843795 mapread 0x0311ba thru 0x03a76d (0x95b4 bytes) 000280 1777421483.854948 trunc from 0x040000 to 0x00703a (0x38fc5 bytes) 000281 1777421483.868339 MIRROR_RESYNC 6 mirrors 000283 1777421484.428323 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) 000287 1777421484.609050 write 0x015ff5 thru 0x01c655 (0x6661 bytes) 000288 1777421484.647306 trunc from 0x01c656 to 0x0146bc (0x7f99 bytes) 000290 1777421484.662061 write 0x03bfc1 thru 0x03fc3e (0x3c7e bytes) 000291 1777421484.663866 mapwrite 0x012b0a thru 0x015700 (0x2bf7 bytes) 000292 1777421484.709787 mapwrite 0x033126 thru 0x03f22b (0xc106 bytes) 000293 1777421484.748431 mapwrite 0x038918 thru 0x03ffff (0x76e8 bytes) 000294 1777421484.792873 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd1f5642e (stale) Mirror 8: 0xd1f5642e (stale) Mirror 10: 0xd1f5642e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd1f5642e (stale) Mirror 8: 0xd1f5642e (stale) Mirror 10: 0xd1f5642e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000295 1777421484.991431 read 0x036857 thru 0x03e346 (0x7af0 bytes) 000296 1777421484.998356 read 0x001325 thru 0x0106a4 (0xf380 bytes) 000297 1777421485.006003 write 0x0208ef thru 0x022696 (0x1da8 bytes) 000298 1777421485.007028 write 0x024864 thru 0x02fe27 (0xb5c4 bytes) 000299 1777421485.011478 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xd1f5642e (stale) Mirror 10: 0xd1f5642e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xd1f5642e (stale) Mirror 10: 0xd1f5642e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000301 1777421485.223971 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xd1f5642e (stale) Mirror 10: 0xd1f5642e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xba1f0b47 Mirror 11: 0xba1f0b47 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000302 1777421485.604008 read 0x00eadd thru 0x01b004 (0xc528 bytes) 000303 1777421485.610052 write 0x00b3fd thru 0x018f6f (0xdb73 bytes) 000304 1777421485.652558 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xd1f5642e (stale) Mirror 10: 0xd1f5642e (stale) Mirror 11: 0xba1f0b47 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb345895f Mirror 12: 0xb345895f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000306 1777421486.065248 read 0x00ddc1 thru 0x00e7d1 (0xa11 bytes) 000309 1777421486.070886 trunc from 0x040000 to 0x0392d2 (0x6d2d bytes) 000310 1777421486.122441 write 0x0223c9 thru 0x0290a0 (0x6cd8 bytes) 000312 1777421486.130941 read 0x01ecf2 thru 0x02d8f3 (0xec02 bytes) 000314 1777421486.138290 trunc from 0x0392d3 to 0x01c62b (0x1cca7 bytes) 000315 1777421486.149165 read 0x015bf2 thru 0x01c62b (0x6a3a bytes) 000316 1777421486.149921 write 0x00e95e thru 0x01120e (0x28b1 bytes) 000318 1777421486.151217 write 0x0149da thru 0x01d4a3 (0x8aca bytes) 000319 1777421486.152384 mapwrite 0x022b5a thru 0x0265e2 (0x3a89 bytes) truncating to largest ever: 0x3ec58 000320 1777421486.189511 trunc from 0x0265e3 to 0x03ec57 (0x18675 bytes) 000321 1777421486.196045 write 0x02f989 thru 0x038410 (0x8a88 bytes) 000324 1777421486.203554 read 0x018928 thru 0x02806a (0xf743 bytes) 000328 1777421486.206013 write 0x003fff thru 0x00fee4 (0xbee6 bytes) 000329 1777421486.207344 write 0x00846b thru 0x0161aa (0xdd40 bytes) 000332 1777421486.210016 write 0x03655c thru 0x03d8b6 (0x735b bytes) 000333 1777421486.211511 write 0x01e4a8 thru 0x023d6c (0x58c5 bytes) 000335 1777421486.213082 read 0x0381e0 thru 0x03c252 (0x4073 bytes) 000337 1777421486.213931 write 0x0358c9 thru 0x038b79 (0x32b1 bytes) 000339 1777421486.215430 mapread 0x022c08 thru 0x027a0d (0x4e06 bytes) 000341 1777421486.217078 mapwrite 0x008cab thru 0x00de45 (0x519b bytes) 000342 1777421486.243706 read 0x00e8d5 thru 0x010d43 (0x246f bytes) 000344 1777421486.245001 mapread 0x031e01 thru 0x03ec57 (0xce57 bytes) 000345 1777421486.246270 trunc from 0x03ec58 to 0x00934b (0x3590c bytes) 000348 1777421486.257076 trunc from 0x00934c to 0x022e21 (0x19ad6 bytes) 000349 1777421486.271001 MIRROR_RESYNC 5 mirrors 000351 1777421486.731739 MIRROR_RESYNC 5 mirrors 000352 1777421486.924884 MIRROR_RESYNC 5 mirrors 000354 1777421487.130722 mapread 0x00d13a thru 0x0196a1 (0xc568 bytes) 000356 1777421487.153320 MIRROR_RESYNC 5 mirrors 000357 1777421487.348959 MIRROR_RESYNC 5 mirrors 000364 1777421487.630818 trunc from 0x022e22 to 0x019f0c (0x8f15 bytes) 000365 1777421487.677609 mapwrite 0x01dced thru 0x025c2b (0x7f3f bytes) 000367 1777421487.732610 mapread 0x00ca17 thru 0x014235 (0x781f bytes) 000368 1777421487.749692 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x25c2c): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25c2c): Mirror 8: 0x1d15ad4a (stale) Mirror 10: 0x1d15ad4a (stale) Mirror 11: 0x1d15ad4a (stale) Mirror 12: 0x1d15ad4a (stale) CRC-32 checksum value for chunk [0, 0x25c2c): Mirror 1: 0x577250e5 Mirror 13: 0x577250e5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25c2c: skipped 000370 1777421488.192919 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x25c2c): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25c2c): Mirror 8: 0x1d15ad4a (stale) Mirror 10: 0x1d15ad4a (stale) Mirror 11: 0x1d15ad4a (stale) Mirror 12: 0x1d15ad4a (stale) CRC-32 checksum value for chunk [0, 0x25c2c): Mirror 1: 0x577250e5 Mirror 13: 0x577250e5 Mirror 14: 0x577250e5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25c2c: skipped 000372 1777421488.632660 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x25c2c): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25c2c): Mirror 8: 0x1d15ad4a (stale) Mirror 10: 0x1d15ad4a (stale) Mirror 11: 0x1d15ad4a (stale) Mirror 12: 0x1d15ad4a (stale) CRC-32 checksum value for chunk [0, 0x25c2c): Mirror 1: 0x577250e5 Mirror 13: 0x577250e5 Mirror 14: 0x577250e5 Mirror 15: 0x577250e5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25c2c: skipped 000374 1777421489.152787 MIRROR_RESYNC 8 mirrors 000375 1777421489.633500 write 0x032647 thru 0x035d93 (0x374d bytes) 000376 1777421489.674224 trunc from 0x035d94 to 0x01c1f7 (0x19b9c bytes) 000377 1777421489.686422 trunc from 0x01c1f8 to 0x0349ec (0x187f5 bytes) 000379 1777421489.699233 mapwrite 0x003371 thru 0x003f09 (0xb99 bytes) 000380 1777421489.722027 mapwrite 0x01337b thru 0x01e270 (0xaef6 bytes) 000382 1777421489.764030 read 0x018888 thru 0x025b15 (0xd28e bytes) 000383 1777421489.765910 mapread 0x025201 thru 0x0349ec (0xf7ec bytes) 000384 1777421489.779953 write 0x036974 thru 0x03ffff (0x968c bytes) 000386 1777421489.781888 mapwrite 0x03111e thru 0x03df74 (0xce57 bytes) 000388 1777421489.821990 trunc from 0x040000 to 0x01df54 (0x220ab bytes) 000389 1777421489.840459 MIRROR_RESYNC 8 mirrors 000390 1777421490.420318 write 0x01747d thru 0x01cf2c (0x5ab0 bytes) 000393 1777421490.463847 trunc from 0x01df55 to 0x01da65 (0x4ef bytes) 000394 1777421490.474414 trunc from 0x01da66 to 0x03e035 (0x205d0 bytes) 000395 1777421490.481486 trunc from 0x03e036 to 0x01f6f3 (0x1e942 bytes) 000396 1777421490.490331 mapread 0x00b722 thru 0x013510 (0x7def bytes) 000398 1777421490.497103 read 0x018d01 thru 0x019ca7 (0xfa7 bytes) 000399 1777421490.497642 mapwrite 0x03b1b0 thru 0x03ffff (0x4e50 bytes) 000401 1777421490.532963 trunc from 0x040000 to 0x034643 (0xb9bc bytes) 000408 1777421490.545837 read 0x00eb11 thru 0x018ad8 (0x9fc8 bytes) 000410 1777421490.552556 mapwrite 0x01bcf4 thru 0x024396 (0x86a3 bytes) 000411 1777421490.581639 read 0x02dfdd thru 0x034643 (0x6667 bytes) 000412 1777421490.582429 write 0x023a9e thru 0x028ca3 (0x5206 bytes) 000413 1777421490.583501 mapread 0x011522 thru 0x015b0e (0x45ed bytes) 000415 1777421490.584537 mapwrite 0x0095c0 thru 0x0139e4 (0xa425 bytes) 000416 1777421490.612938 write 0x01e00d thru 0x02b5b5 (0xd5a9 bytes) 000417 1777421490.614399 trunc from 0x034644 to 0x00f058 (0x255eb bytes) 000418 1777421490.632663 read 0x005f1e thru 0x00f058 (0x913b bytes) 000419 1777421490.633473 mapwrite 0x037d84 thru 0x03ffff (0x827c bytes) 000420 1777421490.684221 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x61bd9b1f (stale) Mirror 11: 0x61bd9b1f (stale) Mirror 12: 0x61bd9b1f (stale) Mirror 13: 0x61bd9b1f (stale) Mirror 14: 0x61bd9b1f (stale) Mirror 15: 0x61bd9b1f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x61bd9b1f (stale) Mirror 11: 0x61bd9b1f (stale) Mirror 12: 0x61bd9b1f (stale) Mirror 13: 0x61bd9b1f (stale) Mirror 14: 0x61bd9b1f (stale) Mirror 15: 0x61bd9b1f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000421 1777421490.876901 write 0x00bebf thru 0x0185df (0xc721 bytes) 000423 1777421490.884576 write 0x01d794 thru 0x02c5c7 (0xee34 bytes) 000425 1777421490.894644 mapwrite 0x0094ca thru 0x010a8a (0x75c1 bytes) 000427 1777421490.922030 write 0x028c58 thru 0x03875a (0xfb03 bytes) 000429 1777421490.926221 mapread 0x02cf9d thru 0x03b18b (0xe1ef bytes) 000430 1777421490.931787 mapread 0x032367 thru 0x03d4fd (0xb197 bytes) 000431 1777421490.932949 mapread 0x00a5a8 thru 0x01a3c5 (0xfe1e bytes) truncating to largest ever: 0x3ff90 000433 1777421490.934522 trunc from 0x040000 to 0x03ff8f (0x70 bytes) 000434 1777421490.945578 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3ff90): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ff90): Mirror 10: 0x5add9a35 (stale) Mirror 11: 0x5add9a35 (stale) Mirror 12: 0x5add9a35 (stale) Mirror 13: 0x5add9a35 (stale) Mirror 14: 0x5add9a35 (stale) Mirror 15: 0x5add9a35 (stale) CRC-32 checksum value for chunk [0, 0x3ff90): Mirror 1: 0x932c5c9 Mirror 16: 0x932c5c9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ff90: skipped 000440 1777421491.384941 mapwrite 0x02d619 thru 0x02dad8 (0x4c0 bytes) 000441 1777421491.462131 mapread 0x002a9f thru 0x010bcb (0xe12d bytes) 000443 1777421491.472581 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ff90): Mirror 10: 0x5add9a35 (stale) Mirror 11: 0x5add9a35 (stale) Mirror 12: 0x5add9a35 (stale) Mirror 13: 0x5add9a35 (stale) Mirror 14: 0x5add9a35 (stale) Mirror 16: 0x932c5c9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ff90): Mirror 10: 0x5add9a35 (stale) Mirror 11: 0x5add9a35 (stale) Mirror 12: 0x5add9a35 (stale) Mirror 13: 0x5add9a35 (stale) Mirror 14: 0x5add9a35 (stale) Mirror 16: 0x932c5c9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ff90: skipped 000444 1777421491.696797 read 0x001973 thru 0x00ea50 (0xd0de bytes) 000445 1777421491.705806 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ff90): Mirror 10: 0x5add9a35 (stale) Mirror 11: 0x5add9a35 (stale) Mirror 12: 0x5add9a35 (stale) Mirror 13: 0x5add9a35 (stale) Mirror 16: 0x932c5c9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ff90): Mirror 10: 0x5add9a35 (stale) Mirror 11: 0x5add9a35 (stale) Mirror 12: 0x5add9a35 (stale) Mirror 13: 0x5add9a35 (stale) Mirror 16: 0x932c5c9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ff90: skipped 000446 1777421491.949950 trunc from 0x03ff90 to 0x01f73c (0x20853 bytes) 000447 1777421491.960468 read 0x01212a thru 0x01c559 (0xa430 bytes) 000449 1777421491.966482 trunc from 0x01f73d to 0x024d3c (0x5600 bytes) 000451 1777421491.974635 trunc from 0x024d3d to 0x009abe (0x1b27e bytes) 000452 1777421491.982926 read 0x0041d8 thru 0x009abe (0x58e7 bytes) 000453 1777421491.988769 mapread 0x007286 thru 0x009abe (0x2839 bytes) 000454 1777421491.989877 trunc from 0x009abf to 0x00a7ce (0xd10 bytes) 000455 1777421491.999679 write 0x019c26 thru 0x023ac9 (0x9ea4 bytes) 000457 1777421492.001725 read 0x0118fe thru 0x018103 (0x6806 bytes) 000459 1777421492.013006 mapwrite 0x0385f1 thru 0x03ffff (0x7a0f bytes) 000464 1777421492.068433 mapwrite 0x031aa1 thru 0x03e799 (0xccf9 bytes) 000467 1777421492.107014 write 0x00dca9 thru 0x0190dc (0xb434 bytes) 000468 1777421492.108267 read 0x0243d5 thru 0x02cbfc (0x8828 bytes) 000470 1777421492.109256 read 0x0261c4 thru 0x02df29 (0x7d66 bytes) 000472 1777421492.110345 write 0x005ec0 thru 0x01567f (0xf7c0 bytes) 000475 1777421492.112467 mapwrite 0x00df33 thru 0x010353 (0x2421 bytes) 000476 1777421492.137804 write 0x039104 thru 0x03daab (0x49a8 bytes) 000477 1777421492.139207 read 0x01c982 thru 0x022c2e (0x62ad bytes) 000478 1777421492.143251 MIRROR_RESYNC 6 mirrors 000479 1777421492.634049 read 0x034bd3 thru 0x03ffff (0xb42d bytes) 000481 1777421492.639738 write 0x00cd5b thru 0x01289d (0x5b43 bytes) 000482 1777421492.679627 read 0x02b239 thru 0x03102a (0x5df2 bytes) 000484 1777421492.686548 read 0x01a3dc thru 0x01b408 (0x102d bytes) 000485 1777421492.687236 mapread 0x0275f6 thru 0x0365f6 (0xf001 bytes) 000488 1777421492.693328 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x7987d96b (stale) Mirror 11: 0x7987d96b (stale) Mirror 13: 0x7987d96b (stale) Mirror 16: 0x7987d96b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x7987d96b (stale) Mirror 11: 0x7987d96b (stale) Mirror 13: 0x7987d96b (stale) Mirror 16: 0x7987d96b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000489 1777421492.911386 mapread 0x01e9ec thru 0x02ae8a (0xc49f bytes) 000490 1777421492.930606 MIRROR_RESYNC 5 mirrors 000492 1777421493.369828 read 0x01e646 thru 0x0215f5 (0x2fb0 bytes) 000493 1777421493.373550 read 0x025228 thru 0x0276bd (0x2496 bytes) 000496 1777421493.381550 trunc from 0x040000 to 0x037ae1 (0x851e bytes) 000497 1777421493.418893 mapwrite 0x00f4fa thru 0x016424 (0x6f2b bytes) 000498 1777421493.472650 write 0x0006d1 thru 0x00aaf0 (0xa420 bytes) 000500 1777421493.480779 read 0x0010f0 thru 0x0104c1 (0xf3d2 bytes) 000501 1777421493.488679 read 0x00f4ef thru 0x011c25 (0x2737 bytes) 000508 1777421493.493260 mapwrite 0x026aa1 thru 0x02f790 (0x8cf0 bytes) 000509 1777421493.523015 mapread 0x002a95 thru 0x008b6c (0x60d8 bytes) 000511 1777421493.524750 mapread 0x037869 thru 0x037ae1 (0x279 bytes) 000512 1777421493.525504 mapread 0x033a23 thru 0x037ae1 (0x40bf bytes) 000516 1777421493.530583 MIRROR_RESYNC 5 mirrors 000518 1777421493.914900 read 0x0288cf thru 0x03453a (0xbc6c bytes) 000520 1777421493.920335 trunc from 0x037ae2 to 0x035dbc (0x1d25 bytes) 000521 1777421493.973730 read 0x0193e3 thru 0x020c43 (0x7861 bytes) 000523 1777421493.981323 mapwrite 0x02fcc3 thru 0x03dbe5 (0xdf23 bytes) 000524 1777421494.032810 mapread 0x021e4a thru 0x02df2b (0xc0e2 bytes) 000525 1777421494.034099 trunc from 0x03dbe6 to 0x020cd6 (0x1cf0f bytes) 000526 1777421494.047727 read 0x02009e thru 0x020cd6 (0xc39 bytes) 000528 1777421494.054208 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20cd7): Mirror 10: 0x1b1a2c4a (stale) Mirror 11: 0x1b1a2c4a (stale) Mirror 16: 0x1b1a2c4a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20cd7): Mirror 10: 0x1b1a2c4a (stale) Mirror 11: 0x1b1a2c4a (stale) Mirror 16: 0x1b1a2c4a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20cd7: skipped 000532 1777421494.258652 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x20cd7): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20cd7): Mirror 10: 0x1b1a2c4a (stale) Mirror 11: 0x1b1a2c4a (stale) Mirror 16: 0x1b1a2c4a (stale) CRC-32 checksum value for chunk [0, 0x20cd7): Mirror 1: 0x1b1a2c4a Mirror 17: 0x1b1a2c4a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20cd7: skipped 000533 1777421494.693805 read 0x015434 thru 0x020cd6 (0xb8a3 bytes) 000534 1777421494.698579 write 0x0028ab thru 0x0047b2 (0x1f08 bytes) 000535 1777421494.742397 trunc from 0x020cd7 to 0x021358 (0x682 bytes) 000537 1777421494.760414 MIRROR_RESYNC 5 mirrors 000539 1777421495.201960 write 0x00a6da thru 0x013e91 (0x97b8 bytes) 000541 1777421495.258801 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21359): Mirror 10: 0xa204efc4 (stale) Mirror 11: 0xa204efc4 (stale) Mirror 17: 0xa204efc4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21359): Mirror 10: 0xa204efc4 (stale) Mirror 11: 0xa204efc4 (stale) Mirror 17: 0xa204efc4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21359: skipped 000542 1777421495.439059 read 0x0074c9 thru 0x013fc0 (0xcaf8 bytes) 000544 1777421495.446531 mapwrite 0x03317e thru 0x038ece (0x5d51 bytes) 000546 1777421495.496300 trunc from 0x038ecf to 0x03b8de (0x2a10 bytes) 000547 1777421495.509490 MIRROR_RESYNC 4 mirrors 000548 1777421495.934035 read 0x00506a thru 0x00eb9a (0x9b31 bytes) 000550 1777421495.944175 MIRROR_RESYNC 4 mirrors 000551 1777421496.133536 read 0x00bc54 thru 0x01ae98 (0xf245 bytes) 000552 1777421496.141454 trunc from 0x03b8df to 0x01d82d (0x1e0b1 bytes) 000553 1777421496.227816 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d82e): Mirror 10: 0xe95f5be1 (stale) Mirror 11: 0xe95f5be1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d82e): Mirror 10: 0xe95f5be1 (stale) Mirror 11: 0xe95f5be1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d82e: skipped 000554 1777421496.415702 write 0x0345b2 thru 0x03bf69 (0x79b8 bytes) 000555 1777421496.417088 mapread 0x008efd thru 0x013884 (0xa988 bytes) 000556 1777421496.440377 mapread 0x00933c thru 0x00ba50 (0x2715 bytes) 000559 1777421496.442513 read 0x003e39 thru 0x006799 (0x2961 bytes) 000560 1777421496.443431 mapread 0x00041c thru 0x003344 (0x2f29 bytes) 000561 1777421496.449453 read 0x013014 thru 0x015103 (0x20f0 bytes) 000562 1777421496.450315 read 0x00550f thru 0x0081c5 (0x2cb7 bytes) 000564 1777421496.451542 write 0x009844 thru 0x011e92 (0x864f bytes) 000565 1777421496.453690 mapread 0x02343d thru 0x031baf (0xe773 bytes) 000566 1777421496.461330 mapread 0x01286b thru 0x0222c4 (0xfa5a bytes) 000569 1777421496.471825 MIRROR_RESYNC 3 mirrors 000570 1777421496.877206 MIRROR_RESYNC 3 mirrors 000571 1777421497.077506 read 0x02179b thru 0x025c32 (0x4498 bytes) 000572 1777421497.083837 mapwrite 0x02362b thru 0x026c6f (0x3645 bytes) 000575 1777421497.182029 write 0x008291 thru 0x011e08 (0x9b78 bytes) 000578 1777421497.195599 mapread 0x01955d thru 0x01a3e3 (0xe87 bytes) 000579 1777421497.203216 write 0x02f84d thru 0x032c93 (0x3447 bytes) 000581 1777421497.208767 trunc from 0x03bf6a to 0x00b6b1 (0x308b8 bytes) 000582 1777421497.244761 mapwrite 0x0125a0 thru 0x01ba2a (0x948b bytes) 000584 1777421497.303511 MIRROR_RESYNC 3 mirrors 000585 1777421497.719459 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) 000586 1777421497.905228 read 0x001e5b thru 0x00b90b (0x9ab1 bytes) 000590 1777421497.912410 mapwrite 0x00905b thru 0x00fa2e (0x69d4 bytes) 000591 1777421498.044229 trunc from 0x01ba2b to 0x005e01 (0x15c29 bytes) 000593 1777421498.069230 trunc from 0x005e02 to 0x03f0bd (0x392bc bytes) 000594 1777421498.078342 write 0x01cc08 thru 0x01fcba (0x30b3 bytes) 000596 1777421498.086434 mapread 0x03848f thru 0x03f0bd (0x6c2f bytes) 000598 1777421498.093371 trunc from 0x03f0be to 0x01f7c9 (0x1f8f4 bytes) 000599 1777421498.107383 mapread 0x01d6c7 thru 0x01f7c9 (0x2103 bytes) 000600 1777421498.108354 read 0x007865 thru 0x00ae1b (0x35b7 bytes) 000601 1777421498.117560 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1f7ca): Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f7ca): Mirror 11: 0x97abc2bb (stale) CRC-32 checksum value for chunk [0, 0x1f7ca): Mirror 1: 0x9c79fea9 Mirror 12: 0x9c79fea9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f7ca: skipped 000604 1777421498.576916 MIRROR_RESYNC 3 mirrors 000605 1777421498.917487 read 0x017bbc thru 0x01f7c9 (0x7c0e bytes) 000608 1777421498.924309 write 0x03ddea thru 0x03ffff (0x2216 bytes) 000610 1777421498.975784 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xdf347b7e (stale) Mirror 12: 0xdf347b7e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x96cdde01 Mirror 13: 0x96cdde01 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000616 1777421499.474967 read 0x010f15 thru 0x0175c3 (0x66af bytes) 000617 1777421499.485511 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xdf347b7e (stale) Mirror 12: 0xdf347b7e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x96cdde01 Mirror 13: 0x96cdde01 Mirror 14: 0x96cdde01 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000619 1777421499.930684 read 0x010460 thru 0x0114b6 (0x1057 bytes) 000620 1777421499.936499 read 0x01275e thru 0x01859f (0x5e42 bytes) 000624 1777421499.948902 mapwrite 0x0357d6 thru 0x03b9e3 (0x620e bytes) 000625 1777421500.057724 write 0x01ca8d thru 0x0245ff (0x7b73 bytes) 000626 1777421500.067898 trunc from 0x040000 to 0x01ff9e (0x20061 bytes) 000629 1777421500.098489 trunc from 0x01ff9f to 0x01dbbc (0x23e2 bytes) 000631 1777421500.114930 MIRROR_RESYNC 5 mirrors 000632 1777421500.612609 MIRROR_EXTEND to 6 mirrors 000636 1777421501.007029 write 0x024a32 thru 0x02ca5f (0x802e bytes) 000637 1777421501.047861 trunc from 0x02ca60 to 0x00bf20 (0x20b3f bytes) 000639 1777421501.061949 trunc from 0x00bf21 to 0x017438 (0xb518 bytes) 000645 1777421501.080578 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x17439): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17439): Mirror 11: 0xba2b2c82 (stale) Mirror 12: 0xba2b2c82 (stale) Mirror 13: 0xba2b2c82 (stale) Mirror 14: 0xba2b2c82 (stale) Mirror 15: 0xba2b2c82 (stale) CRC-32 checksum value for chunk [0, 0x17439): Mirror 1: 0xba2b2c82 Mirror 16: 0xba2b2c82 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17439: skipped 000647 1777421501.570097 mapwrite 0x01bf15 thru 0x02b9a5 (0xfa91 bytes) 000648 1777421501.667331 mapread 0x00d88f thru 0x01bf49 (0xe6bb bytes) 000652 1777421501.678870 trunc from 0x02b9a6 to 0x022c71 (0x8d34 bytes) 000654 1777421501.698799 read 0x01125a thru 0x020a7f (0xf826 bytes) 000655 1777421501.699602 mapread 0x00324b thru 0x007516 (0x42cc bytes) 000656 1777421501.707157 write 0x00718e thru 0x00f350 (0x81c3 bytes) 000657 1777421501.708035 mapread 0x014b41 thru 0x015e5d (0x131d bytes) 000659 1777421501.709196 mapread 0x00ca9d thru 0x01868f (0xbbf3 bytes) 000661 1777421501.710909 mapwrite 0x0012af thru 0x004715 (0x3467 bytes) 000662 1777421501.736614 mapwrite 0x0034de thru 0x00d381 (0x9ea4 bytes) 000663 1777421501.759804 trunc from 0x022c72 to 0x03f890 (0x1cc1f bytes) 000664 1777421501.771469 trunc from 0x03f891 to 0x02bc34 (0x13c5c bytes) 000667 1777421501.781730 mapread 0x00a855 thru 0x00dcc0 (0x346c bytes) 000668 1777421501.782724 mapwrite 0x02ed18 thru 0x031252 (0x253b bytes) 000669 1777421501.828591 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31253): Mirror 12: 0xe5da4a05 (stale) Mirror 13: 0xe5da4a05 (stale) Mirror 14: 0xe5da4a05 (stale) Mirror 15: 0xe5da4a05 (stale) Mirror 16: 0x6366a12 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31253): Mirror 12: 0xe5da4a05 (stale) Mirror 13: 0xe5da4a05 (stale) Mirror 14: 0xe5da4a05 (stale) Mirror 15: 0xe5da4a05 (stale) Mirror 16: 0x6366a12 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31253: skipped 000670 1777421502.059921 trunc from 0x031253 to 0x013f4c (0x1d306 bytes) 000671 1777421502.072856 mapwrite 0x027f1c thru 0x032972 (0xaa57 bytes) 000673 1777421502.146747 write 0x005d2d thru 0x014a8e (0xed62 bytes) 000674 1777421502.159478 mapwrite 0x03ec65 thru 0x03ffff (0x139b bytes) 000675 1777421502.213541 write 0x034b1c thru 0x039aea (0x4fcf bytes) 000676 1777421502.218501 trunc from 0x040000 to 0x00be34 (0x341cb bytes) 000679 1777421502.231399 mapread 0x005bda thru 0x00be34 (0x625b bytes) 000680 1777421502.235417 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xbe35): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbe35): Mirror 12: 0xdb73b1d1 (stale) Mirror 13: 0xdb73b1d1 (stale) Mirror 14: 0xdb73b1d1 (stale) Mirror 15: 0xdb73b1d1 (stale) Mirror 16: 0xdb73b1d1 (stale) CRC-32 checksum value for chunk [0, 0xbe35): Mirror 1: 0xb52d539 Mirror 17: 0xb52d539 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbe35: skipped 000681 1777421502.631667 trunc from 0x00be35 to 0x01ae94 (0xf060 bytes) 000684 1777421502.694405 trunc from 0x01ae95 to 0x032b10 (0x17c7c bytes) 000686 1777421502.711159 mapwrite 0x01fefc thru 0x02e51c (0xe621 bytes) 000687 1777421502.769494 MIRROR_RESYNC 7 mirrors 000690 1777421503.485648 mapwrite 0x004b38 thru 0x00d5c1 (0x8a8a bytes) 000691 1777421503.626677 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32b11): Mirror 13: 0xbe450c6f (stale) Mirror 14: 0xbe450c6f (stale) Mirror 15: 0xbe450c6f (stale) Mirror 16: 0xbe450c6f (stale) Mirror 17: 0xbe450c6f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32b11): Mirror 13: 0xbe450c6f (stale) Mirror 14: 0xbe450c6f (stale) Mirror 15: 0xbe450c6f (stale) Mirror 16: 0xbe450c6f (stale) Mirror 17: 0xbe450c6f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32b11: skipped 000692 1777421503.916171 mapread 0x019900 thru 0x027fa7 (0xe6a8 bytes) 000693 1777421503.937237 mapwrite 0x009cf9 thru 0x00ded9 (0x41e1 bytes) 000694 1777421503.971302 read 0x02d65e thru 0x032b10 (0x54b3 bytes) 000696 1777421503.981410 read 0x00b3df thru 0x012103 (0x6d25 bytes) 000697 1777421503.987731 trunc from 0x032b11 to 0x0149b4 (0x1e15c bytes) 000698 1777421503.998851 write 0x00ab2a thru 0x015caa (0xb181 bytes) 000699 1777421503.999939 write 0x012f28 thru 0x01b103 (0x81dc bytes) 000701 1777421504.002160 trunc from 0x01b104 to 0x02da50 (0x1294d bytes) 000703 1777421504.014333 read 0x003bbd thru 0x00fb88 (0xbfcc bytes) 000705 1777421504.021331 MIRROR_RESYNC 6 mirrors 000706 1777421504.575791 trunc from 0x02da51 to 0x0245fc (0x9454 bytes) 000708 1777421504.652923 mapread 0x00b5cb thru 0x0161de (0xac14 bytes) 000709 1777421504.681358 read 0x001ffa thru 0x00e70c (0xc713 bytes) 000711 1777421504.693096 MIRROR_RESYNC 6 mirrors 000712 1777421505.257282 read 0x01100e thru 0x01a2e4 (0x92d7 bytes) 000713 1777421505.264353 write 0x0090fe thru 0x0113af (0x82b2 bytes) 000715 1777421505.314911 mapwrite 0x02b941 thru 0x0371e7 (0xb8a7 bytes) 000717 1777421505.387033 mapwrite 0x021628 thru 0x02c289 (0xac62 bytes) 000718 1777421505.449544 trunc from 0x0371e8 to 0x01c856 (0x1a991 bytes) 000722 1777421505.469071 read 0x019df1 thru 0x01c856 (0x2a66 bytes) 000723 1777421505.479864 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c857): Mirror 14: 0xfacfb2d1 (stale) Mirror 15: 0xfacfb2d1 (stale) Mirror 16: 0xfacfb2d1 (stale) Mirror 17: 0xfacfb2d1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c857): Mirror 14: 0xfacfb2d1 (stale) Mirror 15: 0xfacfb2d1 (stale) Mirror 16: 0xfacfb2d1 (stale) Mirror 17: 0xfacfb2d1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c857: skipped 000724 1777421505.718675 mapwrite 0x026cd7 thru 0x0279f5 (0xd1f bytes) 000725 1777421505.767180 read 0x005c1e thru 0x006bdb (0xfbe bytes) 000727 1777421505.772499 read 0x023554 thru 0x02768c (0x4139 bytes) 000728 1777421505.781408 mapread 0x0021c3 thru 0x0028a9 (0x6e7 bytes) 000729 1777421505.785764 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x279f6): Mirror 15: 0x7350beac (stale) Mirror 16: 0x7350beac (stale) Mirror 17: 0x7350beac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x279f6): Mirror 15: 0x7350beac (stale) Mirror 16: 0x7350beac (stale) Mirror 17: 0x7350beac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x279f6: skipped 000731 1777421506.014635 mapread 0x017e62 thru 0x024c12 (0xcdb1 bytes) 000732 1777421506.039593 mapwrite 0x02cec4 thru 0x0307ce (0x390b bytes) 000733 1777421506.100941 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x307cf): Mirror 16: 0x8acf57bc (stale) Mirror 17: 0x8acf57bc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x307cf): Mirror 16: 0x8acf57bc (stale) Mirror 17: 0x8acf57bc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x307cf: skipped 000736 1777421506.271706 mapread 0x022484 thru 0x026f6d (0x4aea bytes) 000737 1777421506.284011 write 0x01bc4e thru 0x02043f (0x47f2 bytes) 000739 1777421506.292329 read 0x014730 thru 0x01e7f8 (0xa0c9 bytes) 000740 1777421506.293083 write 0x00d37d thru 0x01196a (0x45ee bytes) 000742 1777421506.299015 mapwrite 0x018c98 thru 0x019e6b (0x11d4 bytes) 000744 1777421506.333725 write 0x018c64 thru 0x01a08f (0x142c bytes) 000745 1777421506.335666 mapread 0x026b89 thru 0x0307ce (0x9c46 bytes) 000746 1777421506.348681 MIRROR_RESYNC 3 mirrors 000747 1777421506.720755 MIRROR_RESYNC 3 mirrors 000749 1777421506.907343 write 0x031893 thru 0x032f21 (0x168f bytes) 000750 1777421506.946301 mapread 0x0235f5 thru 0x024bad (0x15b9 bytes) 000752 1777421506.955340 write 0x03b7fa thru 0x03ffff (0x4806 bytes) 000753 1777421506.960653 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xeb8dd2e7 (stale) Mirror 17: 0xeb8dd2e7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa76bb4d2 Mirror 18: 0xa76bb4d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000756 1777421507.430839 write 0x0149de thru 0x022551 (0xdb74 bytes) 000757 1777421507.496149 write 0x01ef2f thru 0x0218f9 (0x29cb bytes) 000758 1777421507.497195 read 0x038285 thru 0x03aec0 (0x2c3c bytes) 000760 1777421507.503054 read 0x028ceb thru 0x036aea (0xde00 bytes) 000762 1777421507.511864 mapwrite 0x011de5 thru 0x014aad (0x2cc9 bytes) 000764 1777421507.541607 write 0x00b0ed thru 0x017ab9 (0xc9cd bytes) 000766 1777421507.543796 mapread 0x029b3c thru 0x0356a1 (0xbb66 bytes) 000768 1777421507.545356 mapwrite 0x000bfa thru 0x0041f4 (0x35fb bytes) 000769 1777421507.575589 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xeb8dd2e7 (stale) Mirror 18: 0xa76bb4d2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xeb8dd2e7 (stale) Mirror 18: 0xa76bb4d2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000770 1777421507.754376 mapread 0x0264e6 thru 0x02e12e (0x7c49 bytes) 000774 1777421507.774391 mapwrite 0x01c22a thru 0x01cf21 (0xcf8 bytes) 000775 1777421507.805715 read 0x02c9b2 thru 0x034e01 (0x8450 bytes) 000776 1777421507.811199 trunc from 0x040000 to 0x007d65 (0x3829a bytes) 000777 1777421507.824590 mapwrite 0x029f76 thru 0x039d98 (0xfe23 bytes) 000778 1777421507.891630 trunc from 0x039d99 to 0x01c939 (0x1d45f bytes) 000781 1777421507.906394 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1c93a): Mirror 16: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c93a): Mirror 16: 0x63f87c5c (stale) Mirror 18: 0x63f87c5c (stale) CRC-32 checksum value for chunk [0, 0x1c93a): Mirror 1: 0xffffd408 Mirror 19: 0xffffd408 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c93a: skipped 000783 1777421508.359279 read 0x011623 thru 0x01c939 (0xb317 bytes) 000786 1777421508.366596 mapread 0x004cc1 thru 0x00eb20 (0x9e60 bytes) 000789 1777421508.388920 mapwrite 0x023586 thru 0x026a8e (0x3509 bytes) 000790 1777421508.507387 MIRROR_RESYNC 4 mirrors 000791 1777421508.986315 MIRROR_EXTEND to 5 mirrors 000794 1777421509.413758 MIRROR_EXTEND to 6 mirrors 000795 1777421509.802802 mapwrite 0x02c761 thru 0x037fea (0xb88a bytes) 000796 1777421509.909809 mapwrite 0x0162b2 thru 0x01ed4a (0x8a99 bytes) 000798 1777421509.946398 mapread 0x00c018 thru 0x0187bd (0xc7a6 bytes) 000799 1777421509.955156 mapwrite 0x0035df thru 0x00949d (0x5ebf bytes) 000802 1777421509.991490 write 0x0216ac thru 0x028445 (0x6d9a bytes) 000804 1777421509.999233 mapread 0x035a60 thru 0x037fea (0x258b bytes) 000807 1777421510.000904 mapread 0x0370ba thru 0x037fea (0xf31 bytes) 000809 1777421510.006669 MIRROR_RESYNC 6 mirrors 000810 1777421510.564701 read 0x02b829 thru 0x034bc3 (0x939b bytes) 000811 1777421510.573977 trunc from 0x037feb to 0x01ca8f (0x1b55b bytes) 000812 1777421510.621598 mapwrite 0x011e20 thru 0x01a998 (0x8b79 bytes) 000813 1777421510.668833 write 0x029b4b thru 0x033384 (0x983a bytes) 000814 1777421510.670557 read 0x0286a1 thru 0x02d7b2 (0x5112 bytes) 000815 1777421510.677663 read 0x00a8c6 thru 0x010b6d (0x62a8 bytes) 000816 1777421510.678362 read 0x002c5b thru 0x00c18e (0x9534 bytes) 000818 1777421510.679492 read 0x01017b thru 0x0125b3 (0x2439 bytes) 000820 1777421510.680760 mapread 0x00e1c8 thru 0x019529 (0xb362 bytes) 000821 1777421510.682169 mapwrite 0x02a736 thru 0x032986 (0x8251 bytes) 000823 1777421510.707265 mapread 0x02d95e thru 0x033384 (0x5a27 bytes) 000825 1777421510.714523 MIRROR_RESYNC 6 mirrors 000827 1777421511.228629 mapread 0x01832f thru 0x01c59c (0x426e bytes) 000829 1777421511.241874 read 0x027ae9 thru 0x02ba4d (0x3f65 bytes) 000830 1777421511.250167 read 0x013a56 thru 0x0210be (0xd669 bytes) 000832 1777421511.250875 write 0x037d93 thru 0x03ffff (0x826d bytes) 000837 1777421511.291660 read 0x0359f3 thru 0x0395f5 (0x3c03 bytes) 000838 1777421511.297683 mapwrite 0x00c4d2 thru 0x010f2a (0x4a59 bytes) 000839 1777421511.330450 mapwrite 0x005692 thru 0x009531 (0x3ea0 bytes) 000842 1777421511.373410 MIRROR_RESYNC 6 mirrors 000843 1777421511.877256 trunc from 0x040000 to 0x010165 (0x2fe9a bytes) 000844 1777421511.951389 write 0x00ee44 thru 0x012a9b (0x3c58 bytes) 000845 1777421511.958639 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x12a9c): Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12a9c): Mirror 16: 0xd3d51f31 (stale) Mirror 18: 0xd3d51f31 (stale) Mirror 19: 0xd3d51f31 (stale) Mirror 20: 0xd3d51f31 (stale) Mirror 21: 0xd3d51f31 (stale) CRC-32 checksum value for chunk [0, 0x12a9c): Mirror 1: 0x524fff37 Mirror 22: 0x524fff37 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12a9c: skipped 000846 1777421512.467469 read 0x00cdb8 thru 0x012a9b (0x5ce4 bytes) 000848 1777421512.488850 trunc from 0x012a9c to 0x0350ad (0x22612 bytes) 000852 1777421512.557215 read 0x01e075 thru 0x0252ca (0x7256 bytes) 000853 1777421512.567003 read 0x00f6c7 thru 0x014d03 (0x563d bytes) 000854 1777421512.567744 read 0x013660 thru 0x020aa3 (0xd444 bytes) 000855 1777421512.571294 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x350ae): Mirror 18: 0x41804fb9 (stale) Mirror 19: 0x41804fb9 (stale) Mirror 20: 0x41804fb9 (stale) Mirror 21: 0x41804fb9 (stale) Mirror 22: 0x6e3cba68 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x350ae): Mirror 18: 0x41804fb9 (stale) Mirror 19: 0x41804fb9 (stale) Mirror 20: 0x41804fb9 (stale) Mirror 21: 0x41804fb9 (stale) Mirror 22: 0x6e3cba68 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x350ae: skipped 000856 1777421512.799904 write 0x030713 thru 0x034a11 (0x42ff bytes) 000858 1777421512.805710 read 0x02e088 thru 0x02e72e (0x6a7 bytes) 000859 1777421512.809238 mapwrite 0x013609 thru 0x0167de (0x31d6 bytes) 000860 1777421512.835525 mapread 0x005dfb thru 0x011085 (0xb28b bytes) 000861 1777421512.842717 mapwrite 0x021297 thru 0x02b5cf (0xa339 bytes) 000862 1777421512.881235 write 0x00f8cf thru 0x011592 (0x1cc4 bytes) 000863 1777421512.882290 mapwrite 0x00e1db thru 0x011b6e (0x3994 bytes) 000869 1777421512.909708 mapread 0x013b55 thru 0x01c5fd (0x8aa9 bytes) 000870 1777421512.910728 trunc from 0x0350ae to 0x02d45a (0x7c53 bytes) 000872 1777421512.921909 read 0x007ba4 thru 0x009397 (0x17f4 bytes) 000873 1777421512.922542 mapwrite 0x00b902 thru 0x00d44f (0x1b4e bytes) 000874 1777421512.945917 mapread 0x008f59 thru 0x0130d7 (0xa17f bytes) 000876 1777421512.950247 read 0x01bb21 thru 0x024e5e (0x933e bytes) 000877 1777421512.951047 write 0x002df5 thru 0x00b2b5 (0x84c1 bytes) 000879 1777421512.953083 read 0x01e087 thru 0x0243d4 (0x634e bytes) 000881 1777421512.954517 mapread 0x000b91 thru 0x001df3 (0x1263 bytes) 000884 1777421512.961925 mapread 0x01ad6a thru 0x023e3f (0x90d6 bytes) 000885 1777421512.963180 trunc from 0x02d45b to 0x011baa (0x1b8b0 bytes) 000886 1777421512.977043 mapread 0x00b966 thru 0x011baa (0x6245 bytes) 000888 1777421512.979004 read 0x00454d thru 0x0106b2 (0xc166 bytes) 000891 1777421512.980998 mapread 0x00bc59 thru 0x011458 (0x5800 bytes) 000894 1777421512.983194 read 0x011332 thru 0x011baa (0x879 bytes) 000895 1777421512.983818 trunc from 0x011bab to 0x0126bb (0xb11 bytes) 000896 1777421512.994362 mapread 0x00e30a thru 0x0126bb (0x43b2 bytes) 000898 1777421512.999096 trunc from 0x0126bc to 0x03f317 (0x2cc5c bytes) 000900 1777421513.009283 read 0x006b71 thru 0x00c40d (0x589d bytes) 000901 1777421513.009984 write 0x00dc96 thru 0x01843a (0xa7a5 bytes) 000902 1777421513.014865 write 0x0141b5 thru 0x01ad5b (0x6ba7 bytes) 000903 1777421513.023227 mapread 0x011b9b thru 0x017fd5 (0x643b bytes) 000906 1777421513.026843 read 0x0162d2 thru 0x01dc30 (0x795f bytes) 000908 1777421513.028387 write 0x035ff9 thru 0x03b3a6 (0x53ae bytes) 000910 1777421513.033821 MIRROR_RESYNC 6 mirrors 000911 1777421513.594184 read 0x021f6a thru 0x025060 (0x30f7 bytes) 000912 1777421513.600969 mapwrite 0x00e10b thru 0x01dedf (0xfdd5 bytes) 000913 1777421513.733769 mapwrite 0x03ec3c thru 0x03ffff (0x13c4 bytes) 000914 1777421513.804505 mapwrite 0x01a574 thru 0x01ea9f (0x452c bytes) 000915 1777421513.832832 mapread 0x031a7e thru 0x03ee7f (0xd402 bytes) 000917 1777421513.840816 read 0x02d800 thru 0x02fb7b (0x237c bytes) 000918 1777421513.841529 trunc from 0x040000 to 0x01c6b2 (0x2394d bytes) 000919 1777421513.851726 mapwrite 0x03466d thru 0x034e0a (0x79e bytes) 000920 1777421513.906085 mapwrite 0x039db9 thru 0x03bf23 (0x216b bytes) 000921 1777421513.955158 mapread 0x014aa5 thru 0x01b797 (0x6cf3 bytes) 000922 1777421513.960519 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bf24): Mirror 18: 0x6ec94efa (stale) Mirror 19: 0x6ec94efa (stale) Mirror 20: 0x6ec94efa (stale) Mirror 21: 0x6ec94efa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bf24): Mirror 18: 0x6ec94efa (stale) Mirror 19: 0x6ec94efa (stale) Mirror 20: 0x6ec94efa (stale) Mirror 21: 0x6ec94efa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bf24: skipped 000924 1777421514.233270 trunc from 0x03bf24 to 0x004e12 (0x37111 bytes) 000925 1777421514.249555 read 0x001716 thru 0x004085 (0x2970 bytes) 000926 1777421514.257904 write 0x01bc01 thru 0x029d63 (0xe163 bytes) 000928 1777421514.259813 trunc from 0x029d64 to 0x03638b (0xc628 bytes) 000929 1777421514.270452 mapread 0x019c50 thru 0x026761 (0xcb12 bytes) 000931 1777421514.280627 read 0x020cc1 thru 0x02be97 (0xb1d7 bytes) 000932 1777421514.290544 mapwrite 0x004f87 thru 0x0059e5 (0xa5f bytes) 000936 1777421514.326992 trunc from 0x03638c to 0x019b84 (0x1c807 bytes) 000937 1777421514.341844 mapread 0x0026d9 thru 0x01068b (0xdfb3 bytes) 000941 1777421514.346058 mapwrite 0x015519 thru 0x0190aa (0x3b92 bytes) 000942 1777421514.387812 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x19b85): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19b85): Mirror 18: 0xd6aa3e1f (stale) Mirror 19: 0xd6aa3e1f (stale) Mirror 20: 0xd6aa3e1f (stale) Mirror 21: 0xd6aa3e1f (stale) CRC-32 checksum value for chunk [0, 0x19b85): Mirror 1: 0x3b275a5b Mirror 22: 0x3b275a5b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19b85: skipped 000943 1777421514.787552 read 0x008a28 thru 0x017823 (0xedfc bytes) 000949 1777421514.810874 mapread 0x009565 thru 0x014580 (0xb01c bytes) 000951 1777421514.822883 MIRROR_RESYNC 6 mirrors 000956 1777421515.338659 read 0x004954 thru 0x0127c7 (0xde74 bytes) 000957 1777421515.344594 mapread 0x00e946 thru 0x019b84 (0xb23f bytes) 000958 1777421515.354990 write 0x00b508 thru 0x00d82f (0x2328 bytes) 000960 1777421515.418219 write 0x02dde3 thru 0x037c4c (0x9e6a bytes) 000961 1777421515.420188 read 0x027dcf thru 0x02bcde (0x3f10 bytes) 000962 1777421515.427429 trunc from 0x037c4d to 0x0192d4 (0x1e978 bytes) 000964 1777421515.452529 mapwrite 0x03cab9 thru 0x03ffff (0x3547 bytes) 000965 1777421515.514259 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0x6cb94cb8 (stale) Mirror 19: 0x6cb94cb8 (stale) Mirror 20: 0x6cb94cb8 (stale) Mirror 21: 0x6cb94cb8 (stale) Mirror 22: 0x6cb94cb8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x54b6264f Mirror 23: 0x54b6264f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000966 1777421515.993308 write 0x03ceac thru 0x03ffff (0x3154 bytes) 000967 1777421516.051338 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x6cb94cb8 (stale) Mirror 20: 0x6cb94cb8 (stale) Mirror 21: 0x6cb94cb8 (stale) Mirror 22: 0x6cb94cb8 (stale) Mirror 23: 0x54b6264f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x6cb94cb8 (stale) Mirror 20: 0x6cb94cb8 (stale) Mirror 21: 0x6cb94cb8 (stale) Mirror 22: 0x6cb94cb8 (stale) Mirror 23: 0x54b6264f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000968 1777421516.329070 write 0x01e5df thru 0x026312 (0x7d34 bytes) 000970 1777421516.342053 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0x6cb94cb8 (stale) Mirror 21: 0x6cb94cb8 (stale) Mirror 22: 0x6cb94cb8 (stale) Mirror 23: 0x54b6264f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0x6cb94cb8 (stale) Mirror 21: 0x6cb94cb8 (stale) Mirror 22: 0x6cb94cb8 (stale) Mirror 23: 0x54b6264f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000972 1777421516.584869 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0x6cb94cb8 (stale) Mirror 21: 0x6cb94cb8 (stale) Mirror 23: 0x54b6264f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0x6cb94cb8 (stale) Mirror 21: 0x6cb94cb8 (stale) Mirror 23: 0x54b6264f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000974 1777421516.777339 trunc from 0x040000 to 0x001b1a (0x3e4e5 bytes) 000975 1777421516.803814 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b1b): Mirror 20: 0x53caecda (stale) Mirror 21: 0x53caecda (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b1b): Mirror 20: 0x53caecda (stale) Mirror 21: 0x53caecda (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b1b: skipped 000976 1777421516.986299 mapwrite 0x02781e thru 0x02c2cc (0x4aaf bytes) 000977 1777421517.052168 read 0x015efe thru 0x023e3f (0xdf42 bytes) 000978 1777421517.061980 trunc from 0x02c2cd to 0x00007d (0x2c24f bytes) 000979 1777421517.075804 mapread 0x000045 thru 0x00007d (0x39 bytes) 000980 1777421517.082095 trunc from 0x00007e to 0x034281 (0x34204 bytes) 000982 1777421517.095642 read 0x008190 thru 0x0162e3 (0xe154 bytes) 000983 1777421517.105342 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34282): Mirror 21: 0x845f04c3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34282): Mirror 21: 0x845f04c3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34282: skipped 000985 1777421517.267996 mapread 0x00b611 thru 0x00e6b5 (0x30a5 bytes) 000986 1777421517.290940 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2bc:0x0]: Invalid argument (22) 000989 1777421517.373769 mapread 0x02bb37 thru 0x034281 (0x874b bytes) 000990 1777421517.400453 mapwrite 0x022c87 thru 0x0276b9 (0x4a33 bytes) 000991 1777421517.443544 mapread 0x021c13 thru 0x02a018 (0x8406 bytes) 000993 1777421517.446704 mapread 0x023025 thru 0x029621 (0x65fd bytes) 000995 1777421517.448645 mapread 0x003662 thru 0x00da16 (0xa3b5 bytes) 000996 1777421517.456181 write 0x02bed4 thru 0x03647b (0xa5a8 bytes) 000999 1777421517.458839 mapread 0x022f8b thru 0x02d1d6 (0xa24c bytes) 001000 1777421517.462816 MIRROR_EXTEND to 2 mirrors All operations completed A-OK! PASS 70a (49s) == sanity-flr test 71: check mirror extend parent fid ==== 20:12:00 (1777421520) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 72: check append on FLR file ========== 20:12:01 (1777421521) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3d8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.125819 s, 16.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3d5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3d8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091601 s, 11.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3d5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3d9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3d8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3d5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3d9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3d8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3d6:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (4s) == sanity-flr test 200a: stress test ===================== 20:12:05 (1777421525) Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre2 Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4275904 .. Extending file size to 6203648 .. Extending file size to 8507584 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5977408 .. Extending file size to 6789312 .. Extending file size to 9261440 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9057280 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9168192 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 56067 56068 56069 56073 56074 failed 192.168.204.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg438-client.virtnet /mnt/lustre2 (opts:) 192.168.204.138@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg438-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (69s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:13:14 (1777421594) Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre2 Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 544704 .. Extending file size to 6964448 .. failed Extending file size to 2132288 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6696192 .. failed Extending file size to 7750144 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8246976 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7580320 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8292832 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8740864 .. failed Extending file size to 8850144 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9274720 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8779968 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59039 59040 59042 59044 59045 done 192.168.204.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg438-client.virtnet /mnt/lustre2 (opts:) 192.168.204.138@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg438-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (66s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:14:20 (1777421660) Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4480 2203904 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 25600 3743744 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 18432 3750912 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4480 2203904 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 25600 3743744 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 18432 3750912 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0484374 s, 43.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.128719 s, 16.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x528f1a8e (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg438-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:14:27 (1777421667) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0359973 s, 58.3 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 20:14:30 (1777421670) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30e:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3ef:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3ef:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3f0:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x3eb:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:14:33 (1777421673) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0642669 s, 32.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3ec:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3ed:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:14:36 (1777421676) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0669098 s, 31.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3f3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3ee:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3f3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3ee:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3ef:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:14:39 (1777421679) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0803007 s, 26.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3f4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3f0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3f4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3f0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3f1:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:15:04 (1777421704) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg438-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453338 s, 23.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3f5:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3f2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3f5:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3f2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3f6:0x0] } PASS 204d (43s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:15:47 (1777421747) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg438-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3f4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3f5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0667864 s, 31.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3f4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3f5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3f7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3f4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3f5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3f7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (47s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:16:34 (1777421794) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg438-server mds-ost sync done. Creating new pool test_204f oleg438-server: Pool lustre.test_204f created Adding targets to pool oleg438-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg438-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.530744 s, 4.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3f8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3f8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg438-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg438-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg438-server: Pool lustre.test_204f destroyed PASS 204f (77s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:17:51 (1777421871) PASS 205a (4s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:17:55 (1777421875) PASS 205b (5s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:18:00 (1777421880) oleg438-server: Pool lustre.test_206 created Adding targets to pool oleg438-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg438-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg438-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg438-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg438-server: Pool lustre.test_206 destroyed PASS 206 (18s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:18:18 (1777421898) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585237 s, 17.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292494 s, 35.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x30f7154 Mirror 3: 0x30f7154 PASS 207 (6s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:18:24 (1777421904) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:18:26 (1777421906) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:18:28 (1777421908) Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0650938 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1d10000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1a20000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1a20000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1a20000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1070000 Read this (4 bytes): BBBB 192.168.204.138@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg438-client.virtnet /mnt/lustre2 (opts:) PASS 209a (25s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:18:53 (1777421933) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0120943 s, 677 kB/s the very first read read with warm cache PASS 209b (4s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:18:57 (1777421937) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030363 s, 34.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x333 lmm_fid: [0x200000404:0x333:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x333 lmm_fid: [0x200000404:0x333:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x403:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834739 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-28 20:18:58.000000000 -0400 Modify: 2026-04-28 20:18:58.000000000 -0400 Change: 2026-04-28 20:18:59.000000000 -0400 Birth: 2026-04-28 20:18:58.000000000 -0400 PASS 210a (6s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:19:03 (1777421943) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:19:05 (1777421945) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.343927 s, 119 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0340907 s, 120 kB/s size after second write -rw-r--r-- 1 root root 4096 Apr 28 20:19 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x335:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Apr 28 20:19 /mnt/lustre/f211.sanity-flr PASS 211 (5s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:19:10 (1777421950) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (4s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:19:14 (1777421954) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (6s) == sanity-flr test complete, duration 1062 sec =========== 20:19:20 (1777421960) === sanity-flr: start cleanup 20:19:21 (1777421961) === === sanity-flr: finish cleanup 20:19:25 (1777421965) ===