-----============= acceptance-small: sanity-flr ============----- Mon Mar 9 20:21:20 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg150-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 20:21:34 (1773102094) === oleg150-client.virtnet: executing check_config_client /mnt/lustre oleg150-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg150-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa0a602a89800.idle_timeout=debug osc.lustre-OST0001-osc-ffffa0a602a89800.idle_timeout=debug disable quota as required oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 20:22:00 (1773102120) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7569] == sanity-flr test 0a: lfs mirror create with -N option == 20:22:01 (1773102121) PASS 0a (13s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:22:14 (1773102134) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:22:16 (1773102136) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 20:22:19 (1773102139) PASS 0d (17s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:22:37 (1773102157) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:22:39 (1773102159) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 20:22:42 (1773102162) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffffa0a602a89800.stats=clear osc.lustre-OST0001-osc-ffffa0a602a89800.stats=clear PASS 0g (5s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:22:48 (1773102168) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (7s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:22:55 (1773102175) PASS 0j (6s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:23:01 (1773102181) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 20:23:04 (1773102184) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 20:23:12 (1773102192) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:23:19 (1773102199) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:23:22 (1773102202) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:23:28 (1773102208) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (5s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 20:23:36 (1773102216) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:23:42 (1773102222) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.812374 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660606 s, 15.9 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:23:50 (1773102230) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.222808 s, 56.5 MB/s PASS 22 (7s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:23:57 (1773102237) Stopping /mnt/lustre-ost1 (opts:) on oleg150-server oleg150-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa0a602a89800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa0a602a89800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg150-server: oleg150-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg150-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg150-server oleg150-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa0a602a89800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa0a602a89800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg150-server: oleg150-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg150-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (50s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:24:47 (1773102287) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104073 s, 20.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg150-server oleg150-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa0a602a89800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa0a602a89800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg150-server: oleg150-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg150-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (28s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:25:16 (1773102316) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg150-server oleg150-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa0a602a89800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa0a602a89800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg150-server: oleg150-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg150-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg150-server oleg150-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa0a602a89800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa0a602a89800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg150-server: oleg150-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg150-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (51s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:26:08 (1773102368) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.35324 s, 12.0 MB/s -rw-r--r-- 1 root root 52428800 Mar 9 20:26 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.973s user 0m0.001s sys 0m0.203s Stopping /mnt/lustre-ost1 (opts:) on oleg150-server oleg150-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa0a602a89800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa0a602a89800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.923s user 0m0.001s sys 0m0.180s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg150-server: oleg150-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg150-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg150-server oleg150-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa0a602a89800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa0a602a89800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.170s user 0m0.000s sys 0m0.234s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg150-server: oleg150-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg150-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg150-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa0a602a89800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa0a602a89800.ost_server_uuid in FULL state after 0 sec PASS 33b (69s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:27:16 (1773102436) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:27:18 (1773102438) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:27:21 (1773102441) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 20:27:23 (1773102443) PASS 35 (5s) == sanity-flr test 36a: write to mirrored files ========== 20:27:28 (1773102448) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.438063 s, 28.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (10s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:27:38 (1773102458) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268377 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213459 s, 49.1 MB/s PASS 36b (5s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:27:43 (1773102463) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:27:50 (1773102470) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00117954 s, 868 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00119745 s, 855 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00088117 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00265833 s, 385 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 20:27:59 (1773102479) create test files with size 15757135 .. bc773781492466b5f8323a1db2ca50f1 - 1909e94a5b2246b7d6f1711ca2d6b3d6 - f2d58bd8eb8a5a6bcabfd41770efc946 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg150-client.virtnet /mnt/lustre (opts:) Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre PASS 37 (16s) == sanity-flr test 38: resync ============================ 20:28:16 (1773102496) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (9s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:28:26 (1773102506) PASS 39 (5s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:28:31 (1773102511) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111006 s, 9.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148944 s, 70.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (7s) == sanity-flr test 41: lfs mirror resync check =========== 20:28:37 (1773102517) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.173488 s, 12.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.30939 s, 13.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.194533 s, 10.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (12s) == sanity-flr test 42: lfs mirror verify ================= 20:28:49 (1773102529) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 20:28:51 (1773102531) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:28:54 (1773102534) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420273 s, 24.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (5s) == sanity-flr test 44a: lfs mirror split check =========== 20:28:59 (1773102539) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:29:01 (1773102541) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (5s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:29:08 (1773102548) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.216875 s, 48.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg150-client.virtnet /mnt/lustre (opts:) Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (6s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:29:14 (1773102554) llite.lustre-ffffa0a60a10e000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.16576 s, 63.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg150-client.virtnet /mnt/lustre (opts:) Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffffa0a603ee5800.xattr_cache=1 PASS 44e (8s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:29:22 (1773102562) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (27s) == sanity-flr test 46: Verify setstripe --copy option ==== 20:29:49 (1773102589) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (24s) == sanity-flr test 47: Verify mirror obj alloc =========== 20:30:13 (1773102613) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 20:30:15 (1773102615) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.148403 s, 21.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0937678 s, 33.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xf76855ca (stale) Mirror 2: 0xf76855ca (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 1d71c4e3f511d2ef0862573d8db40bfe - mirror 1 checksum: df0f62a970718baf4bf9b58833f9e328 - mirror 2 checksum: 1d71c4e3f511d2ef0862573d8db40bfe - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Mar 9 20:30:16 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Mar 9 20:30:16 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 1d71c4e3f511d2ef0862573d8db40bfe - mirror 1 checksum: df0f62a970718baf4bf9b58833f9e328 - mirror 2 checksum: df0f62a970718baf4bf9b58833f9e328 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Mar 9 20:30:16 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Mar 9 20:30:16 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 48 (7s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:30:23 (1773102623) Failing mds1 on oleg150-server Stopping /mnt/lustre-mds1 (opts:) on oleg150-server 20:30:28 (1773102628) shut down facet: mds1 facet_host: oleg150-server facet_failover_host: oleg150-server Failover mds1 to oleg150-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-MDT0000 20:30:46 (1773102646) targets are mounted 20:30:46 (1773102646) facet_failover done oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029334 s, 35.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } PASS 50A (33s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:30:56 (1773102656) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:30:59 (1773102659) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:31:01 (1773102661) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00893002 s, 1.8 MB/s PASS 50c (5s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:31:06 (1773102666) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:31:08 (1773102668) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:31:10 (1773102670) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (15s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:31:25 (1773102685) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (24s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:31:49 (1773102709) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (21s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:32:10 (1773102730) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (22s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:32:32 (1773102752) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (5s) == sanity-flr test 70: mirror create and split race ====== 20:32:37 (1773102757) mirror create pid 52261 mirror split pid 52262 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52261 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52262 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (65s) == sanity-flr test 70a: flr mode fsx test ================ 20:33:42 (1773102822) Seed set to 2823 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x21786 000001 1773102824.007916 trunc from 00000000 to 0x021785 (0x21786 bytes) 000003 1773102824.022744 mapread 0x01780a thru 0x021785 (0x9f7c bytes) 000004 1773102824.042844 read 0x01c6cd thru 0x021785 (0x50b9 bytes) 000005 1773102824.045520 write 0x03f8bb thru 0x03ffff (0x745 bytes) 000010 1773102824.051522 write 0x0225e6 thru 0x02daa3 (0xb4be bytes) 000011 1773102824.067802 write 0x03096b thru 0x036fa0 (0x6636 bytes) 000013 1773102824.076469 mapwrite 0x02b477 thru 0x03080e (0x5398 bytes) 000016 1773102824.134152 mapread 0x01296f thru 0x01c280 (0x9912 bytes) 000017 1773102824.135664 write 0x02fe4a thru 0x03e7fb (0xe9b2 bytes) 000019 1773102824.141713 mapread 0x0374da thru 0x03ffff (0x8b26 bytes) 000020 1773102824.142872 mapread 0x01828f thru 0x01b0b3 (0x2e25 bytes) 000021 1773102824.144097 write 0x037e0c thru 0x03a0ec (0x22e1 bytes) 000022 1773102824.144974 mapwrite 0x0048d1 thru 0x005638 (0xd68 bytes) 000023 1773102824.200521 write 0x006db5 thru 0x00ceb1 (0x60fd bytes) 000025 1773102824.202225 write 0x03357d thru 0x03ffff (0xca83 bytes) 000027 1773102824.204496 mapread 0x01eb38 thru 0x0273e4 (0x88ad bytes) 000028 1773102824.205465 write 0x02db77 thru 0x03572c (0x7bb6 bytes) 000030 1773102824.208034 write 0x02d677 thru 0x0337ec (0x6176 bytes) 000031 1773102824.210795 mapread 0x009b10 thru 0x012a93 (0x8f84 bytes) 000032 1773102824.219119 mapread 0x023198 thru 0x032f2e (0xfd97 bytes) 000034 1773102824.221015 read 0x02ab64 thru 0x02c8a5 (0x1d42 bytes) truncating to largest ever: 0x37a8d 000036 1773102824.222878 trunc from 0x040000 to 0x037a8c (0x8573 bytes) 000038 1773102824.242993 mapwrite 0x0272e0 thru 0x034bc1 (0xd8e2 bytes) 000042 1773102824.298919 mapread 0x009e7a thru 0x016617 (0xc79e bytes) 000044 1773102824.306368 MIRROR_EXTEND to 2 mirrors 000045 1773102824.713031 MIRROR_EXTEND to 3 mirrors 000046 1773102825.097195 mapread 0x009e42 thru 0x0143e9 (0xa5a8 bytes) 000047 1773102825.127105 mapread 0x028284 thru 0x02984a (0x15c7 bytes) 000049 1773102825.145394 write 0x01ec88 thru 0x0275ea (0x8963 bytes) 000050 1773102825.212573 mapread 0x001382 thru 0x009788 (0x8407 bytes) 000052 1773102825.223106 read 0x02c59a thru 0x037a8c (0xb4f3 bytes) 000056 1773102825.232623 trunc from 0x037a8d to 0x00cae1 (0x2afab bytes) 000058 1773102825.259025 mapread 0x00a8e8 thru 0x00cae1 (0x21fa bytes) 000059 1773102825.262573 write 0x0052a7 thru 0x00ad85 (0x5adf bytes) 000060 1773102825.263841 mapread 0x0015bf thru 0x008373 (0x6db5 bytes) 000062 1773102825.265547 trunc from 0x00cae2 to 0x00a28a (0x2857 bytes) 000063 1773102825.287084 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xa28b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa28b): Mirror 2: 0x89bb31a2 (stale) Mirror 3: 0x89bb31a2 (stale) CRC-32 checksum value for chunk [0, 0xa28b): Mirror 1: 0x34c612a2 Mirror 4: 0x34c612a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa28b: skipped 000064 1773102825.812923 trunc from 0x00a28b to 0x031cc5 (0x27a3b bytes) 000065 1773102825.910746 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x31cc6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31cc6): Mirror 2: 0x13a4e1cd (stale) Mirror 3: 0x13a4e1cd (stale) Mirror 4: 0xed9133ac (stale) CRC-32 checksum value for chunk [0, 0x31cc6): Mirror 1: 0xed9133ac Mirror 5: 0xed9133ac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31cc6: skipped 000066 1773102826.524239 mapread 0x02c4f5 thru 0x031cc5 (0x57d1 bytes) 000068 1773102826.569066 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31cc6): Mirror 3: 0x13a4e1cd (stale) Mirror 4: 0xed9133ac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31cc6): Mirror 3: 0x13a4e1cd (stale) Mirror 4: 0xed9133ac (stale) CRC-32 checksum value for chunk [0, 0x31cc6): Mirror 1: 0xed9133ac Mirror 5: 0xed9133ac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31cc6: skipped 000070 1773102827.154035 write 0x013df7 thru 0x01fb44 (0xbd4e bytes) 000072 1773102827.246251 write 0x006d47 thru 0x015387 (0xe641 bytes) 000073 1773102827.254765 mapread 0x022ac9 thru 0x030444 (0xd97c bytes) 000076 1773102827.275068 trunc from 0x031cc6 to 0x009e6b (0x27e5a bytes) 000077 1773102827.301971 read 0x008b99 thru 0x009e6b (0x12d3 bytes) 000078 1773102827.302945 read 0x002151 thru 0x009e6b (0x7d1b bytes) 000082 1773102827.311244 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9e6c): Mirror 3: 0x772d29a3 (stale) Mirror 4: 0xa130a229 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9e6c): Mirror 3: 0x772d29a3 (stale) Mirror 4: 0xa130a229 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9e6c: skipped 000083 1773102827.527769 write 0x016f25 thru 0x01aa47 (0x3b23 bytes) 000085 1773102827.534587 MIRROR_RESYNC 3 mirrors 000087 1773102828.111660 mapread 0x00d882 thru 0x01aa47 (0xd1c6 bytes) 000088 1773102828.132268 write 0x019484 thru 0x01bbdf (0x275c bytes) 000090 1773102828.192955 write 0x02c7b1 thru 0x038db5 (0xc605 bytes) 000092 1773102828.195715 mapread 0x02a696 thru 0x03562f (0xaf9a bytes) 000094 1773102828.202697 write 0x037bf3 thru 0x03ffff (0x840d bytes) 000095 1773102828.206603 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd32d11cb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd32d11cb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000097 1773102828.414360 mapread 0x02d260 thru 0x03d1ee (0xff8f bytes) 000099 1773102828.444900 read 0x0193b1 thru 0x024aba (0xb70a bytes) 000100 1773102828.457045 mapread 0x0281f8 thru 0x037776 (0xf57f bytes) 000102 1773102828.464996 MIRROR_RESYNC 2 mirrors 000103 1773102829.007411 mapwrite 0x00c5e4 thru 0x014df0 (0x880d bytes) 000104 1773102829.129040 read 0x01f625 thru 0x020686 (0x1062 bytes) 000105 1773102829.135658 mapread 0x02eacc thru 0x0321a4 (0x36d9 bytes) 000106 1773102829.150058 read 0x0154cc thru 0x0227e2 (0xd317 bytes) 000109 1773102829.169859 read 0x029a84 thru 0x02f94f (0x5ecc bytes) 000110 1773102829.170464 read 0x003377 thru 0x008ed6 (0x5b60 bytes) 000111 1773102829.171981 mapread 0x03654a thru 0x03ffff (0x9ab6 bytes) 000114 1773102829.175219 read 0x02715b thru 0x02b14c (0x3ff2 bytes) 000117 1773102829.180679 write 0x025025 thru 0x02c796 (0x7772 bytes) 000118 1773102829.182384 write 0x017dd2 thru 0x01e8eb (0x6b1a bytes) 000120 1773102829.184363 write 0x016674 thru 0x019a20 (0x33ad bytes) 000122 1773102829.186984 trunc from 0x040000 to 0x0110cd (0x2ef32 bytes) 000123 1773102829.235369 mapwrite 0x03591e thru 0x03e3a1 (0x8a84 bytes) 000124 1773102829.322432 read 0x0238f3 thru 0x032d94 (0xf4a2 bytes) 000126 1773102829.323365 read 0x03341c thru 0x03e3a1 (0xaf86 bytes) truncating to largest ever: 0x3deee 000128 1773102829.325409 trunc from 0x03e3a2 to 0x03deed (0x4b4 bytes) 000129 1773102829.336041 write 0x034a65 thru 0x03f3c0 (0xa95c bytes) 000130 1773102829.337365 write 0x00b4d9 thru 0x012dc9 (0x78f1 bytes) 000131 1773102829.338203 mapread 0x0135a4 thru 0x021b8a (0xe5e7 bytes) 000133 1773102829.339873 mapread 0x005533 thru 0x00effd (0x9acb bytes) 000134 1773102829.341070 mapwrite 0x021a49 thru 0x027221 (0x57d9 bytes) 000135 1773102829.368104 mapwrite 0x03a510 thru 0x03ffff (0x5af0 bytes) 000138 1773102829.416099 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa1ce77c9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfe124461 Mirror 4: 0xfe124461 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000139 1773102829.995701 read 0x02afd0 thru 0x03356f (0x85a0 bytes) 000140 1773102830.016950 MIRROR_RESYNC 3 mirrors 000141 1773102830.604505 write 0x027d8b thru 0x02af9c (0x3212 bytes) 000142 1773102830.677688 write 0x03e20d thru 0x03ffff (0x1df3 bytes) 000143 1773102830.689090 mapwrite 0x00da43 thru 0x00ecd1 (0x128f bytes) 000144 1773102830.742949 write 0x01be60 thru 0x027dbb (0xbf5c bytes) 000145 1773102830.757559 read 0x021f1a thru 0x028872 (0x6959 bytes) 000147 1773102830.760283 mapread 0x013e3f thru 0x022da8 (0xef6a bytes) 000148 1773102830.762086 mapread 0x03e6a5 thru 0x03ffff (0x195b bytes) 000151 1773102830.768286 read 0x0246a2 thru 0x02cdb5 (0x8714 bytes) 000152 1773102830.782927 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xfe124461 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xfe124461 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000153 1773102831.030124 write 0x02d7eb thru 0x038e49 (0xb65f bytes) 000155 1773102831.046082 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xfe124461 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7eee4707 Mirror 5: 0x7eee4707 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000156 1773102831.611672 read 0x00dee8 thru 0x01456c (0x6685 bytes) 000157 1773102831.621733 trunc from 0x040000 to 0x018bb6 (0x27449 bytes) 000162 1773102831.674292 read 0x0178f6 thru 0x018bb6 (0x12c1 bytes) 000165 1773102831.684308 mapwrite 0x000bed thru 0x00dc89 (0xd09d bytes) 000166 1773102831.754069 trunc from 0x018bb7 to 0x012c74 (0x5f42 bytes) 000168 1773102831.783944 read 0x0108bc thru 0x012c74 (0x23b9 bytes) 000169 1773102831.791963 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12c75): Mirror 5: 0x55010d1d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12c75): Mirror 5: 0x55010d1d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12c75: skipped 000170 1773102832.018855 trunc from 0x012c75 to 0x02654b (0x138d7 bytes) 000172 1773102832.046233 read 0x021597 thru 0x02654b (0x4fb5 bytes) 000173 1773102832.054933 mapwrite 0x02a877 thru 0x0302a7 (0x5a31 bytes) 000174 1773102832.149632 read 0x00ce6c thru 0x00e92b (0x1ac0 bytes) 000175 1773102832.159345 trunc from 0x0302a8 to 0x011fdd (0x1e2ca bytes) 000176 1773102832.175258 mapread 0x008a0d thru 0x00c12d (0x3721 bytes) 000179 1773102832.182572 trunc from 0x011fde to 0x037cef (0x25d12 bytes) 000180 1773102832.200999 mapread 0x00c74c thru 0x0164f0 (0x9da5 bytes) 000181 1773102832.224054 mapread 0x030dd0 thru 0x037cef (0x6f20 bytes) 000184 1773102832.239744 MIRROR_RESYNC 2 mirrors 000185 1773102832.699786 MIRROR_RESYNC 2 mirrors 000189 1773102832.967920 write 0x002b8f thru 0x00d4a3 (0xa915 bytes) 000190 1773102833.023170 write 0x026e21 thru 0x030af3 (0x9cd3 bytes) 000191 1773102833.040357 trunc from 0x037cf0 to 0x0267f6 (0x114f9 bytes) 000192 1773102833.067911 write 0x025c5c thru 0x0313d2 (0xb777 bytes) 000194 1773102833.076033 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) 000195 1773102833.185151 read 0x004c1e thru 0x010ac7 (0xbeaa bytes) 000196 1773102833.194496 mapread 0x011573 thru 0x0126e7 (0x1175 bytes) 000198 1773102833.205859 mapwrite 0x03ab99 thru 0x03ffff (0x5467 bytes) 000199 1773102833.273992 read 0x009ff4 thru 0x00b99a (0x19a7 bytes) 000200 1773102833.276012 read 0x01c7b4 thru 0x01d3b4 (0xc01 bytes) 000201 1773102833.287519 mapwrite 0x007073 thru 0x01031b (0x92a9 bytes) 000202 1773102833.320817 mapread 0x00f513 thru 0x012370 (0x2e5e bytes) 000204 1773102833.322302 write 0x0208b6 thru 0x025885 (0x4fd0 bytes) 000205 1773102833.324848 trunc from 0x040000 to 0x03699b (0x9664 bytes) 000207 1773102833.339467 mapwrite 0x0127e1 thru 0x01f7ae (0xcfce bytes) 000208 1773102833.375288 mapread 0x022c17 thru 0x02bb82 (0x8f6c bytes) 000209 1773102833.376624 mapwrite 0x03e063 thru 0x03ffff (0x1f9d bytes) 000211 1773102833.444028 mapwrite 0x0374b5 thru 0x03ffff (0x8b4b bytes) 000213 1773102833.535879 read 0x014f0c thru 0x020acd (0xbbc2 bytes) 000214 1773102833.537048 trunc from 0x040000 to 0x020870 (0x1f78f bytes) 000215 1773102833.550581 mapread 0x002229 thru 0x00f964 (0xd73c bytes) 000216 1773102833.552207 write 0x024ccd thru 0x0346ea (0xfa1e bytes) 000217 1773102833.555581 mapwrite 0x0390cc thru 0x03cfc2 (0x3ef7 bytes) 000218 1773102833.620847 read 0x00a8a4 thru 0x00c807 (0x1f64 bytes) 000220 1773102833.622295 write 0x00792c thru 0x00cd88 (0x545d bytes) 000223 1773102833.626672 mapwrite 0x03a07d thru 0x03ffff (0x5f83 bytes) 000224 1773102833.701601 read 0x032b0a thru 0x03f291 (0xc788 bytes) 000226 1773102833.703129 write 0x038594 thru 0x03b49b (0x2f08 bytes) 000227 1773102833.705175 mapwrite 0x01b240 thru 0x02304f (0x7e10 bytes) 000229 1773102833.746926 read 0x033819 thru 0x03ffff (0xc7e7 bytes) 000230 1773102833.747632 mapread 0x01590d thru 0x019512 (0x3c06 bytes) 000231 1773102833.751393 trunc from 0x040000 to 0x03ae16 (0x51e9 bytes) 000232 1773102833.762901 trunc from 0x03ae17 to 0x00ff99 (0x2ae7d bytes) 000235 1773102833.780943 write 0x000d9b thru 0x00fbda (0xee40 bytes) 000236 1773102833.783075 trunc from 0x00ff9a to 0x014e29 (0x4e90 bytes) 000238 1773102833.797751 mapwrite 0x01aed7 thru 0x022a77 (0x7ba1 bytes) 000239 1773102833.882992 trunc from 0x022a78 to 0x02c704 (0x9c8d bytes) 000241 1773102833.902461 read 0x012bd3 thru 0x016c25 (0x4053 bytes) 000243 1773102833.904994 mapwrite 0x00baeb thru 0x01576c (0x9c82 bytes) 000244 1773102834.013788 write 0x035d68 thru 0x03848a (0x2723 bytes) 000245 1773102834.015748 read 0x005455 thru 0x014052 (0xebfe bytes) 000246 1773102834.016727 write 0x022c3a thru 0x02c3c1 (0x9788 bytes) 000248 1773102834.032450 read 0x02f2a9 thru 0x0368ef (0x7647 bytes) 000249 1773102834.042044 mapread 0x027ac6 thru 0x03375c (0xbc97 bytes) 000250 1773102834.044576 write 0x0324a1 thru 0x039ac2 (0x7622 bytes) 000251 1773102834.057370 MIRROR_EXTEND to 2 mirrors 000252 1773102834.574394 read 0x00b590 thru 0x0108c9 (0x533a bytes) 000254 1773102834.583512 mapread 0x034e13 thru 0x038b37 (0x3d25 bytes) 000255 1773102834.598381 mapwrite 0x00e3fe thru 0x01a4d6 (0xc0d9 bytes) 000257 1773102834.738042 MIRROR_RESYNC 2 mirrors 000258 1773102835.157580 read 0x01a716 thru 0x01cd4b (0x2636 bytes) 000259 1773102835.165012 mapread 0x038dad thru 0x039ac2 (0xd16 bytes) 000260 1773102835.181713 write 0x00019f thru 0x00be1b (0xbc7d bytes) 000261 1773102835.236872 trunc from 0x039ac3 to 0x01ea26 (0x1b09c bytes) 000264 1773102835.260509 write 0x0234dd thru 0x025d4a (0x286e bytes) 000266 1773102835.262219 write 0x013ae0 thru 0x02201a (0xe53b bytes) 000268 1773102835.274782 write 0x006981 thru 0x01593c (0xefbc bytes) 000269 1773102835.277059 mapread 0x0010d2 thru 0x010ba2 (0xfad1 bytes) 000270 1773102835.279122 read 0x0161e0 thru 0x020c1e (0xaa3f bytes) 000271 1773102835.279916 mapread 0x000c83 thru 0x00115e (0x4dc bytes) 000274 1773102835.282970 trunc from 0x025d4b to 0x03d0b7 (0x1736d bytes) 000276 1773102835.298901 mapwrite 0x02bbdf thru 0x030eae (0x52d0 bytes) 000277 1773102835.347092 write 0x005d32 thru 0x014ba2 (0xee71 bytes) 000279 1773102835.349343 mapwrite 0x031233 thru 0x034c9f (0x3a6d bytes) 000280 1773102835.397609 mapwrite 0x01226d thru 0x020ef4 (0xec88 bytes) 000281 1773102835.453278 trunc from 0x03d0b8 to 0x0373f2 (0x5cc5 bytes) 000282 1773102835.475649 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) 000283 1773102835.576970 write 0x02491e thru 0x0327a4 (0xde87 bytes) 000285 1773102835.587084 write 0x0130f5 thru 0x013f7f (0xe8b bytes) 000286 1773102835.598118 write 0x0293d6 thru 0x02d01d (0x3c48 bytes) 000289 1773102835.602645 mapwrite 0x0399d6 thru 0x03ffff (0x662a bytes) 000290 1773102835.675482 read 0x030160 thru 0x030525 (0x3c6 bytes) 000293 1773102835.679961 read 0x02531d thru 0x02c4a7 (0x718b bytes) 000294 1773102835.685832 trunc from 0x040000 to 0x024db7 (0x1b248 bytes) 000296 1773102835.706081 trunc from 0x024db8 to 0x01c7e7 (0x85d0 bytes) 000300 1773102835.730300 mapwrite 0x0200c0 thru 0x02f4af (0xf3f0 bytes) 000301 1773102835.808463 mapwrite 0x00e740 thru 0x011d40 (0x3601 bytes) 000302 1773102835.854704 write 0x014dfb thru 0x022d62 (0xdf68 bytes) 000305 1773102835.858361 trunc from 0x02f4b0 to 0x00c325 (0x2318a bytes) 000309 1773102835.875987 write 0x012d79 thru 0x01d6af (0xa937 bytes) 000310 1773102835.879543 read 0x01cba8 thru 0x01d6af (0xb08 bytes) 000311 1773102835.880567 trunc from 0x01d6b0 to 0x037700 (0x1a051 bytes) 000312 1773102835.895867 write 0x02427c thru 0x02d4ff (0x9284 bytes) 000313 1773102835.904874 trunc from 0x037701 to 0x032956 (0x4daa bytes) 000314 1773102835.928791 mapread 0x01b021 thru 0x023b6c (0x8b4c bytes) 000319 1773102835.933011 read 0x022cd3 thru 0x02770f (0x4a3d bytes) 000320 1773102835.933923 trunc from 0x032957 to 0x0225b2 (0x103a4 bytes) 000321 1773102835.950505 trunc from 0x0225b3 to 0x001306 (0x212ac bytes) 000323 1773102835.972297 trunc from 0x001307 to 0x01ca5f (0x1b759 bytes) 000326 1773102835.995981 mapwrite 0x028ba0 thru 0x0316c1 (0x8b22 bytes) 000327 1773102836.066715 read 0x03116f thru 0x0316c1 (0x553 bytes) 000329 1773102836.067786 mapwrite 0x008ba2 thru 0x011948 (0x8da7 bytes) 000330 1773102836.135111 MIRROR_EXTEND to 2 mirrors 000332 1773102836.507732 trunc from 0x0316c2 to 0x03a265 (0x8ba4 bytes) 000333 1773102836.598642 mapwrite 0x039007 thru 0x03ffff (0x6ff9 bytes) 000337 1773102836.681887 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) 000338 1773102836.779811 trunc from 0x040000 to 0x00d721 (0x328de bytes) 000340 1773102836.802937 read 0x00166a thru 0x00d721 (0xc0b8 bytes) 000342 1773102836.811669 write 0x002147 thru 0x0058fc (0x37b6 bytes) 000345 1773102836.814084 write 0x01071e thru 0x013098 (0x297b bytes) 000346 1773102836.815725 write 0x001020 thru 0x00c9cc (0xb9ad bytes) 000348 1773102836.818715 write 0x02c9f7 thru 0x03a3b5 (0xd9bf bytes) 000349 1773102836.820316 mapread 0x00637d thru 0x0063dd (0x61 bytes) 000354 1773102836.826709 trunc from 0x03a3b6 to 0x0049c9 (0x359ec bytes) 000355 1773102836.846797 read 0x000fcc thru 0x0049c9 (0x39fe bytes) 000356 1773102836.850691 write 0x01bcd0 thru 0x02bb88 (0xfeb9 bytes) 000358 1773102836.852790 read 0x01e43f thru 0x0272af (0x8e71 bytes) 000359 1773102836.853702 read 0x00f5b0 thru 0x019c74 (0xa6c5 bytes) 000361 1773102836.865693 mapwrite 0x00c6ea thru 0x01818f (0xbaa6 bytes) 000362 1773102836.909476 mapread 0x00ca46 thru 0x01be4f (0xf40a bytes) 000365 1773102836.911793 read 0x001ee0 thru 0x0042fb (0x241c bytes) 000366 1773102836.912854 write 0x027640 thru 0x028089 (0xa4a bytes) 000368 1773102836.915052 write 0x00c424 thru 0x014843 (0x8420 bytes) 000370 1773102836.918178 mapread 0x00b839 thru 0x01b81d (0xffe5 bytes) 000371 1773102836.923857 mapread 0x021513 thru 0x02a151 (0x8c3f bytes) 000372 1773102836.926077 trunc from 0x02bb89 to 0x00a189 (0x219ff bytes) 000374 1773102836.944279 mapread 0x003eb6 thru 0x009c4c (0x5d97 bytes) 000376 1773102836.946044 mapread 0x0069f3 thru 0x00a189 (0x3797 bytes) 000377 1773102836.947242 write 0x03149d thru 0x03a509 (0x906d bytes) 000378 1773102836.949330 mapwrite 0x02753f thru 0x02ff23 (0x89e5 bytes) 000380 1773102836.998599 trunc from 0x03a50a to 0x03bdca (0x18c1 bytes) 000381 1773102837.009044 mapwrite 0x039e87 thru 0x03ffff (0x6179 bytes) 000385 1773102837.076317 mapread 0x026149 thru 0x027cf9 (0x1bb1 bytes) 000386 1773102837.078601 read 0x024607 thru 0x03322a (0xec24 bytes) 000388 1773102837.080268 mapread 0x0164c7 thru 0x02570b (0xf245 bytes) 000391 1773102837.082631 trunc from 0x040000 to 0x0297bf (0x16840 bytes) 000392 1773102837.101853 mapread 0x010068 thru 0x019a9a (0x9a33 bytes) 000394 1773102837.104280 mapread 0x012fad thru 0x015649 (0x269d bytes) 000395 1773102837.105947 mapwrite 0x016a91 thru 0x01c04c (0x55bc bytes) 000396 1773102837.148994 write 0x022793 thru 0x025ba9 (0x3417 bytes) 000397 1773102837.150496 mapwrite 0x013aba thru 0x01c28c (0x87d3 bytes) 000401 1773102837.179984 read 0x00431e thru 0x01291b (0xe5fe bytes) 000402 1773102837.180601 mapread 0x024d26 thru 0x0297bf (0x4a9a bytes) 000403 1773102837.181458 read 0x0176c7 thru 0x021477 (0x9db1 bytes) 000404 1773102837.181912 trunc from 0x0297c0 to 0x00a07f (0x1f740 bytes) 000408 1773102837.194539 mapwrite 0x0299e1 thru 0x034643 (0xac63 bytes) 000409 1773102837.236737 mapwrite 0x01c2eb thru 0x01e4aa (0x21c0 bytes) 000410 1773102837.264999 MIRROR_EXTEND to 2 mirrors 000411 1773102837.594028 write 0x01fc4c thru 0x0291ac (0x9561 bytes) 000414 1773102837.656091 mapwrite 0x03a4fc thru 0x03ffff (0x5b04 bytes) 000415 1773102837.716789 write 0x00791b thru 0x00fe7e (0x8564 bytes) 000417 1773102837.722868 trunc from 0x040000 to 0x014e06 (0x2b1f9 bytes) 000421 1773102837.739010 trunc from 0x014e07 to 0x027d9a (0x12f94 bytes) 000422 1773102837.759223 write 0x03c89e thru 0x03ffff (0x3762 bytes) 000423 1773102837.760521 mapwrite 0x03c1b4 thru 0x03c57d (0x3ca bytes) 000424 1773102837.802965 write 0x00f895 thru 0x01cfbc (0xd728 bytes) 000426 1773102837.810769 mapwrite 0x001359 thru 0x008589 (0x7231 bytes) 000427 1773102837.864805 read 0x039f47 thru 0x03c936 (0x29f0 bytes) 000430 1773102837.872662 mapread 0x008e3e thru 0x00b486 (0x2649 bytes) 000431 1773102837.873698 mapwrite 0x003a15 thru 0x010258 (0xc844 bytes) 000432 1773102837.916819 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) 000435 1773102838.023595 write 0x030a87 thru 0x03b724 (0xac9e bytes) 000436 1773102838.040980 read 0x034644 thru 0x03f780 (0xb13d bytes) 000438 1773102838.050450 trunc from 0x040000 to 0x00afc4 (0x3503b bytes) 000441 1773102838.069808 read 0x003335 thru 0x00afc4 (0x7c90 bytes) 000444 1773102838.077635 mapwrite 0x01ebba thru 0x023b09 (0x4f50 bytes) 000446 1773102838.140039 MIRROR_EXTEND to 2 mirrors 000453 1773102838.608895 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) 000454 1773102838.719289 mapread 0x007c2f thru 0x00f0ce (0x74a0 bytes) 000455 1773102838.739731 mapread 0x005223 thru 0x00bc50 (0x6a2e bytes) 000457 1773102838.742944 trunc from 0x023b0a to 0x00a4b0 (0x19659 bytes) 000461 1773102838.789491 mapwrite 0x02dac4 thru 0x03a6a0 (0xcbdd bytes) 000462 1773102838.882710 mapwrite 0x00fee0 thru 0x018b31 (0x8c52 bytes) 000464 1773102838.937407 read 0x0079dc thru 0x00f414 (0x7a39 bytes) 000465 1773102838.942080 read 0x010a19 thru 0x018d09 (0x82f1 bytes) 000466 1773102838.945941 MIRROR_EXTEND to 2 mirrors 000470 1773102839.421019 write 0x006e09 thru 0x00cb61 (0x5d59 bytes) 000471 1773102839.490788 mapread 0x01416a thru 0x0227a2 (0xe639 bytes) 000473 1773102839.514804 read 0x01596a thru 0x0167ac (0xe43 bytes) 000475 1773102839.516528 write 0x03efd6 thru 0x03ffff (0x102a bytes) 000483 1773102839.527869 mapread 0x01b5ee thru 0x0295b4 (0xdfc7 bytes) 000487 1773102839.539317 read 0x0229cc thru 0x022bcd (0x202 bytes) 000488 1773102839.539978 mapwrite 0x01291c thru 0x01a2eb (0x79d0 bytes) 000491 1773102839.601868 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8b822525 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7c87bf56 Mirror 3: 0x7c87bf56 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000492 1773102840.215478 write 0x011883 thru 0x01c0ff (0xa87d bytes) 000493 1773102840.285945 write 0x0141a5 thru 0x01d341 (0x919d bytes) 000495 1773102840.295502 mapwrite 0x012f4d thru 0x013895 (0x949 bytes) 000496 1773102840.332923 write 0x014e9c thru 0x01f43c (0xa5a1 bytes) 000499 1773102840.342447 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7c87bf56 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7c87bf56 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000501 1773102840.639073 mapread 0x01c843 thru 0x0282cf (0xba8d bytes) 000503 1773102840.680920 trunc from 0x040000 to 0x013dc1 (0x2c23e bytes) 000504 1773102840.732585 trunc from 0x013dc2 to 0x03162b (0x1d86a bytes) 000505 1773102840.755418 trunc from 0x03162c to 0x02d620 (0x400b bytes) 000507 1773102840.771630 mapwrite 0x032741 thru 0x039117 (0x69d7 bytes) 000508 1773102840.844643 write 0x01bdf9 thru 0x028460 (0xc668 bytes) 000510 1773102840.863446 mapwrite 0x027c0b thru 0x02bb24 (0x3f1a bytes) 000512 1773102840.922810 trunc from 0x039118 to 0x00a00f (0x2f108 bytes) 000513 1773102840.948160 mapread 0x00728d thru 0x00a00f (0x2d83 bytes) 000516 1773102840.960879 trunc from 0x00a010 to 0x001a96 (0x8579 bytes) 000520 1773102841.004744 MIRROR_RESYNC 2 mirrors 000522 1773102841.533305 write 0x00521d thru 0x013e3b (0xec1f bytes) 000524 1773102841.578393 write 0x025851 thru 0x030be7 (0xb397 bytes) 000525 1773102841.588920 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x30be8): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30be8): Mirror 3: 0xe8a049e7 (stale) CRC-32 checksum value for chunk [0, 0x30be8): Mirror 1: 0x78c48268 Mirror 4: 0x78c48268 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30be8: skipped 000526 1773102842.261844 write 0x02ad2b thru 0x032eba (0x8190 bytes) 000529 1773102842.336012 mapwrite 0x03e6cf thru 0x03ffff (0x1931 bytes) 000530 1773102842.429271 write 0x00f9c6 thru 0x016927 (0x6f62 bytes) 000532 1773102842.438267 read 0x01c36d thru 0x020e47 (0x4adb bytes) 000534 1773102842.451824 trunc from 0x040000 to 0x0316e9 (0xe916 bytes) 000535 1773102842.470093 mapread 0x0076bd thru 0x009f58 (0x289c bytes) 000539 1773102842.481868 write 0x036f39 thru 0x03ffff (0x90c7 bytes) 000542 1773102842.485498 mapwrite 0x01b0df thru 0x025b19 (0xaa3b bytes) 000543 1773102842.578775 mapwrite 0x01bf58 thru 0x01f9f5 (0x3a9e bytes) 000544 1773102842.625845 read 0x02d0f5 thru 0x03bcab (0xebb7 bytes) 000545 1773102842.653251 MIRROR_RESYNC 3 mirrors 000546 1773102843.301845 read 0x038f23 thru 0x03ffff (0x70dd bytes) 000547 1773102843.321514 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) 000550 1773102843.627666 mapread 0x03e88f thru 0x03ffff (0x1771 bytes) 000551 1773102843.637480 trunc from 0x040000 to 0x020c2a (0x1f3d5 bytes) 000554 1773102843.727403 read 0x01bdff thru 0x01be59 (0x5b bytes) 000555 1773102843.732768 read 0x019e2e thru 0x020c2a (0x6dfd bytes) 000560 1773102843.751038 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x20c2b): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20c2b): Mirror 3: 0xf33b482e (stale) CRC-32 checksum value for chunk [0, 0x20c2b): Mirror 1: 0xf33b482e Mirror 4: 0xf33b482e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20c2b: skipped 000561 1773102844.511504 mapread 0x012bbf thru 0x017fdd (0x541f bytes) 000562 1773102844.563621 MIRROR_RESYNC 3 mirrors 000563 1773102845.190872 mapwrite 0x015d2b thru 0x01736a (0x1640 bytes) 000564 1773102845.345058 write 0x02a391 thru 0x0355d3 (0xb243 bytes) 000566 1773102845.357927 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x355d4): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x355d4): Mirror 3: 0x5d9cf779 (stale) Mirror 4: 0x5d9cf779 (stale) CRC-32 checksum value for chunk [0, 0x355d4): Mirror 1: 0xd01233cd Mirror 5: 0xd01233cd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x355d4: skipped 000567 1773102846.124958 write 0x019036 thru 0x01ace6 (0x1cb1 bytes) 000568 1773102846.204683 mapread 0x01179f thru 0x0151e8 (0x3a4a bytes) 000571 1773102846.215944 read 0x027129 thru 0x02f823 (0x86fb bytes) 000572 1773102846.233330 MIRROR_RESYNC 4 mirrors 000573 1773102846.906685 mapwrite 0x0026b1 thru 0x007c3b (0x558b bytes) 000576 1773102847.128702 mapread 0x003080 thru 0x00824f (0x51d0 bytes) 000578 1773102847.130583 write 0x000078 thru 0x007d21 (0x7caa bytes) 000580 1773102847.149378 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x355d4): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x355d4): Mirror 3: 0xb7f163f6 (stale) Mirror 4: 0xb7f163f6 (stale) Mirror 5: 0xb7f163f6 (stale) CRC-32 checksum value for chunk [0, 0x355d4): Mirror 1: 0xad8aec4d Mirror 6: 0xad8aec4d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x355d4: skipped 000582 1773102847.891993 mapwrite 0x034184 thru 0x03ffff (0xbe7c bytes) 000585 1773102848.052055 mapread 0x03d193 thru 0x03e8ab (0x1719 bytes) 000587 1773102848.071784 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1e512b53 (stale) Mirror 5: 0x1e512b53 (stale) Mirror 6: 0x9369ded3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1e512b53 (stale) Mirror 5: 0x1e512b53 (stale) Mirror 6: 0x9369ded3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000590 1773102848.562849 MIRROR_RESYNC 4 mirrors 000593 1773102849.354768 mapwrite 0x013603 thru 0x01fefb (0xc8f9 bytes) 000594 1773102849.542378 trunc from 0x040000 to 0x021500 (0x1eaff bytes) 000596 1773102849.582364 mapread 0x01b656 thru 0x021500 (0x5eab bytes) 000597 1773102849.585008 write 0x0240d6 thru 0x025dfa (0x1d25 bytes) 000599 1773102849.587050 read 0x01c0bb thru 0x01d285 (0x11cb bytes) 000601 1773102849.597010 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x25dfb): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25dfb): Mirror 4: 0x5868ab44 (stale) Mirror 5: 0x5868ab44 (stale) Mirror 6: 0x5868ab44 (stale) CRC-32 checksum value for chunk [0, 0x25dfb): Mirror 1: 0xd834449b Mirror 7: 0xd834449b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25dfb: skipped 000606 1773102850.337912 trunc from 0x025dfb to 0x0179c3 (0xe437 bytes) 000607 1773102850.445601 mapwrite 0x023fe5 thru 0x02df32 (0x9f4e bytes) 000609 1773102850.545624 trunc from 0x02df33 to 0x031f5f (0x402d bytes) 000610 1773102850.567745 mapwrite 0x0255e4 thru 0x029947 (0x4364 bytes) 000613 1773102850.606353 mapread 0x0195d7 thru 0x01e18b (0x4bb5 bytes) 000619 1773102850.639758 mapread 0x02fe0a thru 0x031f5f (0x2156 bytes) 000620 1773102850.647291 write 0x007827 thru 0x013957 (0xc131 bytes) 000621 1773102850.657478 mapread 0x0044df thru 0x00e9e4 (0xa506 bytes) 000622 1773102850.659035 read 0x01206e thru 0x0157c5 (0x3758 bytes) 000623 1773102850.661000 write 0x021cbc thru 0x02f86e (0xdbb3 bytes) 000626 1773102850.665653 mapwrite 0x03a4e3 thru 0x03ffff (0x5b1d bytes) 000629 1773102850.769026 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x68176dd0 (stale) Mirror 6: 0x68176dd0 (stale) Mirror 7: 0x91d1122d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x68176dd0 (stale) Mirror 6: 0x68176dd0 (stale) Mirror 7: 0x91d1122d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000630 1773102851.190363 trunc from 0x040000 to 0x0254f2 (0x1ab0d bytes) 000631 1773102851.206830 mapread 0x00ea0b thru 0x018c8c (0xa282 bytes) 000633 1773102851.251796 read 0x019e45 thru 0x01a2d6 (0x492 bytes) 000636 1773102851.267269 write 0x01faf2 thru 0x023462 (0x3971 bytes) 000637 1773102851.281800 trunc from 0x0254f3 to 0x037564 (0x12072 bytes) 000639 1773102851.309057 read 0x0306ea thru 0x037564 (0x6e7b bytes) 000640 1773102851.339010 MIRROR_RESYNC 4 mirrors 000641 1773102852.132967 mapwrite 0x01217e thru 0x015237 (0x30ba bytes) 000643 1773102852.313966 mapread 0x00fecc thru 0x01dd2d (0xde62 bytes) 000644 1773102852.330935 mapread 0x01606f thru 0x02525f (0xf1f1 bytes) 000645 1773102852.342872 mapwrite 0x026168 thru 0x031059 (0xaef2 bytes) 000647 1773102852.421104 write 0x01ca5f thru 0x0200e1 (0x3683 bytes) 000648 1773102852.422018 mapwrite 0x031894 thru 0x031d00 (0x46d bytes) 000649 1773102852.462971 write 0x000470 thru 0x003590 (0x3121 bytes) 000650 1773102852.471505 mapwrite 0x029d5b thru 0x02d98c (0x3c32 bytes) 000652 1773102852.528682 write 0x015fde thru 0x021cdb (0xbcfe bytes) 000655 1773102852.537729 trunc from 0x037565 to 0x001a4e (0x35b16 bytes) 000657 1773102852.596960 mapread 0x0017d4 thru 0x001a4e (0x27b bytes) 000658 1773102852.599943 mapread 0x000e3a thru 0x001a4e (0xc15 bytes) 000660 1773102852.603723 mapread 0x000b3e thru 0x000d62 (0x225 bytes) 000661 1773102852.606120 write 0x02ca1c thru 0x033c0f (0x71f4 bytes) 000666 1773102852.612480 mapwrite 0x030cf7 thru 0x03f760 (0xea6a bytes) 000667 1773102852.717015 mapread 0x038891 thru 0x03f760 (0x6ed0 bytes) 000668 1773102852.719998 trunc from 0x03f761 to 0x020810 (0x1ef50 bytes) 000672 1773102852.759705 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x20811): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20811): Mirror 4: 0x14bfed8c (stale) Mirror 6: 0x14bfed8c (stale) Mirror 7: 0x14bfed8c (stale) CRC-32 checksum value for chunk [0, 0x20811): Mirror 1: 0xc440ed61 Mirror 8: 0xc440ed61 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20811: skipped 000675 1773102853.648823 trunc from 0x020811 to 0x025ba4 (0x5394 bytes) 000678 1773102853.783985 read 0x01068a thru 0x0195e0 (0x8f57 bytes) 000680 1773102853.802767 trunc from 0x025ba5 to 0x030160 (0xa5bc bytes) 000681 1773102853.833927 mapwrite 0x00a80b thru 0x011395 (0x6b8b bytes) 000682 1773102853.947940 mapwrite 0x032641 thru 0x03ffff (0xd9bf bytes) 000683 1773102854.061848 trunc from 0x040000 to 0x033233 (0xcdcc bytes) 000686 1773102854.090498 mapread 0x02ef3b thru 0x033233 (0x42f9 bytes) 000687 1773102854.096473 read 0x00d2eb thru 0x0163c6 (0x90dc bytes) 000690 1773102854.098678 write 0x02115d thru 0x02ed41 (0xdbe5 bytes) 000691 1773102854.110976 write 0x03d0a1 thru 0x03d6b7 (0x617 bytes) 000692 1773102854.112680 mapread 0x001544 thru 0x00ec77 (0xd734 bytes) 000693 1773102854.115874 mapwrite 0x00c19c thru 0x01843b (0xc2a0 bytes) 000694 1773102854.174185 write 0x009194 thru 0x00caf5 (0x3962 bytes) 000695 1773102854.180590 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d6b8): Mirror 4: 0x303a262b (stale) Mirror 6: 0x303a262b (stale) Mirror 8: 0xf21dde12 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d6b8): Mirror 4: 0x303a262b (stale) Mirror 6: 0x303a262b (stale) Mirror 8: 0xf21dde12 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d6b8: skipped 000696 1773102854.604266 trunc from 0x03d6b8 to 0x00e40b (0x2f2ac bytes) 000697 1773102854.617562 mapread 0x008948 thru 0x00e40b (0x5ac4 bytes) 000698 1773102854.638502 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe40c): Mirror 4: 0x8b6b5141 (stale) Mirror 8: 0x7af17832 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe40c): Mirror 4: 0x8b6b5141 (stale) Mirror 8: 0x7af17832 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe40c: skipped 000700 1773102854.964504 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xe40c): Mirror 4: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe40c): Mirror 4: 0x8b6b5141 (stale) Mirror 8: 0x7af17832 (stale) CRC-32 checksum value for chunk [0, 0xe40c): Mirror 1: 0xf851fb08 Mirror 9: 0xf851fb08 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe40c: skipped 000701 1773102855.735773 MIRROR_RESYNC 4 mirrors 000702 1773102856.462436 mapwrite 0x0021c3 thru 0x00a7dd (0x861b bytes) 000705 1773102856.626582 mapread 0x0036e7 thru 0x00c426 (0x8d40 bytes) 000706 1773102856.628268 mapread 0x00032b thru 0x009298 (0x8f6e bytes) 000711 1773102856.650814 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe40c): Mirror 4: 0xf851fb08 (stale) Mirror 8: 0xf851fb08 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe40c): Mirror 4: 0xf851fb08 (stale) Mirror 8: 0xf851fb08 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe40c: skipped 000712 1773102857.016218 mapread 0x009ac2 thru 0x00e40b (0x494a bytes) 000714 1773102857.043806 mapwrite 0x02f3af thru 0x03217a (0x2dcc bytes) 000716 1773102857.137047 mapwrite 0x0279b2 thru 0x0341ec (0xc83b bytes) 000717 1773102857.192844 read 0x0147fe thru 0x01f6a3 (0xaea6 bytes) truncating to largest ever: 0x3ea96 000718 1773102857.200221 trunc from 0x0341ed to 0x03ea95 (0xa8a9 bytes) 000719 1773102857.214066 trunc from 0x03ea96 to 0x02da14 (0x11081 bytes) 000720 1773102857.233813 mapwrite 0x007ee3 thru 0x00c21c (0x433a bytes) 000721 1773102857.275783 mapread 0x005aaf thru 0x00ad6c (0x52be bytes) 000722 1773102857.277791 read 0x02b667 thru 0x02da14 (0x23ae bytes) 000723 1773102857.281778 mapwrite 0x00b316 thru 0x014c6c (0x9957 bytes) 000724 1773102857.331370 mapread 0x00169c thru 0x004039 (0x299e bytes) 000726 1773102857.333783 write 0x00d023 thru 0x0189d7 (0xb9b5 bytes) 000727 1773102857.335460 trunc from 0x02da15 to 0x01e9b9 (0xf05b bytes) 000728 1773102857.355809 mapwrite 0x00f3b6 thru 0x012748 (0x3393 bytes) 000730 1773102857.403307 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1e9ba): Mirror 4: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e9ba): Mirror 4: 0x49b137ee (stale) Mirror 8: 0x49b137ee (stale) CRC-32 checksum value for chunk [0, 0x1e9ba): Mirror 1: 0x4b129412 Mirror 9: 0x4b129412 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e9ba: skipped 000732 1773102858.276182 write 0x03d16a thru 0x03ffff (0x2e96 bytes) 000733 1773102858.339853 mapread 0x018a72 thru 0x023ffc (0xb58b bytes) 000734 1773102858.377422 mapwrite 0x025cad thru 0x03416e (0xe4c2 bytes) 000739 1773102858.443586 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xc7df6445 (stale) Mirror 8: 0xc7df6445 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xc7df6445 (stale) Mirror 8: 0xc7df6445 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000740 1773102858.745878 read 0x018966 thru 0x0223d3 (0x9a6e bytes) 000742 1773102858.770902 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xc7df6445 (stale) Mirror 8: 0xc7df6445 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa7505063 Mirror 9: 0xa7505063 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000743 1773102859.518336 write 0x038cfd thru 0x03b6e4 (0x29e8 bytes) 000748 1773102859.591970 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xc7df6445 (stale) Mirror 8: 0xc7df6445 (stale) Mirror 9: 0xa7505063 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf4d563ed Mirror 10: 0xf4d563ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000749 1773102860.401896 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xc7df6445 (stale) Mirror 8: 0xc7df6445 (stale) Mirror 9: 0xa7505063 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf4d563ed Mirror 10: 0xf4d563ed Mirror 11: 0xf4d563ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000751 1773102861.287915 mapwrite 0x00b681 thru 0x0128bb (0x723b bytes) 000752 1773102861.492496 trunc from 0x040000 to 0x02bd8d (0x14272 bytes) 000754 1773102861.518322 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2bd8e): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bd8e): Mirror 4: 0x34d477d2 (stale) Mirror 8: 0x34d477d2 (stale) Mirror 9: 0x6f881ca (stale) Mirror 10: 0x6f881ca (stale) Mirror 11: 0x6f881ca (stale) CRC-32 checksum value for chunk [0, 0x2bd8e): Mirror 1: 0xa9c99903 Mirror 12: 0xa9c99903 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bd8e: skipped 000755 1773102862.292564 trunc from 0x02bd8e to 0x019b48 (0x12245 bytes) 000756 1773102862.381307 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x19b49): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19b49): Mirror 4: 0xa27744ad (stale) Mirror 8: 0xa27744ad (stale) Mirror 9: 0x153b1695 (stale) Mirror 10: 0x153b1695 (stale) Mirror 11: 0x153b1695 (stale) Mirror 12: 0x2e6d9975 (stale) CRC-32 checksum value for chunk [0, 0x19b49): Mirror 1: 0x2e6d9975 Mirror 13: 0x2e6d9975 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19b49: skipped 000757 1773102863.276569 write 0x031a77 thru 0x03e3de (0xc968 bytes) 000760 1773102863.346426 MIRROR_RESYNC 8 mirrors 000761 1773102864.654691 read 0x03d9f4 thru 0x03e3de (0x9eb bytes) 000763 1773102864.662685 read 0x00e90e thru 0x01b332 (0xca25 bytes) 000764 1773102864.689115 MIRROR_RESYNC 8 mirrors 000765 1773102865.229671 mapread 0x038032 thru 0x03e3de (0x63ad bytes) 000766 1773102865.234177 MIRROR_EXTEND to 9 mirrors 000767 1773102865.945362 read 0x03c349 thru 0x03e3de (0x2096 bytes) 000768 1773102865.954310 trunc from 0x03e3df to 0x035c04 (0x87da bytes) 000770 1773102866.055446 MIRROR_RESYNC 9 mirrors 000771 1773102867.388044 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) 000772 1773102867.739618 mapread 0x009a7b thru 0x00f97f (0x5f05 bytes) 000773 1773102867.764372 MIRROR_EXTEND to 9 mirrors 000775 1773102868.505430 trunc from 0x035c05 to 0x03ad49 (0x5145 bytes) 000777 1773102868.636909 MIRROR_RESYNC 9 mirrors 000779 1773102870.117039 mapread 0x02134f thru 0x026629 (0x52db bytes) 000780 1773102870.138988 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) 000781 1773102870.603380 mapread 0x039cb3 thru 0x03ad49 (0x1097 bytes) 000782 1773102870.630293 write 0x016953 thru 0x026506 (0xfbb4 bytes) 000783 1773102870.703526 mapread 0x03491c thru 0x03ad49 (0x642e bytes) 000784 1773102870.712938 trunc from 0x03ad4a to 0x00e7e2 (0x2c567 bytes) 000785 1773102870.735182 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0xe7e3): Mirror 4: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe7e3): Mirror 4: 0x6a35d5 (stale) Mirror 9: 0x6a35d5 (stale) Mirror 11: 0x6a35d5 (stale) Mirror 12: 0x6a35d5 (stale) Mirror 13: 0x6a35d5 (stale) Mirror 14: 0x6a35d5 (stale) Mirror 15: 0x6a35d5 (stale) CRC-32 checksum value for chunk [0, 0xe7e3): Mirror 1: 0x6a35d5 Mirror 16: 0x6a35d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe7e3: skipped 000787 1773102871.602737 mapwrite 0x0301a3 thru 0x03857d (0x83db bytes) 000788 1773102871.775115 read 0x00b52e thru 0x00ed20 (0x37f3 bytes) 000789 1773102871.780200 read 0x00b67d thru 0x01449c (0x8e20 bytes) 000794 1773102871.788881 mapread 0x035628 thru 0x03857d (0x2f56 bytes) 000798 1773102871.796704 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3857e): Mirror 9: 0x52581527 (stale) Mirror 11: 0x52581527 (stale) Mirror 12: 0x52581527 (stale) Mirror 13: 0x52581527 (stale) Mirror 14: 0x52581527 (stale) Mirror 15: 0x52581527 (stale) Mirror 16: 0xeb2a82be (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3857e): Mirror 9: 0x52581527 (stale) Mirror 11: 0x52581527 (stale) Mirror 12: 0x52581527 (stale) Mirror 13: 0x52581527 (stale) Mirror 14: 0x52581527 (stale) Mirror 15: 0x52581527 (stale) Mirror 16: 0xeb2a82be (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3857e: skipped 000799 1773102872.412785 mapread 0x021cfc thru 0x02545e (0x3763 bytes) 000800 1773102872.452665 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3857e): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3857e): Mirror 9: 0x52581527 (stale) Mirror 11: 0x52581527 (stale) Mirror 12: 0x52581527 (stale) Mirror 13: 0x52581527 (stale) Mirror 14: 0x52581527 (stale) Mirror 15: 0x52581527 (stale) Mirror 16: 0xeb2a82be (stale) CRC-32 checksum value for chunk [0, 0x3857e): Mirror 1: 0xcc3e499e Mirror 17: 0xcc3e499e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3857e: skipped 000801 1773102873.471229 mapwrite 0x022093 thru 0x023b29 (0x1a97 bytes) 000802 1773102873.651108 trunc from 0x03857e to 0x0192ef (0x1f28e bytes) 000803 1773102873.673454 trunc from 0x0192f0 to 0x028fe9 (0xfcfa bytes) 000804 1773102873.687909 mapwrite 0x0009ae thru 0x00ac41 (0xa294 bytes) 000808 1773102873.744510 mapread 0x00970b thru 0x00ff20 (0x6816 bytes) 000810 1773102873.746462 mapread 0x00e1ce thru 0x019829 (0xb65c bytes) 000811 1773102873.768541 mapwrite 0x00af71 thru 0x01504e (0xa0de bytes) 000812 1773102873.815972 trunc from 0x028fea to 0x0036fb (0x258ee bytes) 000813 1773102873.837420 write 0x02ee1b thru 0x031f30 (0x3116 bytes) 000814 1773102873.840168 mapread 0x01b096 thru 0x021b87 (0x6af2 bytes) 000816 1773102873.853286 mapwrite 0x02cf45 thru 0x0318a2 (0x495e bytes) 000817 1773102873.923918 MIRROR_RESYNC 9 mirrors 000819 1773102875.016018 mapread 0x01623c thru 0x022bb3 (0xc978 bytes) 000820 1773102875.049896 mapread 0x0126db thru 0x01e8cc (0xc1f2 bytes) 000821 1773102875.051022 trunc from 0x031f31 to 0x039449 (0x7519 bytes) 000822 1773102875.126206 trunc from 0x03944a to 0x025bdd (0x1386c bytes) 000823 1773102875.146227 mapread 0x001f00 thru 0x00c49b (0xa59c bytes) 000824 1773102875.155875 read 0x017592 thru 0x0256e6 (0xe155 bytes) 000826 1773102875.164398 mapread 0x0244bf thru 0x025bdd (0x171f bytes) 000829 1773102875.173833 MIRROR_RESYNC 9 mirrors 000831 1773102876.279587 MIRROR_RESYNC 9 mirrors 000836 1773102876.779539 trunc from 0x025bde to 0x03da44 (0x17e67 bytes) 000837 1773102876.916412 mapwrite 0x008074 thru 0x00e0c5 (0x6052 bytes) 000838 1773102876.985261 mapwrite 0x033ced thru 0x036647 (0x295b bytes) 000840 1773102877.022095 write 0x02e0c9 thru 0x031ea4 (0x3ddc bytes) 000841 1773102877.023036 mapread 0x031e6a thru 0x03b877 (0x9a0e bytes) 000843 1773102877.025480 trunc from 0x03da45 to 0x0050d3 (0x38971 bytes) 000844 1773102877.040609 write 0x003c06 thru 0x00f5d2 (0xb9cd bytes) 000847 1773102877.044052 trunc from 0x00f5d3 to 0x03c87e (0x2d2ac bytes) 000848 1773102877.060538 mapread 0x021762 thru 0x02384f (0x20ee bytes) 000850 1773102877.079630 MIRROR_RESYNC 9 mirrors 000851 1773102878.062559 read 0x028b6c thru 0x03471d (0xbbb2 bytes) 000852 1773102878.069862 write 0x0393fe thru 0x03ffff (0x6c02 bytes) 000853 1773102878.142413 read 0x01f397 thru 0x021d34 (0x299e bytes) 000854 1773102878.161232 write 0x011015 thru 0x020f06 (0xfef2 bytes) 000857 1773102878.163900 mapread 0x016800 thru 0x016830 (0x31 bytes) 000858 1773102878.166580 mapwrite 0x00f1ca thru 0x01d966 (0xe79d bytes) 000860 1773102878.214442 write 0x022e67 thru 0x026b97 (0x3d31 bytes) 000862 1773102878.216537 write 0x00378b thru 0x00b82c (0x80a2 bytes) 000864 1773102878.219828 write 0x00eadc thru 0x01e7a2 (0xfcc7 bytes) 000866 1773102878.224417 read 0x01b804 thru 0x02a902 (0xf0ff bytes) 000867 1773102878.225856 read 0x025035 thru 0x026455 (0x1421 bytes) 000870 1773102878.228337 read 0x0277c8 thru 0x028a03 (0x123c bytes) 000871 1773102878.230017 mapread 0x009567 thru 0x00a70b (0x11a5 bytes) 000873 1773102878.236950 write 0x00cb6b thru 0x011be9 (0x507f bytes) 000874 1773102878.247306 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x243f8878 (stale) Mirror 12: 0x243f8878 (stale) Mirror 13: 0x243f8878 (stale) Mirror 14: 0x243f8878 (stale) Mirror 15: 0x243f8878 (stale) Mirror 16: 0x243f8878 (stale) Mirror 17: 0x243f8878 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x243f8878 (stale) Mirror 12: 0x243f8878 (stale) Mirror 13: 0x243f8878 (stale) Mirror 14: 0x243f8878 (stale) Mirror 15: 0x243f8878 (stale) Mirror 16: 0x243f8878 (stale) Mirror 17: 0x243f8878 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000877 1773102878.675511 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x243f8878 (stale) Mirror 12: 0x243f8878 (stale) Mirror 13: 0x243f8878 (stale) Mirror 14: 0x243f8878 (stale) Mirror 16: 0x243f8878 (stale) Mirror 17: 0x243f8878 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x243f8878 (stale) Mirror 12: 0x243f8878 (stale) Mirror 13: 0x243f8878 (stale) Mirror 14: 0x243f8878 (stale) Mirror 16: 0x243f8878 (stale) Mirror 17: 0x243f8878 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000878 1773102879.108642 mapwrite 0x0287e3 thru 0x028c95 (0x4b3 bytes) 000879 1773102879.169117 read 0x01dca2 thru 0x02ab31 (0xce90 bytes) 000881 1773102879.185410 mapwrite 0x02df55 thru 0x03d7fb (0xf8a7 bytes) 000882 1773102879.247605 write 0x01e1d4 thru 0x02d903 (0xf730 bytes) 000884 1773102879.249451 write 0x015868 thru 0x01745d (0x1bf6 bytes) 000886 1773102879.257914 read 0x00824f thru 0x013ec5 (0xbc77 bytes) 000890 1773102879.267324 mapread 0x0315d2 thru 0x0357cf (0x41fe bytes) 000891 1773102879.268844 trunc from 0x040000 to 0x033b59 (0xc4a6 bytes) 000892 1773102879.291463 trunc from 0x033b5a to 0x020a21 (0x13138 bytes) 000893 1773102879.305294 mapwrite 0x039f20 thru 0x03ffff (0x60e0 bytes) 000894 1773102879.398337 mapread 0x02e7ea thru 0x03301f (0x4836 bytes) 000895 1773102879.432466 read 0x00ee1b thru 0x01415c (0x5342 bytes) 000897 1773102879.434817 mapread 0x0163f1 thru 0x0169dc (0x5ec bytes) 000898 1773102879.436419 mapwrite 0x00b4ee thru 0x01ac31 (0xf744 bytes) 000901 1773102879.491003 mapwrite 0x02dcb2 thru 0x038a64 (0xadb3 bytes) 000902 1773102879.560190 mapread 0x01ee58 thru 0x0298bc (0xaa65 bytes) 000904 1773102879.568278 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x243f8878 (stale) Mirror 12: 0x243f8878 (stale) Mirror 13: 0x243f8878 (stale) Mirror 14: 0x243f8878 (stale) Mirror 16: 0x243f8878 (stale) Mirror 17: 0x243f8878 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x72d4b1e7 Mirror 18: 0x72d4b1e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000905 1773102880.346458 trunc from 0x040000 to 0x02c64c (0x139b3 bytes) 000906 1773102880.428687 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c64d): Mirror 11: 0xd5bc8e86 (stale) Mirror 12: 0xd5bc8e86 (stale) Mirror 13: 0xd5bc8e86 (stale) Mirror 14: 0xd5bc8e86 (stale) Mirror 16: 0xd5bc8e86 (stale) Mirror 18: 0x7a79bfa7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c64d): Mirror 11: 0xd5bc8e86 (stale) Mirror 12: 0xd5bc8e86 (stale) Mirror 13: 0xd5bc8e86 (stale) Mirror 14: 0xd5bc8e86 (stale) Mirror 16: 0xd5bc8e86 (stale) Mirror 18: 0x7a79bfa7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c64d: skipped 000907 1773102880.834922 write 0x014b0d thru 0x017f47 (0x343b bytes) 000908 1773102880.847423 read 0x01e373 thru 0x021bdf (0x386d bytes) 000909 1773102880.858838 trunc from 0x02c64d to 0x02374c (0x8f00 bytes) 000910 1773102880.896824 MIRROR_RESYNC 7 mirrors 000911 1773102881.781613 write 0x01d88e thru 0x0267de (0x8f51 bytes) 000915 1773102881.873918 mapread 0x01f02b thru 0x0267de (0x77b4 bytes) 000918 1773102881.877843 mapwrite 0x02cde2 thru 0x03c04a (0xf269 bytes) 000920 1773102881.988622 mapwrite 0x027616 thru 0x035e0c (0xe7f7 bytes) 000922 1773102882.048219 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3c04b): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c04b): Mirror 11: 0xb68eed79 (stale) Mirror 12: 0xb68eed79 (stale) Mirror 13: 0xb68eed79 (stale) Mirror 14: 0xb68eed79 (stale) Mirror 16: 0xb68eed79 (stale) Mirror 18: 0xb68eed79 (stale) CRC-32 checksum value for chunk [0, 0x3c04b): Mirror 1: 0x649fcda2 Mirror 19: 0x649fcda2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c04b: skipped 000923 1773102882.874296 MIRROR_RESYNC 8 mirrors 000924 1773102883.867324 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) 000925 1773102884.189665 trunc from 0x03c04b to 0x025131 (0x16f19 bytes) 000926 1773102884.308842 write 0x020bd0 thru 0x02310c (0x253d bytes) 000927 1773102884.326968 read 0x01a1e9 thru 0x025131 (0xaf49 bytes) 000928 1773102884.339678 read 0x0052cd thru 0x00c530 (0x7264 bytes) 000929 1773102884.353075 write 0x002ad6 thru 0x00ed45 (0xc270 bytes) 000930 1773102884.356200 read 0x018619 thru 0x01c7d2 (0x41ba bytes) 000931 1773102884.361787 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x25132): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25132): Mirror 11: 0x13a39ae6 (stale) Mirror 12: 0x13a39ae6 (stale) Mirror 13: 0x13a39ae6 (stale) Mirror 16: 0x13a39ae6 (stale) Mirror 18: 0x13a39ae6 (stale) Mirror 19: 0x13a39ae6 (stale) CRC-32 checksum value for chunk [0, 0x25132): Mirror 1: 0x8888a100 Mirror 20: 0x8888a100 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25132: skipped 000933 1773102885.079504 write 0x01cdcb thru 0x0287b9 (0xb9ef bytes) 000935 1773102885.163837 mapread 0x026ad9 thru 0x0287b9 (0x1ce1 bytes) 000936 1773102885.174248 MIRROR_RESYNC 8 mirrors 000937 1773102886.228072 trunc from 0x0287ba to 0x03db03 (0x1534a bytes) 000938 1773102886.347210 mapwrite 0x00d40c thru 0x01bc37 (0xe82c bytes) 000940 1773102886.462953 read 0x01a478 thru 0x023bed (0x9776 bytes) 000942 1773102886.471044 read 0x03c381 thru 0x03dad3 (0x1753 bytes) 000943 1773102886.489123 write 0x01f5e5 thru 0x02571c (0x6138 bytes) 000945 1773102886.501427 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3db04): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3db04): Mirror 11: 0x98fd508 (stale) Mirror 12: 0x98fd508 (stale) Mirror 13: 0x98fd508 (stale) Mirror 16: 0x98fd508 (stale) Mirror 18: 0x98fd508 (stale) Mirror 19: 0x98fd508 (stale) Mirror 20: 0x98fd508 (stale) CRC-32 checksum value for chunk [0, 0x3db04): Mirror 1: 0xe38cd4a6 Mirror 21: 0xe38cd4a6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3db04: skipped 000947 1773102887.197340 mapwrite 0x03bc2c thru 0x03c3ee (0x7c3 bytes) 000948 1773102887.346665 mapwrite 0x009967 thru 0x01942e (0xfac8 bytes) 000949 1773102887.426874 mapwrite 0x005544 thru 0x00f762 (0xa21f bytes) 000950 1773102887.492253 trunc from 0x03db04 to 0x0293ad (0x14756 bytes) 000952 1773102887.545835 mapwrite 0x025e05 thru 0x027289 (0x1485 bytes) 000954 1773102887.587287 trunc from 0x0293ae to 0x011d5e (0x1764f bytes) 000955 1773102887.599329 write 0x03e9f4 thru 0x03ffff (0x160c bytes) 000957 1773102887.601589 trunc from 0x040000 to 0x0151b6 (0x2ae49 bytes) 000958 1773102887.610001 mapwrite 0x016fe6 thru 0x01df77 (0x6f92 bytes) 000960 1773102887.662766 write 0x00f2b8 thru 0x01e415 (0xf15e bytes) 000962 1773102887.665206 mapread 0x008b72 thru 0x012dc6 (0xa255 bytes) 000963 1773102887.668932 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e416): Mirror 11: 0x40ea9d88 (stale) Mirror 12: 0x40ea9d88 (stale) Mirror 13: 0x40ea9d88 (stale) Mirror 16: 0x40ea9d88 (stale) Mirror 19: 0x40ea9d88 (stale) Mirror 20: 0x40ea9d88 (stale) Mirror 21: 0x8e457eca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e416): Mirror 11: 0x40ea9d88 (stale) Mirror 12: 0x40ea9d88 (stale) Mirror 13: 0x40ea9d88 (stale) Mirror 16: 0x40ea9d88 (stale) Mirror 19: 0x40ea9d88 (stale) Mirror 20: 0x40ea9d88 (stale) Mirror 21: 0x8e457eca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e416: skipped 000965 1773102888.048317 read 0x019d1b thru 0x01a9ac (0xc92 bytes) 000966 1773102888.053038 mapread 0x01aa95 thru 0x01e415 (0x3981 bytes) 000967 1773102888.071819 mapread 0x001900 thru 0x00f17f (0xd880 bytes) 000973 1773102888.087512 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e416): Mirror 11: 0x40ea9d88 (stale) Mirror 12: 0x40ea9d88 (stale) Mirror 13: 0x40ea9d88 (stale) Mirror 16: 0x40ea9d88 (stale) Mirror 20: 0x40ea9d88 (stale) Mirror 21: 0x8e457eca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e416): Mirror 11: 0x40ea9d88 (stale) Mirror 12: 0x40ea9d88 (stale) Mirror 13: 0x40ea9d88 (stale) Mirror 16: 0x40ea9d88 (stale) Mirror 20: 0x40ea9d88 (stale) Mirror 21: 0x8e457eca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e416: skipped 000976 1773102888.418292 trunc from 0x01e416 to 0x03bc6d (0x1d858 bytes) 000977 1773102888.451499 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bc6e): Mirror 11: 0x5f74abed (stale) Mirror 12: 0x5f74abed (stale) Mirror 16: 0x5f74abed (stale) Mirror 20: 0x5f74abed (stale) Mirror 21: 0x64b54131 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bc6e): Mirror 11: 0x5f74abed (stale) Mirror 12: 0x5f74abed (stale) Mirror 16: 0x5f74abed (stale) Mirror 20: 0x5f74abed (stale) Mirror 21: 0x64b54131 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bc6e: skipped 000978 1773102888.781440 mapwrite 0x03facd thru 0x03ffff (0x533 bytes) 000980 1773102888.858029 trunc from 0x040000 to 0x01abbe (0x25441 bytes) 000981 1773102888.874536 trunc from 0x01abbf to 0x031742 (0x16b84 bytes) 000982 1773102888.897330 trunc from 0x031743 to 0x01d61b (0x14127 bytes) 000983 1773102888.917777 read 0x01910c thru 0x01adda (0x1ccf bytes) 000984 1773102888.924955 write 0x00add7 thru 0x00edf2 (0x401c bytes) 000986 1773102888.939808 mapwrite 0x03e947 thru 0x03ffff (0x16b9 bytes) 000987 1773102889.006435 trunc from 0x040000 to 0x00ccaf (0x33350 bytes) 000990 1773102889.029767 trunc from 0x00ccb0 to 0x029e3c (0x1d18d bytes) 000991 1773102889.047873 write 0x017aae thru 0x024bd4 (0xd127 bytes) 000992 1773102889.063523 mapread 0x012a90 thru 0x022974 (0xfee5 bytes) 000994 1773102889.075102 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x237:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29e3d): Mirror 11: 0x6f931ff2 (stale) Mirror 12: 0x6f931ff2 (stale) Mirror 16: 0x6f931ff2 (stale) Mirror 20: 0x6f931ff2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29e3d): Mirror 11: 0x6f931ff2 (stale) Mirror 12: 0x6f931ff2 (stale) Mirror 16: 0x6f931ff2 (stale) Mirror 20: 0x6f931ff2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29e3d: skipped 000995 1773102889.448110 read 0x016a5e thru 0x021d31 (0xb2d4 bytes) 000996 1773102889.458207 mapread 0x0064fd thru 0x011eca (0xb9ce bytes) All operations completed A-OK! PASS 70a (70s) == sanity-flr test 71: check mirror extend parent fid ==== 20:34:53 (1773102893) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (2s) == sanity-flr test 72: check append on FLR file ========== 20:34:55 (1773102895) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.198186 s, 10.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157068 s, 6.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32b:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32b:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x330:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (5s) == sanity-flr test 200a: stress test ===================== 20:35:00 (1773102900) Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre2 Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5743040 .. Extending file size to 1730368 .. Extending file size to 1880544 .. Extending file size to 7594432 .. Extending file size to 5996224 .. Extending file size to 6443840 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8768704 .. Extending file size to 7938592 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8932960 .. Extending file size to 9198016 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9153632 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 55794 55795 55797 55798 55802 done 192.168.201.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg150-client.virtnet /mnt/lustre2 (opts:) 192.168.201.150@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg150-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (75s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:36:16 (1773102976) Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre2 Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2050176 .. Extending file size to 1728608 .. done Extending file size to 8634624 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5558816 .. failed Extending file size to 7531200 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8829184 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8923360 .. Extending file size to 8975200 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 57791 57792 57794 57795 57797 failed 192.168.201.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg150-client.virtnet /mnt/lustre2 (opts:) 192.168.201.150@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg150-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (69s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:37:25 (1773103045) Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4096 2204288 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 18432 3750912 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 25600 3743744 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4096 2204288 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 18432 3750912 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 25600 3743744 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0579307 s, 36.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.14881 s, 14.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xc0237985 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg150-client.virtnet /mnt/lustre2 (opts:) PASS 200c (10s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:37:36 (1773103056) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0620486 s, 33.8 MB/s PASS 202 (4s) == sanity-flr test 203: mirror file preserve mirror ID === 20:37:40 (1773103060) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x286:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x341:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x341:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x342:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x345:0x0] } PASS 203 (5s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:37:45 (1773103065) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.132587 s, 15.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x343:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x346:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x347:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (4s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:37:49 (1773103069) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0994531 s, 21.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x344:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x345:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x348:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x344:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x345:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x348:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x349:0x0] } PASS 204b (5s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:37:55 (1773103075) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.10053 s, 20.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x346:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x346:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34b:0x0] } PASS 204c (28s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:38:22 (1773103102) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg150-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463511 s, 22.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x347:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x347:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x348:0x0] } PASS 204d (53s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:39:15 (1773103155) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg150-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x349:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100661 s, 20.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x349:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x349:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (50s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:40:05 (1773103205) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg150-server mds-ost sync done. Creating new pool test_204f oleg150-server: Pool lustre.test_204f created Adding targets to pool oleg150-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg150-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Updated after 2s: want 'lustre-OST0000_UUID lustre-OST0001_UUID ' got 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.102171 s, 20.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg150-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg150-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg150-server: Pool lustre.test_204f destroyed PASS 204f (66s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:41:11 (1773103271) PASS 205a (4s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:41:15 (1773103275) PASS 205b (4s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:41:19 (1773103279) oleg150-server: Pool lustre.test_206 created Adding targets to pool oleg150-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg150-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg150-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg150-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg150-server: Pool lustre.test_206 destroyed PASS 206 (19s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:41:38 (1773103298) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314956 s, 33.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201704 s, 52.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x7e34adee Mirror 3: 0x7e34adee PASS 207 (5s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:41:43 (1773103303) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:41:45 (1773103305) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:41:47 (1773103307) Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.08216 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2190000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1720000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1720000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1720000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1ff0000 Read this (4 bytes): BBBB 192.168.201.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg150-client.virtnet /mnt/lustre2 (opts:) PASS 209a (22s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:42:09 (1773103329) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0142242 s, 576 kB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:42:12 (1773103332) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238179 s, 44.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x2ab lmm_fid: [0x200000404:0x2ab:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x2ab lmm_fid: [0x200000404:0x2ab:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x35a:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834603 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-09 20:42:13.000000000 -0400 Modify: 2026-03-09 20:42:13.000000000 -0400 Change: 2026-03-09 20:42:14.000000000 -0400 Birth: 2026-03-09 20:42:13.000000000 -0400 PASS 210a (5s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:42:17 (1773103337) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:42:18 (1773103338) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.210537 s, 195 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0266808 s, 154 kB/s size after second write -rw-r--r-- 1 root root 4096 Mar 9 20:42 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2ad:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Mar 9 20:42 /mnt/lustre/f211.sanity-flr PASS 211 (4s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:42:22 (1773103342) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (4s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:42:26 (1773103346) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (4s) == sanity-flr test complete, duration 1269 sec =========== 20:42:30 (1773103350) === sanity-flr: start cleanup 20:42:31 (1773103351) === === sanity-flr: finish cleanup 20:42:33 (1773103353) ===