-----============= acceptance-small: sanity-flr ============----- Tue Nov 25 16:26:25 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg118-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 16:26:33 (1764105993) === oleg118-client.virtnet: executing check_config_client /mnt/lustre oleg118-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg118-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88f1040ba800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88f1040ba800.idle_timeout=debug disable quota as required oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 16:26:47 (1764106007) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7552] == sanity-flr test 0a: lfs mirror create with -N option == 16:26:48 (1764106008) PASS 0a (7s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 16:26:55 (1764106015) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 16:26:57 (1764106017) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 16:26:58 (1764106018) PASS 0d (10s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 16:27:08 (1764106028) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 16:27:09 (1764106029) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 16:27:10 (1764106030) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88f1040ba800.stats=clear osc.lustre-OST0001-osc-ffff88f1040ba800.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 16:27:14 (1764106034) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 16:27:19 (1764106039) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 16:27:22 (1764106042) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 16:27:24 (1764106044) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (5s) == sanity-flr test 2: create components from existing files ========================================================== 16:27:29 (1764106049) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 16:27:32 (1764106052) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 16:27:34 (1764106054) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 16:27:37 (1764106057) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 16:27:42 (1764106062) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 16:27:45 (1764106065) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.705281 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691697 s, 15.2 MB/s PASS 21 (5s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 16:27:51 (1764106071) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 0.16414 s, 89.4 MB/s PASS 22 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 16:27:54 (1764106074) Stopping /mnt/lustre-ost1 (opts:) on oleg118-server oleg118-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88f1040ba800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88f1040ba800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg118-server: oleg118-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg118-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg118-server oleg118-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88f1040ba800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88f1040ba800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg118-server: oleg118-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg118-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (33s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 16:28:28 (1764106108) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.2322 s, 40.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg118-server oleg118-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88f1040ba800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88f1040ba800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg118-server: oleg118-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg118-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (20s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 16:28:48 (1764106128) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg118-server oleg118-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88f1040ba800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88f1040ba800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg118-server: oleg118-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg118-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg118-server oleg118-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88f1040ba800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88f1040ba800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg118-server: oleg118-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg118-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (29s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 16:29:17 (1764106157) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.98502 s, 26.4 MB/s -rw-r--r-- 1 root root 52428800 Nov 25 16:29 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.342s user 0m0.000s sys 0m0.161s Stopping /mnt/lustre-ost1 (opts:) on oleg118-server oleg118-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88f1040ba800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88f1040ba800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.403s user 0m0.000s sys 0m0.175s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg118-server: oleg118-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg118-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg118-server oleg118-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88f1040ba800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88f1040ba800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.516s user 0m0.000s sys 0m0.159s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg118-server: oleg118-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg118-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg118-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88f1040ba800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88f1040ba800.ost_server_uuid in FULL state after 0 sec PASS 33b (38s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 16:29:55 (1764106195) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 16:29:57 (1764106197) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 16:29:59 (1764106199) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 16:30:01 (1764106201) PASS 35 (4s) == sanity-flr test 36a: write to mirrored files ========== 16:30:05 (1764106205) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.299334 s, 42.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (7s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 16:30:12 (1764106212) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215553 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164546 s, 63.7 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 16:30:15 (1764106215) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 16:30:19 (1764106219) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00088432 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000918478 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000730947 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000802358 s, 1.3 MB/s flv=10 olv=10 PASS 36d (5s) == sanity-flr test 37: mirror I/O API verification ======= 16:30:24 (1764106224) create test files with size 15747413 .. 6e7f6ce0cecdcdd7a9bae7b90e19b160 - 6ff0d389c14b34e4deeb87a661a7d3c8 - 28acda97d5422a769e056bfa5be5c441 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg118-client.virtnet /mnt/lustre (opts:) Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre PASS 37 (8s) == sanity-flr test 38: resync ============================ 16:30:32 (1764106232) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (6s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 16:30:38 (1764106238) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 16:30:41 (1764106241) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514963 s, 20.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169977 s, 61.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 16:30:45 (1764106245) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0525533 s, 39.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.147296 s, 28.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0836916 s, 25.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (7s) == sanity-flr test 42: lfs mirror verify ================= 16:30:52 (1764106252) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 16:30:54 (1764106254) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 16:30:56 (1764106256) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163326 s, 64.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 16:30:59 (1764106259) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 16:31:01 (1764106261) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4e:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 16:31:05 (1764106265) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.132094 s, 79.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg118-client.virtnet /mnt/lustre (opts:) Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (15s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 16:31:20 (1764106280) llite.lustre-ffff88f11214e000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.112188 s, 93.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 Stopping client oleg118-client.virtnet /mnt/lustre (opts:) Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 llite.lustre-ffff88f1040bb000.xattr_cache=1 PASS 44e (6s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 16:31:26 (1764106286) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (15s) == sanity-flr test 46: Verify setstripe --copy option ==== 16:31:41 (1764106301) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (15s) == sanity-flr test 47: Verify mirror obj alloc =========== 16:31:56 (1764106316) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 16:31:57 (1764106317) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0770043 s, 40.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0637668 s, 49.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xc3bb2293 (stale) Mirror 2: 0xc3bb2293 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: e510b9c0f106af36396010ddcbe0cb45 - mirror 1 checksum: 11566c1ad33de26bb1b95a8d0c4df7b5 - mirror 2 checksum: e510b9c0f106af36396010ddcbe0cb45 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Nov 25 16:31:58 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Nov 25 16:31:58 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: e510b9c0f106af36396010ddcbe0cb45 - mirror 1 checksum: 11566c1ad33de26bb1b95a8d0c4df7b5 - mirror 2 checksum: 11566c1ad33de26bb1b95a8d0c4df7b5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Nov 25 16:31:58 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Nov 25 16:31:58 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } PASS 48 (4s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 16:32:02 (1764106322) Failing mds1 on oleg118-server Stopping /mnt/lustre-mds1 (opts:) on oleg118-server 16:32:05 (1764106325) shut down facet: mds1 facet_host: oleg118-server facet_failover_host: oleg118-server Failover mds1 to oleg118-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0000 16:32:19 (1764106339) targets are mounted 16:32:19 (1764106339) facet_failover done oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170762 s, 61.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 50A (25s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 16:32:27 (1764106347) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 16:32:28 (1764106348) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 16:32:30 (1764106350) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00755942 s, 2.2 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 16:32:33 (1764106353) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 16:32:34 (1764106354) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 16:32:36 (1764106356) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (9s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 16:32:45 (1764106365) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (21s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 16:33:06 (1764106386) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 16:33:24 (1764106404) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 16:33:43 (1764106423) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 16:33:46 (1764106426) mirror create pid 52203 mirror split pid 52206 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 52203 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 52206 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 16:34:49 (1764106489) Seed set to 6489 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0xf2e3 000001 1764106489.924967 trunc from 00000000 to 0x00f2e2 (0xf2e3 bytes) 000002 1764106489.935644 read 0x008f66 thru 0x00f2e2 (0x637d bytes) 000008 1764106489.942682 read 0x00dde1 thru 0x00f2e2 (0x1502 bytes) 000009 1764106489.946132 MIRROR_EXTEND to 2 mirrors 000011 1764106490.302775 trunc from 0x00f2e3 to 0x008b7d (0x6765 bytes) 000013 1764106490.355833 mapwrite 0x0153b3 thru 0x0201f0 (0xae3e bytes) 000014 1764106490.420879 mapwrite 0x032a6c thru 0x03bb4f (0x90e4 bytes) 000016 1764106490.483444 mapwrite 0x01ba39 thru 0x0294cf (0xda97 bytes) 000017 1764106490.510803 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3bb50): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb50): Mirror 2: 0xbd1e949a (stale) CRC-32 checksum value for chunk [0, 0x3bb50): Mirror 1: 0xce7d431d Mirror 3: 0xce7d431d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb50: skipped 000020 1764106490.870480 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40b:0x0]: Invalid argument (22) 000025 1764106491.024972 MIRROR_RESYNC 2 mirrors 000026 1764106491.195907 read 0x001cb5 thru 0x00d766 (0xbab2 bytes) 000028 1764106491.200908 trunc from 0x03bb50 to 0x007810 (0x3433f bytes) 000032 1764106491.248019 write 0x0344c6 thru 0x03c912 (0x844d bytes) 000034 1764106491.250326 mapread 0x018ec9 thru 0x01b2f7 (0x242f bytes) 000035 1764106491.265392 MIRROR_RESYNC 2 mirrors 000036 1764106491.603452 mapread 0x02c5dc thru 0x036820 (0xa245 bytes) 000038 1764106491.618849 read 0x00dabe thru 0x0172cc (0x980f bytes) 000040 1764106491.624909 read 0x01031a thru 0x014e9c (0x4b83 bytes) 000042 1764106491.625703 mapwrite 0x0115a0 thru 0x0198c8 (0x8329 bytes) 000043 1764106491.714860 mapwrite 0x0393b5 thru 0x03ffff (0x6c4b bytes) 000045 1764106491.780123 mapread 0x0365c3 thru 0x03ffff (0x9a3d bytes) 000046 1764106491.781209 write 0x00e479 thru 0x01b04b (0xcbd3 bytes) 000048 1764106491.786071 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe117bd44 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x645843da Mirror 4: 0x645843da lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000051 1764106492.207711 read 0x01494e thru 0x0184d0 (0x3b83 bytes) 000052 1764106492.214232 write 0x019c0c thru 0x0226f5 (0x8aea bytes) 000053 1764106492.259043 mapwrite 0x022be5 thru 0x02c38a (0x97a6 bytes) 000054 1764106492.306841 mapread 0x030e54 thru 0x037ff5 (0x71a2 bytes) 000056 1764106492.313675 write 0x01dca4 thru 0x02321c (0x5579 bytes) 000057 1764106492.314933 trunc from 0x040000 to 0x002a27 (0x3d5d8 bytes) 000058 1764106492.353561 MIRROR_RESYNC 3 mirrors 000060 1764106492.709741 read 0x001b53 thru 0x002a27 (0xed5 bytes) 000062 1764106492.713914 mapread 0x00261e thru 0x002a27 (0x40a bytes) truncating to largest ever: 0xfaff 000064 1764106492.714884 trunc from 0x002a28 to 0x00fafe (0xd0d7 bytes) 000065 1764106492.752865 read 0x0026cc thru 0x00fafe (0xd433 bytes) 000066 1764106492.761598 MIRROR_RESYNC 3 mirrors 000067 1764106493.111202 write 0x016a74 thru 0x026a0e (0xff9b bytes) 000068 1764106493.146490 trunc from 0x026a0f to 0x00606f (0x2099f bytes) 000071 1764106493.177111 trunc from 0x006070 to 0x00366f (0x2a00 bytes) 000072 1764106493.188973 mapread 0x002991 thru 0x00366f (0xcdf bytes) 000073 1764106493.198596 mapwrite 0x020fd9 thru 0x026a25 (0x5a4d bytes) 000074 1764106493.246292 MIRROR_RESYNC 3 mirrors 000076 1764106493.566448 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40b:0x0]: Invalid argument (22) 000078 1764106493.745436 mapread 0x017549 thru 0x01febb (0x8973 bytes) truncating to largest ever: 0x27261 000080 1764106493.756673 trunc from 0x026a26 to 0x027260 (0x83b bytes) 000083 1764106493.804462 mapwrite 0x0279e4 thru 0x030def (0x940c bytes) 000086 1764106493.857888 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x30df0): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30df0): Mirror 3: 0x3cf2987d (stale) CRC-32 checksum value for chunk [0, 0x30df0): Mirror 1: 0xd42b5300 Mirror 4: 0xd42b5300 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30df0: skipped 000087 1764106494.274824 read 0x014ca7 thru 0x0199c0 (0x4d1a bytes) 000088 1764106494.280477 mapwrite 0x00f9f3 thru 0x019d4f (0xa35d bytes) 000089 1764106494.391834 write 0x01ce76 thru 0x029413 (0xc59e bytes) 000090 1764106494.396321 read 0x0185b6 thru 0x01d761 (0x51ac bytes) 000091 1764106494.396907 mapread 0x02d4a0 thru 0x02dfb7 (0xb18 bytes) 000092 1764106494.406079 read 0x00e247 thru 0x01db1b (0xf8d5 bytes) 000094 1764106494.411160 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x30df0): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30df0): Mirror 3: 0x3cf2987d (stale) Mirror 4: 0xd42b5300 (stale) CRC-32 checksum value for chunk [0, 0x30df0): Mirror 1: 0x8586eddf Mirror 5: 0x8586eddf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30df0: skipped 000096 1764106494.837487 read 0x001962 thru 0x010c08 (0xf2a7 bytes) 000097 1764106494.850279 MIRROR_RESYNC 4 mirrors 000099 1764106495.177749 mapread 0x014c81 thru 0x023b43 (0xeec3 bytes) 000100 1764106495.196696 mapread 0x010e87 thru 0x015bda (0x4d54 bytes) 000101 1764106495.200423 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40b:0x0]: Invalid argument (22) 000102 1764106495.347743 mapwrite 0x0273d2 thru 0x0338db (0xc50a bytes) 000103 1764106495.438147 trunc from 0x0338dc to 0x01a6d6 (0x19205 bytes) 000105 1764106495.448815 mapwrite 0x022503 thru 0x022f84 (0xa82 bytes) 000106 1764106495.479778 read 0x011aa6 thru 0x01ff17 (0xe472 bytes) truncating to largest ever: 0x33c3a 000107 1764106495.485008 trunc from 0x022f85 to 0x033c39 (0x10cb5 bytes) 000108 1764106495.493678 write 0x03015c thru 0x03a9ca (0xa86f bytes) 000109 1764106495.499238 write 0x02a667 thru 0x038f33 (0xe8cd bytes) 000110 1764106495.503941 MIRROR_RESYNC 3 mirrors 000111 1764106495.814283 mapread 0x016669 thru 0x0253a9 (0xed41 bytes) 000113 1764106495.831042 write 0x00f0e4 thru 0x01710f (0x802c bytes) 000114 1764106495.877431 write 0x036338 thru 0x03ffff (0x9cc8 bytes) 000118 1764106495.884857 read 0x006f06 thru 0x00cb23 (0x5c1e bytes) 000119 1764106495.888858 read 0x02672b thru 0x02c0fb (0x59d1 bytes) 000120 1764106495.895433 mapwrite 0x01ad11 thru 0x021255 (0x6545 bytes) 000121 1764106495.951079 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x9d15836a (stale) Mirror 5: 0x9d15836a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc1cbbc7 Mirror 6: 0xc1cbbc7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000122 1764106496.406569 trunc from 0x040000 to 0x015906 (0x2a6f9 bytes) 000123 1764106496.458320 read 0x01265e thru 0x015906 (0x32a9 bytes) 000124 1764106496.466319 MIRROR_RESYNC 4 mirrors 000125 1764106496.911465 write 0x02bab9 thru 0x0329d3 (0x6f1b bytes) 000126 1764106496.951448 mapread 0x0092ac thru 0x0167bd (0xd512 bytes) 000127 1764106496.981060 mapread 0x013afd thru 0x01efe0 (0xb4e4 bytes) 000129 1764106496.994173 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x329d4): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x329d4): Mirror 4: 0x9e74de73 (stale) Mirror 5: 0x9e74de73 (stale) Mirror 6: 0x9e74de73 (stale) CRC-32 checksum value for chunk [0, 0x329d4): Mirror 1: 0x3e1662b5 Mirror 7: 0x3e1662b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x329d4: skipped 000131 1764106497.564831 mapwrite 0x02090e thru 0x025838 (0x4f2b bytes) 000132 1764106497.693119 mapread 0x01735f thru 0x01e0c5 (0x6d67 bytes) 000133 1764106497.699650 write 0x00646a thru 0x0105f7 (0xa18e bytes) 000135 1764106497.711574 mapwrite 0x03ce38 thru 0x03ffff (0x31c8 bytes) 000136 1764106497.770788 mapread 0x0005b1 thru 0x002d77 (0x27c7 bytes) 000137 1764106497.776687 trunc from 0x040000 to 0x03323e (0xcdc1 bytes) 000140 1764106497.788667 read 0x0127b1 thru 0x019740 (0x6f90 bytes) 000141 1764106497.789438 trunc from 0x03323f to 0x00045d (0x32de1 bytes) 000147 1764106497.801389 mapwrite 0x006b7e thru 0x0084fe (0x1981 bytes) 000148 1764106497.857499 mapwrite 0x00f3de thru 0x00fda5 (0x9c8 bytes) 000149 1764106497.889777 mapwrite 0x012c6f thru 0x01eeb3 (0xc245 bytes) 000151 1764106497.943724 write 0x031832 thru 0x03ffff (0xe7ce bytes) 000152 1764106497.945381 read 0x02f71a thru 0x031373 (0x1c5a bytes) 000153 1764106497.954287 write 0x022a58 thru 0x023ae7 (0x1090 bytes) 000154 1764106497.955230 trunc from 0x040000 to 0x00308e (0x3cf71 bytes) 000155 1764106497.968712 mapwrite 0x039acd thru 0x03baaf (0x1fe3 bytes) 000156 1764106498.015734 mapwrite 0x01feb2 thru 0x02e632 (0xe781 bytes) 000160 1764106498.104871 mapread 0x0145c2 thru 0x0200ed (0xbb2c bytes) 000162 1764106498.153315 mapread 0x00a398 thru 0x017f3e (0xdba7 bytes) 000167 1764106498.190817 trunc from 0x03bab0 to 0x02a103 (0x119ac bytes) 000168 1764106498.199769 mapread 0x0287bb thru 0x02a103 (0x1949 bytes) 000172 1764106498.202093 trunc from 0x02a104 to 0x019d12 (0x103f1 bytes) 000174 1764106498.212510 mapwrite 0x03f14c thru 0x03ffff (0xeb4 bytes) 000177 1764106498.248543 mapread 0x021735 thru 0x02d524 (0xbdf0 bytes) 000178 1764106498.290860 write 0x03abfa thru 0x03c276 (0x167d bytes) 000179 1764106498.298679 write 0x036180 thru 0x03ffff (0x9e80 bytes) 000180 1764106498.304078 mapread 0x01abd9 thru 0x0274a2 (0xc8ca bytes) 000181 1764106498.337813 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xff8399a6 (stale) Mirror 5: 0xff8399a6 (stale) Mirror 6: 0xff8399a6 (stale) Mirror 7: 0x83b2b7f2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3f65d869 Mirror 8: 0x3f65d869 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000182 1764106498.719844 write 0x03b173 thru 0x03ffff (0x4e8d bytes) 000183 1764106498.753157 write 0x037155 thru 0x038d1e (0x1bca bytes) 000184 1764106498.760629 mapread 0x00feeb thru 0x01a031 (0xa147 bytes) 000188 1764106498.776183 write 0x02d76d thru 0x034060 (0x68f4 bytes) 000189 1764106498.784308 trunc from 0x040000 to 0x01d7c5 (0x2283a bytes) 000190 1764106498.808129 mapread 0x01cf5a thru 0x01d7c5 (0x86c bytes) 000193 1764106498.810053 mapread 0x00a7cc thru 0x00aaf2 (0x327 bytes) 000194 1764106498.810912 mapread 0x00c83b thru 0x017117 (0xa8dd bytes) 000195 1764106498.811788 mapread 0x000123 thru 0x0095ba (0x9498 bytes) 000196 1764106498.822876 MIRROR_RESYNC 6 mirrors 000197 1764106499.387097 trunc from 0x01d7c6 to 0x0296c2 (0xbefd bytes) 000198 1764106499.444817 write 0x03c33b thru 0x03ffff (0x3cc5 bytes) 000200 1764106499.446346 write 0x03e614 thru 0x03ffff (0x19ec bytes) 000201 1764106499.447113 mapwrite 0x02410d thru 0x0241d0 (0xc4 bytes) 000205 1764106499.478969 read 0x00b941 thru 0x00bbaf (0x26f bytes) 000206 1764106499.483050 mapwrite 0x01c711 thru 0x029eb2 (0xd7a2 bytes) 000207 1764106499.528355 mapread 0x000a5d thru 0x00e331 (0xd8d5 bytes) 000208 1764106499.538099 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000211 1764106499.738444 read 0x039524 thru 0x03ffff (0x6adc bytes) 000212 1764106499.743334 read 0x032108 thru 0x034571 (0x246a bytes) 000213 1764106499.750667 write 0x01c982 thru 0x0287f1 (0xbe70 bytes) 000216 1764106499.753185 read 0x0348a9 thru 0x0358e9 (0x1041 bytes) 000218 1764106499.754128 write 0x026a14 thru 0x035fd1 (0xf5be bytes) 000220 1764106499.759166 MIRROR_RESYNC 5 mirrors 000222 1764106500.318312 read 0x03ce52 thru 0x03ffff (0x31ae bytes) 000224 1764106500.322331 mapwrite 0x009dad thru 0x00c995 (0x2be9 bytes) 000227 1764106500.418645 mapread 0x001ef0 thru 0x0069fb (0x4b0c bytes) 000229 1764106500.427178 MIRROR_RESYNC 5 mirrors 000234 1764106500.897751 MIRROR_EXTEND to 6 mirrors 000235 1764106501.310463 read 0x035246 thru 0x03ffff (0xadba bytes) 000237 1764106501.320713 write 0x035b15 thru 0x03ffff (0xa4eb bytes) 000238 1764106501.390774 read 0x037d14 thru 0x03db62 (0x5e4f bytes) 000239 1764106501.391405 mapread 0x02800c thru 0x03522f (0xd224 bytes) 000244 1764106501.402208 trunc from 0x040000 to 0x014ed6 (0x2b129 bytes) 000245 1764106501.435836 mapwrite 0x039a05 thru 0x03d06a (0x3666 bytes) 000247 1764106501.518657 mapwrite 0x01aa6f thru 0x025689 (0xac1b bytes) 000250 1764106501.606082 mapread 0x03a43c thru 0x03d06a (0x2c2f bytes) 000252 1764106501.607592 mapread 0x0226c0 thru 0x02dc91 (0xb5d2 bytes) 000253 1764106501.650196 mapread 0x039efe thru 0x03cea1 (0x2fa4 bytes) 000254 1764106501.651215 mapwrite 0x0058ff thru 0x00ba07 (0x6109 bytes) 000256 1764106501.703711 write 0x03d831 thru 0x03ffff (0x27cf bytes) 000257 1764106501.704619 mapread 0x000f4a thru 0x00a0c3 (0x917a bytes) 000258 1764106501.705484 trunc from 0x040000 to 0x0284e6 (0x17b19 bytes) 000260 1764106501.721543 mapread 0x020d03 thru 0x0284e6 (0x77e4 bytes) 000261 1764106501.722823 mapwrite 0x024559 thru 0x03078a (0xc232 bytes) 000263 1764106501.817242 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3078b): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3078b): Mirror 5: 0x775f553c (stale) Mirror 6: 0x775f553c (stale) Mirror 7: 0x775f553c (stale) Mirror 8: 0x775f553c (stale) Mirror 9: 0x775f553c (stale) CRC-32 checksum value for chunk [0, 0x3078b): Mirror 1: 0x60727a9b Mirror 10: 0x60727a9b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3078b: skipped 000266 1764106502.297950 trunc from 0x03078b to 0x00e35e (0x2242c bytes) 000268 1764106502.350083 trunc from 0x00e35f to 0x00d3c8 (0xf96 bytes) 000271 1764106502.366677 mapread 0x001703 thru 0x007b72 (0x6470 bytes) 000272 1764106502.392521 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd3c9): Mirror 6: 0xfc388bf0 (stale) Mirror 7: 0xfc388bf0 (stale) Mirror 8: 0xfc388bf0 (stale) Mirror 9: 0xfc388bf0 (stale) Mirror 10: 0x546faffb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd3c9): Mirror 6: 0xfc388bf0 (stale) Mirror 7: 0xfc388bf0 (stale) Mirror 8: 0xfc388bf0 (stale) Mirror 9: 0xfc388bf0 (stale) Mirror 10: 0x546faffb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd3c9: skipped 000273 1764106502.610933 MIRROR_RESYNC 6 mirrors 000274 1764106503.127432 mapwrite 0x00f739 thru 0x01069c (0xf64 bytes) 000275 1764106503.246251 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1069d): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1069d): Mirror 6: 0x7d8633d1 (stale) Mirror 7: 0x7d8633d1 (stale) Mirror 8: 0x7d8633d1 (stale) Mirror 9: 0x7d8633d1 (stale) Mirror 10: 0x7d8633d1 (stale) CRC-32 checksum value for chunk [0, 0x1069d): Mirror 1: 0x27d75165 Mirror 11: 0x27d75165 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1069d: skipped 000277 1764106503.671444 mapread 0x00c12e thru 0x01069c (0x456f bytes) 000278 1764106503.694203 read 0x00a6b0 thru 0x01069c (0x5fed bytes) 000279 1764106503.701244 read 0x00c849 thru 0x01069c (0x3e54 bytes) 000281 1764106503.711852 read 0x003595 thru 0x01069c (0xd108 bytes) 000283 1764106503.726109 mapwrite 0x028fc8 thru 0x02f601 (0x663a bytes) 000285 1764106503.827321 write 0x028c4f thru 0x032d86 (0xa138 bytes) 000286 1764106503.834689 MIRROR_RESYNC 7 mirrors 000287 1764106504.516731 write 0x038d67 thru 0x03ffff (0x7299 bytes) 000288 1764106504.567268 write 0x01d92a thru 0x02bf12 (0xe5e9 bytes) 000289 1764106504.581295 trunc from 0x040000 to 0x028ab5 (0x1754a bytes) 000293 1764106504.603653 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28ab6): Mirror 7: 0x365af60 (stale) Mirror 8: 0x365af60 (stale) Mirror 9: 0x365af60 (stale) Mirror 10: 0x365af60 (stale) Mirror 11: 0x365af60 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28ab6): Mirror 7: 0x365af60 (stale) Mirror 8: 0x365af60 (stale) Mirror 9: 0x365af60 (stale) Mirror 10: 0x365af60 (stale) Mirror 11: 0x365af60 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28ab6: skipped 000294 1764106504.942021 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x28ab6): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28ab6): Mirror 7: 0x365af60 (stale) Mirror 8: 0x365af60 (stale) Mirror 9: 0x365af60 (stale) Mirror 10: 0x365af60 (stale) Mirror 11: 0x365af60 (stale) CRC-32 checksum value for chunk [0, 0x28ab6): Mirror 1: 0x19e13355 Mirror 12: 0x19e13355 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28ab6: skipped 000295 1764106505.461420 mapread 0x027f4f thru 0x028ab5 (0xb67 bytes) 000298 1764106505.513290 MIRROR_RESYNC 7 mirrors 000299 1764106506.078182 mapread 0x0240c1 thru 0x028ab5 (0x49f5 bytes) 000300 1764106506.092643 read 0x026545 thru 0x0277c6 (0x1282 bytes) 000301 1764106506.093038 write 0x03f22d thru 0x03ffff (0xdd3 bytes) 000302 1764106506.127495 mapwrite 0x01ca15 thru 0x022eed (0x64d9 bytes) 000303 1764106506.180552 write 0x005f4a thru 0x009431 (0x34e8 bytes) 000307 1764106506.191047 mapwrite 0x017224 thru 0x018c7a (0x1a57 bytes) 000309 1764106506.226691 write 0x00f7f6 thru 0x0101de (0x9e9 bytes) truncating to largest ever: 0x3eca7 000310 1764106506.227695 trunc from 0x040000 to 0x03eca6 (0x1359 bytes) 000311 1764106506.245050 mapwrite 0x01786c thru 0x01e592 (0x6d27 bytes) 000312 1764106506.275510 write 0x03fbb0 thru 0x03ffff (0x450 bytes) 000313 1764106506.276359 mapread 0x03b061 thru 0x03f9c9 (0x4969 bytes) 000314 1764106506.282709 mapread 0x0155c7 thru 0x01a4ee (0x4f28 bytes) 000316 1764106506.283945 mapread 0x021a58 thru 0x02723f (0x57e8 bytes) 000317 1764106506.291210 write 0x02d415 thru 0x0366ee (0x92da bytes) 000319 1764106506.297050 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x82b61844 (stale) Mirror 8: 0x82b61844 (stale) Mirror 9: 0x82b61844 (stale) Mirror 11: 0x82b61844 (stale) Mirror 12: 0x82b61844 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x82b61844 (stale) Mirror 8: 0x82b61844 (stale) Mirror 9: 0x82b61844 (stale) Mirror 11: 0x82b61844 (stale) Mirror 12: 0x82b61844 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000320 1764106506.530817 mapread 0x01f16e thru 0x0215b7 (0x244a bytes) 000321 1764106506.544948 write 0x01700d thru 0x0173fc (0x3f0 bytes) 000322 1764106506.553654 read 0x0236cc thru 0x02d3e2 (0x9d17 bytes) 000323 1764106506.559184 trunc from 0x040000 to 0x02aa63 (0x1559c bytes) 000324 1764106506.574540 read 0x029f5a thru 0x02aa63 (0xb0a bytes) 000325 1764106506.575498 mapwrite 0x0249c0 thru 0x02b16e (0x67af bytes) 000328 1764106506.638587 write 0x0137e1 thru 0x01cc09 (0x9429 bytes) 000334 1764106506.646883 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2b16f): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b16f): Mirror 7: 0x95b2c007 (stale) Mirror 8: 0x95b2c007 (stale) Mirror 9: 0x95b2c007 (stale) Mirror 11: 0x95b2c007 (stale) Mirror 12: 0x95b2c007 (stale) CRC-32 checksum value for chunk [0, 0x2b16f): Mirror 1: 0x3820674a Mirror 13: 0x3820674a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b16f: skipped 000337 1764106507.052203 mapwrite 0x01e21f thru 0x01e8ea (0x6cc bytes) 000338 1764106507.148621 read 0x01945c thru 0x019eee (0xa93 bytes) 000343 1764106507.153614 write 0x015fb7 thru 0x024559 (0xe5a3 bytes) 000344 1764106507.164801 read 0x024205 thru 0x02b16e (0x6f6a bytes) 000347 1764106507.176160 read 0x006f99 thru 0x008180 (0x11e8 bytes) 000348 1764106507.176776 write 0x034672 thru 0x03d1a7 (0x8b36 bytes) 000351 1764106507.180138 write 0x00d12e thru 0x0154a0 (0x8373 bytes) 000352 1764106507.181259 write 0x0209c3 thru 0x02754e (0x6b8c bytes) 000353 1764106507.183934 write 0x007063 thru 0x01388d (0xc82b bytes) 000354 1764106507.185489 read 0x018de0 thru 0x01e17a (0x539b bytes) 000355 1764106507.191964 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3d1a8): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d1a8): Mirror 7: 0xcbddb911 (stale) Mirror 8: 0xcbddb911 (stale) Mirror 9: 0xcbddb911 (stale) Mirror 11: 0xcbddb911 (stale) Mirror 12: 0xcbddb911 (stale) Mirror 13: 0x2ae4ad5e (stale) CRC-32 checksum value for chunk [0, 0x3d1a8): Mirror 1: 0xa7447e44 Mirror 14: 0xa7447e44 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d1a8: skipped 000358 1764106507.587556 mapread 0x029e38 thru 0x02e81b (0x49e4 bytes) 000359 1764106507.604358 write 0x0206ba thru 0x023c9e (0x35e5 bytes) 000360 1764106507.649300 mapwrite 0x0393c3 thru 0x03ffff (0x6c3d bytes) 000363 1764106507.722455 read 0x023300 thru 0x026594 (0x3295 bytes) 000365 1764106507.727350 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x82b61844 (stale) Mirror 8: 0x82b61844 (stale) Mirror 9: 0x82b61844 (stale) Mirror 11: 0x82b61844 (stale) Mirror 12: 0x82b61844 (stale) Mirror 13: 0xf9170687 (stale) Mirror 14: 0xafd04710 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7df61760 Mirror 15: 0x7df61760 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000367 1764106508.109171 write 0x0169ce thru 0x020647 (0x9c7a bytes) 000368 1764106508.155266 read 0x000b83 thru 0x00ef5a (0xe3d8 bytes) 000370 1764106508.161432 trunc from 0x040000 to 0x00e1f6 (0x31e09 bytes) 000371 1764106508.184488 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0xe1f7): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe1f7): Mirror 7: 0x97480a59 (stale) Mirror 8: 0x97480a59 (stale) Mirror 9: 0x97480a59 (stale) Mirror 11: 0x97480a59 (stale) Mirror 12: 0x97480a59 (stale) Mirror 13: 0x482de30b (stale) Mirror 14: 0xb2e597a1 (stale) Mirror 15: 0xb2e597a1 (stale) CRC-32 checksum value for chunk [0, 0xe1f7): Mirror 1: 0xb2e597a1 Mirror 16: 0xb2e597a1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe1f7: skipped 000372 1764106508.569660 read 0x00ab3a thru 0x00e1f6 (0x36bd bytes) 000374 1764106508.578293 write 0x018c1a thru 0x022322 (0x9709 bytes) 000376 1764106508.611421 mapread 0x018106 thru 0x01a273 (0x216e bytes) 000377 1764106508.612358 write 0x010dff thru 0x01b66d (0xa86f bytes) 000381 1764106508.620290 MIRROR_RESYNC 10 mirrors 000382 1764106509.288363 mapread 0x009944 thru 0x011405 (0x7ac2 bytes) 000383 1764106509.302628 mapread 0x00edf1 thru 0x016769 (0x7979 bytes) 000384 1764106509.308423 write 0x010a55 thru 0x014af0 (0x409c bytes) 000389 1764106509.347652 mapread 0x0207af thru 0x022322 (0x1b74 bytes) 000390 1764106509.353232 trunc from 0x022323 to 0x006a4f (0x1b8d3 bytes) 000391 1764106509.369632 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6a50): Mirror 7: 0xa7bd7b43 (stale) Mirror 8: 0xa7bd7b43 (stale) Mirror 9: 0xa7bd7b43 (stale) Mirror 11: 0xa7bd7b43 (stale) Mirror 12: 0xa7bd7b43 (stale) Mirror 14: 0xa7bd7b43 (stale) Mirror 15: 0xa7bd7b43 (stale) Mirror 16: 0xa7bd7b43 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6a50): Mirror 7: 0xa7bd7b43 (stale) Mirror 8: 0xa7bd7b43 (stale) Mirror 9: 0xa7bd7b43 (stale) Mirror 11: 0xa7bd7b43 (stale) Mirror 12: 0xa7bd7b43 (stale) Mirror 14: 0xa7bd7b43 (stale) Mirror 15: 0xa7bd7b43 (stale) Mirror 16: 0xa7bd7b43 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6a50: skipped 000393 1764106509.570439 read 0x003193 thru 0x003412 (0x280 bytes) 000396 1764106509.575155 trunc from 0x006a50 to 0x01e844 (0x17df5 bytes) 000397 1764106509.588347 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e845): Mirror 7: 0x4bda3acd (stale) Mirror 8: 0x4bda3acd (stale) Mirror 9: 0x4bda3acd (stale) Mirror 11: 0x4bda3acd (stale) Mirror 12: 0x4bda3acd (stale) Mirror 14: 0x4bda3acd (stale) Mirror 15: 0x4bda3acd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e845): Mirror 7: 0x4bda3acd (stale) Mirror 8: 0x4bda3acd (stale) Mirror 9: 0x4bda3acd (stale) Mirror 11: 0x4bda3acd (stale) Mirror 12: 0x4bda3acd (stale) Mirror 14: 0x4bda3acd (stale) Mirror 15: 0x4bda3acd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e845: skipped 000398 1764106509.861776 write 0x03f928 thru 0x03ffff (0x6d8 bytes) 000400 1764106509.863548 mapread 0x03adb4 thru 0x03b7c5 (0xa12 bytes) 000401 1764106509.870237 mapwrite 0x02d7ac thru 0x02de7f (0x6d4 bytes) 000402 1764106509.899513 mapread 0x023137 thru 0x02cc40 (0x9b0a bytes) 000404 1764106509.905592 mapread 0x013e87 thru 0x01c101 (0x827b bytes) 000406 1764106509.915878 MIRROR_RESYNC 8 mirrors 000408 1764106510.553423 write 0x00dad5 thru 0x01c607 (0xeb33 bytes) 000409 1764106510.609185 trunc from 0x040000 to 0x01641a (0x29be5 bytes) 000410 1764106510.621223 trunc from 0x01641b to 0x0181b3 (0x1d99 bytes) 000416 1764106510.631289 read 0x01131f thru 0x017352 (0x6034 bytes) 000417 1764106510.637443 MIRROR_RESYNC 8 mirrors 000418 1764106511.329747 trunc from 0x0181b4 to 0x00ffe0 (0x81d3 bytes) 000420 1764106511.381100 read 0x00b99e thru 0x00ff35 (0x4598 bytes) 000421 1764106511.388427 mapread 0x0099c6 thru 0x00c2a9 (0x28e4 bytes) 000424 1764106511.404794 read 0x00f13c thru 0x00ffe0 (0xea5 bytes) 000426 1764106511.406014 write 0x012d21 thru 0x01cca2 (0x9f82 bytes) 000427 1764106511.407440 trunc from 0x01cca3 to 0x00b132 (0x11b70 bytes) 000428 1764106511.427573 trunc from 0x00b133 to 0x010265 (0x5133 bytes) 000431 1764106511.443777 trunc from 0x010266 to 0x02485e (0x145f9 bytes) 000432 1764106511.468130 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2485f): Mirror 7: 0x78f956a (stale) Mirror 8: 0x78f956a (stale) Mirror 11: 0x78f956a (stale) Mirror 12: 0x78f956a (stale) Mirror 14: 0x78f956a (stale) Mirror 15: 0x78f956a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2485f): Mirror 7: 0x78f956a (stale) Mirror 8: 0x78f956a (stale) Mirror 11: 0x78f956a (stale) Mirror 12: 0x78f956a (stale) Mirror 14: 0x78f956a (stale) Mirror 15: 0x78f956a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2485f: skipped 000434 1764106511.775745 mapwrite 0x0073e4 thru 0x00a8f5 (0x3512 bytes) 000435 1764106511.842201 write 0x010321 thru 0x011b67 (0x1847 bytes) 000436 1764106511.843117 mapwrite 0x00aeaf thru 0x00c049 (0x119b bytes) 000441 1764106511.892898 write 0x029750 thru 0x02cc2a (0x34db bytes) 000442 1764106511.897330 mapread 0x017b19 thru 0x026cd6 (0xf1be bytes) 000443 1764106511.914764 read 0x0246c3 thru 0x02cc2a (0x8568 bytes) 000444 1764106511.915410 write 0x03438f thru 0x03ffff (0xbc71 bytes) 000447 1764106511.917683 write 0x02c0a5 thru 0x033a0b (0x7967 bytes) 000448 1764106511.922074 mapread 0x03fccb thru 0x03ffff (0x335 bytes) 000449 1764106511.922875 mapwrite 0x002e9e thru 0x00be6d (0x8fd0 bytes) 000450 1764106511.974585 write 0x022023 thru 0x02660d (0x45eb bytes) 000451 1764106511.975597 trunc from 0x040000 to 0x038f7f (0x7080 bytes) 000452 1764106511.993419 write 0x03d201 thru 0x03ffff (0x2dff bytes) 000453 1764106511.994434 trunc from 0x040000 to 0x021153 (0x1eeac bytes) 000454 1764106512.011095 read 0x011d4e thru 0x01515f (0x3412 bytes) 000458 1764106512.013109 write 0x00472c thru 0x00d6f6 (0x8fcb bytes) 000459 1764106512.014451 mapwrite 0x00ef92 thru 0x01951e (0xa58d bytes) 000460 1764106512.059868 trunc from 0x021154 to 0x033c5b (0x12b08 bytes) 000461 1764106512.079111 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x33c5c): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33c5c): Mirror 7: 0xa49127bc (stale) Mirror 8: 0xa49127bc (stale) Mirror 11: 0xa49127bc (stale) Mirror 12: 0xa49127bc (stale) Mirror 14: 0xa49127bc (stale) Mirror 15: 0xa49127bc (stale) CRC-32 checksum value for chunk [0, 0x33c5c): Mirror 1: 0xc409fc10 Mirror 16: 0xc409fc10 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33c5c: skipped 000463 1764106512.642304 read 0x015143 thru 0x01935c (0x421a bytes) 000464 1764106512.652171 write 0x0081db thru 0x011a8d (0x98b3 bytes) 000466 1764106512.717597 mapwrite 0x02e00c thru 0x03bbfd (0xdbf2 bytes) 000467 1764106512.791420 mapwrite 0x0286fc thru 0x02dff4 (0x58f9 bytes) 000472 1764106512.833334 MIRROR_RESYNC 8 mirrors 000473 1764106513.676494 mapread 0x01048a thru 0x01d1b8 (0xcd2f bytes) 000476 1764106513.701618 MIRROR_EXTEND to 9 mirrors 000477 1764106514.126674 mapwrite 0x007156 thru 0x01362b (0xc4d6 bytes) 000480 1764106514.237599 trunc from 0x03bbfe to 0x005b4c (0x360b1 bytes) 000481 1764106514.253602 trunc from 0x005b4d to 0x02d107 (0x275bb bytes) 000484 1764106514.266331 MIRROR_RESYNC 9 mirrors 000487 1764106514.959323 trunc from 0x02d108 to 0x01a288 (0x12e7f bytes) 000488 1764106515.020172 read 0x0077a2 thru 0x014008 (0xc867 bytes) 000489 1764106515.028854 write 0x01f519 thru 0x02f4b5 (0xff9d bytes) 000490 1764106515.030381 mapwrite 0x035965 thru 0x039225 (0x38c1 bytes) 000491 1764106515.099278 read 0x00f179 thru 0x0190fb (0x9f83 bytes) 000492 1764106515.107516 write 0x012245 thru 0x01db5d (0xb919 bytes) 000494 1764106515.112507 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x39226): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39226): Mirror 7: 0x7be25b98 (stale) Mirror 8: 0x7be25b98 (stale) Mirror 11: 0x7be25b98 (stale) Mirror 12: 0x7be25b98 (stale) Mirror 14: 0x7be25b98 (stale) Mirror 15: 0x7be25b98 (stale) Mirror 16: 0x7be25b98 (stale) Mirror 17: 0x7be25b98 (stale) CRC-32 checksum value for chunk [0, 0x39226): Mirror 1: 0x41974b53 Mirror 18: 0x41974b53 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39226: skipped 000495 1764106515.626246 mapwrite 0x003b7b thru 0x008bd9 (0x505f bytes) 000496 1764106515.742160 read 0x0353aa thru 0x035d3a (0x991 bytes) 000497 1764106515.749168 mapwrite 0x037cbf thru 0x03ffff (0x8341 bytes) 000502 1764106515.833344 mapwrite 0x02baa8 thru 0x03a9ba (0xef13 bytes) 000503 1764106515.873962 write 0x01e010 thru 0x026755 (0x8746 bytes) 000504 1764106515.881379 write 0x0160c1 thru 0x022fdb (0xcf1b bytes) 000508 1764106515.890659 write 0x00c060 thru 0x00eec0 (0x2e61 bytes) 000509 1764106515.894693 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd636222e (stale) Mirror 8: 0xd636222e (stale) Mirror 11: 0xd636222e (stale) Mirror 14: 0xd636222e (stale) Mirror 15: 0xd636222e (stale) Mirror 16: 0xd636222e (stale) Mirror 17: 0xd636222e (stale) Mirror 18: 0x98f4f0ba (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd636222e (stale) Mirror 8: 0xd636222e (stale) Mirror 11: 0xd636222e (stale) Mirror 14: 0xd636222e (stale) Mirror 15: 0xd636222e (stale) Mirror 16: 0xd636222e (stale) Mirror 17: 0xd636222e (stale) Mirror 18: 0x98f4f0ba (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000511 1764106516.222111 mapread 0x00b59b thru 0x01537f (0x9de5 bytes) 000512 1764106516.240523 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd636222e (stale) Mirror 8: 0xd636222e (stale) Mirror 11: 0xd636222e (stale) Mirror 14: 0xd636222e (stale) Mirror 15: 0xd636222e (stale) Mirror 16: 0xd636222e (stale) Mirror 17: 0xd636222e (stale) Mirror 18: 0x98f4f0ba (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcd0bafd Mirror 19: 0xcd0bafd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000513 1764106516.781097 trunc from 0x040000 to 0x01b853 (0x247ac bytes) 000514 1764106516.845664 write 0x0009f3 thru 0x006da6 (0x63b4 bytes) 000515 1764106516.852862 trunc from 0x01b854 to 0x039aaa (0x1e257 bytes) 000516 1764106516.869776 write 0x023ea8 thru 0x029d69 (0x5ec2 bytes) 000517 1764106516.882492 write 0x017bc4 thru 0x026d04 (0xf141 bytes) 000519 1764106516.891098 write 0x01d0a4 thru 0x02806e (0xafcb bytes) 000520 1764106516.892546 write 0x023301 thru 0x02a2e0 (0x6fe0 bytes) 000521 1764106516.909930 MIRROR_RESYNC 10 mirrors 000523 1764106517.782489 trunc from 0x039aab to 0x03a57f (0xad5 bytes) 000524 1764106517.820195 trunc from 0x03a580 to 0x021791 (0x18dee bytes) 000527 1764106517.833485 mapwrite 0x038ba2 thru 0x03ffff (0x745e bytes) 000528 1764106517.903497 write 0x017bbe thru 0x022413 (0xa856 bytes) 000529 1764106517.916407 trunc from 0x040000 to 0x00f9c9 (0x30636 bytes) 000530 1764106517.928826 read 0x009990 thru 0x00de2a (0x449b bytes) 000531 1764106517.936201 trunc from 0x00f9ca to 0x005873 (0xa156 bytes) 000535 1764106517.953350 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x5874): Mirror 7: 0x7f39b059 (stale) Mirror 8: 0x7f39b059 (stale) Mirror 11: 0x7f39b059 (stale) Mirror 14: 0x7f39b059 (stale) Mirror 15: 0x7f39b059 (stale) Mirror 16: 0x7f39b059 (stale) Mirror 17: 0x7f39b059 (stale) Mirror 18: 0x7f39b059 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5874): Mirror 7: 0x7f39b059 (stale) Mirror 8: 0x7f39b059 (stale) Mirror 11: 0x7f39b059 (stale) Mirror 14: 0x7f39b059 (stale) Mirror 15: 0x7f39b059 (stale) Mirror 16: 0x7f39b059 (stale) Mirror 17: 0x7f39b059 (stale) Mirror 18: 0x7f39b059 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5874: skipped 000536 1764106518.262854 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x5874): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5874): Mirror 7: 0x7f39b059 (stale) Mirror 8: 0x7f39b059 (stale) Mirror 11: 0x7f39b059 (stale) Mirror 14: 0x7f39b059 (stale) Mirror 15: 0x7f39b059 (stale) Mirror 16: 0x7f39b059 (stale) Mirror 17: 0x7f39b059 (stale) Mirror 18: 0x7f39b059 (stale) CRC-32 checksum value for chunk [0, 0x5874): Mirror 1: 0x7f39b059 Mirror 19: 0x7f39b059 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5874: skipped 000537 1764106518.835324 read 0x0026fd thru 0x005873 (0x3177 bytes) 000540 1764106518.853678 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x5874): Mirror 8: 0x7f39b059 (stale) Mirror 11: 0x7f39b059 (stale) Mirror 14: 0x7f39b059 (stale) Mirror 15: 0x7f39b059 (stale) Mirror 16: 0x7f39b059 (stale) Mirror 17: 0x7f39b059 (stale) Mirror 18: 0x7f39b059 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5874): Mirror 8: 0x7f39b059 (stale) Mirror 11: 0x7f39b059 (stale) Mirror 14: 0x7f39b059 (stale) Mirror 15: 0x7f39b059 (stale) Mirror 16: 0x7f39b059 (stale) Mirror 17: 0x7f39b059 (stale) Mirror 18: 0x7f39b059 (stale) CRC-32 checksum value for chunk [0, 0x5874): Mirror 1: 0x7f39b059 Mirror 19: 0x7f39b059 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5874: skipped 000542 1764106519.240715 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x5874): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5874): Mirror 8: 0x7f39b059 (stale) Mirror 11: 0x7f39b059 (stale) Mirror 14: 0x7f39b059 (stale) Mirror 15: 0x7f39b059 (stale) Mirror 16: 0x7f39b059 (stale) Mirror 17: 0x7f39b059 (stale) Mirror 18: 0x7f39b059 (stale) CRC-32 checksum value for chunk [0, 0x5874): Mirror 1: 0x7f39b059 Mirror 19: 0x7f39b059 Mirror 20: 0x7f39b059 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5874: skipped 000543 1764106519.846635 MIRROR_RESYNC 10 mirrors 000544 1764106520.556561 mapread 0x000a2c thru 0x00112b (0x700 bytes) 000545 1764106520.563522 write 0x012930 thru 0x01cbff (0xa2d0 bytes) 000546 1764106520.607254 write 0x01db5f thru 0x022c2d (0x50cf bytes) 000548 1764106520.608719 mapwrite 0x02fdc7 thru 0x0321a7 (0x23e1 bytes) 000549 1764106520.651865 mapwrite 0x003f34 thru 0x00cb85 (0x8c52 bytes) 000550 1764106520.686693 read 0x0191f1 thru 0x025a78 (0xc888 bytes) 000551 1764106520.691049 write 0x037960 thru 0x0380f5 (0x796 bytes) 000556 1764106520.693712 trunc from 0x0380f6 to 0x0383de (0x2e9 bytes) 000557 1764106520.706256 mapread 0x022e7f thru 0x03206d (0xf1ef bytes) 000560 1764106520.715311 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x383df): Mirror 8: 0x711489d2 (stale) Mirror 11: 0x711489d2 (stale) Mirror 14: 0x711489d2 (stale) Mirror 15: 0x711489d2 (stale) Mirror 16: 0x711489d2 (stale) Mirror 18: 0x711489d2 (stale) Mirror 19: 0x711489d2 (stale) Mirror 20: 0x711489d2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x383df): Mirror 8: 0x711489d2 (stale) Mirror 11: 0x711489d2 (stale) Mirror 14: 0x711489d2 (stale) Mirror 15: 0x711489d2 (stale) Mirror 16: 0x711489d2 (stale) Mirror 18: 0x711489d2 (stale) Mirror 19: 0x711489d2 (stale) Mirror 20: 0x711489d2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x383df: skipped 000562 1764106521.007473 mapread 0x033279 thru 0x0383de (0x5166 bytes) 000563 1764106521.020584 read 0x0280df thru 0x031d04 (0x9c26 bytes) 000564 1764106521.026934 read 0x004b7d thru 0x01174a (0xcbce bytes) 000565 1764106521.033788 read 0x00a7f2 thru 0x019651 (0xee60 bytes) 000567 1764106521.034885 mapwrite 0x00b5df thru 0x014448 (0x8e6a bytes) 000569 1764106521.068864 write 0x03df21 thru 0x03ffff (0x20df bytes) 000571 1764106521.070799 write 0x00a981 thru 0x01149c (0x6b1c bytes) 000573 1764106521.072699 mapwrite 0x00d599 thru 0x01251b (0x4f83 bytes) 000575 1764106521.102118 mapread 0x004fcd thru 0x00d938 (0x896c bytes) 000577 1764106521.103529 mapread 0x014362 thru 0x01db00 (0x979f bytes) 000578 1764106521.104416 mapread 0x03589a thru 0x03ffff (0xa766 bytes) 000579 1764106521.109747 mapread 0x03268a thru 0x03bf2b (0x98a2 bytes) 000581 1764106521.111185 mapread 0x00028f thru 0x0016f6 (0x1468 bytes) 000582 1764106521.111931 mapwrite 0x025796 thru 0x032fd3 (0xd83e bytes) 000583 1764106521.139531 read 0x001372 thru 0x00a8a1 (0x9530 bytes) 000585 1764106521.140644 read 0x030e90 thru 0x03c99b (0xbb0c bytes) 000586 1764106521.141282 mapwrite 0x00a541 thru 0x00ed58 (0x4818 bytes) 000587 1764106521.172514 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x650d763b (stale) Mirror 11: 0x650d763b (stale) Mirror 14: 0x650d763b (stale) Mirror 15: 0x650d763b (stale) Mirror 16: 0x650d763b (stale) Mirror 19: 0x650d763b (stale) Mirror 20: 0x650d763b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x650d763b (stale) Mirror 11: 0x650d763b (stale) Mirror 14: 0x650d763b (stale) Mirror 15: 0x650d763b (stale) Mirror 16: 0x650d763b (stale) Mirror 19: 0x650d763b (stale) Mirror 20: 0x650d763b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000589 1764106521.456962 write 0x00b82c thru 0x012cd8 (0x74ad bytes) 000590 1764106521.472374 read 0x01035b thru 0x01bb93 (0xb839 bytes) 000591 1764106521.481369 read 0x025fb7 thru 0x0267ff (0x849 bytes) 000592 1764106521.496333 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x650d763b (stale) Mirror 14: 0x650d763b (stale) Mirror 15: 0x650d763b (stale) Mirror 16: 0x650d763b (stale) Mirror 19: 0x650d763b (stale) Mirror 20: 0x650d763b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x650d763b (stale) Mirror 14: 0x650d763b (stale) Mirror 15: 0x650d763b (stale) Mirror 16: 0x650d763b (stale) Mirror 19: 0x650d763b (stale) Mirror 20: 0x650d763b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000593 1764106521.776210 mapwrite 0x02d60e thru 0x031367 (0x3d5a bytes) 000594 1764106521.821887 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x650d763b (stale) Mirror 15: 0x650d763b (stale) Mirror 16: 0x650d763b (stale) Mirror 19: 0x650d763b (stale) Mirror 20: 0x650d763b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x650d763b (stale) Mirror 15: 0x650d763b (stale) Mirror 16: 0x650d763b (stale) Mirror 19: 0x650d763b (stale) Mirror 20: 0x650d763b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000596 1764106522.034107 mapwrite 0x03036d thru 0x03e721 (0xe3b5 bytes) 000598 1764106522.095930 write 0x023631 thru 0x03263a (0xf00a bytes) 000599 1764106522.104128 mapwrite 0x001bf5 thru 0x0115c2 (0xf9ce bytes) 000602 1764106522.169725 read 0x03a4a1 thru 0x03ffff (0x5b5f bytes) 000603 1764106522.174744 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x650d763b (stale) Mirror 16: 0x650d763b (stale) Mirror 19: 0x650d763b (stale) Mirror 20: 0x650d763b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x650d763b (stale) Mirror 16: 0x650d763b (stale) Mirror 19: 0x650d763b (stale) Mirror 20: 0x650d763b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000605 1764106522.400670 trunc from 0x040000 to 0x0097db (0x36824 bytes) 000607 1764106522.412180 write 0x03e83f thru 0x03ffff (0x17c1 bytes) 000608 1764106522.413606 mapread 0x024ff7 thru 0x02d4da (0x84e4 bytes) 000609 1764106522.432491 read 0x00a46a thru 0x011a7c (0x7613 bytes) 000612 1764106522.439252 mapwrite 0x02f0f2 thru 0x03792d (0x883c bytes) 000615 1764106522.495796 mapread 0x03c244 thru 0x03ffff (0x3dbc bytes) 000616 1764106522.502452 read 0x008e55 thru 0x00e2d2 (0x547e bytes) 000619 1764106522.512014 read 0x015274 thru 0x01c743 (0x74d0 bytes) 000621 1764106522.517810 MIRROR_RESYNC 5 mirrors 000623 1764106523.000098 write 0x030db9 thru 0x03ffff (0xf247 bytes) 000624 1764106523.050515 trunc from 0x040000 to 0x035426 (0xabd9 bytes) 000626 1764106523.087343 trunc from 0x035427 to 0x03b6a7 (0x6281 bytes) 000627 1764106523.097598 trunc from 0x03b6a8 to 0x00f378 (0x2c32f bytes) 000629 1764106523.109755 mapread 0x004f72 thru 0x00f378 (0xa407 bytes) 000631 1764106523.120629 trunc from 0x00f379 to 0x039f35 (0x2abbd bytes) 000632 1764106523.134388 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39f36): Mirror 15: 0x197c8052 (stale) Mirror 16: 0x197c8052 (stale) Mirror 20: 0x197c8052 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39f36): Mirror 15: 0x197c8052 (stale) Mirror 16: 0x197c8052 (stale) Mirror 20: 0x197c8052 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39f36: skipped 000633 1764106523.346354 trunc from 0x039f36 to 0x03e639 (0x4704 bytes) 000634 1764106523.357017 mapwrite 0x013bff thru 0x01cc41 (0x9043 bytes) 000635 1764106523.404907 mapwrite 0x039222 thru 0x03ffff (0x6dde bytes) 000637 1764106523.464728 trunc from 0x040000 to 0x02ebaa (0x11455 bytes) 000638 1764106523.481466 mapread 0x01d125 thru 0x02aeda (0xddb6 bytes) 000639 1764106523.499264 write 0x028fe4 thru 0x031e7d (0x8e9a bytes) 000641 1764106523.501390 mapread 0x02c5c5 thru 0x031e7d (0x58b9 bytes) 000642 1764106523.502285 read 0x011de4 thru 0x021bc3 (0xfde0 bytes) 000644 1764106523.503275 write 0x034def thru 0x03fb71 (0xad83 bytes) 000645 1764106523.504788 mapread 0x0196a9 thru 0x01f5a2 (0x5efa bytes) 000646 1764106523.505714 write 0x0040d4 thru 0x004f37 (0xe64 bytes) 000653 1764106523.516361 mapread 0x01250a thru 0x014589 (0x2080 bytes) 000655 1764106523.518095 trunc from 0x03fb72 to 0x01f95b (0x20216 bytes) 000657 1764106523.538916 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1f95c): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f95c): Mirror 15: 0x539dfd77 (stale) Mirror 16: 0x539dfd77 (stale) Mirror 20: 0x539dfd77 (stale) CRC-32 checksum value for chunk [0, 0x1f95c): Mirror 1: 0x208a5fa5 Mirror 21: 0x208a5fa5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f95c: skipped 000659 1764106524.064617 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f95c): Mirror 15: 0x539dfd77 (stale) Mirror 16: 0x539dfd77 (stale) Mirror 20: 0x539dfd77 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f95c): Mirror 15: 0x539dfd77 (stale) Mirror 16: 0x539dfd77 (stale) Mirror 20: 0x539dfd77 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f95c: skipped 000664 1764106524.329247 write 0x015d12 thru 0x0197a2 (0x3a91 bytes) 000665 1764106524.351788 read 0x0122e7 thru 0x01f95b (0xd675 bytes) 000669 1764106524.360347 trunc from 0x01f95c to 0x002d95 (0x1cbc6 bytes) 000671 1764106524.379575 trunc from 0x002d96 to 0x027507 (0x24772 bytes) 000672 1764106524.393251 mapread 0x000e46 thru 0x00c813 (0xb9ce bytes) 000678 1764106524.409866 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27508): Mirror 16: 0x25904510 (stale) Mirror 20: 0x25904510 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27508): Mirror 16: 0x25904510 (stale) Mirror 20: 0x25904510 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27508: skipped 000679 1764106524.596370 mapread 0x0055b6 thru 0x00cc3b (0x7686 bytes) 000680 1764106524.609534 trunc from 0x027508 to 0x00223d (0x252ca bytes) 000685 1764106524.622226 trunc from 0x00223e to 0x02615f (0x23f22 bytes) truncating to largest ever: 0x3f521 000687 1764106524.636482 trunc from 0x026160 to 0x03f520 (0x193c1 bytes) 000689 1764106524.651011 mapwrite 0x0364f2 thru 0x03dfbc (0x7acb bytes) 000690 1764106524.695275 trunc from 0x03f521 to 0x038de2 (0x673e bytes) 000691 1764106524.712405 mapwrite 0x028192 thru 0x029daa (0x1c19 bytes) 000692 1764106524.755439 trunc from 0x038de3 to 0x02f1df (0x9c03 bytes) 000695 1764106524.769631 mapwrite 0x013649 thru 0x0153bc (0x1d74 bytes) 000696 1764106524.801197 mapwrite 0x011d8f thru 0x01a9a1 (0x8c13 bytes) 000697 1764106524.835664 mapread 0x027017 thru 0x02f1df (0x81c9 bytes) 000699 1764106524.837354 mapwrite 0x009e00 thru 0x00d436 (0x3637 bytes) 000701 1764106524.879098 mapwrite 0x007627 thru 0x00a9e3 (0x33bd bytes) 000702 1764106524.913384 mapwrite 0x00bc6b thru 0x01051c (0x48b2 bytes) 000703 1764106524.939612 mapwrite 0x007324 thru 0x00ae87 (0x3b64 bytes) 000705 1764106524.968261 trunc from 0x02f1e0 to 0x004263 (0x2af7c bytes) 000706 1764106524.987396 read 0x00298c thru 0x004263 (0x18d8 bytes) 000707 1764106524.988022 mapwrite 0x025ec3 thru 0x02ab70 (0x4cae bytes) 000708 1764106525.039352 write 0x000ff5 thru 0x010bca (0xfbd6 bytes) 000710 1764106525.051389 mapread 0x006f81 thru 0x013040 (0xc0c0 bytes) 000711 1764106525.058033 write 0x018767 thru 0x023fe4 (0xb87e bytes) 000713 1764106525.067364 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2ab71): Mirror 16: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ab71): Mirror 16: 0x9e781c25 (stale) Mirror 20: 0x9e781c25 (stale) CRC-32 checksum value for chunk [0, 0x2ab71): Mirror 1: 0xddb84a09 Mirror 21: 0xddb84a09 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ab71: skipped 000716 1764106525.506340 read 0x00d2fe thru 0x01a5b2 (0xd2b5 bytes) 000717 1764106525.516220 MIRROR_RESYNC 4 mirrors 000718 1764106525.914649 write 0x016ac3 thru 0x026018 (0xf556 bytes) 000724 1764106525.959261 read 0x00261c thru 0x00782d (0x5212 bytes) 000726 1764106525.966867 mapread 0x017e0a thru 0x022298 (0xa48f bytes) 000729 1764106525.972972 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2ab71): Mirror 16: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ab71): Mirror 16: 0xddb84a09 (stale) Mirror 20: 0xddb84a09 (stale) Mirror 21: 0xddb84a09 (stale) CRC-32 checksum value for chunk [0, 0x2ab71): Mirror 1: 0xd7eea590 Mirror 22: 0xd7eea590 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ab71: skipped 000730 1764106526.432105 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2ab71): Mirror 16: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ab71): Mirror 16: 0xddb84a09 (stale) Mirror 20: 0xddb84a09 (stale) Mirror 21: 0xddb84a09 (stale) CRC-32 checksum value for chunk [0, 0x2ab71): Mirror 1: 0xd7eea590 Mirror 22: 0xd7eea590 Mirror 23: 0xd7eea590 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ab71: skipped 000732 1764106526.807287 mapread 0x01a972 thru 0x0298c6 (0xef55 bytes) 000733 1764106526.820080 write 0x01e2ab thru 0x02a243 (0xbf99 bytes) 000734 1764106526.858381 trunc from 0x02ab71 to 0x037a17 (0xcea7 bytes) 000736 1764106526.871147 MIRROR_RESYNC 6 mirrors 000738 1764106527.313448 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40b:0x0]: Invalid argument (22) 000741 1764106527.490269 write 0x03f62d thru 0x03ffff (0x9d3 bytes) 000745 1764106527.521300 write 0x00da05 thru 0x01722c (0x9828 bytes) 000746 1764106527.527028 write 0x031133 thru 0x0391b9 (0x8087 bytes) 000747 1764106527.531545 mapwrite 0x02f008 thru 0x03c951 (0xd94a bytes) 000749 1764106527.559817 read 0x026ccb thru 0x03053b (0x9871 bytes) 000750 1764106527.563562 write 0x009861 thru 0x00d1e7 (0x3987 bytes) 000753 1764106527.575065 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x282e3569 (stale) Mirror 20: 0x282e3569 (stale) Mirror 21: 0x282e3569 (stale) Mirror 23: 0x282e3569 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4347cad8 Mirror 24: 0x4347cad8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000754 1764106527.920142 MIRROR_RESYNC 6 mirrors 000756 1764106528.308974 mapread 0x01ab94 thru 0x01affa (0x467 bytes) 000757 1764106528.312065 mapread 0x03cb90 thru 0x03ffff (0x3470 bytes) 000758 1764106528.318655 mapread 0x0201c0 thru 0x029bf8 (0x9a39 bytes) 000759 1764106528.326396 read 0x019c07 thru 0x0213f8 (0x77f2 bytes) 000760 1764106528.326754 mapwrite 0x01aa6f thru 0x01bca4 (0x1236 bytes) 000763 1764106528.384136 read 0x023cd6 thru 0x02bbbc (0x7ee7 bytes) 000764 1764106528.390526 read 0x002c23 thru 0x010a25 (0xde03 bytes) 000765 1764106528.391095 read 0x01e471 thru 0x029eb9 (0xba49 bytes) 000769 1764106528.392788 mapread 0x01e452 thru 0x0211da (0x2d89 bytes) 000770 1764106528.393526 mapwrite 0x01bcb6 thru 0x01f142 (0x348d bytes) 000771 1764106528.415060 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x4347cad8 (stale) Mirror 20: 0x4347cad8 (stale) Mirror 21: 0x4347cad8 (stale) Mirror 23: 0x4347cad8 (stale) Mirror 24: 0x4347cad8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3d68422e Mirror 25: 0x3d68422e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000772 1764106528.765115 mapread 0x027674 thru 0x02bcb4 (0x4641 bytes) 000775 1764106528.784128 read 0x0032b2 thru 0x012425 (0xf174 bytes) 000776 1764106528.790661 mapread 0x03a233 thru 0x03ffff (0x5dcd bytes) 000777 1764106528.799315 trunc from 0x040000 to 0x020112 (0x1feed bytes) 000779 1764106528.857959 mapwrite 0x026bee thru 0x03297b (0xbd8e bytes) 000780 1764106528.919206 MIRROR_RESYNC 7 mirrors 000781 1764106529.418129 write 0x014146 thru 0x019b84 (0x5a3f bytes) 000788 1764106529.460919 mapwrite 0x01a8a8 thru 0x0223b7 (0x7b10 bytes) 000789 1764106529.500221 mapwrite 0x03d605 thru 0x03fade (0x24da bytes) 000792 1764106529.532581 trunc from 0x03fadf to 0x017820 (0x282be bytes) 000793 1764106529.541150 mapwrite 0x02fc66 thru 0x03fade (0xfe79 bytes) 000794 1764106529.593010 mapwrite 0x0081b5 thru 0x0137ee (0xb63a bytes) 000795 1764106529.628549 mapwrite 0x015e73 thru 0x016277 (0x405 bytes) 000797 1764106529.652360 write 0x02a360 thru 0x034feb (0xac8c bytes) 000798 1764106529.653509 mapwrite 0x03d336 thru 0x03ffff (0x2cca bytes) 000800 1764106529.696899 trunc from 0x040000 to 0x0176f7 (0x28908 bytes) 000801 1764106529.706472 mapread 0x013748 thru 0x0176f7 (0x3fb0 bytes) 000803 1764106529.707924 write 0x01ec66 thru 0x02cfd9 (0xe374 bytes) 000805 1764106529.710060 trunc from 0x02cfda to 0x0379f7 (0xaa1e bytes) 000808 1764106529.722532 trunc from 0x0379f8 to 0x01a4db (0x1d51c bytes) 000809 1764106529.733956 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1a4dc): Mirror 16: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a4dc): Mirror 16: 0x97b7e70b (stale) Mirror 20: 0x97b7e70b (stale) Mirror 21: 0x97b7e70b (stale) Mirror 23: 0x97b7e70b (stale) Mirror 24: 0x97b7e70b (stale) Mirror 25: 0x97b7e70b (stale) CRC-32 checksum value for chunk [0, 0x1a4dc): Mirror 1: 0x7d011ee2 Mirror 26: 0x7d011ee2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a4dc: skipped 000810 1764106530.110110 mapread 0x0075a9 thru 0x00a7d9 (0x3231 bytes) 000813 1764106530.147319 write 0x0284a2 thru 0x0290fd (0xc5c bytes) 000814 1764106530.194132 trunc from 0x0290fe to 0x03613d (0xd040 bytes) 000816 1764106530.208276 MIRROR_RESYNC 8 mirrors 000817 1764106530.718109 mapwrite 0x021dbd thru 0x02323b (0x147f bytes) 000819 1764106530.792825 mapwrite 0x03ce76 thru 0x03ffff (0x318a bytes) 000821 1764106530.831464 mapread 0x03af90 thru 0x03ffff (0x5070 bytes) 000822 1764106530.832367 trunc from 0x040000 to 0x02df40 (0x120bf bytes) 000823 1764106530.838697 mapread 0x025a1d thru 0x02d3f2 (0x79d6 bytes) 000824 1764106530.842293 read 0x0172db thru 0x022c81 (0xb9a7 bytes) 000825 1764106530.845635 write 0x02c71a thru 0x03ae68 (0xe74f bytes) 000827 1764106530.849763 MIRROR_RESYNC 8 mirrors 000831 1764106531.370287 mapread 0x02fe6d thru 0x03ae68 (0xaffc bytes) 000834 1764106531.390054 mapread 0x00df68 thru 0x01905d (0xb0f6 bytes) 000835 1764106531.402316 mapread 0x00ba4e thru 0x00c3ba (0x96d bytes) 000836 1764106531.402793 write 0x00b858 thru 0x00bf84 (0x72d bytes) 000840 1764106531.451742 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ae69): Mirror 20: 0x3278bc6d (stale) Mirror 21: 0x3278bc6d (stale) Mirror 23: 0x3278bc6d (stale) Mirror 24: 0x3278bc6d (stale) Mirror 25: 0x3278bc6d (stale) Mirror 26: 0x3278bc6d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ae69): Mirror 20: 0x3278bc6d (stale) Mirror 21: 0x3278bc6d (stale) Mirror 23: 0x3278bc6d (stale) Mirror 24: 0x3278bc6d (stale) Mirror 25: 0x3278bc6d (stale) Mirror 26: 0x3278bc6d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ae69: skipped 000841 1764106531.689781 mapwrite 0x0206e7 thru 0x02804e (0x7968 bytes) 000842 1764106531.735476 MIRROR_RESYNC 7 mirrors 000844 1764106532.303430 write 0x023f20 thru 0x02aa45 (0x6b26 bytes) 000847 1764106532.340918 write 0x02c654 thru 0x0329e4 (0x6391 bytes) 000849 1764106532.346711 write 0x02ac0d thru 0x02b07f (0x473 bytes) 000851 1764106532.347897 mapread 0x0210c1 thru 0x0237b3 (0x26f3 bytes) 000853 1764106532.353432 read 0x01c43b thru 0x028c36 (0xc7fc bytes) 000854 1764106532.354085 write 0x00e5cb thru 0x014742 (0x6178 bytes) 000857 1764106532.360210 trunc from 0x03ae69 to 0x00154c (0x3991c bytes) 000859 1764106532.372811 trunc from 0x00154d to 0x014992 (0x13446 bytes) 000860 1764106532.384159 trunc from 0x014993 to 0x00acb6 (0x9cdc bytes) 000862 1764106532.399443 MIRROR_RESYNC 7 mirrors 000863 1764106532.935619 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40b:0x0]: Invalid argument (22) 000865 1764106533.172380 read 0x0021ae thru 0x006ded (0x4c40 bytes) 000867 1764106533.178278 read 0x0018fb thru 0x0031b2 (0x18b8 bytes) 000868 1764106533.186363 write 0x031601 thru 0x03165d (0x5d bytes) 000870 1764106533.245411 mapread 0x00f0b0 thru 0x0183d2 (0x9323 bytes) 000872 1764106533.255709 mapread 0x0294b8 thru 0x031300 (0x7e49 bytes) 000873 1764106533.259168 MIRROR_RESYNC 6 mirrors 000874 1764106533.763133 mapread 0x019235 thru 0x01c394 (0x3160 bytes) 000878 1764106533.774724 write 0x038e66 thru 0x03b2c5 (0x2460 bytes) 000879 1764106533.815504 MIRROR_RESYNC 6 mirrors 000881 1764106534.328256 trunc from 0x03b2c6 to 0x003740 (0x37b85 bytes) 000882 1764106534.366652 mapwrite 0x00ae66 thru 0x016663 (0xb7fe bytes) 000883 1764106534.429428 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x16664): Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16664): Mirror 21: 0xd6d62db1 (stale) Mirror 23: 0xd6d62db1 (stale) Mirror 24: 0xd6d62db1 (stale) Mirror 25: 0xd6d62db1 (stale) Mirror 26: 0xd6d62db1 (stale) CRC-32 checksum value for chunk [0, 0x16664): Mirror 1: 0x6ed0691 Mirror 27: 0x6ed0691 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16664: skipped 000884 1764106534.814704 MIRROR_RESYNC 7 mirrors 000887 1764106535.437141 trunc from 0x016664 to 0x03a72b (0x240c8 bytes) 000890 1764106535.495193 mapwrite 0x02b16e thru 0x03ac1a (0xfaad bytes) 000891 1764106535.564888 read 0x009159 thru 0x016668 (0xd510 bytes) 000892 1764106535.573602 mapread 0x036766 thru 0x03ac1a (0x44b5 bytes) 000895 1764106535.579150 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ac1b): Mirror 21: 0x5e35cb37 (stale) Mirror 23: 0x5e35cb37 (stale) Mirror 24: 0x5e35cb37 (stale) Mirror 25: 0x5e35cb37 (stale) Mirror 26: 0x5e35cb37 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ac1b): Mirror 21: 0x5e35cb37 (stale) Mirror 23: 0x5e35cb37 (stale) Mirror 24: 0x5e35cb37 (stale) Mirror 25: 0x5e35cb37 (stale) Mirror 26: 0x5e35cb37 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ac1b: skipped 000896 1764106535.809894 read 0x00ad31 thru 0x00c41d (0x16ed bytes) 000897 1764106535.816912 MIRROR_RESYNC 6 mirrors 000898 1764106536.293445 mapwrite 0x0298e9 thru 0x034225 (0xa93d bytes) 000899 1764106536.394492 write 0x00ccbb thru 0x0108e2 (0x3c28 bytes) 000900 1764106536.399330 trunc from 0x03ac1b to 0x0222d0 (0x1894a bytes) 000901 1764106536.412311 read 0x01d13a thru 0x0222d0 (0x5197 bytes) 000903 1764106536.420405 write 0x029f6a thru 0x0364fc (0xc593 bytes) 000904 1764106536.421707 mapwrite 0x0164d0 thru 0x017fe7 (0x1b18 bytes) 000907 1764106536.466084 mapwrite 0x032c22 thru 0x03ffff (0xd3de bytes) 000908 1764106536.521345 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: 0xd2b64931 (stale) Mirror 24: 0xd2b64931 (stale) Mirror 25: 0xd2b64931 (stale) Mirror 26: 0xd2b64931 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: 0xd2b64931 (stale) Mirror 24: 0xd2b64931 (stale) Mirror 25: 0xd2b64931 (stale) Mirror 26: 0xd2b64931 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000909 1764106536.734615 mapread 0x010fbe thru 0x014206 (0x3249 bytes) 000911 1764106536.747424 mapread 0x038097 thru 0x03fc47 (0x7bb1 bytes) 000912 1764106536.753264 mapread 0x023524 thru 0x02af83 (0x7a60 bytes) 000913 1764106536.760586 write 0x03338a thru 0x03986c (0x64e3 bytes) 000914 1764106536.761911 write 0x001593 thru 0x00a6df (0x914d bytes) 000915 1764106536.769405 read 0x0070a2 thru 0x00d3a3 (0x6302 bytes) 000916 1764106536.770136 trunc from 0x040000 to 0x009dea (0x36215 bytes) 000920 1764106536.785333 write 0x0088ad thru 0x00c5a1 (0x3cf5 bytes) 000921 1764106536.786352 mapwrite 0x002688 thru 0x00c9ae (0xa327 bytes) 000922 1764106536.837431 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xc9af): Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc9af): Mirror 23: 0x85afd2be (stale) Mirror 24: 0x85afd2be (stale) Mirror 25: 0x85afd2be (stale) Mirror 26: 0x85afd2be (stale) CRC-32 checksum value for chunk [0, 0xc9af): Mirror 1: 0x92305b0b Mirror 27: 0x92305b0b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc9af: skipped 000923 1764106537.258896 read 0x009154 thru 0x00ad81 (0x1c2e bytes) 000925 1764106537.262170 read 0x008969 thru 0x00c9ae (0x4046 bytes) 000927 1764106537.269488 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc9af): Mirror 23: 0x85afd2be (stale) Mirror 24: 0x85afd2be (stale) Mirror 25: 0x85afd2be (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc9af): Mirror 23: 0x85afd2be (stale) Mirror 24: 0x85afd2be (stale) Mirror 25: 0x85afd2be (stale) CRC-32 checksum value for chunk [0, 0xc9af): Mirror 1: 0x92305b0b Mirror 27: 0x92305b0b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc9af: skipped 000928 1764106537.545116 trunc from 0x00c9af to 0x037147 (0x2a799 bytes) 000930 1764106537.579580 write 0x032d88 thru 0x036b74 (0x3ded bytes) 000931 1764106537.585507 mapwrite 0x02d259 thru 0x038ad1 (0xb879 bytes) 000934 1764106537.634274 MIRROR_RESYNC 5 mirrors 000939 1764106538.008930 mapread 0x02e6ad thru 0x0321d9 (0x3b2d bytes) 000940 1764106538.020152 mapread 0x0219a3 thru 0x02b9a1 (0x9fff bytes) 000943 1764106538.025548 read 0x00937a thru 0x00b769 (0x23f0 bytes) 000945 1764106538.029570 mapwrite 0x033b41 thru 0x037f32 (0x43f2 bytes) 000948 1764106538.112407 trunc from 0x038ad2 to 0x024285 (0x1484c bytes) 000951 1764106538.135939 write 0x02303f thru 0x02cb6e (0x9b30 bytes) 000953 1764106538.141887 read 0x00441e thru 0x008202 (0x3de5 bytes) 000954 1764106538.147743 mapread 0x0052b1 thru 0x0124fe (0xd24e bytes) 000956 1764106538.149138 write 0x02f214 thru 0x03c6ad (0xd49a bytes) 000961 1764106538.152270 read 0x008269 thru 0x0094eb (0x1283 bytes) 000962 1764106538.152828 mapread 0x024800 thru 0x034405 (0xfc06 bytes) 000963 1764106538.158014 read 0x001372 thru 0x00ccad (0xb93c bytes) 000964 1764106538.161738 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x40b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c6ae): Mirror 23: 0xb0daf900 (stale) Mirror 24: 0xb0daf900 (stale) Mirror 27: 0xb0daf900 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c6ae): Mirror 23: 0xb0daf900 (stale) Mirror 24: 0xb0daf900 (stale) Mirror 27: 0xb0daf900 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c6ae: skipped 000966 1764106538.356140 read 0x0143b5 thru 0x0165cb (0x2217 bytes) 000968 1764106538.360570 write 0x00bada thru 0x00eb6b (0x3092 bytes) 000970 1764106538.369241 trunc from 0x03c6ae to 0x016389 (0x26324 bytes) 000975 1764106538.382217 read 0x012080 thru 0x016389 (0x430a bytes) 000976 1764106538.388946 write 0x03ce17 thru 0x03ffff (0x31e9 bytes) 000978 1764106538.390694 mapread 0x00a580 thru 0x019c73 (0xf6f4 bytes) 000979 1764106538.396874 mapwrite 0x03455a thru 0x03e47b (0x9f22 bytes) 000980 1764106538.422350 read 0x025da7 thru 0x034a47 (0xeca1 bytes) 000982 1764106538.423657 mapread 0x03aac8 thru 0x03ffff (0x5538 bytes) 000983 1764106538.424559 trunc from 0x040000 to 0x00ec71 (0x3138e bytes) 000990 1764106538.439918 MIRROR_RESYNC 4 mirrors 000991 1764106538.795188 mapwrite 0x007a73 thru 0x013a77 (0xc005 bytes) 000993 1764106538.879288 write 0x0160cf thru 0x0192ec (0x321e bytes) 000995 1764106538.880817 mapread 0x0096cc thru 0x01480e (0xb143 bytes) 000996 1764106538.885106 trunc from 0x0192ed to 0x01775e (0x1b8e bytes) 000999 1764106538.898291 trunc from 0x01775f to 0x00c789 (0xafd5 bytes) All operations completed A-OK! PASS 70a (52s) == sanity-flr test 71: check mirror extend parent fid ==== 16:35:41 (1764106541) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 200a: stress test ===================== 16:35:42 (1764106542) Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre2 Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4032704 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6223040 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7133888 .. done Extending file size to 7867616 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8243584 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8684384 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8997984 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9169344 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8771328 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 56344 56345 56347 56349 56350 failed 192.168.201.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg118-client.virtnet /mnt/lustre2 (opts:) 192.168.201.118@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg118-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (68s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 16:36:50 (1764106610) Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre2 Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5080192 .. Extending file size to 3847008 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8142400 .. Extending file size to 5525344 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5645696 .. Extending file size to 8575040 .. Extending file size to 8695232 .. Extending file size to 8581760 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9289376 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8786080 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9100864 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59393 59394 59396 59397 59399 done 192.168.201.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg118-client.virtnet /mnt/lustre2 (opts:) 192.168.201.118@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg118-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (69s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 16:37:59 (1764106679) Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 4608 2203648 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 4608 2203648 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0509626 s, 41.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.133707 s, 15.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xba198ad4 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg118-client.virtnet /mnt/lustre2 (opts:) PASS 200c (7s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 16:38:07 (1764106687) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0466054 s, 45.0 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 16:38:10 (1764106690) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x454:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5b3:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5b3:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5b4:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x5b6:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 16:38:13 (1764106693) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0726227 s, 28.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5b5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5b7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5b8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 16:38:16 (1764106696) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.14375 s, 14.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5b6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5b7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5b9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5b6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5b7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5b9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5ba:0x0] } PASS 204b (4s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 16:38:20 (1764106700) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.122103 s, 17.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5b8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5bb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5b8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5bb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5bc:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 16:38:45 (1764106725) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg118-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080583 s, 13.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5b9:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5be:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5bd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5b9:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5be:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5bd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5ba:0x0] } PASS 204d (52s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 16:39:37 (1764106777) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg118-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5bb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5bc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0881506 s, 23.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5bb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5bc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5bf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5bb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5bc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5bf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 16:40:23 (1764106823) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg118-server mds-ost sync done. Creating new pool test_204f oleg118-server: Pool lustre.test_204f created Adding targets to pool oleg118-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg118-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0541321 s, 38.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5bd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5c0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5bd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5c0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg118-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg118-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg118-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (58s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 16:41:21 (1764106881) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 16:41:24 (1764106884) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 16:41:27 (1764106887) oleg118-server: Pool lustre.test_206 created Adding targets to pool oleg118-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg118-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg118-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg118-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg118-server: Pool lustre.test_206 destroyed PASS 206 (16s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 16:41:43 (1764106903) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030844 s, 34.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129887 s, 80.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x25d0b9d7 Mirror 3: 0x25d0b9d7 PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 16:41:46 (1764106906) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 16:41:48 (1764106908) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 16:41:49 (1764106909) Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0238913 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1080000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1b50000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1b50000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1b50000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1140000 Read this (4 bytes): BBBB 192.168.201.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg118-client.virtnet /mnt/lustre2 (opts:) PASS 209a (23s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 16:42:12 (1764106932) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0338385 s, 242 kB/s the very first read read with warm cache PASS 209b (10s) == sanity-flr test 210a: handle broken mirrored lovea ==== 16:42:22 (1764106942) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448775 s, 23.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x479 lmm_fid: [0x200000404:0x479:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x479 lmm_fid: [0x200000404:0x479:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5cb:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835065 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-25 16:42:26.000000000 -0500 Modify: 2025-11-25 16:42:26.000000000 -0500 Change: 2025-11-25 16:42:29.000000000 -0500 Birth: 2025-11-25 16:42:25.000000000 -0500 PASS 210a (19s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 16:42:41 (1764106961) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (8s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 16:42:49 (1764106969) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.907525 s, 45.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0852084 s, 48.1 kB/s size after second write -rw-r--r-- 1 root root 4096 Nov 25 16:42 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x47b:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Nov 25 16:42 /mnt/lustre/f211.sanity-flr PASS 211 (17s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 16:43:07 (1764106987) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (11s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 16:43:19 (1764106999) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1023 sec =========== 16:43:29 (1764107009) === sanity-flr: start cleanup 16:43:31 (1764107011) === === sanity-flr: finish cleanup 16:43:37 (1764107017) ===