-----============= acceptance-small: sanity-flr ============----- Fri Apr 10 20:11:50 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg153-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 20:11:58 (1775866318) === oleg153-client.virtnet: executing check_config_client /mnt/lustre oleg153-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg153-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa00cc7e2b000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa00cc7e2b000.idle_timeout=debug disable quota as required oleg153-server: oleg153-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 20:12:44 (1775866364) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7560] == sanity-flr test 0a: lfs mirror create with -N option == 20:12:47 (1775866367) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:13:02 (1775866382) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (5s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:13:07 (1775866387) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (6s) == sanity-flr test 0d: lfs mirror extend with -N option == 20:13:14 (1775866394) PASS 0d (33s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:13:47 (1775866427) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:13:51 (1775866431) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 20:13:55 (1775866435) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffffa00cc7e2b000.stats=clear osc.lustre-OST0001-osc-ffffa00cc7e2b000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:14:03 (1775866443) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:14:15 (1775866455) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:14:24 (1775866464) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 20:14:27 (1775866467) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 20:14:36 (1775866476) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 20:14:45 (1775866485) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:14:51 (1775866491) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:14:55 (1775866495) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:15:03 (1775866503) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 20:15:13 (1775866513) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:15:21 (1775866521) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277056 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210632 s, 5.0 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:15:32 (1775866532) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.278869 s, 22.6 MB/s PASS 22 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:15:41 (1775866541) Stopping /mnt/lustre-ost1 (opts:) on oleg153-server oleg153-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa00cc7e2b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa00cc7e2b000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg153-server: oleg153-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg153-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg153-server oleg153-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa00cc7e2b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa00cc7e2b000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg153-server: oleg153-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg153-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (73s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:16:54 (1775866614) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.611195 s, 15.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg153-server oleg153-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa00cc7e2b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa00cc7e2b000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg153-server: oleg153-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg153-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (40s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:17:34 (1775866654) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg153-server oleg153-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa00cc7e2b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa00cc7e2b000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg153-server: oleg153-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg153-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg153-server oleg153-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa00cc7e2b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa00cc7e2b000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg153-server: oleg153-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg153-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (80s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:18:55 (1775866735) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.37339 s, 9.8 MB/s -rw-r--r-- 1 root root 52428800 Apr 10 20:19 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.522s user 0m0.000s sys 0m0.163s Stopping /mnt/lustre-ost1 (opts:) on oleg153-server oleg153-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa00cc7e2b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa00cc7e2b000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.251s user 0m0.000s sys 0m0.175s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg153-server: oleg153-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg153-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg153-server oleg153-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa00cc7e2b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa00cc7e2b000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.469s user 0m0.002s sys 0m0.242s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg153-server: oleg153-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg153-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg153-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa00cc7e2b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa00cc7e2b000.ost_server_uuid in FULL state after 0 sec PASS 33b (92s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:20:27 (1775866827) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:20:31 (1775866831) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:20:34 (1775866834) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 20:20:38 (1775866838) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 20:20:47 (1775866847) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.6897 s, 18.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:21:01 (1775866861) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259654 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252904 s, 41.5 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:21:09 (1775866869) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:21:18 (1775866878) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000897559 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00579197 s, 177 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00101606 s, 1.0 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00292495 s, 350 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 20:21:29 (1775866889) create test files with size 15746596 .. 31c4e1fe8cb8a52ebe9e72fbe590816b - 2c7c21a5e1d7c67ee133db5f57e343b5 - 0f0cfedffae925ac1a1a4d7cd45625cd - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg153-client.virtnet /mnt/lustre (opts:) Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 20:21:55 (1775866915) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:22:11 (1775866931) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:22:18 (1775866938) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133018 s, 7.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4d:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234009 s, 44.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x50:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x51:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4e:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 20:22:26 (1775866946) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.137203 s, 15.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.353062 s, 11.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.174452 s, 12.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 20:22:41 (1775866961) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 20:22:44 (1775866964) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:22:48 (1775866968) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281712 s, 37.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 20:22:55 (1775866975) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:22:58 (1775866978) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x58:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x55:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x55:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:23:08 (1775866988) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.311161 s, 33.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg153-client.virtnet /mnt/lustre (opts:) Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:23:20 (1775867000) llite.lustre-ffffa00ccb2e7800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.383595 s, 27.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg153-client.virtnet /mnt/lustre (opts:) Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffffa00ccc250000.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:23:37 (1775867017) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 20:24:15 (1775867055) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (32s) == sanity-flr test 47: Verify mirror obj alloc =========== 20:24:48 (1775867088) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 20:24:51 (1775867091) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.257429 s, 12.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.128604 s, 24.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x7d2afbe5 (stale) Mirror 2: 0x7d2afbe5 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 731316c31d9661ae13aa3f2931cb4729 - mirror 1 checksum: e3b3af215f63b56971a77552e6929c62 - mirror 2 checksum: 731316c31d9661ae13aa3f2931cb4729 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Apr 10 20:24:54 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Apr 10 20:24:54 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 731316c31d9661ae13aa3f2931cb4729 - mirror 1 checksum: e3b3af215f63b56971a77552e6929c62 - mirror 2 checksum: e3b3af215f63b56971a77552e6929c62 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Apr 10 20:24:54 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Apr 10 20:24:54 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:25:05 (1775867105) Failing mds1 on oleg153-server Stopping /mnt/lustre-mds1 (opts:) on oleg153-server 20:25:12 (1775867112) shut down facet: mds1 facet_host: oleg153-server facet_failover_host: oleg153-server Failover mds1 to oleg153-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg153-server: oleg153-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-MDT0000 20:25:36 (1775867136) targets are mounted 20:25:36 (1775867136) facet_failover done oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576421 s, 18.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6c:0x0] } PASS 50A (47s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:25:51 (1775867151) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:25:55 (1775867155) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:25:58 (1775867158) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.01595 s, 1.0 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:26:05 (1775867165) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:26:08 (1775867168) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:26:11 (1775867171) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:26:32 (1775867192) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:27:00 (1775867220) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:27:23 (1775867243) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:27:46 (1775867266) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 20:27:52 (1775867272) mirror create pid 52877 mirror split pid 52878 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52877 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52878 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 20:29:01 (1775867341) Seed set to 7344 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1775867345.070041 mapwrite 0x01463c thru 0x01ba2d (0x73f2 bytes) 000002 1775867345.252809 mapwrite 0x02ac80 thru 0x037304 (0xc685 bytes) 000004 1775867345.466777 write 0x0133c5 thru 0x01b12a (0x7d66 bytes) 000005 1775867345.479268 write 0x027e71 thru 0x02883f (0x9cf bytes) 000007 1775867345.494887 mapread 0x0147e9 thru 0x0161b3 (0x19cb bytes) 000013 1775867345.513949 read 0x01f894 thru 0x020f8d (0x16fa bytes) truncating to largest ever: 0xb851 000015 1775867345.526923 trunc from 0x037305 to 0x00b850 (0x2bab4 bytes) 000016 1775867345.560119 mapwrite 0x03fafc thru 0x03ffff (0x504 bytes) 000017 1775867345.787638 mapread 0x0317be thru 0x03aa82 (0x92c5 bytes) 000019 1775867345.860648 read 0x0303ff thru 0x0305b6 (0x1b8 bytes) 000020 1775867345.863736 mapwrite 0x01332a thru 0x017ac3 (0x479a bytes) 000025 1775867346.093123 mapread 0x03c8ac thru 0x03f36e (0x2ac3 bytes) truncating to largest ever: 0x38ee5 000026 1775867346.100453 trunc from 0x040000 to 0x038ee4 (0x711b bytes) 000027 1775867346.173169 write 0x023b21 thru 0x031c5f (0xe13f bytes) truncating to largest ever: 0x39bd0 000028 1775867346.174843 trunc from 0x038ee5 to 0x039bcf (0xceb bytes) 000029 1775867346.242699 read 0x00b2c2 thru 0x00bb53 (0x892 bytes) 000030 1775867346.248520 mapwrite 0x01c654 thru 0x024d33 (0x86e0 bytes) 000032 1775867346.424201 mapwrite 0x01133f thru 0x013fc2 (0x2c84 bytes) 000033 1775867346.548494 mapread 0x0321ab thru 0x0380a3 (0x5ef9 bytes) 000035 1775867346.558079 mapwrite 0x031ab2 thru 0x037669 (0x5bb8 bytes) 000036 1775867346.685898 MIRROR_EXTEND to 2 mirrors 000039 1775867347.614409 trunc from 0x039bd0 to 0x003eb8 (0x35d17 bytes) 000040 1775867347.713508 MIRROR_RESYNC 2 mirrors 000041 1775867348.813367 write 0x00b45f thru 0x011187 (0x5d29 bytes) 000045 1775867348.912846 read 0x007bc2 thru 0x010fa6 (0x93e5 bytes) 000046 1775867348.919397 mapread 0x004da4 thru 0x0083ab (0x3608 bytes) 000049 1775867348.941584 write 0x002314 thru 0x005031 (0x2d1e bytes) 000051 1775867348.949282 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x11188): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11188): Mirror 2: 0x387c5c24 (stale) CRC-32 checksum value for chunk [0, 0x11188): Mirror 1: 0xdf7cb4d2 Mirror 3: 0xdf7cb4d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11188: skipped 000052 1775867350.021420 mapread 0x0031f3 thru 0x0103ab (0xd1b9 bytes) 000053 1775867350.056628 read 0x00c909 thru 0x011187 (0x487f bytes) 000054 1775867350.069897 write 0x025fbe thru 0x02a2f3 (0x4336 bytes) truncating to largest ever: 0x3fc95 000058 1775867350.231893 trunc from 0x02a2f4 to 0x03fc94 (0x159a1 bytes) 000059 1775867350.306918 mapwrite 0x01e914 thru 0x02e153 (0xf840 bytes) 000061 1775867350.495063 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fc95): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fc95): Mirror 2: 0xff42c880 (stale) Mirror 3: 0x151b1cc (stale) CRC-32 checksum value for chunk [0, 0x3fc95): Mirror 1: 0x8ec1823 Mirror 4: 0x8ec1823 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fc95: skipped 000062 1775867351.978287 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3fc95): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fc95): Mirror 2: 0xff42c880 (stale) Mirror 3: 0x151b1cc (stale) CRC-32 checksum value for chunk [0, 0x3fc95): Mirror 1: 0x8ec1823 Mirror 4: 0x8ec1823 Mirror 5: 0x8ec1823 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fc95: skipped 000063 1775867353.717394 trunc from 0x03fc95 to 0x0031d1 (0x3cac3 bytes) 000064 1775867353.910807 trunc from 0x0031d2 to 0x02ed44 (0x2bb73 bytes) 000065 1775867353.962021 read 0x017e1e thru 0x027491 (0xf674 bytes) 000066 1775867353.979003 mapwrite 0x02259b thru 0x023862 (0x12c8 bytes) 000069 1775867354.085655 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ed45): Mirror 2: 0x74b6e229 (stale) Mirror 3: 0x242c8ee0 (stale) Mirror 4: 0xef37dc24 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ed45): Mirror 2: 0x74b6e229 (stale) Mirror 3: 0x242c8ee0 (stale) Mirror 4: 0xef37dc24 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ed45: skipped 000070 1775867354.560290 mapread 0x028fc6 thru 0x02ed44 (0x5d7f bytes) 000072 1775867354.611714 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ed45): Mirror 3: 0x242c8ee0 (stale) Mirror 4: 0xef37dc24 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ed45): Mirror 3: 0x242c8ee0 (stale) Mirror 4: 0xef37dc24 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ed45: skipped 000073 1775867355.492171 read 0x013874 thru 0x01da5f (0xa1ec bytes) 000075 1775867355.511547 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2ed45): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ed45): Mirror 3: 0x242c8ee0 (stale) Mirror 4: 0xef37dc24 (stale) CRC-32 checksum value for chunk [0, 0x2ed45): Mirror 1: 0xaffa06fd Mirror 5: 0xaffa06fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ed45: skipped 000078 1775867357.094527 mapread 0x00fd7a thru 0x012a7b (0x2d02 bytes) 000080 1775867357.188067 mapwrite 0x00136c thru 0x001458 (0xed bytes) 000081 1775867357.470918 write 0x0344fc thru 0x03de35 (0x993a bytes) 000082 1775867357.472412 trunc from 0x03de36 to 0x025cdd (0x18158 bytes) 000083 1775867357.498799 mapread 0x022370 thru 0x025cdd (0x396e bytes) 000084 1775867357.513531 read 0x01f414 thru 0x025cdd (0x68ca bytes) 000087 1775867357.521216 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25cde): Mirror 4: 0x39482870 (stale) Mirror 5: 0xa8741a0c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25cde): Mirror 4: 0x39482870 (stale) Mirror 5: 0xa8741a0c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25cde: skipped 000088 1775867357.872798 mapwrite 0x023f85 thru 0x02bc0a (0x7c86 bytes) 000089 1775867357.994543 write 0x024fc5 thru 0x0263ba (0x13f6 bytes) 000091 1775867357.997666 mapread 0x0137b2 thru 0x016ef2 (0x3741 bytes) 000092 1775867358.014095 trunc from 0x02bc0b to 0x0321dc (0x65d2 bytes) 000093 1775867358.047312 mapwrite 0x02d6b5 thru 0x037e24 (0xa770 bytes) 000097 1775867358.139642 write 0x03cbc1 thru 0x03ffff (0x343f bytes) 000099 1775867358.152339 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x88c51cf7 (stale) Mirror 5: 0xde5f1d65 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc0640424 Mirror 6: 0xc0640424 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000100 1775867359.257601 mapread 0x0206b1 thru 0x024da5 (0x46f5 bytes) 000102 1775867359.303423 mapread 0x015910 thru 0x01b41f (0x5b10 bytes) 000105 1775867359.333047 trunc from 0x040000 to 0x00b351 (0x34cae bytes) 000106 1775867359.510179 read 0x0096a6 thru 0x00b351 (0x1cac bytes) 000109 1775867359.516145 trunc from 0x00b352 to 0x029d18 (0x1e9c7 bytes) 000115 1775867359.568766 MIRROR_RESYNC 4 mirrors 000117 1775867360.833079 write 0x0393d7 thru 0x03ffff (0x6c29 bytes) 000118 1775867360.963518 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xad817bd4 (stale) Mirror 5: 0xad817bd4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xad817bd4 (stale) Mirror 5: 0xad817bd4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000119 1775867361.418611 read 0x0125f4 thru 0x01e983 (0xc390 bytes) 000123 1775867361.443092 write 0x01e541 thru 0x0223fa (0x3eba bytes) 000124 1775867361.449155 read 0x036968 thru 0x03b7c5 (0x4e5e bytes) 000125 1775867361.458996 write 0x00505f thru 0x005b8e (0xb30 bytes) 000126 1775867361.465575 MIRROR_RESYNC 3 mirrors 000127 1775867362.595955 mapread 0x0391c5 thru 0x03ffff (0x6e3b bytes) 000128 1775867362.623853 mapwrite 0x0209c9 thru 0x02b3a4 (0xa9dc bytes) 000129 1775867362.828420 read 0x022967 thru 0x030aa6 (0xe140 bytes) 000130 1775867362.829729 write 0x015c46 thru 0x0162bf (0x67a bytes) 000131 1775867362.841729 read 0x01c5ab thru 0x028002 (0xba58 bytes) 000134 1775867362.844728 read 0x035c84 thru 0x0379e9 (0x1d66 bytes) 000135 1775867362.869715 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x1627409b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x1627409b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000136 1775867363.199517 write 0x012613 thru 0x018634 (0x6022 bytes) 000137 1775867363.216294 read 0x004873 thru 0x012689 (0xde17 bytes) 000143 1775867363.234439 trunc from 0x040000 to 0x0326b1 (0xd94e bytes) 000146 1775867363.355531 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) 000147 1775867363.563374 mapread 0x025deb thru 0x02f752 (0x9968 bytes) 000148 1775867363.595364 write 0x012693 thru 0x0135e1 (0xf4f bytes) 000150 1775867363.608084 trunc from 0x0326b2 to 0x009738 (0x28f79 bytes) 000154 1775867363.642474 read 0x008439 thru 0x009738 (0x1300 bytes) 000155 1775867363.647698 write 0x01b5ca thru 0x02184a (0x6281 bytes) 000157 1775867363.650729 write 0x0031d2 thru 0x007357 (0x4186 bytes) 000158 1775867363.661163 mapread 0x0144b9 thru 0x020c6c (0xc7b4 bytes) 000160 1775867363.672651 read 0x0017cc thru 0x007b12 (0x6347 bytes) 000161 1775867363.673990 write 0x000096 thru 0x007e50 (0x7dbb bytes) 000163 1775867363.683504 trunc from 0x02184b to 0x023570 (0x1d26 bytes) 000164 1775867363.705504 mapread 0x00cd8b thru 0x00f123 (0x2399 bytes) 000165 1775867363.707920 mapread 0x01844b thru 0x023570 (0xb126 bytes) 000169 1775867363.721203 trunc from 0x023571 to 0x027480 (0x3f10 bytes) 000171 1775867363.753005 write 0x0207b3 thru 0x02ec74 (0xe4c2 bytes) 000172 1775867363.755342 read 0x00c84f thru 0x01011e (0x38d0 bytes) 000173 1775867363.757293 write 0x025442 thru 0x034cb2 (0xf871 bytes) 000174 1775867363.764183 MIRROR_EXTEND to 2 mirrors 000175 1775867364.534990 mapwrite 0x003fc8 thru 0x00d538 (0x9571 bytes) 000177 1775867364.909726 read 0x028dd1 thru 0x02ae37 (0x2067 bytes) 000178 1775867364.922927 trunc from 0x034cb3 to 0x01feae (0x14e04 bytes) 000179 1775867364.979553 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) 000180 1775867365.185093 mapwrite 0x01e8d7 thru 0x029cfe (0xb428 bytes) 000182 1775867365.462284 read 0x0244da thru 0x029cfe (0x5825 bytes) 000183 1775867365.465111 mapwrite 0x012c15 thru 0x014f8b (0x2377 bytes) 000184 1775867365.526992 mapread 0x024b5b thru 0x029cfe (0x51a4 bytes) 000187 1775867365.530609 mapread 0x00eec1 thru 0x01e5f8 (0xf738 bytes) 000189 1775867365.533605 trunc from 0x029cff to 0x01e2bc (0xba42 bytes) 000190 1775867365.559558 read 0x01a2b8 thru 0x01e2bc (0x4005 bytes) 000191 1775867365.560957 mapwrite 0x034112 thru 0x0367c8 (0x26b7 bytes) 000192 1775867365.635095 write 0x02fbdd thru 0x03df48 (0xe36c bytes) 000196 1775867365.640495 trunc from 0x03df49 to 0x008b62 (0x353e6 bytes) 000199 1775867365.662314 trunc from 0x008b63 to 0x02dd65 (0x25203 bytes) 000200 1775867365.674986 read 0x028949 thru 0x02dd65 (0x541d bytes) 000201 1775867365.689982 trunc from 0x02dd66 to 0x00aa60 (0x23305 bytes) 000204 1775867365.711002 trunc from 0x00aa61 to 0x00fefc (0x549c bytes) 000206 1775867365.736590 mapwrite 0x028df7 thru 0x0338a4 (0xaaae bytes) 000209 1775867365.935805 write 0x03c51d thru 0x03ffff (0x3ae3 bytes) 000210 1775867365.937335 write 0x007a84 thru 0x00e4eb (0x6a68 bytes) 000211 1775867365.948419 write 0x020213 thru 0x029a0a (0x97f8 bytes) 000214 1775867365.957299 mapwrite 0x035588 thru 0x03b2d6 (0x5d4f bytes) 000215 1775867366.107184 trunc from 0x040000 to 0x03706c (0x8f93 bytes) 000217 1775867366.136779 write 0x01b013 thru 0x028ea1 (0xde8f bytes) 000219 1775867366.144523 mapwrite 0x010ef3 thru 0x0111f6 (0x304 bytes) 000220 1775867366.204916 mapwrite 0x03a68b thru 0x03ffff (0x5975 bytes) 000221 1775867366.339895 MIRROR_EXTEND to 2 mirrors 000222 1775867367.046534 mapread 0x0240ee thru 0x0293e2 (0x52f5 bytes) 000224 1775867367.069657 mapread 0x029ead thru 0x03678c (0xc8e0 bytes) 000225 1775867367.086121 MIRROR_EXTEND to 3 mirrors 000227 1775867367.653794 trunc from 0x040000 to 0x031753 (0xe8ac bytes) 000229 1775867367.780876 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31754): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31754): Mirror 2: 0x6e9bb559 (stale) Mirror 3: 0x6e9bb559 (stale) CRC-32 checksum value for chunk [0, 0x31754): Mirror 1: 0x6e9bb559 Mirror 4: 0x6e9bb559 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31754: skipped 000232 1775867368.732331 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31754): Mirror 2: 0x6e9bb559 (stale) Mirror 3: 0x6e9bb559 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31754): Mirror 2: 0x6e9bb559 (stale) Mirror 3: 0x6e9bb559 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31754: skipped 000235 1775867369.045614 mapread 0x000846 thru 0x003e86 (0x3641 bytes) 000236 1775867369.059330 read 0x0182e4 thru 0x01f46c (0x7189 bytes) 000237 1775867369.072913 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31754): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31754): Mirror 2: 0x6e9bb559 (stale) Mirror 3: 0x6e9bb559 (stale) CRC-32 checksum value for chunk [0, 0x31754): Mirror 1: 0x6e9bb559 Mirror 4: 0x6e9bb559 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31754: skipped 000238 1775867370.043988 read 0x02e283 thru 0x031753 (0x34d1 bytes) 000239 1775867370.061962 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31754): Mirror 2: 0x6e9bb559 (stale) Mirror 3: 0x6e9bb559 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31754): Mirror 2: 0x6e9bb559 (stale) Mirror 3: 0x6e9bb559 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31754: skipped 000240 1775867370.500899 trunc from 0x031754 to 0x023ec6 (0xd88d bytes) 000241 1775867370.519505 write 0x03edfe thru 0x03ffff (0x1202 bytes) 000242 1775867370.523071 MIRROR_RESYNC 3 mirrors 000243 1775867371.603527 MIRROR_EXTEND to 4 mirrors 000244 1775867372.674940 read 0x01c566 thru 0x02bff6 (0xfa91 bytes) 000246 1775867372.686187 read 0x01d9b8 thru 0x0216e5 (0x3d2e bytes) 000247 1775867372.692194 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) 000248 1775867373.380793 read 0x013769 thru 0x01adf1 (0x7689 bytes) 000249 1775867373.408431 write 0x019f67 thru 0x02647e (0xc518 bytes) 000251 1775867373.685137 trunc from 0x040000 to 0x008329 (0x37cd6 bytes) 000253 1775867373.768397 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x832a): Mirror 4: 0x130bde8e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x832a): Mirror 4: 0x130bde8e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x832a: skipped 000254 1775867374.369459 trunc from 0x00832a to 0x0095b7 (0x128e bytes) 000256 1775867374.412658 mapwrite 0x012691 thru 0x01ea39 (0xc3a9 bytes) 000257 1775867374.716266 mapwrite 0x03bc61 thru 0x03ffff (0x439f bytes) 000258 1775867374.897513 trunc from 0x040000 to 0x00e462 (0x31b9d bytes) 000259 1775867374.939903 trunc from 0x00e463 to 0x029d82 (0x1b920 bytes) 000260 1775867374.988699 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x29d83): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29d83): Mirror 4: 0x781db661 (stale) CRC-32 checksum value for chunk [0, 0x29d83): Mirror 1: 0xcfe87690 Mirror 5: 0xcfe87690 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29d83: skipped 000261 1775867376.278629 trunc from 0x029d83 to 0x014bbb (0x151c7 bytes) 000262 1775867376.484554 read 0x002432 thru 0x00cc34 (0xa803 bytes) 000263 1775867376.525208 trunc from 0x014bbc to 0x018f32 (0x4377 bytes) 000264 1775867376.577980 mapread 0x011ab0 thru 0x018f32 (0x7483 bytes) 000267 1775867376.623989 mapwrite 0x00137d thru 0x003d80 (0x2a04 bytes) 000268 1775867376.706760 mapwrite 0x02862a thru 0x02fce5 (0x76bc bytes) 000269 1775867376.880618 mapread 0x018d8e thru 0x02687b (0xdaee bytes) 000270 1775867376.896895 mapwrite 0x034ff2 thru 0x03f202 (0xa211 bytes) 000271 1775867377.117742 mapwrite 0x01a271 thru 0x024250 (0x9fe0 bytes) 000273 1775867377.182547 mapwrite 0x02048b thru 0x023faf (0x3b25 bytes) 000274 1775867377.232045 trunc from 0x03f203 to 0x007e31 (0x373d1 bytes) 000276 1775867377.256977 mapread 0x0077b5 thru 0x007e31 (0x67d bytes) 000277 1775867377.257999 read 0x006f1b thru 0x00785f (0x945 bytes) 000279 1775867377.260424 read 0x006111 thru 0x007e31 (0x1d21 bytes) 000281 1775867377.262512 read 0x001f5d thru 0x005d20 (0x3dc4 bytes) 000288 1775867377.269445 read 0x0021a4 thru 0x007e31 (0x5c8e bytes) 000289 1775867377.270428 write 0x019c0a thru 0x023e09 (0xa200 bytes) 000290 1775867377.273541 mapwrite 0x03cdc8 thru 0x03ffff (0x3238 bytes) 000291 1775867377.385762 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x4e046353 (stale) Mirror 5: 0x76275e60 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4b0df39a Mirror 6: 0x4b0df39a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000293 1775867378.566663 mapwrite 0x03a462 thru 0x03d606 (0x31a5 bytes) 000294 1775867378.855687 trunc from 0x040000 to 0x02b449 (0x14bb6 bytes) 000295 1775867378.912870 mapread 0x016574 thru 0x01699e (0x42b bytes) 000296 1775867378.939006 mapwrite 0x03427d thru 0x03dd72 (0x9af6 bytes) 000298 1775867379.046597 read 0x0009c1 thru 0x00586c (0x4eac bytes) 000300 1775867379.058757 mapwrite 0x01fac0 thru 0x02ec8f (0xf1d0 bytes) 000302 1775867379.170164 mapread 0x03a92c thru 0x03aac9 (0x19e bytes) 000303 1775867379.171886 read 0x01242e thru 0x0179c3 (0x5596 bytes) 000304 1775867379.178497 MIRROR_RESYNC 4 mirrors 000307 1775867380.415722 read 0x01679a thru 0x026186 (0xf9ed bytes) 000308 1775867380.441838 read 0x00f78f thru 0x01e6e8 (0xef5a bytes) 000309 1775867380.457025 trunc from 0x03dd73 to 0x00d034 (0x30d3e bytes) 000310 1775867380.621941 write 0x00de38 thru 0x014151 (0x631a bytes) 000311 1775867380.631880 mapread 0x00b2ed thru 0x00e643 (0x3357 bytes) 000312 1775867380.645855 mapwrite 0x017f35 thru 0x01b89c (0x3968 bytes) 000314 1775867380.803677 read 0x018234 thru 0x01b89c (0x3669 bytes) 000315 1775867380.806705 trunc from 0x01b89d to 0x0320df (0x16843 bytes) 000316 1775867380.843054 write 0x00fd81 thru 0x01626e (0x64ee bytes) 000317 1775867380.860228 read 0x02a04b thru 0x0320df (0x8095 bytes) 000319 1775867380.874819 read 0x02e2d4 thru 0x0320df (0x3e0c bytes) 000320 1775867380.875522 read 0x03003c thru 0x0320df (0x20a4 bytes) 000321 1775867380.876109 write 0x023387 thru 0x03327e (0xfef8 bytes) 000322 1775867380.884737 read 0x02bc4a thru 0x03327e (0x7635 bytes) 000324 1775867380.889201 write 0x00b5f8 thru 0x01361b (0x8024 bytes) 000327 1775867380.897048 trunc from 0x03327f to 0x005d0e (0x2d570 bytes) 000329 1775867380.944947 write 0x00c369 thru 0x0111ce (0x4e66 bytes) 000332 1775867380.951613 trunc from 0x0111cf to 0x033a84 (0x228b6 bytes) 000333 1775867381.003821 mapwrite 0x016414 thru 0x022204 (0xbdf1 bytes) 000334 1775867381.221044 MIRROR_RESYNC 4 mirrors 000335 1775867383.515670 read 0x02f085 thru 0x033a84 (0x4a00 bytes) 000336 1775867383.528044 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) 000337 1775867384.250729 mapwrite 0x03a39e thru 0x03ffff (0x5c62 bytes) 000338 1775867384.471277 read 0x01cbb8 thru 0x029fb8 (0xd401 bytes) 000341 1775867384.511593 MIRROR_RESYNC 3 mirrors 000342 1775867386.151869 mapwrite 0x02586b thru 0x034fb7 (0xf74d bytes) 000348 1775867386.616651 trunc from 0x040000 to 0x029596 (0x16a69 bytes) 000349 1775867386.641797 trunc from 0x029597 to 0x00afba (0x1e5dc bytes) 000350 1775867386.664486 read 0x00a30f thru 0x00afba (0xcac bytes) 000351 1775867386.670769 mapread 0x003837 thru 0x00afba (0x7784 bytes) 000352 1775867386.682171 read 0x007c57 thru 0x00afba (0x3364 bytes) 000355 1775867386.688882 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xafbb): Mirror 6: 0xcad00a9a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xafbb): Mirror 6: 0xcad00a9a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xafbb: skipped 000357 1775867387.174790 mapread 0x001903 thru 0x0098fb (0x7ff9 bytes) 000358 1775867387.218902 read 0x000ab1 thru 0x00afba (0xa50a bytes) 000359 1775867387.240499 mapread 0x0078b8 thru 0x00afba (0x3703 bytes) 000360 1775867387.244200 write 0x020cff thru 0x028031 (0x7333 bytes) 000361 1775867387.248399 mapwrite 0x028565 thru 0x02d773 (0x520f bytes) 000363 1775867387.401218 mapwrite 0x000e2d thru 0x00bcaf (0xae83 bytes) 000365 1775867387.598856 mapread 0x01d175 thru 0x01f7c6 (0x2652 bytes) 000367 1775867387.625224 mapread 0x00915b thru 0x00d230 (0x40d6 bytes) 000368 1775867387.630837 trunc from 0x02d774 to 0x0203cd (0xd3a6 bytes) 000369 1775867387.692208 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) 000370 1775867387.839545 write 0x0343b8 thru 0x03f238 (0xae81 bytes) 000371 1775867387.845702 MIRROR_EXTEND to 2 mirrors 000373 1775867388.752632 trunc from 0x03f239 to 0x000602 (0x3ec36 bytes) 000376 1775867388.872973 MIRROR_RESYNC 2 mirrors 000379 1775867389.505751 read 0x00029f thru 0x000602 (0x364 bytes) 000380 1775867389.515412 write 0x032dbb thru 0x039fe9 (0x722f bytes) 000381 1775867389.630742 MIRROR_RESYNC 2 mirrors 000382 1775867390.581511 write 0x02f13b thru 0x03796b (0x8831 bytes) 000385 1775867390.690484 mapread 0x018ca5 thru 0x02826c (0xf5c8 bytes) 000386 1775867390.717461 read 0x01048f thru 0x01a9d1 (0xa543 bytes) 000388 1775867390.726439 mapread 0x035afd thru 0x039fe9 (0x44ed bytes) 000390 1775867390.750748 write 0x01c6bc thru 0x02310f (0x6a54 bytes) 000392 1775867390.753455 mapread 0x02bd19 thru 0x032d7a (0x7062 bytes) 000393 1775867390.769801 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x39fea): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39fea): Mirror 2: 0xf003e260 (stale) CRC-32 checksum value for chunk [0, 0x39fea): Mirror 1: 0xe97d35af Mirror 3: 0xe97d35af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39fea: skipped 000394 1775867391.903840 mapread 0x025f99 thru 0x028c5b (0x2cc3 bytes) 000400 1775867391.934178 MIRROR_RESYNC 3 mirrors 000401 1775867393.215916 read 0x013211 thru 0x01df9a (0xad8a bytes) 000402 1775867393.242104 write 0x035a5c thru 0x03ffff (0xa5a4 bytes) 000403 1775867393.384548 write 0x00ec44 thru 0x01b5bd (0xc97a bytes) 000404 1775867393.425995 read 0x02b2ec thru 0x0354d3 (0xa1e8 bytes) 000405 1775867393.446487 trunc from 0x040000 to 0x00f2d8 (0x30d27 bytes) 000409 1775867393.493898 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf2d9): Mirror 2: 0xac9e39e4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf2d9): Mirror 2: 0xac9e39e4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf2d9: skipped 000410 1775867393.960169 trunc from 0x00f2d9 to 0x01def5 (0xec1d bytes) 000412 1775867393.999208 trunc from 0x01def6 to 0x027cfa (0x9e05 bytes) 000413 1775867394.039674 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x27cfb): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27cfb): Mirror 2: 0xb91e017f (stale) CRC-32 checksum value for chunk [0, 0x27cfb): Mirror 1: 0xa9ce5778 Mirror 3: 0xa9ce5778 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27cfb: skipped 000415 1775867395.283726 trunc from 0x027cfb to 0x022bf2 (0x5108 bytes) 000417 1775867395.474720 write 0x03ffce thru 0x03ffff (0x32 bytes) 000418 1775867395.483223 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xce60e22f (stale) Mirror 3: 0x77dac87a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb54adf66 Mirror 4: 0xb54adf66 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000419 1775867396.620712 write 0x01b1bd thru 0x020e05 (0x5c49 bytes) 000422 1775867396.724199 MIRROR_RESYNC 4 mirrors 000423 1775867398.127996 MIRROR_RESYNC 4 mirrors 000424 1775867398.640518 mapwrite 0x00daa5 thru 0x019917 (0xbe73 bytes) 000426 1775867398.902673 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8d569f5d (stale) Mirror 3: 0x8d569f5d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8d569f5d (stale) Mirror 3: 0x8d569f5d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000429 1775867399.436583 mapread 0x030a92 thru 0x03ffff (0xf56e bytes) 000430 1775867399.495988 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8d569f5d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8d569f5d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000432 1775867399.931471 mapwrite 0x03c17d thru 0x03ffff (0x3e83 bytes) 000433 1775867400.048824 trunc from 0x040000 to 0x019384 (0x26c7b bytes) 000434 1775867400.086943 mapread 0x00a97c thru 0x01414f (0x97d4 bytes) 000435 1775867400.173396 write 0x01906f thru 0x01bd3e (0x2cd0 bytes) 000436 1775867400.175706 write 0x02a61a thru 0x03364f (0x9036 bytes) 000440 1775867400.194185 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x33650): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33650): Mirror 3: 0xe2370161 (stale) CRC-32 checksum value for chunk [0, 0x33650): Mirror 1: 0x7fd24549 Mirror 4: 0x7fd24549 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33650: skipped 000441 1775867401.381008 read 0x02bca9 thru 0x03364f (0x79a7 bytes) 000444 1775867401.400229 mapwrite 0x03112f thru 0x0334c9 (0x239b bytes) 000445 1775867401.662988 write 0x00c8c7 thru 0x015206 (0x8940 bytes) 000452 1775867401.697593 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x33650): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33650): Mirror 3: 0xe2370161 (stale) Mirror 4: 0x7fd24549 (stale) CRC-32 checksum value for chunk [0, 0x33650): Mirror 1: 0x5f6f9e39 Mirror 5: 0x5f6f9e39 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33650: skipped 000454 1775867402.984157 trunc from 0x033650 to 0x011d7f (0x218d0 bytes) 000456 1775867403.159228 write 0x039048 thru 0x03ec29 (0x5be2 bytes) 000458 1775867403.163701 trunc from 0x03ec2a to 0x006c9b (0x37f8e bytes) 000459 1775867403.191547 mapwrite 0x00471d thru 0x00ea13 (0xa2f7 bytes) 000460 1775867403.287720 mapread 0x00019a thru 0x00e5ca (0xe431 bytes) 000462 1775867403.296572 mapwrite 0x0108c6 thru 0x01e066 (0xd7a1 bytes) 000463 1775867403.391440 trunc from 0x01e067 to 0x00ba7f (0x125e7 bytes) 000465 1775867403.416485 mapread 0x00512e thru 0x0054c7 (0x39a bytes) 000467 1775867403.420892 read 0x00b866 thru 0x00ba7f (0x21a bytes) 000468 1775867403.423481 mapwrite 0x03aa5a thru 0x03ffff (0x55a6 bytes) 000469 1775867403.496389 write 0x039ee1 thru 0x03ffff (0x611f bytes) 000471 1775867403.499488 mapread 0x001845 thru 0x00e78c (0xcf48 bytes) 000472 1775867403.511417 read 0x014a4c thru 0x0235c8 (0xeb7d bytes) 000473 1775867403.521583 read 0x00c5b4 thru 0x015f61 (0x99ae bytes) 000474 1775867403.522412 read 0x03b347 thru 0x03ffff (0x4cb9 bytes) 000475 1775867403.523041 mapwrite 0x031f66 thru 0x0377b4 (0x584f bytes) 000476 1775867403.569835 mapwrite 0x00f22a thru 0x010cd7 (0x1aae bytes) 000477 1775867403.620702 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x2b525cea (stale) Mirror 5: 0x9be714aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x2b525cea (stale) Mirror 5: 0x9be714aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000480 1775867404.002034 trunc from 0x040000 to 0x016f06 (0x290f9 bytes) 000481 1775867404.035476 read 0x008ba4 thru 0x0160dc (0xd539 bytes) 000483 1775867404.056565 trunc from 0x016f07 to 0x00cd2d (0xa1d9 bytes) 000484 1775867404.122127 mapwrite 0x004d36 thru 0x00a160 (0x542b bytes) 000488 1775867404.262348 read 0x0069c4 thru 0x00cd2d (0x636a bytes) 000491 1775867404.266012 read 0x00b389 thru 0x00cd2d (0x19a5 bytes) 000493 1775867404.266999 mapwrite 0x006cb9 thru 0x00e196 (0x74de bytes) 000494 1775867404.343101 write 0x006418 thru 0x014cc2 (0xe8ab bytes) 000495 1775867404.345046 mapwrite 0x0352ba thru 0x03711d (0x1e64 bytes) 000497 1775867404.471635 mapread 0x00f2cf thru 0x011e06 (0x2b38 bytes) 000498 1775867404.474091 trunc from 0x03711e to 0x025821 (0x118fc bytes) 000500 1775867404.502871 mapwrite 0x03d7dc thru 0x03ffff (0x2824 bytes) 000503 1775867404.611858 trunc from 0x040000 to 0x026452 (0x19bad bytes) 000504 1775867404.635816 mapwrite 0x01399a thru 0x016840 (0x2ea7 bytes) 000506 1775867404.694652 mapwrite 0x0101f5 thru 0x012b71 (0x297d bytes) 000507 1775867404.734706 mapread 0x021084 thru 0x026452 (0x53cf bytes) 000508 1775867404.743307 mapread 0x00c78a thru 0x013085 (0x68fc bytes) 000511 1775867404.748571 trunc from 0x026453 to 0x00bee9 (0x1a569 bytes) 000512 1775867404.770558 read 0x00affb thru 0x00bee9 (0xeef bytes) 000513 1775867404.772306 write 0x002600 thru 0x00db49 (0xb54a bytes) 000514 1775867404.782932 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xdb4a): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdb4a): Mirror 4: 0x27314a82 (stale) Mirror 5: 0xef8d2c87 (stale) CRC-32 checksum value for chunk [0, 0xdb4a): Mirror 1: 0x358e3ef9 Mirror 6: 0x358e3ef9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdb4a: skipped 000515 1775867405.885880 trunc from 0x00db4a to 0x038b3f (0x2aff6 bytes) 000516 1775867406.007119 read 0x01f470 thru 0x020fc8 (0x1b59 bytes) 000518 1775867406.016951 write 0x005247 thru 0x014290 (0xf04a bytes) 000520 1775867406.033122 write 0x038013 thru 0x03f352 (0x7340 bytes) 000521 1775867406.060884 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3f353): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f353): Mirror 4: 0x769c2196 (stale) Mirror 5: 0xbfab1cf6 (stale) Mirror 6: 0x384544fc (stale) CRC-32 checksum value for chunk [0, 0x3f353): Mirror 1: 0x4faa14f1 Mirror 7: 0x4faa14f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f353: skipped 000522 1775867406.869286 trunc from 0x03f353 to 0x0228e0 (0x1ca72 bytes) 000524 1775867407.007089 read 0x01ed97 thru 0x0228e0 (0x3b4a bytes) 000525 1775867407.020904 read 0x014aaa thru 0x0185d4 (0x3b2b bytes) 000526 1775867407.035472 mapread 0x0091ae thru 0x00cd1d (0x3b70 bytes) 000529 1775867407.039603 read 0x000e3d thru 0x01007b (0xf23f bytes) 000532 1775867407.042214 mapwrite 0x02c005 thru 0x0325da (0x65d6 bytes) 000534 1775867407.162465 trunc from 0x0325db to 0x00e196 (0x24444 bytes) 000535 1775867407.193892 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xe197): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe197): Mirror 4: 0xc6382e6c (stale) Mirror 5: 0xf38c9b9c (stale) Mirror 6: 0x59d1ca45 (stale) Mirror 7: 0x19f06de (stale) CRC-32 checksum value for chunk [0, 0xe197): Mirror 1: 0x19f06de Mirror 8: 0x19f06de lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe197: skipped 000536 1775867408.141351 mapwrite 0x0371e6 thru 0x03ffff (0x8e1a bytes) 000537 1775867408.384460 mapwrite 0x017d0c thru 0x01a2d5 (0x25ca bytes) 000538 1775867408.467801 MIRROR_RESYNC 6 mirrors 000539 1775867409.585387 mapwrite 0x0216ea thru 0x0288e6 (0x71fd bytes) 000540 1775867409.803660 mapwrite 0x0142c8 thru 0x01705e (0x2d97 bytes) 000542 1775867409.890335 write 0x0266ef thru 0x029636 (0x2f48 bytes) 000549 1775867409.898346 mapwrite 0x01260b thru 0x01581a (0x3210 bytes) 000550 1775867410.052468 mapwrite 0x03fa02 thru 0x03ffff (0x5fe bytes) 000551 1775867410.186119 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x2ff8ecb0 (stale) Mirror 5: 0x2ff8ecb0 (stale) Mirror 6: 0x2ff8ecb0 (stale) Mirror 7: 0x2ff8ecb0 (stale) Mirror 8: 0x2ff8ecb0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x76c16d51 Mirror 9: 0x76c16d51 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000553 1775867411.190469 mapwrite 0x01a095 thru 0x01f42b (0x5397 bytes) 000554 1775867411.680135 trunc from 0x040000 to 0x027cf9 (0x18306 bytes) 000558 1775867411.783752 mapwrite 0x002e3e thru 0x010ff0 (0xe1b3 bytes) 000560 1775867411.857059 trunc from 0x027cfa to 0x02c203 (0x450a bytes) 000562 1775867411.898349 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2c204): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c204): Mirror 4: 0xa40b9160 (stale) Mirror 5: 0xa40b9160 (stale) Mirror 6: 0xa40b9160 (stale) Mirror 7: 0xa40b9160 (stale) Mirror 8: 0xa40b9160 (stale) Mirror 9: 0x6c2c991c (stale) CRC-32 checksum value for chunk [0, 0x2c204): Mirror 1: 0x7144042d Mirror 10: 0x7144042d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c204: skipped 000563 1775867413.356976 write 0x0179e2 thru 0x023f3b (0xc55a bytes) 000564 1775867413.551088 read 0x0272fb thru 0x02c203 (0x4f09 bytes) 000565 1775867413.566468 mapread 0x001b8f thru 0x00ca39 (0xaeab bytes) 000570 1775867413.585514 read 0x0205fc thru 0x02b845 (0xb24a bytes) 000571 1775867413.608284 mapwrite 0x03a070 thru 0x03ffff (0x5f90 bytes) 000572 1775867413.791334 mapread 0x03462a thru 0x03ffff (0xb9d6 bytes) 000577 1775867413.819773 read 0x03d33c thru 0x03ffff (0x2cc4 bytes) 000579 1775867413.826084 write 0x018e71 thru 0x0236e0 (0xa870 bytes) 000581 1775867413.831390 mapwrite 0x003927 thru 0x0120c9 (0xe7a3 bytes) 000582 1775867414.014792 read 0x013b92 thru 0x01f32b (0xb79a bytes) 000584 1775867414.021745 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x2ff8ecb0 (stale) Mirror 5: 0x2ff8ecb0 (stale) Mirror 6: 0x2ff8ecb0 (stale) Mirror 7: 0x2ff8ecb0 (stale) Mirror 8: 0x2ff8ecb0 (stale) Mirror 10: 0x89ca4300 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x2ff8ecb0 (stale) Mirror 5: 0x2ff8ecb0 (stale) Mirror 6: 0x2ff8ecb0 (stale) Mirror 7: 0x2ff8ecb0 (stale) Mirror 8: 0x2ff8ecb0 (stale) Mirror 10: 0x89ca4300 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000585 1775867414.494953 write 0x033cb1 thru 0x03ffff (0xc34f bytes) 000586 1775867414.508012 trunc from 0x040000 to 0x02c62d (0x139d2 bytes) 000587 1775867414.550532 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c62e): Mirror 5: 0xd68facc (stale) Mirror 6: 0xd68facc (stale) Mirror 7: 0xd68facc (stale) Mirror 8: 0xd68facc (stale) Mirror 10: 0x3ec4d2e2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c62e): Mirror 5: 0xd68facc (stale) Mirror 6: 0xd68facc (stale) Mirror 7: 0xd68facc (stale) Mirror 8: 0xd68facc (stale) Mirror 10: 0x3ec4d2e2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c62e: skipped 000589 1775867415.165534 mapread 0x01f045 thru 0x023017 (0x3fd3 bytes) 000590 1775867415.195355 trunc from 0x02c62e to 0x01d0f2 (0xf53b bytes) 000591 1775867415.237101 mapwrite 0x01ea7f thru 0x01fd8d (0x130f bytes) 000593 1775867415.341708 write 0x0243a9 thru 0x02578e (0x13e6 bytes) 000595 1775867415.350962 mapread 0x018a09 thru 0x020ebe (0x84b6 bytes) 000597 1775867415.369651 MIRROR_RESYNC 6 mirrors 000599 1775867416.418845 mapread 0x01dd7d thru 0x0248dd (0x6b61 bytes) 000600 1775867416.441717 mapwrite 0x03c96e thru 0x03ffff (0x3692 bytes) 000601 1775867416.544308 write 0x011d08 thru 0x013dd1 (0x20ca bytes) 000602 1775867416.560933 write 0x026f06 thru 0x02b43b (0x4536 bytes) 000603 1775867416.575854 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x8db3291d (stale) Mirror 6: 0x8db3291d (stale) Mirror 7: 0x8db3291d (stale) Mirror 8: 0x8db3291d (stale) Mirror 10: 0x8db3291d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfd6b5eb5 Mirror 11: 0xfd6b5eb5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000605 1775867417.486102 mapwrite 0x02d4da thru 0x03a22c (0xcd53 bytes) 000607 1775867417.706453 trunc from 0x040000 to 0x03aeea (0x5115 bytes) 000609 1775867417.757288 MIRROR_RESYNC 7 mirrors 000611 1775867420.044022 mapread 0x027512 thru 0x031e2a (0xa919 bytes) 000613 1775867420.077558 mapwrite 0x02d37c thru 0x032f06 (0x5b8b bytes) 000615 1775867420.330233 read 0x00e85d thru 0x011c42 (0x33e6 bytes) 000616 1775867420.353367 mapwrite 0x0205ff thru 0x027991 (0x7393 bytes) 000617 1775867420.484760 write 0x03b41c thru 0x03ffff (0x4be4 bytes) 000618 1775867420.486016 trunc from 0x040000 to 0x008081 (0x37f7e bytes) 000619 1775867420.536332 mapwrite 0x001b4e thru 0x007b24 (0x5fd7 bytes) 000621 1775867420.663232 mapwrite 0x030ac8 thru 0x03848f (0x79c8 bytes) 000622 1775867420.893635 MIRROR_RESYNC 7 mirrors 000623 1775867423.186785 trunc from 0x038490 to 0x030d3a (0x7755 bytes) 000625 1775867423.353361 read 0x00bfc7 thru 0x00c719 (0x753 bytes) 000626 1775867423.365790 mapwrite 0x015128 thru 0x01bdd4 (0x6cad bytes) 000627 1775867423.457065 trunc from 0x030d3b to 0x0241ac (0xcb8e bytes) 000628 1775867423.512046 mapwrite 0x0398f5 thru 0x03d5a8 (0x3cb4 bytes) 000629 1775867423.739386 trunc from 0x03d5a9 to 0x00d432 (0x30176 bytes) 000631 1775867423.793425 mapwrite 0x03239c thru 0x03c853 (0xa4b8 bytes) 000632 1775867424.009888 read 0x0175ff thru 0x01d0eb (0x5aed bytes) 000633 1775867424.033362 read 0x027a0b thru 0x02fc72 (0x8268 bytes) 000635 1775867424.037224 write 0x027b8c thru 0x02f097 (0x750c bytes) 000636 1775867424.043216 mapread 0x02ee46 thru 0x0364cc (0x7687 bytes) 000640 1775867424.052056 read 0x025a1d thru 0x035707 (0xfceb bytes) 000644 1775867424.065216 read 0x032631 thru 0x033bad (0x157d bytes) 000645 1775867424.071743 trunc from 0x03c854 to 0x00d207 (0x2f64c bytes) 000648 1775867424.130995 mapwrite 0x00ae84 thru 0x00c270 (0x13ed bytes) 000649 1775867424.236428 write 0x00c881 thru 0x00cc45 (0x3c5 bytes) 000650 1775867424.239776 read 0x00c157 thru 0x00d1f9 (0x10a3 bytes) 000651 1775867424.242943 mapwrite 0x01a242 thru 0x0213d0 (0x718f bytes) 000652 1775867424.412957 mapwrite 0x03f1a6 thru 0x03ffff (0xe5a bytes) 000653 1775867424.638566 trunc from 0x040000 to 0x00aa8d (0x35572 bytes) 000654 1775867424.671007 read 0x0067e5 thru 0x00aa8d (0x42a9 bytes) 000656 1775867424.677323 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xaa8e): Mirror 5: 0x90b1124f (stale) Mirror 6: 0x90b1124f (stale) Mirror 7: 0x90b1124f (stale) Mirror 8: 0x90b1124f (stale) Mirror 11: 0x90b1124f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xaa8e): Mirror 5: 0x90b1124f (stale) Mirror 6: 0x90b1124f (stale) Mirror 7: 0x90b1124f (stale) Mirror 8: 0x90b1124f (stale) Mirror 11: 0x90b1124f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xaa8e: skipped 000660 1775867425.458649 mapwrite 0x029e3a thru 0x036e9c (0xd063 bytes) 000661 1775867425.741325 trunc from 0x036e9d to 0x02036d (0x16b2f bytes) 000662 1775867425.783821 read 0x002933 thru 0x00ef03 (0xc5d1 bytes) 000663 1775867425.807943 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2036e): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2036e): Mirror 5: 0x20a124a7 (stale) Mirror 6: 0x20a124a7 (stale) Mirror 7: 0x20a124a7 (stale) Mirror 8: 0x20a124a7 (stale) Mirror 11: 0x20a124a7 (stale) CRC-32 checksum value for chunk [0, 0x2036e): Mirror 1: 0x20a124a7 Mirror 12: 0x20a124a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2036e: skipped 000664 1775867427.894983 mapwrite 0x03b79f thru 0x03ffff (0x4861 bytes) 000665 1775867428.350967 trunc from 0x040000 to 0x0070d0 (0x38f2f bytes) 000667 1775867428.432516 write 0x03bb28 thru 0x03ffff (0x44d8 bytes) 000674 1775867428.459729 write 0x02ae2b thru 0x037a10 (0xcbe6 bytes) 000675 1775867428.523319 mapwrite 0x0373c1 thru 0x03fbf7 (0x8837 bytes) 000677 1775867428.737022 MIRROR_RESYNC 7 mirrors 000679 1775867430.589451 MIRROR_RESYNC 7 mirrors 000680 1775867431.250611 mapwrite 0x008231 thru 0x01148f (0x925f bytes) 000681 1775867431.516478 mapread 0x01c028 thru 0x020068 (0x4041 bytes) 000682 1775867431.554226 write 0x034faa thru 0x0354a4 (0x4fb bytes) 000683 1775867431.594897 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x7125ea6 (stale) Mirror 6: 0x7125ea6 (stale) Mirror 7: 0x7125ea6 (stale) Mirror 8: 0x7125ea6 (stale) Mirror 11: 0x7125ea6 (stale) Mirror 12: 0x7125ea6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3eb00aa Mirror 13: 0x3eb00aa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000687 1775867433.869436 write 0x0388d3 thru 0x03ffff (0x772d bytes) 000689 1775867433.950755 mapread 0x036629 thru 0x037b08 (0x14e0 bytes) 000692 1775867433.969656 write 0x009652 thru 0x012c17 (0x95c6 bytes) 000697 1775867433.990728 trunc from 0x040000 to 0x0319b8 (0xe647 bytes) 000698 1775867434.015280 trunc from 0x0319b9 to 0x039d24 (0x836c bytes) 000699 1775867434.039927 mapread 0x006dab thru 0x013e9f (0xd0f5 bytes) 000701 1775867434.082530 trunc from 0x039d25 to 0x03f641 (0x591d bytes) 000702 1775867434.141920 trunc from 0x03f642 to 0x02602c (0x19615 bytes) 000704 1775867434.206553 write 0x03e94b thru 0x03ffff (0x16b5 bytes) 000705 1775867434.207805 mapwrite 0x008328 thru 0x009431 (0x110a bytes) 000707 1775867434.306150 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x7125ea6 (stale) Mirror 6: 0x7125ea6 (stale) Mirror 8: 0x7125ea6 (stale) Mirror 11: 0x7125ea6 (stale) Mirror 12: 0x7125ea6 (stale) Mirror 13: 0x3eb00aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x7125ea6 (stale) Mirror 6: 0x7125ea6 (stale) Mirror 8: 0x7125ea6 (stale) Mirror 11: 0x7125ea6 (stale) Mirror 12: 0x7125ea6 (stale) Mirror 13: 0x3eb00aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000710 1775867434.882189 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x7125ea6 (stale) Mirror 8: 0x7125ea6 (stale) Mirror 11: 0x7125ea6 (stale) Mirror 12: 0x7125ea6 (stale) Mirror 13: 0x3eb00aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x7125ea6 (stale) Mirror 8: 0x7125ea6 (stale) Mirror 11: 0x7125ea6 (stale) Mirror 12: 0x7125ea6 (stale) Mirror 13: 0x3eb00aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000711 1775867435.475807 write 0x01d5fb thru 0x02304c (0x5a52 bytes) 000712 1775867435.495349 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x7125ea6 (stale) Mirror 8: 0x7125ea6 (stale) Mirror 11: 0x7125ea6 (stale) Mirror 12: 0x7125ea6 (stale) Mirror 13: 0x3eb00aa (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9b987b2 Mirror 14: 0x9b987b2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000713 1775867436.950925 mapread 0x00f1aa thru 0x01c784 (0xd5db bytes) 000715 1775867437.065841 mapread 0x03012c thru 0x03a469 (0xa33e bytes) 000718 1775867437.141048 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x7125ea6 (stale) Mirror 8: 0x7125ea6 (stale) Mirror 11: 0x7125ea6 (stale) Mirror 12: 0x7125ea6 (stale) Mirror 13: 0x3eb00aa (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9b987b2 Mirror 14: 0x9b987b2 Mirror 15: 0x9b987b2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000720 1775867438.325991 write 0x015685 thru 0x01eff1 (0x996d bytes) 000722 1775867438.437126 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x7125ea6 (stale) Mirror 8: 0x7125ea6 (stale) Mirror 11: 0x7125ea6 (stale) Mirror 12: 0x7125ea6 (stale) Mirror 13: 0x3eb00aa (stale) Mirror 14: 0x9b987b2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x7125ea6 (stale) Mirror 8: 0x7125ea6 (stale) Mirror 11: 0x7125ea6 (stale) Mirror 12: 0x7125ea6 (stale) Mirror 13: 0x3eb00aa (stale) Mirror 14: 0x9b987b2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000723 1775867438.944848 trunc from 0x040000 to 0x02e9ed (0x11612 bytes) 000727 1775867438.977602 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e9ee): Mirror 6: 0xcd139a6e (stale) Mirror 8: 0xcd139a6e (stale) Mirror 11: 0xcd139a6e (stale) Mirror 13: 0xf4fc8554 (stale) Mirror 14: 0x946531eb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e9ee): Mirror 6: 0xcd139a6e (stale) Mirror 8: 0xcd139a6e (stale) Mirror 11: 0xcd139a6e (stale) Mirror 13: 0xf4fc8554 (stale) Mirror 14: 0x946531eb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e9ee: skipped 000728 1775867439.417846 MIRROR_RESYNC 6 mirrors 000730 1775867440.560089 read 0x01983e thru 0x01a5e0 (0xda3 bytes) 000733 1775867440.571808 mapwrite 0x0093a4 thru 0x00ecce (0x592b bytes) 000734 1775867440.776742 mapread 0x015807 thru 0x020e5f (0xb659 bytes) 000735 1775867440.812570 mapwrite 0x021285 thru 0x029d67 (0x8ae3 bytes) 000736 1775867440.918876 mapread 0x0082d8 thru 0x01496d (0xc696 bytes) 000738 1775867440.920913 mapwrite 0x03f65e thru 0x03ffff (0x9a2 bytes) 000739 1775867441.033060 trunc from 0x040000 to 0x00018b (0x3fe74 bytes) 000740 1775867441.077216 read 0x00011e thru 0x00018b (0x6e bytes) 000742 1775867441.094371 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x18c): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18c): Mirror 6: 0x64f1bd4c (stale) Mirror 8: 0x64f1bd4c (stale) Mirror 11: 0x64f1bd4c (stale) Mirror 13: 0x64f1bd4c (stale) Mirror 14: 0x64f1bd4c (stale) CRC-32 checksum value for chunk [0, 0x18c): Mirror 1: 0x64f1bd4c Mirror 15: 0x64f1bd4c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18c: skipped 000744 1775867442.264139 mapwrite 0x001ad5 thru 0x008394 (0x68c0 bytes) 000745 1775867442.470697 MIRROR_RESYNC 7 mirrors 000747 1775867443.439641 read 0x00225c thru 0x007801 (0x55a6 bytes) 000748 1775867443.450186 write 0x02c9b0 thru 0x038cf6 (0xc347 bytes) 000749 1775867443.519788 write 0x036b7f thru 0x03ffff (0x9481 bytes) 000751 1775867443.522162 mapread 0x00c72e thru 0x017f14 (0xb7e7 bytes) 000753 1775867443.566325 MIRROR_RESYNC 7 mirrors 000754 1775867444.956061 read 0x0152fa thru 0x0228c0 (0xd5c7 bytes) 000755 1775867444.973935 mapread 0x01aede thru 0x01c389 (0x14ac bytes) 000756 1775867444.974702 trunc from 0x040000 to 0x037a9a (0x8565 bytes) 000757 1775867445.094971 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x37a9b): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a9b): Mirror 6: 0x8ea20359 (stale) Mirror 8: 0x8ea20359 (stale) Mirror 11: 0x8ea20359 (stale) Mirror 13: 0x8ea20359 (stale) Mirror 14: 0x8ea20359 (stale) Mirror 15: 0x8ea20359 (stale) CRC-32 checksum value for chunk [0, 0x37a9b): Mirror 1: 0x8ea20359 Mirror 16: 0x8ea20359 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a9b: skipped 000758 1775867446.223514 trunc from 0x037a9b to 0x01566f (0x2242b bytes) 000759 1775867446.327877 read 0x007da4 thru 0x0142bc (0xc519 bytes) 000760 1775867446.337305 trunc from 0x015670 to 0x02b4a4 (0x15e35 bytes) 000761 1775867446.383747 MIRROR_RESYNC 8 mirrors 000763 1775867448.185791 MIRROR_RESYNC 8 mirrors 000764 1775867448.724904 mapwrite 0x02d86b thru 0x031682 (0x3e18 bytes) 000765 1775867448.904707 write 0x006c30 thru 0x015516 (0xe8e7 bytes) 000767 1775867448.924458 trunc from 0x031683 to 0x03ab53 (0x94d1 bytes) 000769 1775867448.954630 mapread 0x00ddd8 thru 0x0175ad (0x97d6 bytes) 000772 1775867448.968949 mapwrite 0x019fea thru 0x024347 (0xa35e bytes) 000774 1775867449.060625 write 0x02a1dc thru 0x0345b6 (0xa3db bytes) 000775 1775867449.068747 read 0x030ab0 thru 0x034844 (0x3d95 bytes) 000776 1775867449.076128 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ab54): Mirror 6: 0x415e8c67 (stale) Mirror 8: 0x415e8c67 (stale) Mirror 11: 0x415e8c67 (stale) Mirror 14: 0x415e8c67 (stale) Mirror 15: 0x415e8c67 (stale) Mirror 16: 0x415e8c67 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ab54): Mirror 6: 0x415e8c67 (stale) Mirror 8: 0x415e8c67 (stale) Mirror 11: 0x415e8c67 (stale) Mirror 14: 0x415e8c67 (stale) Mirror 15: 0x415e8c67 (stale) Mirror 16: 0x415e8c67 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ab54: skipped 000777 1775867449.594691 mapread 0x002003 thru 0x009b69 (0x7b67 bytes) 000778 1775867449.624906 read 0x00f632 thru 0x01313e (0x3b0d bytes) 000779 1775867449.629730 trunc from 0x03ab54 to 0x008388 (0x327cb bytes) 000780 1775867449.647378 write 0x00bbfc thru 0x019c55 (0xe05a bytes) 000782 1775867449.650687 write 0x00f538 thru 0x012680 (0x3149 bytes) 000784 1775867449.652875 write 0x01484f thru 0x01ce0a (0x85bc bytes) 000786 1775867449.655700 read 0x00a690 thru 0x015ba8 (0xb519 bytes) 000787 1775867449.668084 MIRROR_RESYNC 7 mirrors 000788 1775867450.864140 read 0x003a35 thru 0x004c2d (0x11f9 bytes) 000790 1775867450.869349 write 0x000ac4 thru 0x004ca5 (0x41e2 bytes) 000791 1775867450.948908 write 0x031e40 thru 0x03ffff (0xe1c0 bytes) 000792 1775867450.951873 read 0x02c1ce thru 0x02d2c6 (0x10f9 bytes) 000794 1775867450.964611 read 0x01b4ec thru 0x023557 (0x806c bytes) 000796 1775867450.966436 read 0x0284c3 thru 0x037eab (0xf9e9 bytes) 000798 1775867450.967830 mapread 0x028746 thru 0x029668 (0xf23 bytes) 000800 1775867450.970525 mapwrite 0x028b6c thru 0x031895 (0x8d2a bytes) 000804 1775867451.083370 read 0x039374 thru 0x03a972 (0x15ff bytes) 000805 1775867451.084341 trunc from 0x040000 to 0x01ce21 (0x231de bytes) 000806 1775867451.135420 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ce22): Mirror 6: 0xd2f99587 (stale) Mirror 8: 0xd2f99587 (stale) Mirror 11: 0xd2f99587 (stale) Mirror 15: 0xd2f99587 (stale) Mirror 16: 0xd2f99587 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ce22): Mirror 6: 0xd2f99587 (stale) Mirror 8: 0xd2f99587 (stale) Mirror 11: 0xd2f99587 (stale) Mirror 15: 0xd2f99587 (stale) Mirror 16: 0xd2f99587 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ce22: skipped 000807 1775867451.727907 mapwrite 0x00ec5f thru 0x0125fc (0x399e bytes) 000809 1775867451.836539 mapread 0x018ee5 thru 0x01c190 (0x32ac bytes) 000810 1775867451.844654 read 0x007a7e thru 0x007e73 (0x3f6 bytes) 000814 1775867451.854959 read 0x01b23e thru 0x01ce21 (0x1be4 bytes) 000817 1775867451.859037 write 0x00bb48 thru 0x00db0f (0x1fc8 bytes) 000818 1775867451.860759 trunc from 0x01ce22 to 0x00d686 (0xf79b bytes) 000819 1775867451.882333 mapread 0x008685 thru 0x00d686 (0x5002 bytes) 000821 1775867451.898051 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd687): Mirror 8: 0x15ff4592 (stale) Mirror 11: 0x15ff4592 (stale) Mirror 15: 0x15ff4592 (stale) Mirror 16: 0x15ff4592 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd687): Mirror 8: 0x15ff4592 (stale) Mirror 11: 0x15ff4592 (stale) Mirror 15: 0x15ff4592 (stale) Mirror 16: 0x15ff4592 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd687: skipped 000825 1775867452.541478 read 0x003fd8 thru 0x00d686 (0x96af bytes) 000826 1775867452.573909 trunc from 0x00d687 to 0x03b99e (0x2e318 bytes) 000827 1775867452.644870 mapwrite 0x0319bb thru 0x03443d (0x2a83 bytes) 000828 1775867452.757459 mapwrite 0x01b06e thru 0x020dd7 (0x5d6a bytes) 000829 1775867452.835632 mapread 0x0135e2 thru 0x017437 (0x3e56 bytes) 000831 1775867452.854550 trunc from 0x03b99f to 0x003cdf (0x37cbf bytes) 000832 1775867452.884333 trunc from 0x003ce0 to 0x0009d7 (0x3308 bytes) 000833 1775867452.909584 trunc from 0x0009d8 to 0x026f42 (0x2656b bytes) 000834 1775867452.925731 trunc from 0x026f43 to 0x00c1de (0x1ad64 bytes) 000836 1775867452.950670 read 0x0078e8 thru 0x00c1de (0x48f7 bytes) 000837 1775867452.966705 mapread 0x005c1a thru 0x00c1de (0x65c5 bytes) 000838 1775867452.973348 mapread 0x00bc5f thru 0x00c1de (0x580 bytes) 000839 1775867452.979948 mapread 0x0062ba thru 0x009787 (0x34ce bytes) 000840 1775867452.994583 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xc1df): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc1df): Mirror 8: 0xc9acd4c8 (stale) Mirror 11: 0xc9acd4c8 (stale) Mirror 15: 0xc9acd4c8 (stale) Mirror 16: 0xc9acd4c8 (stale) CRC-32 checksum value for chunk [0, 0xc1df): Mirror 1: 0x5de5ca47 Mirror 17: 0x5de5ca47 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc1df: skipped 000841 1775867454.221624 mapread 0x000b0e thru 0x007439 (0x692c bytes) 000843 1775867454.263826 mapread 0x00109e thru 0x0031d9 (0x213c bytes) 000845 1775867454.265137 trunc from 0x00c1df to 0x02cd61 (0x20b83 bytes) 000846 1775867454.381201 mapwrite 0x028d1c thru 0x02dafc (0x4de1 bytes) 000847 1775867454.466556 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2dafd): Mirror 11: 0x70b2e605 (stale) Mirror 15: 0x70b2e605 (stale) Mirror 16: 0x70b2e605 (stale) Mirror 17: 0x30e5393a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dafd): Mirror 11: 0x70b2e605 (stale) Mirror 15: 0x70b2e605 (stale) Mirror 16: 0x70b2e605 (stale) Mirror 17: 0x30e5393a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dafd: skipped 000849 1775867455.079978 write 0x011ccb thru 0x020b69 (0xee9f bytes) 000850 1775867455.101813 mapread 0x013999 thru 0x0201b7 (0xc81f bytes) 000851 1775867455.103340 trunc from 0x02dafd to 0x013d03 (0x19df9 bytes) 000852 1775867455.148995 mapwrite 0x03de68 thru 0x03ffff (0x2198 bytes) 000853 1775867455.269209 read 0x00938c thru 0x012ff2 (0x9c67 bytes) 000856 1775867455.282435 write 0x003a96 thru 0x00af90 (0x74fb bytes) 000857 1775867455.296165 read 0x0127da thru 0x0137fd (0x1024 bytes) 000858 1775867455.296838 trunc from 0x040000 to 0x021fa3 (0x1e05c bytes) 000860 1775867455.321831 mapwrite 0x029c6b thru 0x032c88 (0x901e bytes) 000861 1775867455.419919 mapwrite 0x015d77 thru 0x0229ac (0xcc36 bytes) 000863 1775867455.479074 mapread 0x0268df thru 0x026a1a (0x13c bytes) 000865 1775867455.482821 write 0x01f7db thru 0x023a46 (0x426c bytes) 000866 1775867455.484554 mapwrite 0x02b731 thru 0x0344b1 (0x8d81 bytes) 000868 1775867455.562065 trunc from 0x0344b2 to 0x00a29d (0x2a214 bytes) 000870 1775867455.594941 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa29e): Mirror 11: 0x70884a18 (stale) Mirror 15: 0x70884a18 (stale) Mirror 16: 0x70884a18 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa29e): Mirror 11: 0x70884a18 (stale) Mirror 15: 0x70884a18 (stale) Mirror 16: 0x70884a18 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa29e: skipped 000871 1775867455.987613 mapwrite 0x00c082 thru 0x01618f (0xa10e bytes) 000872 1775867456.196916 mapwrite 0x03e0eb thru 0x03ffff (0x1f15 bytes) 000875 1775867456.337362 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xf9180896 (stale) Mirror 15: 0xf9180896 (stale) Mirror 16: 0xf9180896 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x79e928ed Mirror 17: 0x79e928ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000876 1775867457.240374 mapread 0x03fe7d thru 0x03ffff (0x183 bytes) 000877 1775867457.255630 trunc from 0x040000 to 0x0136ad (0x2c952 bytes) 000879 1775867457.379574 mapwrite 0x023d8a thru 0x030f8a (0xd201 bytes) 000880 1775867457.470345 trunc from 0x030f8b to 0x01098e (0x205fc bytes) 000882 1775867457.489898 read 0x007021 thru 0x01098e (0x996e bytes) 000883 1775867457.501227 trunc from 0x01098f to 0x018e3b (0x84ad bytes) 000887 1775867457.527406 write 0x028786 thru 0x038740 (0xffbb bytes) 000888 1775867457.532585 read 0x004cd8 thru 0x011470 (0xc799 bytes) 000890 1775867457.546498 read 0x00aad2 thru 0x0154e4 (0xaa13 bytes) 000892 1775867457.556368 MIRROR_RESYNC 5 mirrors 000893 1775867458.825497 MIRROR_RESYNC 5 mirrors 000895 1775867459.288279 trunc from 0x038741 to 0x0095c6 (0x2f17a bytes) 000897 1775867459.404799 mapread 0x008147 thru 0x0095c6 (0x1480 bytes) 000898 1775867459.423510 write 0x023b2c thru 0x028141 (0x4616 bytes) 000900 1775867459.431629 write 0x016bf8 thru 0x02601b (0xf424 bytes) 000902 1775867459.448367 trunc from 0x028142 to 0x013fad (0x14194 bytes) 000904 1775867459.499552 mapwrite 0x0292c9 thru 0x036aac (0xd7e4 bytes) 000905 1775867459.661868 trunc from 0x036aad to 0x00482b (0x32281 bytes) 000907 1775867459.681372 write 0x00bebb thru 0x010e3e (0x4f84 bytes) 000909 1775867459.684967 trunc from 0x010e3f to 0x036bd2 (0x25d94 bytes) 000910 1775867459.705757 read 0x006160 thru 0x00e747 (0x85e8 bytes) 000911 1775867459.715676 write 0x00117c thru 0x00bee6 (0xad6b bytes) 000912 1775867459.727289 trunc from 0x036bd3 to 0x030833 (0x639f bytes) 000913 1775867459.754528 mapread 0x00d68e thru 0x01503d (0x79b0 bytes) 000915 1775867459.771866 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30834): Mirror 11: 0x856f2f18 (stale) Mirror 15: 0x856f2f18 (stale) Mirror 16: 0x856f2f18 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30834): Mirror 11: 0x856f2f18 (stale) Mirror 15: 0x856f2f18 (stale) Mirror 16: 0x856f2f18 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30834: skipped 000916 1775867460.138618 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30834): Mirror 15: 0x856f2f18 (stale) Mirror 16: 0x856f2f18 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30834): Mirror 15: 0x856f2f18 (stale) Mirror 16: 0x856f2f18 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30834: skipped 000919 1775867460.594080 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x30834): Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30834): Mirror 15: 0x856f2f18 (stale) Mirror 16: 0x856f2f18 (stale) CRC-32 checksum value for chunk [0, 0x30834): Mirror 1: 0x942ec6af Mirror 17: 0x942ec6af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30834: skipped 000920 1775867461.694975 mapread 0x014e82 thru 0x01e8fb (0x9a7a bytes) 000921 1775867461.758875 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30834): Mirror 16: 0x856f2f18 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30834): Mirror 16: 0x856f2f18 (stale) CRC-32 checksum value for chunk [0, 0x30834): Mirror 1: 0x942ec6af Mirror 17: 0x942ec6af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30834: skipped 000922 1775867462.442718 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) 000923 1775867462.814539 trunc from 0x030834 to 0x02385b (0xcfd8 bytes) 000924 1775867462.994608 write 0x035e0e thru 0x03ffff (0xa1f2 bytes) 000925 1775867462.996166 mapwrite 0x02f446 thru 0x0318da (0x2495 bytes) 000928 1775867463.162865 mapwrite 0x0235bc thru 0x024375 (0xdba bytes) 000930 1775867463.285537 mapwrite 0x00523a thru 0x00d421 (0x81e8 bytes) 000931 1775867463.427861 mapread 0x011594 thru 0x01c231 (0xac9e bytes) 000934 1775867463.464675 read 0x035c07 thru 0x03ffff (0xa3f9 bytes) 000938 1775867463.469946 trunc from 0x040000 to 0x00dcaf (0x32350 bytes) 000941 1775867463.526232 write 0x0375a4 thru 0x03ffff (0x8a5c bytes) 000942 1775867463.542143 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) 000944 1775867463.730819 write 0x025890 thru 0x031280 (0xb9f1 bytes) 000946 1775867463.746555 trunc from 0x040000 to 0x01f5d8 (0x20a27 bytes) 000949 1775867463.772507 read 0x0037ce thru 0x00d59d (0x9dd0 bytes) 000950 1775867463.783486 mapread 0x0015cd thru 0x005ad1 (0x4505 bytes) 000951 1775867463.806843 MIRROR_EXTEND to 2 mirrors 000954 1775867464.527134 MIRROR_EXTEND to 3 mirrors 000955 1775867465.302358 mapwrite 0x0022e6 thru 0x00f508 (0xd223 bytes) 000956 1775867465.430444 mapread 0x01a70c thru 0x01eb45 (0x443a bytes) 000959 1775867465.438056 mapwrite 0x03a992 thru 0x03ffff (0x566e bytes) 000966 1775867465.550083 trunc from 0x040000 to 0x027fa9 (0x18056 bytes) 000967 1775867465.573818 read 0x01083a thru 0x0157ed (0x4fb4 bytes) 000969 1775867465.581775 mapread 0x002b0b thru 0x00340e (0x904 bytes) 000970 1775867465.582821 mapread 0x0153eb thru 0x024a85 (0xf69b bytes) 000974 1775867465.596071 mapread 0x00d688 thru 0x013e9f (0x6818 bytes) 000976 1775867465.598518 mapread 0x006026 thru 0x008b44 (0x2b1f bytes) 000977 1775867465.604351 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x160:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27faa): Mirror 3: 0xee8b9eb2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27faa): Mirror 3: 0xee8b9eb2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27faa: skipped 000978 1775867465.877388 write 0x03896c thru 0x03ffff (0x7694 bytes) 000981 1775867465.882391 mapread 0x003935 thru 0x004782 (0xe4e bytes) 000982 1775867465.902041 mapread 0x030096 thru 0x03f0ea (0xf055 bytes) 000983 1775867465.914383 mapwrite 0x000012 thru 0x009b63 (0x9b52 bytes) 000986 1775867466.043951 trunc from 0x040000 to 0x003e16 (0x3c1e9 bytes) 000987 1775867466.076633 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e17): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e17): Mirror 3: 0xb26403a (stale) CRC-32 checksum value for chunk [0, 0x3e17): Mirror 1: 0x39a923bd Mirror 4: 0x39a923bd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e17: skipped 000988 1775867466.778899 mapread 0x00091e thru 0x003e16 (0x34f9 bytes) 000989 1775867466.806193 mapread 0x0002ad thru 0x0036a7 (0x33fb bytes) 000990 1775867466.807292 mapwrite 0x032521 thru 0x03a661 (0x8141 bytes) 000992 1775867466.967603 mapread 0x022cb4 thru 0x023d12 (0x105f bytes) 000997 1775867466.985546 MIRROR_RESYNC 3 mirrors 000999 1775867467.689983 write 0x0260d3 thru 0x032086 (0xbfb4 bytes) All operations completed A-OK! PASS 70a (131s) == sanity-flr test 71: check mirror extend parent fid ==== 20:31:12 (1775867472) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 20:31:15 (1775867475) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x205:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x221:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.274112 s, 7.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x205:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x206:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x221:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213346 s, 4.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x205:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x206:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x222:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x221:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x205:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x206:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x222:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x221:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x207:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 20:31:24 (1775867484) Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre2 Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5343616 .. Extending file size to 6631072 .. Extending file size to 7801312 .. Extending file size to 5627680 .. Extending file size to 5943808 .. Extending file size to 8402976 .. Extending file size to 8453472 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9141024 .. Extending file size to 8755712 .. Extending file size to 9217568 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 55937 55938 55939 55942 55944 done 192.168.201.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg153-client.virtnet /mnt/lustre2 (opts:) 192.168.201.153@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg153-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (94s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:32:57 (1775867577) Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre2 Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 1952864 .. Extending file size to 7772192 .. Extending file size to 4282048 .. Extending file size to 8323776 .. Extending file size to 8376416 .. Extending file size to 7792704 .. Extending file size to 8982144 .. Extending file size to 8041504 .. Extending file size to 8799904 .. Extending file size to 8976384 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9027040 .. Waiting 57369 57370 57373 57374 57377 done 192.168.201.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg153-client.virtnet /mnt/lustre2 (opts:) 192.168.201.153@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg153-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:34:18 (1775867658) Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.112397 s, 18.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.420099 s, 5.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x2e168372 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg153-client.virtnet /mnt/lustre2 (opts:) PASS 200c (22s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:34:43 (1775867683) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.173636 s, 12.1 MB/s PASS 202 (12s) == sanity-flr test 203: mirror file preserve mirror ID === 20:34:54 (1775867694) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bb:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x238:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x238:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x239:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x21c:0x0] } PASS 203 (10s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:35:04 (1775867704) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.320189 s, 6.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x23a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x21d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x21e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:35:14 (1775867714) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.373815 s, 5.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x23b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x23c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x21f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x23b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x23c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x21f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x220:0x0] } PASS 204b (12s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:35:26 (1775867726) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.328153 s, 6.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x23d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x221:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x23d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x221:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x222:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:36:00 (1775867760) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg153-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274338 s, 3.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x23e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x224:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x223:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x23e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x224:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x223:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x23f:0x0] } PASS 204d (64s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:37:04 (1775867824) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg153-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x240:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x241:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.304774 s, 6.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x240:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x241:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x225:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x240:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x241:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x225:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (62s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:38:06 (1775867886) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg153-server mds-ost sync done. Creating new pool test_204f oleg153-server: Pool lustre.test_204f created Adding targets to pool oleg153-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg153-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.296364 s, 7.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x242:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x226:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x242:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x226:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg153-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg153-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg153-server: Pool lustre.test_204f destroyed PASS 204f (84s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:39:30 (1775867970) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:39:37 (1775867977) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:39:46 (1775867986) oleg153-server: Pool lustre.test_206 created Adding targets to pool oleg153-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg153-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg153-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg153-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg153-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:40:12 (1775868012) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874371 s, 12.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401638 s, 26.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x8df4724a Mirror 3: 0x8df4724a PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:40:26 (1775868026) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:40:30 (1775868030) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:40:35 (1775868035) Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.21619 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1e90000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xc10000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xc10000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xc10000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1040000 Read this (4 bytes): BBBB 192.168.201.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg153-client.virtnet /mnt/lustre2 (opts:) PASS 209a (50s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:41:25 (1775868085) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0114951 s, 713 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:41:33 (1775868093) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072259 s, 14.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1e0 lmm_fid: [0x200000404:0x1e0:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1e0 lmm_fid: [0x200000404:0x1e0:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x230:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834400 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-10 20:41:35.000000000 -0400 Modify: 2026-04-10 20:41:35.000000000 -0400 Change: 2026-04-10 20:41:37.000000000 -0400 Birth: 2026-04-10 20:41:35.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:41:43 (1775868103) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:41:47 (1775868107) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.719242 s, 56.9 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0546753 s, 74.9 kB/s size after second write -rw-r--r-- 1 root root 4096 Apr 10 20:41 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e2:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Apr 10 20:41 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:41:57 (1775868117) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:42:05 (1775868125) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1825 sec =========== 20:42:15 (1775868135) === sanity-flr: start cleanup 20:42:16 (1775868136) === === sanity-flr: finish cleanup 20:42:23 (1775868143) ===