-----============= acceptance-small: sanity-flr ============----- Tue Aug 11 16:31:29 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg148-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 16:31:49 (1786480309) === oleg148-client.virtnet: executing check_config_client /mnt/lustre oleg148-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg148-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8cb3c8603800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8cb3c8603800.idle_timeout=debug disable quota as required oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 16:32:28 (1786480348) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7624] == sanity-flr test 0a: lfs mirror create with -N option == 16:32:30 (1786480350) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 16:32:48 (1786480368) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 16:32:52 (1786480372) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 16:32:56 (1786480376) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 16:33:24 (1786480404) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 16:33:29 (1786480409) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 16:33:32 (1786480412) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8cb3c8603800.stats=clear osc.lustre-OST0001-osc-ffff8cb3c8603800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 16:33:40 (1786480420) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 16:33:51 (1786480431) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 16:34:01 (1786480441) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 16:34:04 (1786480444) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 16:34:12 (1786480452) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 16:34:23 (1786480463) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 16:34:29 (1786480469) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 16:34:33 (1786480473) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 16:34:39 (1786480479) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 16:34:49 (1786480489) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 16:34:55 (1786480495) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 2.02944 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133711 s, 7.8 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 16:35:06 (1786480506) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.373151 s, 36.5 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 16:35:14 (1786480514) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 16:35:23 (1786480523) Stopping /mnt/lustre-ost1 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8cb3c8603800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8cb3c8603800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg148-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8cb3c8603800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8cb3c8603800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg148-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (69s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 16:36:32 (1786480592) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.294552 s, 21.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8cb3c8603800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8cb3c8603800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg148-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (35s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 16:37:07 (1786480627) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8cb3c8603800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8cb3c8603800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg148-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8cb3c8603800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8cb3c8603800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg148-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (74s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 16:38:21 (1786480701) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.4045 s, 11.9 MB/s -rw-r--r-- 1 root root 52428800 Aug 11 16:38 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.148s user 0m0.001s sys 0m0.179s Stopping /mnt/lustre-ost1 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8cb3c8603800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8cb3c8603800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.885s user 0m0.000s sys 0m0.191s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg148-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8cb3c8603800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8cb3c8603800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.283s user 0m0.001s sys 0m0.157s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg148-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg148-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8cb3c8603800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8cb3c8603800.ost_server_uuid in FULL state after 0 sec PASS 33b (77s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 16:39:38 (1786480778) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 16:39:41 (1786480781) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 16:39:44 (1786480784) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 16:39:47 (1786480787) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 16:39:53 (1786480793) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.538898 s, 23.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 16:40:04 (1786480804) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391624 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586 s, 17.9 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 16:40:11 (1786480811) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 16:40:18 (1786480818) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0017745 s, 577 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00382447 s, 268 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0011697 s, 875 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0017755 s, 577 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 16:40:28 (1786480828) create test files with size 15752150 .. f8cb93ca9f96dbf81254e999ed09dcdc - b82e5cb2e93bbbfe7c7ad68980b0dfd0 - 2a55b58b2fbc8c94560200de3f39b1d0 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre PASS 37 (21s) == sanity-flr test 38: resync ============================ 16:40:48 (1786480848) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (11s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 16:40:59 (1786480859) PASS 39 (5s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 16:41:04 (1786480864) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131773 s, 8.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271236 s, 38.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xda:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 16:41:12 (1786480872) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.173346 s, 12.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.344966 s, 12.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.152429 s, 13.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 16:41:26 (1786480886) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 16:41:28 (1786480888) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 16:41:31 (1786480891) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374995 s, 28.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 16:41:38 (1786480898) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 16:41:41 (1786480901) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xbf:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe1:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe1:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (6s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 16:41:48 (1786480908) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.23131 s, 45.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 16:41:58 (1786480918) llite.lustre-ffff8cb3c639c000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.310116 s, 33.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8cb3c7475000.xattr_cache=1 PASS 44e (11s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 16:42:09 (1786480929) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (30s) == sanity-flr test 46: Verify setstripe --copy option ==== 16:42:39 (1786480959) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (34s) == sanity-flr test 47: Verify mirror obj alloc =========== 16:43:14 (1786480994) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 16:43:16 (1786480996) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.241019 s, 13.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.199428 s, 15.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x123f95b (nosync,stale) Mirror 2: 0x123f95b (nosync,stale) ** verify mirror 2 doesn't change original checksum: 95694d625c14c2b938729faa4ddcd3e8 - mirror 1 checksum: 2286d82a33145bfd1b8e3ab6892890dd - mirror 2 checksum: 95694d625c14c2b938729faa4ddcd3e8 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-11T16:43:20-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-11T16:43:20-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd6:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 95694d625c14c2b938729faa4ddcd3e8 - mirror 1 checksum: 2286d82a33145bfd1b8e3ab6892890dd - mirror 2 checksum: 2286d82a33145bfd1b8e3ab6892890dd - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-11T16:43:20-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-11T16:43:20-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd6:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 16:43:28 (1786481008) Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 16:43:34 (1786481014) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 16:43:56 (1786481036) targets are mounted 16:43:56 (1786481036) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030254 s, 34.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } PASS 50A (42s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 16:44:10 (1786481050) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 16:44:13 (1786481053) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 16:44:16 (1786481056) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00512603 s, 3.2 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 16:44:23 (1786481063) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 16:44:26 (1786481066) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.013679 s, 299 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.082365 s, 49.7 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 16:44:33 (1786481073) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (19s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 16:44:53 (1786481093) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 16:45:19 (1786481119) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 16:45:41 (1786481141) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (21s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 16:46:03 (1786481163) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (5s) == sanity-flr test 70: mirror create and split race ====== 16:46:07 (1786481167) mirror create pid 54281 mirror split pid 54282 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7817: 54282 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54281 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 16:47:16 (1786481236) Seed set to 1239 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000003 1786481240.074211 write 0x026a2b thru 0x02efdf (0x85b5 bytes) 000004 1786481240.082882 mapread 0x014c73 thru 0x01a062 (0x53f0 bytes) truncating to largest ever: 0x2bd02 000005 1786481240.100196 trunc from 0x02efe0 to 0x02bd01 (0x32de bytes) 000007 1786481240.121198 MIRROR_EXTEND to 2 mirrors 000009 1786481241.565642 mapwrite 0x0395b9 thru 0x03ef95 (0x59dd bytes) 000011 1786481241.861627 read 0x038f9c thru 0x03ad86 (0x1deb bytes) 000012 1786481241.863028 mapread 0x014526 thru 0x021ed9 (0xd9b4 bytes) 000014 1786481241.903512 mapread 0x02c022 thru 0x036982 (0xa961 bytes) 000015 1786481241.940666 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ef96): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ef96): Mirror 2: 0xc2398113 (stale) CRC-32 checksum value for chunk [0, 0x3ef96): Mirror 1: 0x85857a58 Mirror 3: 0x85857a58 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ef96: skipped truncating to largest ever: 0x326dc 000018 1786481243.986533 trunc from 0x03ef96 to 0x0326db (0xc8ba bytes) 000019 1786481244.104501 mapwrite 0x003292 thru 0x0119f4 (0xe763 bytes) 000022 1786481244.281474 mapread 0x01a04a thru 0x01a65b (0x612 bytes) 000023 1786481244.333496 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x326dc): Mirror 3: 0x255dc3de (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x326dc): Mirror 3: 0x255dc3de (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x326dc: skipped 000024 1786481244.901466 mapread 0x000e27 thru 0x00fcbb (0xee95 bytes) truncating to largest ever: 0x396e0 000025 1786481244.981416 trunc from 0x0326dc to 0x0396df (0x7004 bytes) 000027 1786481245.014288 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x396e0): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x396e0): Mirror 3: 0x8fc7cfa (stale) CRC-32 checksum value for chunk [0, 0x396e0): Mirror 1: 0xfb7ece01 Mirror 4: 0xfb7ece01 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x396e0: skipped 000029 1786481246.761423 MIRROR_RESYNC 3 mirrors 000030 1786481248.050054 MIRROR_RESYNC 3 mirrors 000031 1786481248.585099 read 0x01459a thru 0x0177dc (0x3243 bytes) 000032 1786481248.596172 MIRROR_EXTEND to 4 mirrors 000033 1786481249.826034 write 0x02ad5b thru 0x02efac (0x4252 bytes) 000035 1786481250.074279 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x396e0): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x396e0): Mirror 3: 0xfb7ece01 (stale) Mirror 4: 0xfb7ece01 (stale) Mirror 5: 0xfb7ece01 (stale) CRC-32 checksum value for chunk [0, 0x396e0): Mirror 1: 0x6ba47459 Mirror 6: 0x6ba47459 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x396e0: skipped 000036 1786481252.403197 mapread 0x005ec5 thru 0x00bb7c (0x5cb8 bytes) 000037 1786481252.520000 mapwrite 0x017108 thru 0x01f27c (0x8175 bytes) 000039 1786481252.961660 read 0x002440 thru 0x00758b (0x514c bytes) 000041 1786481252.980178 read 0x01303f thru 0x01513c (0x20fe bytes) 000042 1786481252.982874 mapread 0x0013d7 thru 0x00ab3f (0x9769 bytes) 000043 1786481253.097309 mapwrite 0x00cde3 thru 0x016ee5 (0xa103 bytes) 000044 1786481253.258371 trunc from 0x0396e0 to 0x029a27 (0xfcb8 bytes) 000046 1786481253.333855 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29a28): Mirror 3: 0x29744e2d (stale) Mirror 4: 0x29744e2d (stale) Mirror 6: 0x29744e2d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29a28): Mirror 3: 0x29744e2d (stale) Mirror 4: 0x29744e2d (stale) Mirror 6: 0x29744e2d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29a28: skipped 000047 1786481254.057750 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x29a28): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29a28): Mirror 3: 0x29744e2d (stale) Mirror 4: 0x29744e2d (stale) Mirror 6: 0x29744e2d (stale) CRC-32 checksum value for chunk [0, 0x29a28): Mirror 1: 0x5b602712 Mirror 7: 0x5b602712 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29a28: skipped 000048 1786481255.368266 trunc from 0x029a28 to 0x020a4a (0x8fdd bytes) 000050 1786481255.460704 trunc from 0x020a4b to 0x02468b (0x3c41 bytes) 000051 1786481255.486081 read 0x009aaa thru 0x01567a (0xbbd1 bytes) 000052 1786481255.496370 read 0x011a65 thru 0x0183b0 (0x694c bytes) 000057 1786481255.509697 mapread 0x0100a8 thru 0x01749d (0x73f6 bytes) 000058 1786481255.511174 mapwrite 0x003f7b thru 0x007716 (0x379c bytes) 000060 1786481255.574870 MIRROR_RESYNC 5 mirrors 000061 1786481257.182487 read 0x00a067 thru 0x015a88 (0xba22 bytes) 000062 1786481257.240699 mapread 0x022e60 thru 0x023464 (0x605 bytes) 000064 1786481257.278704 mapwrite 0x00a3f3 thru 0x00e3e8 (0x3ff6 bytes) 000065 1786481257.513258 mapwrite 0x00cbf9 thru 0x01383c (0x6c44 bytes) 000066 1786481257.597139 mapwrite 0x03c7f3 thru 0x03ffff (0x380d bytes) 000067 1786481257.683326 trunc from 0x040000 to 0x002566 (0x3da99 bytes) 000070 1786481257.706463 trunc from 0x002567 to 0x01d6f7 (0x1b191 bytes) 000071 1786481257.729906 trunc from 0x01d6f8 to 0x0051db (0x1851c bytes) 000074 1786481257.757937 mapwrite 0x00b33e thru 0x00f20d (0x3ed0 bytes) 000076 1786481257.835349 trunc from 0x00f20e to 0x01c1cd (0xcfc0 bytes) 000078 1786481257.865115 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1c1ce): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c1ce): Mirror 3: 0x690a51ca (stale) Mirror 4: 0x690a51ca (stale) Mirror 6: 0x690a51ca (stale) Mirror 7: 0x690a51ca (stale) CRC-32 checksum value for chunk [0, 0x1c1ce): Mirror 1: 0xbf92e714 Mirror 8: 0xbf92e714 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c1ce: skipped 000080 1786481259.015194 MIRROR_RESYNC 6 mirrors 000081 1786481260.669434 mapwrite 0x03575e thru 0x0372c1 (0x1b64 bytes) 000083 1786481260.874269 MIRROR_RESYNC 6 mirrors 000084 1786481262.881365 write 0x033d17 thru 0x03e01b (0xa305 bytes) 000086 1786481262.945732 write 0x014d74 thru 0x015635 (0x8c2 bytes) 000087 1786481262.956204 trunc from 0x03e01c to 0x003a7f (0x3a59c bytes) 000088 1786481262.982195 write 0x02de6a thru 0x039b6f (0xbd06 bytes) 000089 1786481262.992728 mapwrite 0x03c783 thru 0x03ffff (0x387d bytes) 000090 1786481263.125709 mapread 0x019e43 thru 0x0208f3 (0x6ab1 bytes) 000091 1786481263.145374 write 0x02dfff thru 0x034711 (0x6713 bytes) 000092 1786481263.152415 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x44b7a53c (stale) Mirror 4: 0x44b7a53c (stale) Mirror 6: 0x44b7a53c (stale) Mirror 7: 0x44b7a53c (stale) Mirror 8: 0x44b7a53c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9f3af030 Mirror 9: 0x9f3af030 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000093 1786481264.299984 mapread 0x03f4b8 thru 0x03ffff (0xb48 bytes) 000094 1786481264.368692 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x44b7a53c (stale) Mirror 4: 0x44b7a53c (stale) Mirror 6: 0x44b7a53c (stale) Mirror 7: 0x44b7a53c (stale) Mirror 8: 0x44b7a53c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9f3af030 Mirror 9: 0x9f3af030 Mirror 10: 0x9f3af030 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000095 1786481265.446785 trunc from 0x040000 to 0x0071bf (0x38e40 bytes) 000096 1786481265.579126 mapread 0x00142b thru 0x0071bf (0x5d95 bytes) 000097 1786481265.612792 mapwrite 0x03fa3c thru 0x03ffff (0x5c4 bytes) 000098 1786481265.751369 mapread 0x03bb08 thru 0x03f00f (0x3508 bytes) 000100 1786481265.754176 mapwrite 0x002728 thru 0x005f22 (0x37fb bytes) 000101 1786481265.821001 write 0x02d703 thru 0x03901f (0xb91d bytes) 000102 1786481265.828906 mapread 0x027649 thru 0x0329ff (0xb3b7 bytes) 000103 1786481265.840719 trunc from 0x040000 to 0x002b77 (0x3d488 bytes) 000105 1786481265.863641 write 0x00acc0 thru 0x017b8e (0xcecf bytes) 000107 1786481265.871997 trunc from 0x017b8f to 0x01f846 (0x7cb8 bytes) 000108 1786481265.901888 trunc from 0x01f847 to 0x00b247 (0x145ff bytes) 000109 1786481265.923421 write 0x03fa34 thru 0x03ffff (0x5cc bytes) 000110 1786481265.925965 trunc from 0x040000 to 0x0055d2 (0x3aa2d bytes) 000112 1786481265.950931 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x55d3): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x55d3): Mirror 3: 0xde2ba889 (stale) Mirror 4: 0xde2ba889 (stale) Mirror 6: 0xde2ba889 (stale) Mirror 7: 0xde2ba889 (stale) Mirror 8: 0xde2ba889 (stale) Mirror 9: 0xde2ba889 (stale) Mirror 10: 0xde2ba889 (stale) CRC-32 checksum value for chunk [0, 0x55d3): Mirror 1: 0x21f1aa1 Mirror 11: 0x21f1aa1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x55d3: skipped 000114 1786481266.874959 MIRROR_RESYNC 9 mirrors 000115 1786481268.108327 write 0x02ce1e thru 0x031898 (0x4a7b bytes) 000117 1786481268.191075 write 0x02ecf7 thru 0x032676 (0x3980 bytes) 000120 1786481268.198173 mapread 0x00b232 thru 0x00fc09 (0x49d8 bytes) 000121 1786481268.216057 mapread 0x01cecb thru 0x023b04 (0x6c3a bytes) 000123 1786481268.229044 mapwrite 0x024c00 thru 0x026959 (0x1d5a bytes) 000124 1786481268.291290 write 0x0136c0 thru 0x01b56e (0x7eaf bytes) 000125 1786481268.293162 mapread 0x02ea7c thru 0x032676 (0x3bfb bytes) 000127 1786481268.295930 write 0x011ff9 thru 0x01ef80 (0xcf88 bytes) 000128 1786481268.298797 read 0x00a61a thru 0x00aa9f (0x486 bytes) 000131 1786481268.301950 read 0x02fadd thru 0x032676 (0x2b9a bytes) 000133 1786481268.303974 write 0x01a101 thru 0x02067d (0x657d bytes) 000134 1786481268.306097 write 0x0014fa thru 0x003953 (0x245a bytes) 000137 1786481268.323486 mapread 0x001971 thru 0x006be4 (0x5274 bytes) 000138 1786481268.326157 mapread 0x02e7cb thru 0x02fd22 (0x1558 bytes) 000139 1786481268.332506 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x32677): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32677): Mirror 3: 0x12b005be (stale) Mirror 4: 0x12b005be (stale) Mirror 6: 0x12b005be (stale) Mirror 7: 0x12b005be (stale) Mirror 8: 0x12b005be (stale) Mirror 9: 0x12b005be (stale) Mirror 10: 0x12b005be (stale) Mirror 11: 0x12b005be (stale) CRC-32 checksum value for chunk [0, 0x32677): Mirror 1: 0xcbf474c5 Mirror 12: 0xcbf474c5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32677: skipped 000140 1786481269.598959 trunc from 0x032677 to 0x01c102 (0x16574 bytes) 000143 1786481269.801754 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c103): Mirror 3: 0xeb58d9ae (stale) Mirror 4: 0xeb58d9ae (stale) Mirror 6: 0xeb58d9ae (stale) Mirror 8: 0xeb58d9ae (stale) Mirror 9: 0xeb58d9ae (stale) Mirror 10: 0xeb58d9ae (stale) Mirror 11: 0xeb58d9ae (stale) Mirror 12: 0x87eeec6e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c103): Mirror 3: 0xeb58d9ae (stale) Mirror 4: 0xeb58d9ae (stale) Mirror 6: 0xeb58d9ae (stale) Mirror 8: 0xeb58d9ae (stale) Mirror 9: 0xeb58d9ae (stale) Mirror 10: 0xeb58d9ae (stale) Mirror 11: 0xeb58d9ae (stale) Mirror 12: 0x87eeec6e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c103: skipped 000146 1786481270.359057 trunc from 0x01c103 to 0x0297cc (0xd6ca bytes) 000147 1786481270.377329 trunc from 0x0297cd to 0x002b35 (0x26c97 bytes) 000148 1786481270.391944 mapread 0x001fdc thru 0x002b35 (0xb5a bytes) 000149 1786481270.405695 trunc from 0x002b36 to 0x01b735 (0x18c00 bytes) truncating to largest ever: 0x3be8d 000150 1786481270.425207 trunc from 0x01b736 to 0x03be8c (0x20757 bytes) 000151 1786481270.453119 write 0x024f7b thru 0x02c113 (0x7199 bytes) 000152 1786481270.465823 read 0x00cd3c thru 0x01a6b0 (0xd975 bytes) 000153 1786481270.487793 MIRROR_RESYNC 9 mirrors 000154 1786481272.991047 mapwrite 0x00599f thru 0x0066d4 (0xd36 bytes) 000155 1786481273.188588 trunc from 0x03be8d to 0x001dfc (0x3a090 bytes) 000156 1786481273.219991 write 0x006c15 thru 0x00e459 (0x7845 bytes) 000157 1786481273.230343 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0xe45a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe45a): Mirror 3: 0x289f7ff1 (stale) Mirror 4: 0x289f7ff1 (stale) Mirror 6: 0x289f7ff1 (stale) Mirror 8: 0x289f7ff1 (stale) Mirror 9: 0x289f7ff1 (stale) Mirror 10: 0x289f7ff1 (stale) Mirror 11: 0x289f7ff1 (stale) Mirror 12: 0x289f7ff1 (stale) CRC-32 checksum value for chunk [0, 0xe45a): Mirror 1: 0x8c3a0fab Mirror 13: 0x8c3a0fab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe45a: skipped 000158 1786481274.687122 mapwrite 0x009b6e thru 0x00ea90 (0x4f23 bytes) 000160 1786481274.851085 read 0x003b00 thru 0x00ea90 (0xaf91 bytes) 000162 1786481274.865187 mapwrite 0x0278eb thru 0x029eb9 (0x25cf bytes) 000163 1786481274.974928 read 0x026b84 thru 0x029eb9 (0x3336 bytes) 000166 1786481274.978254 mapwrite 0x009b3b thru 0x00b501 (0x19c7 bytes) 000170 1786481275.024980 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29eba): Mirror 3: 0xfb0df17a (stale) Mirror 4: 0xfb0df17a (stale) Mirror 6: 0xfb0df17a (stale) Mirror 9: 0xfb0df17a (stale) Mirror 10: 0xfb0df17a (stale) Mirror 11: 0xfb0df17a (stale) Mirror 12: 0xfb0df17a (stale) Mirror 13: 0x7e2f7a2f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29eba): Mirror 3: 0xfb0df17a (stale) Mirror 4: 0xfb0df17a (stale) Mirror 6: 0xfb0df17a (stale) Mirror 9: 0xfb0df17a (stale) Mirror 10: 0xfb0df17a (stale) Mirror 11: 0xfb0df17a (stale) Mirror 12: 0xfb0df17a (stale) Mirror 13: 0x7e2f7a2f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29eba: skipped 000171 1786481275.508166 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29eba): Mirror 3: 0xfb0df17a (stale) Mirror 4: 0xfb0df17a (stale) Mirror 9: 0xfb0df17a (stale) Mirror 10: 0xfb0df17a (stale) Mirror 11: 0xfb0df17a (stale) Mirror 12: 0xfb0df17a (stale) Mirror 13: 0x7e2f7a2f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29eba): Mirror 3: 0xfb0df17a (stale) Mirror 4: 0xfb0df17a (stale) Mirror 9: 0xfb0df17a (stale) Mirror 10: 0xfb0df17a (stale) Mirror 11: 0xfb0df17a (stale) Mirror 12: 0xfb0df17a (stale) Mirror 13: 0x7e2f7a2f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29eba: skipped 000174 1786481276.047291 mapread 0x021e03 thru 0x029d2d (0x7f2b bytes) 000176 1786481276.081494 write 0x010fc0 thru 0x0119b1 (0x9f2 bytes) 000177 1786481276.095989 mapwrite 0x03801d thru 0x03ffff (0x7fe3 bytes) 000181 1786481276.227455 read 0x009a2b thru 0x009c4a (0x220 bytes) 000183 1786481276.256710 MIRROR_RESYNC 8 mirrors 000184 1786481278.695944 trunc from 0x040000 to 0x023689 (0x1c976 bytes) 000186 1786481278.773088 mapwrite 0x03c06a thru 0x03ffff (0x3f96 bytes) 000187 1786481278.863950 mapread 0x03e188 thru 0x03ffff (0x1e78 bytes) 000188 1786481278.865123 mapwrite 0x009774 thru 0x01166d (0x7efa bytes) 000189 1786481278.916194 read 0x0112cf thru 0x01cee8 (0xbc1a bytes) 000191 1786481278.933275 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf05277a5 (stale) Mirror 4: 0xf05277a5 (stale) Mirror 9: 0xf05277a5 (stale) Mirror 10: 0xf05277a5 (stale) Mirror 11: 0xf05277a5 (stale) Mirror 12: 0xf05277a5 (stale) Mirror 13: 0xf05277a5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2599bb52 Mirror 14: 0x2599bb52 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000192 1786481280.116105 trunc from 0x040000 to 0x026fec (0x19013 bytes) 000194 1786481280.204837 mapread 0x01f910 thru 0x023b84 (0x4275 bytes) 000195 1786481280.221883 trunc from 0x026fed to 0x01298a (0x14662 bytes) 000196 1786481280.246339 read 0x004a2e thru 0x009dc8 (0x539b bytes) 000198 1786481280.265126 MIRROR_RESYNC 9 mirrors 000199 1786481281.686781 read 0x00fbcb thru 0x01298a (0x2dc0 bytes) 000201 1786481281.693996 trunc from 0x01298b to 0x032d06 (0x2037c bytes) 000204 1786481281.766027 write 0x030cb4 thru 0x032ce4 (0x2031 bytes) 000205 1786481281.774799 trunc from 0x032d07 to 0x017661 (0x1b6a5 bytes) 000207 1786481281.801237 mapwrite 0x011d7f thru 0x01a69a (0x891c bytes) 000208 1786481281.913585 trunc from 0x01a69b to 0x02c55d (0x11ec3 bytes) 000209 1786481281.931657 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c55e): Mirror 3: 0xa787c477 (stale) Mirror 4: 0xa787c477 (stale) Mirror 10: 0xa787c477 (stale) Mirror 11: 0xa787c477 (stale) Mirror 12: 0xa787c477 (stale) Mirror 13: 0xa787c477 (stale) Mirror 14: 0xa787c477 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c55e): Mirror 3: 0xa787c477 (stale) Mirror 4: 0xa787c477 (stale) Mirror 10: 0xa787c477 (stale) Mirror 11: 0xa787c477 (stale) Mirror 12: 0xa787c477 (stale) Mirror 13: 0xa787c477 (stale) Mirror 14: 0xa787c477 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c55e: skipped 000210 1786481282.523894 mapwrite 0x038bd3 thru 0x03ffff (0x742d bytes) 000211 1786481282.685494 write 0x0322a8 thru 0x037cfa (0x5a53 bytes) 000213 1786481282.701671 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2e9b5a5b (stale) Mirror 4: 0x2e9b5a5b (stale) Mirror 10: 0x2e9b5a5b (stale) Mirror 11: 0x2e9b5a5b (stale) Mirror 12: 0x2e9b5a5b (stale) Mirror 13: 0x2e9b5a5b (stale) Mirror 14: 0x2e9b5a5b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xab7e9910 Mirror 15: 0xab7e9910 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000214 1786481284.117477 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2e9b5a5b (stale) Mirror 4: 0x2e9b5a5b (stale) Mirror 10: 0x2e9b5a5b (stale) Mirror 11: 0x2e9b5a5b (stale) Mirror 12: 0x2e9b5a5b (stale) Mirror 13: 0x2e9b5a5b (stale) Mirror 14: 0x2e9b5a5b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xab7e9910 Mirror 15: 0xab7e9910 Mirror 16: 0xab7e9910 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped truncating to largest ever: 0x3c3f9 000216 1786481285.335006 trunc from 0x040000 to 0x03c3f8 (0x3c07 bytes) 000217 1786481285.430270 mapwrite 0x01ca33 thru 0x021bc2 (0x5190 bytes) 000222 1786481285.501297 mapread 0x02ac85 thru 0x034ecd (0xa249 bytes) 000223 1786481285.524862 mapread 0x01b1fa thru 0x024e9c (0x9ca3 bytes) 000224 1786481285.526226 read 0x02c34a thru 0x02d663 (0x131a bytes) 000225 1786481285.526959 trunc from 0x03c3f9 to 0x024843 (0x17bb5 bytes) 000226 1786481285.547254 write 0x0054cf thru 0x00f781 (0xa2b3 bytes) 000227 1786481285.558340 trunc from 0x024844 to 0x02182d (0x3016 bytes) 000229 1786481285.578179 read 0x01cae1 thru 0x02182d (0x4d4d bytes) 000230 1786481285.578962 mapwrite 0x0272a4 thru 0x02e85b (0x75b8 bytes) 000231 1786481285.652607 mapwrite 0x0369e8 thru 0x03ffff (0x9618 bytes) 000233 1786481285.728843 mapread 0x039399 thru 0x03ffff (0x6c67 bytes) 000235 1786481285.737984 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2e9b5a5b (stale) Mirror 10: 0x2e9b5a5b (stale) Mirror 11: 0x2e9b5a5b (stale) Mirror 12: 0x2e9b5a5b (stale) Mirror 13: 0x2e9b5a5b (stale) Mirror 14: 0x2e9b5a5b (stale) Mirror 15: 0xab7e9910 (stale) Mirror 16: 0xab7e9910 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2e9b5a5b (stale) Mirror 10: 0x2e9b5a5b (stale) Mirror 11: 0x2e9b5a5b (stale) Mirror 12: 0x2e9b5a5b (stale) Mirror 13: 0x2e9b5a5b (stale) Mirror 14: 0x2e9b5a5b (stale) Mirror 15: 0xab7e9910 (stale) Mirror 16: 0xab7e9910 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000238 1786481286.260012 mapread 0x0015df thru 0x00a0a4 (0x8ac6 bytes) 000241 1786481286.291236 mapread 0x023489 thru 0x0317e2 (0xe35a bytes) 000242 1786481286.319045 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2e9b5a5b (stale) Mirror 10: 0x2e9b5a5b (stale) Mirror 11: 0x2e9b5a5b (stale) Mirror 12: 0x2e9b5a5b (stale) Mirror 13: 0x2e9b5a5b (stale) Mirror 14: 0x2e9b5a5b (stale) Mirror 15: 0xab7e9910 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2e9b5a5b (stale) Mirror 10: 0x2e9b5a5b (stale) Mirror 11: 0x2e9b5a5b (stale) Mirror 12: 0x2e9b5a5b (stale) Mirror 13: 0x2e9b5a5b (stale) Mirror 14: 0x2e9b5a5b (stale) Mirror 15: 0xab7e9910 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000243 1786481286.832820 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2e9b5a5b (stale) Mirror 10: 0x2e9b5a5b (stale) Mirror 11: 0x2e9b5a5b (stale) Mirror 12: 0x2e9b5a5b (stale) Mirror 13: 0x2e9b5a5b (stale) Mirror 14: 0x2e9b5a5b (stale) Mirror 15: 0xab7e9910 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcefb3a6b Mirror 16: 0xcefb3a6b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000244 1786481287.873325 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2e9b5a5b (stale) Mirror 10: 0x2e9b5a5b (stale) Mirror 11: 0x2e9b5a5b (stale) Mirror 12: 0x2e9b5a5b (stale) Mirror 13: 0x2e9b5a5b (stale) Mirror 14: 0x2e9b5a5b (stale) Mirror 15: 0xab7e9910 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcefb3a6b Mirror 16: 0xcefb3a6b Mirror 17: 0xcefb3a6b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000246 1786481289.059938 MIRROR_RESYNC 10 mirrors 000248 1786481290.448228 trunc from 0x040000 to 0x02fcc6 (0x10339 bytes) 000249 1786481290.588686 trunc from 0x02fcc7 to 0x036c0e (0x6f48 bytes) 000250 1786481290.642736 mapread 0x029634 thru 0x036c0e (0xd5db bytes) 000254 1786481290.698475 trunc from 0x036c0f to 0x007ff7 (0x2ec17 bytes) 000256 1786481290.761262 MIRROR_RESYNC 10 mirrors 000258 1786481292.336637 mapwrite 0x030e05 thru 0x033710 (0x290c bytes) 000264 1786481292.640279 MIRROR_RESYNC 10 mirrors 000266 1786481295.306151 mapread 0x00e8e8 thru 0x016c02 (0x831b bytes) 000268 1786481295.326516 read 0x006c82 thru 0x00bacf (0x4e4e bytes) 000269 1786481295.337644 MIRROR_EXTEND to 11 mirrors 000270 1786481296.405073 mapread 0x01ed71 thru 0x026123 (0x73b3 bytes) 000271 1786481296.464229 write 0x0371ec thru 0x03ffff (0x8e14 bytes) 000272 1786481296.580270 mapread 0x0183dd thru 0x021c5d (0x9881 bytes) 000274 1786481296.598498 read 0x01f486 thru 0x023e79 (0x49f4 bytes) 000275 1786481296.605317 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x789f3cea (stale) Mirror 10: 0x789f3cea (stale) Mirror 11: 0x789f3cea (stale) Mirror 12: 0x789f3cea (stale) Mirror 13: 0x789f3cea (stale) Mirror 14: 0x789f3cea (stale) Mirror 15: 0x789f3cea (stale) Mirror 16: 0x789f3cea (stale) Mirror 18: 0x789f3cea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x789f3cea (stale) Mirror 10: 0x789f3cea (stale) Mirror 11: 0x789f3cea (stale) Mirror 12: 0x789f3cea (stale) Mirror 13: 0x789f3cea (stale) Mirror 14: 0x789f3cea (stale) Mirror 15: 0x789f3cea (stale) Mirror 16: 0x789f3cea (stale) Mirror 18: 0x789f3cea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000276 1786481297.256394 write 0x00503f thru 0x00eddc (0x9d9e bytes) 000282 1786481297.283890 write 0x020c0f thru 0x02ecad (0xe09f bytes) 000283 1786481297.304653 mapwrite 0x023715 thru 0x031d0d (0xe5f9 bytes) 000284 1786481297.518423 write 0x03f507 thru 0x03ffff (0xaf9 bytes) 000285 1786481297.530598 trunc from 0x040000 to 0x01c403 (0x23bfc bytes) 000287 1786481297.565662 trunc from 0x01c404 to 0x0363ed (0x19fea bytes) 000290 1786481297.620491 read 0x01ba22 thru 0x026e73 (0xb452 bytes) 000291 1786481297.636878 read 0x01b4b6 thru 0x02117e (0x5cc9 bytes) 000293 1786481297.647953 MIRROR_RESYNC 10 mirrors 000294 1786481299.597186 mapread 0x02d567 thru 0x033038 (0x5ad2 bytes) 000295 1786481299.620622 write 0x00a2e8 thru 0x00c9fa (0x2713 bytes) 000297 1786481299.700012 mapread 0x0133e8 thru 0x01b9c3 (0x85dc bytes) 000300 1786481299.711017 trunc from 0x0363ee to 0x02fcf0 (0x66fd bytes) 000301 1786481299.752626 trunc from 0x02fcf1 to 0x0326fe (0x2a0e bytes) 000302 1786481299.802424 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x326ff): Mirror 3: 0x3d59394c (stale) Mirror 10: 0x3d59394c (stale) Mirror 11: 0x3d59394c (stale) Mirror 12: 0x3d59394c (stale) Mirror 13: 0x3d59394c (stale) Mirror 14: 0x3d59394c (stale) Mirror 16: 0x3d59394c (stale) Mirror 18: 0x3d59394c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x326ff): Mirror 3: 0x3d59394c (stale) Mirror 10: 0x3d59394c (stale) Mirror 11: 0x3d59394c (stale) Mirror 12: 0x3d59394c (stale) Mirror 13: 0x3d59394c (stale) Mirror 14: 0x3d59394c (stale) Mirror 16: 0x3d59394c (stale) Mirror 18: 0x3d59394c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x326ff: skipped 000303 1786481300.460166 trunc from 0x0326ff to 0x02cf7b (0x5783 bytes) 000306 1786481300.499953 write 0x01e3cc thru 0x02d0e6 (0xed1b bytes) 000308 1786481300.510076 write 0x036193 thru 0x03ffff (0x9e6d bytes) 000309 1786481300.513458 read 0x0032a1 thru 0x00efd3 (0xbd33 bytes) 000310 1786481300.529620 write 0x033cb3 thru 0x03ffff (0xc34d bytes) 000311 1786481300.538800 read 0x0174de thru 0x017c6f (0x792 bytes) 000312 1786481300.551110 mapwrite 0x011fc5 thru 0x0215bb (0xf5f7 bytes) 000313 1786481300.656621 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x95354b8b (stale) Mirror 11: 0x95354b8b (stale) Mirror 12: 0x95354b8b (stale) Mirror 13: 0x95354b8b (stale) Mirror 14: 0x95354b8b (stale) Mirror 16: 0x95354b8b (stale) Mirror 18: 0x95354b8b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x95354b8b (stale) Mirror 11: 0x95354b8b (stale) Mirror 12: 0x95354b8b (stale) Mirror 13: 0x95354b8b (stale) Mirror 14: 0x95354b8b (stale) Mirror 16: 0x95354b8b (stale) Mirror 18: 0x95354b8b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000314 1786481301.206470 write 0x00800d thru 0x00fe05 (0x7df9 bytes) 000318 1786481301.222326 mapread 0x00066a thru 0x003bd0 (0x3567 bytes) 000319 1786481301.248004 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x95354b8b (stale) Mirror 11: 0x95354b8b (stale) Mirror 12: 0x95354b8b (stale) Mirror 13: 0x95354b8b (stale) Mirror 14: 0x95354b8b (stale) Mirror 16: 0x95354b8b (stale) Mirror 18: 0x95354b8b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x50d44e3e Mirror 19: 0x50d44e3e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000320 1786481302.140361 trunc from 0x040000 to 0x01e7e8 (0x21817 bytes) 000321 1786481302.285518 MIRROR_RESYNC 9 mirrors 000324 1786481304.001190 write 0x039d65 thru 0x03ffff (0x629b bytes) 000325 1786481304.072354 mapread 0x028d4a thru 0x02b8eb (0x2ba2 bytes) 000326 1786481304.094029 write 0x02c5e5 thru 0x038079 (0xba95 bytes) 000328 1786481304.102112 read 0x0378f7 thru 0x03ffff (0x8709 bytes) 000329 1786481304.103290 write 0x023b2c thru 0x024d95 (0x126a bytes) 000330 1786481304.104453 read 0x00b80a thru 0x01afb0 (0xf7a7 bytes) 000331 1786481304.126128 mapwrite 0x0005b8 thru 0x00ab5d (0xa5a6 bytes) 000332 1786481304.182365 mapwrite 0x00eb55 thru 0x01d6b7 (0xeb63 bytes) 000334 1786481304.244899 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x36b38118 (stale) Mirror 11: 0x36b38118 (stale) Mirror 12: 0x36b38118 (stale) Mirror 13: 0x36b38118 (stale) Mirror 14: 0x36b38118 (stale) Mirror 16: 0x36b38118 (stale) Mirror 18: 0x36b38118 (stale) Mirror 19: 0x36b38118 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x126ca87 Mirror 20: 0x126ca87 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000335 1786481305.263299 trunc from 0x040000 to 0x007e22 (0x381dd bytes) 000336 1786481305.408545 read 0x006319 thru 0x007e22 (0x1b0a bytes) 000337 1786481305.415282 mapwrite 0x001c0b thru 0x00f9a0 (0xdd96 bytes) 000340 1786481305.524933 mapwrite 0x0224f5 thru 0x030a0b (0xe517 bytes) 000341 1786481305.654215 MIRROR_RESYNC 10 mirrors 000343 1786481308.104980 MIRROR_RESYNC 10 mirrors 000345 1786481308.707352 read 0x02f24e thru 0x030a0b (0x17be bytes) 000346 1786481308.720442 mapwrite 0x012051 thru 0x01f187 (0xd137 bytes) 000348 1786481308.931971 trunc from 0x030a0c to 0x03518e (0x4783 bytes) 000349 1786481308.956212 mapread 0x02404a thru 0x0312fa (0xd2b1 bytes) 000351 1786481308.981237 trunc from 0x03518f to 0x00b296 (0x29ef8 bytes) 000353 1786481309.002657 trunc from 0x00b297 to 0x02de5c (0x22bc6 bytes) 000355 1786481309.023115 mapread 0x010b17 thru 0x015fbd (0x54a7 bytes) 000356 1786481309.034315 mapread 0x0188ba thru 0x021038 (0x877f bytes) 000359 1786481309.045689 trunc from 0x02de5d to 0x022726 (0xb736 bytes) 000361 1786481309.070222 read 0x01028e thru 0x017cbb (0x7a2e bytes) 000362 1786481309.071491 mapwrite 0x01dc04 thru 0x022d68 (0x5165 bytes) 000363 1786481309.154259 mapread 0x000ed4 thru 0x00df46 (0xd073 bytes) 000364 1786481309.165931 MIRROR_RESYNC 10 mirrors 000365 1786481311.685518 mapread 0x0196cf thru 0x022d68 (0x969a bytes) 000366 1786481311.705987 mapwrite 0x002335 thru 0x00635e (0x402a bytes) 000368 1786481311.913204 mapread 0x00eae8 thru 0x017c23 (0x913c bytes) 000369 1786481311.927887 mapwrite 0x032bc9 thru 0x03ffff (0xd437 bytes) 000370 1786481312.108223 mapread 0x0057a5 thru 0x00ff1a (0xa776 bytes) 000372 1786481312.114309 read 0x008f7e thru 0x00ae50 (0x1ed3 bytes) 000374 1786481312.124828 mapwrite 0x01e8bb thru 0x02b118 (0xc85e bytes) 000375 1786481312.208934 mapwrite 0x00f71d thru 0x01e218 (0xeafc bytes) 000380 1786481312.307448 write 0x018b80 thru 0x01dd10 (0x5191 bytes) 000382 1786481312.320542 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xc3a07eda (stale) Mirror 11: 0xc3a07eda (stale) Mirror 12: 0xc3a07eda (stale) Mirror 13: 0xc3a07eda (stale) Mirror 14: 0xc3a07eda (stale) Mirror 16: 0xc3a07eda (stale) Mirror 19: 0xc3a07eda (stale) Mirror 20: 0xc3a07eda (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xc3a07eda (stale) Mirror 11: 0xc3a07eda (stale) Mirror 12: 0xc3a07eda (stale) Mirror 13: 0xc3a07eda (stale) Mirror 14: 0xc3a07eda (stale) Mirror 16: 0xc3a07eda (stale) Mirror 19: 0xc3a07eda (stale) Mirror 20: 0xc3a07eda (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000383 1786481313.256079 mapread 0x0103ca thru 0x01dd61 (0xd998 bytes) 000384 1786481313.303135 trunc from 0x040000 to 0x00f9b8 (0x30647 bytes) 000385 1786481313.321015 mapwrite 0x015d72 thru 0x02555d (0xf7ec bytes) 000387 1786481313.430316 trunc from 0x02555e to 0x021ba3 (0x39ba bytes) 000388 1786481313.455492 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21ba4): Mirror 10: 0xe0d2d6b3 (stale) Mirror 11: 0xe0d2d6b3 (stale) Mirror 12: 0xe0d2d6b3 (stale) Mirror 14: 0xe0d2d6b3 (stale) Mirror 16: 0xe0d2d6b3 (stale) Mirror 19: 0xe0d2d6b3 (stale) Mirror 20: 0xe0d2d6b3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21ba4): Mirror 10: 0xe0d2d6b3 (stale) Mirror 11: 0xe0d2d6b3 (stale) Mirror 12: 0xe0d2d6b3 (stale) Mirror 14: 0xe0d2d6b3 (stale) Mirror 16: 0xe0d2d6b3 (stale) Mirror 19: 0xe0d2d6b3 (stale) Mirror 20: 0xe0d2d6b3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21ba4: skipped 000389 1786481314.024119 read 0x01dba3 thru 0x021ba3 (0x4001 bytes) 000390 1786481314.037898 read 0x02196e thru 0x021ba3 (0x236 bytes) 000391 1786481314.038915 mapwrite 0x003ce7 thru 0x0094cc (0x57e6 bytes) 000395 1786481314.115616 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21ba4): Mirror 10: 0xe0d2d6b3 (stale) Mirror 11: 0xe0d2d6b3 (stale) Mirror 14: 0xe0d2d6b3 (stale) Mirror 16: 0xe0d2d6b3 (stale) Mirror 19: 0xe0d2d6b3 (stale) Mirror 20: 0xe0d2d6b3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21ba4): Mirror 10: 0xe0d2d6b3 (stale) Mirror 11: 0xe0d2d6b3 (stale) Mirror 14: 0xe0d2d6b3 (stale) Mirror 16: 0xe0d2d6b3 (stale) Mirror 19: 0xe0d2d6b3 (stale) Mirror 20: 0xe0d2d6b3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21ba4: skipped 000397 1786481314.668536 trunc from 0x021ba4 to 0x0314ce (0xf92b bytes) 000398 1786481314.690830 mapread 0x00c8b4 thru 0x0148b5 (0x8002 bytes) 000399 1786481314.718931 mapread 0x02fa10 thru 0x0314ce (0x1abf bytes) 000402 1786481314.739192 read 0x012639 thru 0x01b6b8 (0x9080 bytes) 000408 1786481314.759756 write 0x00ec2e thru 0x01818b (0x955e bytes) 000411 1786481314.769933 write 0x01c426 thru 0x01e80f (0x23ea bytes) 000414 1786481314.789428 mapread 0x016a71 thru 0x023d62 (0xd2f2 bytes) 000415 1786481314.803541 read 0x014384 thru 0x0143c6 (0x43 bytes) 000416 1786481314.810539 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x314cf): Mirror 10: 0x2f89c634 (stale) Mirror 11: 0x2f89c634 (stale) Mirror 16: 0x2f89c634 (stale) Mirror 19: 0x2f89c634 (stale) Mirror 20: 0x2f89c634 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x314cf): Mirror 10: 0x2f89c634 (stale) Mirror 11: 0x2f89c634 (stale) Mirror 16: 0x2f89c634 (stale) Mirror 19: 0x2f89c634 (stale) Mirror 20: 0x2f89c634 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x314cf: skipped 000417 1786481315.308944 mapread 0x000b91 thru 0x0083a9 (0x7819 bytes) 000419 1786481315.332037 mapwrite 0x00d87b thru 0x011225 (0x39ab bytes) 000420 1786481315.381890 mapwrite 0x039d4f thru 0x03ffff (0x62b1 bytes) 000421 1786481315.454141 mapwrite 0x031879 thru 0x03ffff (0xe787 bytes) 000422 1786481315.507067 mapread 0x02c610 thru 0x038c29 (0xc61a bytes) 000423 1786481315.508043 write 0x02bab3 thru 0x030251 (0x479f bytes) 000425 1786481315.509497 write 0x00f263 thru 0x00f8a1 (0x63f bytes) 000427 1786481315.513192 mapwrite 0x03a7ae thru 0x03dc05 (0x3458 bytes) 000428 1786481315.575223 mapwrite 0x036dfe thru 0x03b6d9 (0x48dc bytes) 000429 1786481315.662623 write 0x00cda5 thru 0x00e369 (0x15c5 bytes) 000430 1786481315.663700 trunc from 0x040000 to 0x00010e (0x3fef1 bytes) 000433 1786481315.698636 mapwrite 0x01f04f thru 0x02ab95 (0xbb47 bytes) 000434 1786481315.848936 write 0x003650 thru 0x005148 (0x1af9 bytes) 000435 1786481315.864066 write 0x03d10a thru 0x03ffff (0x2ef6 bytes) 000437 1786481315.867761 write 0x036269 thru 0x03ffff (0x9d97 bytes) 000438 1786481315.880681 MIRROR_RESYNC 6 mirrors 000446 1786481317.833757 mapwrite 0x007cb9 thru 0x00a126 (0x246e bytes) 000448 1786481317.998093 read 0x01147b thru 0x0145c7 (0x314d bytes) 000450 1786481318.007728 write 0x03a3cf thru 0x03ffff (0x5c31 bytes) 000453 1786481318.016308 write 0x0270a4 thru 0x029078 (0x1fd5 bytes) 000455 1786481318.028524 mapwrite 0x00c626 thru 0x00f037 (0x2a12 bytes) 000457 1786481318.071364 trunc from 0x040000 to 0x008af5 (0x3750a bytes) 000460 1786481318.101961 mapread 0x004ac7 thru 0x00806f (0x35a9 bytes) 000461 1786481318.104036 write 0x01e675 thru 0x02875c (0xa0e8 bytes) 000462 1786481318.111959 MIRROR_RESYNC 6 mirrors 000464 1786481319.751505 trunc from 0x02875d to 0x02ba3d (0x32e1 bytes) 000466 1786481319.817671 trunc from 0x02ba3e to 0x023379 (0x86c4 bytes) 000467 1786481319.832659 mapread 0x019f2c thru 0x022233 (0x8308 bytes) 000468 1786481319.846914 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2337a): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2337a): Mirror 10: 0x10a63547 (stale) Mirror 11: 0x10a63547 (stale) Mirror 16: 0x10a63547 (stale) Mirror 19: 0x10a63547 (stale) Mirror 20: 0x10a63547 (stale) CRC-32 checksum value for chunk [0, 0x2337a): Mirror 1: 0x10a63547 Mirror 21: 0x10a63547 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2337a: skipped 000469 1786481320.954840 mapwrite 0x009e20 thru 0x00a267 (0x448 bytes) 000470 1786481321.220243 mapwrite 0x03ab5b thru 0x03e1fa (0x36a0 bytes) 000473 1786481321.354010 write 0x03a55c thru 0x03ffff (0x5aa4 bytes) 000478 1786481321.362959 mapread 0x00e524 thru 0x01602b (0x7b08 bytes) 000479 1786481321.378125 mapwrite 0x00e0e4 thru 0x01d715 (0xf632 bytes) 000480 1786481321.457226 read 0x007914 thru 0x0088aa (0xf97 bytes) 000482 1786481321.465396 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xb1aee63f (stale) Mirror 11: 0xb1aee63f (stale) Mirror 19: 0xb1aee63f (stale) Mirror 20: 0xb1aee63f (stale) Mirror 21: 0xf02fab21 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xb1aee63f (stale) Mirror 11: 0xb1aee63f (stale) Mirror 19: 0xb1aee63f (stale) Mirror 20: 0xb1aee63f (stale) Mirror 21: 0xf02fab21 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000484 1786481322.159829 read 0x015464 thru 0x017d2e (0x28cb bytes) 000486 1786481322.178046 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xb1aee63f (stale) Mirror 11: 0xb1aee63f (stale) Mirror 19: 0xb1aee63f (stale) Mirror 20: 0xb1aee63f (stale) Mirror 21: 0xf02fab21 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4cd060e6 Mirror 22: 0x4cd060e6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000487 1786481323.544115 mapwrite 0x00e669 thru 0x019237 (0xabcf bytes) 000488 1786481323.814181 mapwrite 0x02a11d thru 0x033bd3 (0x9ab7 bytes) 000489 1786481323.920132 trunc from 0x040000 to 0x030973 (0xf68c bytes) 000490 1786481323.947883 write 0x03298b thru 0x039f1f (0x7595 bytes) 000491 1786481323.950011 read 0x0340a7 thru 0x039f1f (0x5e79 bytes) 000492 1786481323.951211 write 0x03caa1 thru 0x03ffff (0x355f bytes) 000493 1786481323.953353 write 0x02f2fd thru 0x036c89 (0x798d bytes) 000494 1786481323.955407 write 0x00aa17 thru 0x016743 (0xbd2d bytes) 000495 1786481323.962791 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xb1aee63f (stale) Mirror 19: 0xb1aee63f (stale) Mirror 20: 0xb1aee63f (stale) Mirror 21: 0xf02fab21 (stale) Mirror 22: 0x4cd060e6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xb1aee63f (stale) Mirror 19: 0xb1aee63f (stale) Mirror 20: 0xb1aee63f (stale) Mirror 21: 0xf02fab21 (stale) Mirror 22: 0x4cd060e6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000496 1786481324.532541 read 0x03513d thru 0x039e70 (0x4d34 bytes) 000497 1786481324.546126 mapwrite 0x001949 thru 0x00b7f5 (0x9ead bytes) 000499 1786481324.639230 write 0x010bdd thru 0x01623f (0x5663 bytes) 000501 1786481324.647722 mapread 0x02969e thru 0x037555 (0xdeb8 bytes) 000505 1786481324.669216 trunc from 0x040000 to 0x031160 (0xee9f bytes) 000506 1786481324.686940 read 0x012aef thru 0x01e3a1 (0xb8b3 bytes) 000511 1786481324.701192 mapread 0x024890 thru 0x02704e (0x27bf bytes) 000512 1786481324.702851 write 0x00a93a thru 0x017e97 (0xd55e bytes) 000513 1786481324.711671 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31161): Mirror 11: 0x784689a4 (stale) Mirror 19: 0x784689a4 (stale) Mirror 20: 0x784689a4 (stale) Mirror 21: 0x809d6027 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31161): Mirror 11: 0x784689a4 (stale) Mirror 19: 0x784689a4 (stale) Mirror 20: 0x784689a4 (stale) Mirror 21: 0x809d6027 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31161: skipped 000514 1786481325.170768 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x31161): Mirror 11: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31161): Mirror 11: 0x784689a4 (stale) Mirror 19: 0x784689a4 (stale) Mirror 20: 0x784689a4 (stale) Mirror 21: 0x809d6027 (stale) CRC-32 checksum value for chunk [0, 0x31161): Mirror 1: 0x4f2b3197 Mirror 22: 0x4f2b3197 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31161: skipped 000517 1786481326.217625 mapwrite 0x018f45 thru 0x023f31 (0xafed bytes) 000518 1786481326.494117 read 0x017f7f thru 0x022a71 (0xaaf3 bytes) 000520 1786481326.509217 MIRROR_RESYNC 6 mirrors 000521 1786481327.753852 read 0x011fc9 thru 0x01439e (0x23d6 bytes) 000522 1786481327.767998 mapread 0x00c4bc thru 0x014aa9 (0x85ee bytes) 000527 1786481327.818461 mapread 0x006d3e thru 0x008928 (0x1beb bytes) 000529 1786481327.828400 mapread 0x015354 thru 0x018643 (0x32f0 bytes) 000530 1786481327.845066 mapwrite 0x017d49 thru 0x020f6a (0x9222 bytes) 000532 1786481328.042869 mapread 0x02b565 thru 0x031160 (0x5bfc bytes) 000534 1786481328.064065 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x31161): Mirror 11: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31161): Mirror 11: 0x11ede9b8 (stale) Mirror 19: 0x11ede9b8 (stale) Mirror 20: 0x11ede9b8 (stale) Mirror 21: 0x11ede9b8 (stale) Mirror 22: 0x11ede9b8 (stale) CRC-32 checksum value for chunk [0, 0x31161): Mirror 1: 0xc81fb1b8 Mirror 23: 0xc81fb1b8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31161: skipped 000535 1786481329.137904 read 0x00adb1 thru 0x00e691 (0x38e1 bytes) 000536 1786481329.171837 mapread 0x00529e thru 0x00bb65 (0x68c8 bytes) 000539 1786481329.286534 mapread 0x027d35 thru 0x02b239 (0x3505 bytes) 000540 1786481329.328136 mapread 0x02bc84 thru 0x031160 (0x54dd bytes) 000545 1786481329.403426 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31161): Mirror 11: 0x11ede9b8 (stale) Mirror 19: 0x11ede9b8 (stale) Mirror 21: 0x11ede9b8 (stale) Mirror 22: 0x11ede9b8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31161): Mirror 11: 0x11ede9b8 (stale) Mirror 19: 0x11ede9b8 (stale) Mirror 21: 0x11ede9b8 (stale) Mirror 22: 0x11ede9b8 (stale) CRC-32 checksum value for chunk [0, 0x31161): Mirror 1: 0xc81fb1b8 Mirror 23: 0xc81fb1b8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31161: skipped 000546 1786481330.303542 mapwrite 0x03827c thru 0x03ffff (0x7d84 bytes) 000547 1786481330.658856 trunc from 0x040000 to 0x0351fe (0xae01 bytes) 000548 1786481330.697440 mapread 0x0061d0 thru 0x00888d (0x26be bytes) 000551 1786481330.712000 write 0x03aaf7 thru 0x03ffff (0x5509 bytes) 000552 1786481330.716100 mapread 0x03376d thru 0x03ffff (0xc893 bytes) 000553 1786481330.749473 mapwrite 0x01334d thru 0x01614a (0x2dfe bytes) 000554 1786481330.869293 read 0x034274 thru 0x038b5e (0x48eb bytes) 000556 1786481330.870767 mapread 0x012512 thru 0x01d0eb (0xabda bytes) 000557 1786481330.884366 mapwrite 0x031ce4 thru 0x03b40d (0x972a bytes) 000558 1786481331.033891 mapwrite 0x02af40 thru 0x02b87d (0x93e bytes) 000560 1786481331.107231 write 0x03fa5f thru 0x03ffff (0x5a1 bytes) 000562 1786481331.110669 mapwrite 0x029387 thru 0x03736a (0xdfe4 bytes) 000564 1786481331.161228 read 0x007ce4 thru 0x010e9b (0x91b8 bytes) 000566 1786481331.164302 read 0x039fd5 thru 0x03b765 (0x1791 bytes) 000569 1786481331.171697 MIRROR_RESYNC 6 mirrors 000570 1786481332.509036 mapwrite 0x012780 thru 0x019ccc (0x754d bytes) 000572 1786481332.691644 read 0x0071eb thru 0x008b2e (0x1944 bytes) 000573 1786481332.707462 trunc from 0x040000 to 0x0336b8 (0xc947 bytes) 000574 1786481332.741373 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x336b9): Mirror 11: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x336b9): Mirror 11: 0xe617bc5d (stale) Mirror 19: 0xe617bc5d (stale) Mirror 21: 0xe617bc5d (stale) Mirror 22: 0xe617bc5d (stale) Mirror 23: 0xe617bc5d (stale) CRC-32 checksum value for chunk [0, 0x336b9): Mirror 1: 0xc4ca2bad Mirror 24: 0xc4ca2bad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x336b9: skipped 000575 1786481334.063480 trunc from 0x0336b9 to 0x02da92 (0x5c26 bytes) 000576 1786481334.194478 trunc from 0x02da93 to 0x002106 (0x2b98c bytes) 000577 1786481334.242398 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2107): Mirror 11: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2107): Mirror 11: 0x667d2d4c (stale) Mirror 19: 0x667d2d4c (stale) Mirror 21: 0x667d2d4c (stale) Mirror 22: 0x667d2d4c (stale) Mirror 23: 0x667d2d4c (stale) Mirror 24: 0x667d2d4c (stale) CRC-32 checksum value for chunk [0, 0x2107): Mirror 1: 0x667d2d4c Mirror 25: 0x667d2d4c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2107: skipped 000578 1786481335.196525 write 0x024996 thru 0x032dea (0xe455 bytes) 000579 1786481335.324375 mapread 0x01f6dc thru 0x021c5c (0x2581 bytes) 000580 1786481335.348709 read 0x000439 thru 0x00d3ff (0xcfc7 bytes) 000581 1786481335.359993 write 0x016248 thru 0x017a71 (0x182a bytes) 000582 1786481335.384180 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x32deb): Mirror 11: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32deb): Mirror 11: 0xe54dd86a (stale) Mirror 19: 0xe54dd86a (stale) Mirror 21: 0xe54dd86a (stale) Mirror 22: 0xe54dd86a (stale) Mirror 23: 0xe54dd86a (stale) Mirror 24: 0x424e5144 (stale) Mirror 25: 0x36fe4f11 (stale) CRC-32 checksum value for chunk [0, 0x32deb): Mirror 1: 0xd977b377 Mirror 26: 0xd977b377 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32deb: skipped 000584 1786481336.927952 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x32deb): Mirror 11: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32deb): Mirror 11: 0xe54dd86a (stale) Mirror 19: 0xe54dd86a (stale) Mirror 21: 0xe54dd86a (stale) Mirror 22: 0xe54dd86a (stale) Mirror 23: 0xe54dd86a (stale) Mirror 24: 0x424e5144 (stale) Mirror 25: 0x36fe4f11 (stale) CRC-32 checksum value for chunk [0, 0x32deb): Mirror 1: 0xd977b377 Mirror 26: 0xd977b377 Mirror 27: 0xd977b377 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32deb: skipped 000585 1786481338.518403 write 0x01dc95 thru 0x028863 (0xabcf bytes) 000587 1786481338.704488 trunc from 0x032deb to 0x02c0da (0x6d10 bytes) 000590 1786481338.793710 mapwrite 0x019077 thru 0x0233cf (0xa359 bytes) 000591 1786481338.893742 write 0x03fd36 thru 0x03ffff (0x2ca bytes) 000592 1786481338.896138 trunc from 0x040000 to 0x024634 (0x1b9cb bytes) 000593 1786481338.922389 write 0x034c9b thru 0x03ffff (0xb365 bytes) 000594 1786481338.932304 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xc987d0c (stale) Mirror 19: 0xc987d0c (stale) Mirror 21: 0xc987d0c (stale) Mirror 23: 0xc987d0c (stale) Mirror 24: 0x9659dc54 (stale) Mirror 25: 0xb4d25a67 (stale) Mirror 26: 0xf2177158 (stale) Mirror 27: 0xf2177158 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xc987d0c (stale) Mirror 19: 0xc987d0c (stale) Mirror 21: 0xc987d0c (stale) Mirror 23: 0xc987d0c (stale) Mirror 24: 0x9659dc54 (stale) Mirror 25: 0xb4d25a67 (stale) Mirror 26: 0xf2177158 (stale) Mirror 27: 0xf2177158 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000595 1786481339.710839 mapwrite 0x028817 thru 0x02c1d8 (0x39c2 bytes) 000596 1786481339.806459 trunc from 0x040000 to 0x02bc89 (0x14376 bytes) 000597 1786481339.831955 read 0x025417 thru 0x02bc89 (0x6873 bytes) 000598 1786481339.833601 write 0x00d7b4 thru 0x012cf1 (0x553e bytes) 000599 1786481339.850258 mapwrite 0x01a211 thru 0x02938b (0xf17b bytes) 000600 1786481339.934498 mapwrite 0x00271f thru 0x00e39e (0xbc80 bytes) 000602 1786481340.013745 MIRROR_RESYNC 9 mirrors 000604 1786481341.697687 mapwrite 0x00ffe4 thru 0x01b639 (0xb656 bytes) 000605 1786481342.007800 mapwrite 0x035d3f thru 0x035fef (0x2b1 bytes) 000606 1786481342.121790 read 0x000909 thru 0x004e06 (0x44fe bytes) 000608 1786481342.136317 write 0x007a49 thru 0x00fa3a (0x7ff2 bytes) 000609 1786481342.143979 mapread 0x031e0f thru 0x035fef (0x41e1 bytes) 000610 1786481342.145295 mapread 0x02ab0d thru 0x035fef (0xb4e3 bytes) 000612 1786481342.162061 read 0x0215ce thru 0x02bced (0xa720 bytes) 000613 1786481342.173789 read 0x007e55 thru 0x012eb9 (0xb065 bytes) 000614 1786481342.174596 write 0x03fe93 thru 0x03ffff (0x16d bytes) 000615 1786481342.177512 mapwrite 0x024eac thru 0x03159e (0xc6f3 bytes) 000617 1786481342.276544 mapread 0x018d33 thru 0x01fab7 (0x6d85 bytes) 000620 1786481342.285356 mapread 0x025664 thru 0x0267de (0x117b bytes) 000622 1786481342.287787 write 0x01f7ef thru 0x02c576 (0xcd88 bytes) 000623 1786481342.295304 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x9a6917ea (stale) Mirror 19: 0x9a6917ea (stale) Mirror 21: 0x9a6917ea (stale) Mirror 23: 0x9a6917ea (stale) Mirror 24: 0x9a6917ea (stale) Mirror 25: 0x9a6917ea (stale) Mirror 26: 0x9a6917ea (stale) Mirror 27: 0x9a6917ea (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x14c2f7c3 Mirror 28: 0x14c2f7c3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000632 1786481343.569795 MIRROR_RESYNC 10 mirrors 000636 1786481345.610221 trunc from 0x040000 to 0x002a00 (0x3d5ff bytes) 000638 1786481345.714700 MIRROR_RESYNC 10 mirrors 000640 1786481347.228255 write 0x006832 thru 0x00a56e (0x3d3d bytes) 000641 1786481347.326969 write 0x012874 thru 0x01808d (0x581a bytes) 000642 1786481347.328650 write 0x02bd39 thru 0x038e3f (0xd107 bytes) 000643 1786481347.339635 write 0x02fe04 thru 0x03fc5c (0xfe59 bytes) 000645 1786481347.347488 write 0x01e5b8 thru 0x02b7b4 (0xd1fd bytes) 000647 1786481347.365945 mapwrite 0x03f656 thru 0x03fa30 (0x3db bytes) 000648 1786481347.439441 MIRROR_RESYNC 10 mirrors 000649 1786481349.127009 trunc from 0x03fc5d to 0x006a21 (0x3923b bytes) 000650 1786481349.194796 write 0x01c65a thru 0x020ac6 (0x446d bytes) 000652 1786481349.196535 mapwrite 0x03a454 thru 0x03ffff (0x5bac bytes) 000653 1786481349.309657 write 0x00362f thru 0x0131d1 (0xfba3 bytes) 000655 1786481349.324910 mapread 0x0036a7 thru 0x0113e5 (0xdd3f bytes) 000657 1786481349.328515 mapread 0x0042c4 thru 0x01254d (0xe28a bytes) 000658 1786481349.331494 mapwrite 0x013dff thru 0x0154a2 (0x16a4 bytes) 000659 1786481349.405902 write 0x00fb0b thru 0x01a599 (0xaa8f bytes) 000663 1786481349.417110 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x80b62af2 (stale) Mirror 19: 0x80b62af2 (stale) Mirror 21: 0x80b62af2 (stale) Mirror 23: 0x80b62af2 (stale) Mirror 24: 0x80b62af2 (stale) Mirror 25: 0x80b62af2 (stale) Mirror 26: 0x80b62af2 (stale) Mirror 27: 0x80b62af2 (stale) Mirror 28: 0x80b62af2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x510a8f2c Mirror 29: 0x510a8f2c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000664 1786481351.043266 MIRROR_RESYNC 11 mirrors 000665 1786481353.761519 read 0x03e55f thru 0x03ea31 (0x4d3 bytes) 000667 1786481353.774763 mapread 0x00f79e thru 0x01b838 (0xc09b bytes) 000668 1786481353.821870 write 0x00cb73 thru 0x01c22f (0xf6bd bytes) 000671 1786481353.940370 read 0x030da9 thru 0x0373f3 (0x664b bytes) 000672 1786481353.951993 mapwrite 0x0067b4 thru 0x0105f9 (0x9e46 bytes) 000673 1786481353.988090 trunc from 0x040000 to 0x01ce26 (0x231d9 bytes) 000674 1786481354.011718 mapread 0x019081 thru 0x019ee4 (0xe64 bytes) 000679 1786481354.018197 mapwrite 0x029eb4 thru 0x02c853 (0x29a0 bytes) 000682 1786481354.147176 trunc from 0x02c854 to 0x022f19 (0x993a bytes) 000687 1786481354.181074 trunc from 0x022f1a to 0x014911 (0xe608 bytes) 000688 1786481354.200384 mapwrite 0x03b908 thru 0x03ffff (0x46f8 bytes) 000696 1786481354.307392 write 0x0172f5 thru 0x01e34f (0x705b bytes) 000697 1786481354.322066 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x510a8f2c (stale) Mirror 21: 0x510a8f2c (stale) Mirror 23: 0x510a8f2c (stale) Mirror 24: 0x510a8f2c (stale) Mirror 25: 0x510a8f2c (stale) Mirror 26: 0x510a8f2c (stale) Mirror 27: 0x510a8f2c (stale) Mirror 28: 0x510a8f2c (stale) Mirror 29: 0x510a8f2c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x510a8f2c (stale) Mirror 21: 0x510a8f2c (stale) Mirror 23: 0x510a8f2c (stale) Mirror 24: 0x510a8f2c (stale) Mirror 25: 0x510a8f2c (stale) Mirror 26: 0x510a8f2c (stale) Mirror 27: 0x510a8f2c (stale) Mirror 28: 0x510a8f2c (stale) Mirror 29: 0x510a8f2c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000699 1786481355.014059 read 0x03f693 thru 0x03ffff (0x96d bytes) 000700 1786481355.038880 write 0x0154d5 thru 0x01e08e (0x8bba bytes) 000701 1786481355.065510 trunc from 0x040000 to 0x02daaa (0x12555 bytes) 000703 1786481355.142192 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2daab): Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2daab): Mirror 19: 0x273efd2c (stale) Mirror 21: 0x273efd2c (stale) Mirror 23: 0x273efd2c (stale) Mirror 24: 0x273efd2c (stale) Mirror 25: 0x273efd2c (stale) Mirror 26: 0x273efd2c (stale) Mirror 27: 0x273efd2c (stale) Mirror 28: 0x273efd2c (stale) Mirror 29: 0x273efd2c (stale) CRC-32 checksum value for chunk [0, 0x2daab): Mirror 1: 0x348ebad1 Mirror 30: 0x348ebad1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2daab: skipped 000705 1786481356.893330 write 0x00c5be thru 0x019574 (0xcfb7 bytes) 000708 1786481357.099295 write 0x027e8e thru 0x02d6c2 (0x5835 bytes) 000709 1786481357.120524 mapwrite 0x0071c7 thru 0x008609 (0x1443 bytes) 000711 1786481357.222255 write 0x035ffe thru 0x03ffff (0xa002 bytes) 000712 1786481357.228014 trunc from 0x040000 to 0x016d9f (0x29260 bytes) 000713 1786481357.293348 read 0x00c48f thru 0x00fc8b (0x37fd bytes) 000715 1786481357.297205 mapread 0x00b20f thru 0x012717 (0x7509 bytes) 000717 1786481357.322216 MIRROR_RESYNC 11 mirrors 000718 1786481359.000920 MIRROR_EXTEND to 12 mirrors 000719 1786481359.776124 write 0x01384d thru 0x015c54 (0x2408 bytes) 000721 1786481359.865981 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16da0): Mirror 19: 0x3fce8e4c (stale) Mirror 21: 0x3fce8e4c (stale) Mirror 23: 0x3fce8e4c (stale) Mirror 24: 0x3fce8e4c (stale) Mirror 25: 0x3fce8e4c (stale) Mirror 26: 0x3fce8e4c (stale) Mirror 28: 0x3fce8e4c (stale) Mirror 29: 0x3fce8e4c (stale) Mirror 30: 0x3fce8e4c (stale) Mirror 31: 0x3fce8e4c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16da0): Mirror 19: 0x3fce8e4c (stale) Mirror 21: 0x3fce8e4c (stale) Mirror 23: 0x3fce8e4c (stale) Mirror 24: 0x3fce8e4c (stale) Mirror 25: 0x3fce8e4c (stale) Mirror 26: 0x3fce8e4c (stale) Mirror 28: 0x3fce8e4c (stale) Mirror 29: 0x3fce8e4c (stale) Mirror 30: 0x3fce8e4c (stale) Mirror 31: 0x3fce8e4c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16da0: skipped 000722 1786481360.719719 mapwrite 0x03dc5d thru 0x03ffff (0x23a3 bytes) 000725 1786481360.817633 trunc from 0x040000 to 0x014a26 (0x2b5d9 bytes) 000729 1786481360.842442 write 0x0288b3 thru 0x02ba81 (0x31cf bytes) 000731 1786481360.845590 trunc from 0x02ba82 to 0x0024a1 (0x295e0 bytes) 000733 1786481360.868512 mapread 0x001a75 thru 0x0024a1 (0xa2d bytes) 000735 1786481360.884438 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x24a2): Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24a2): Mirror 19: 0xb48cdfb9 (stale) Mirror 21: 0xb48cdfb9 (stale) Mirror 23: 0xb48cdfb9 (stale) Mirror 24: 0xb48cdfb9 (stale) Mirror 25: 0xb48cdfb9 (stale) Mirror 26: 0xb48cdfb9 (stale) Mirror 28: 0xb48cdfb9 (stale) Mirror 29: 0xb48cdfb9 (stale) Mirror 30: 0xb48cdfb9 (stale) Mirror 31: 0xb48cdfb9 (stale) CRC-32 checksum value for chunk [0, 0x24a2): Mirror 1: 0xb48cdfb9 Mirror 32: 0xb48cdfb9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24a2: skipped 000736 1786481361.992301 mapwrite 0x005f90 thru 0x011841 (0xb8b2 bytes) 000737 1786481362.238241 mapwrite 0x03bb04 thru 0x03ffff (0x44fc bytes) 000739 1786481362.357406 mapwrite 0x033b1a thru 0x038634 (0x4b1b bytes) 000741 1786481362.420267 write 0x0008c3 thru 0x007f65 (0x76a3 bytes) 000742 1786481362.434110 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x4364e73d (stale) Mirror 21: 0x4364e73d (stale) Mirror 23: 0x4364e73d (stale) Mirror 24: 0x4364e73d (stale) Mirror 25: 0x4364e73d (stale) Mirror 26: 0x4364e73d (stale) Mirror 28: 0x4364e73d (stale) Mirror 29: 0x4364e73d (stale) Mirror 30: 0x4364e73d (stale) Mirror 31: 0x4364e73d (stale) Mirror 32: 0xb4d25a67 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc2521717 Mirror 33: 0xc2521717 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000743 1786481364.007240 MIRROR_RESYNC 13 mirrors 000744 1786481367.419411 mapwrite 0x023c65 thru 0x029222 (0x55be bytes) 000745 1786481367.646337 write 0x005ff2 thru 0x01411a (0xe129 bytes) 000746 1786481367.675323 mapread 0x013b92 thru 0x01ac72 (0x70e1 bytes) 000749 1786481367.722727 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0xc2521717 (stale) Mirror 21: 0xc2521717 (stale) Mirror 23: 0xc2521717 (stale) Mirror 24: 0xc2521717 (stale) Mirror 25: 0xc2521717 (stale) Mirror 26: 0xc2521717 (stale) Mirror 28: 0xc2521717 (stale) Mirror 29: 0xc2521717 (stale) Mirror 30: 0xc2521717 (stale) Mirror 31: 0xc2521717 (stale) Mirror 32: 0xc2521717 (stale) Mirror 33: 0xc2521717 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5c82e994 Mirror 34: 0x5c82e994 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000754 1786481369.475102 MIRROR_RESYNC 14 mirrors 000755 1786481372.370781 MIRROR_RESYNC 14 mirrors 000756 1786481373.415612 mapwrite 0x0247e0 thru 0x03433f (0xfb60 bytes) 000757 1786481373.721183 read 0x0167c0 thru 0x0178e0 (0x1121 bytes) 000758 1786481373.728618 trunc from 0x040000 to 0x0052e3 (0x3ad1c bytes) 000760 1786481373.769395 trunc from 0x0052e4 to 0x03701c (0x31d39 bytes) 000763 1786481373.789929 mapread 0x033313 thru 0x03701c (0x3d0a bytes) 000764 1786481373.812543 mapread 0x0004ea thru 0x00f0f9 (0xec10 bytes) 000766 1786481373.825714 mapwrite 0x03b492 thru 0x03f7e0 (0x434f bytes) 000768 1786481373.904459 write 0x0001c4 thru 0x00e512 (0xe34f bytes) 000769 1786481373.911996 MIRROR_RESYNC 14 mirrors truncating to largest ever: 0x3d49c 000770 1786481376.822655 trunc from 0x03f7e1 to 0x03d49b (0x2345 bytes) 000771 1786481376.921297 mapread 0x001180 thru 0x00f332 (0xe1b3 bytes) 000772 1786481376.949247 write 0x01ef97 thru 0x02167e (0x26e8 bytes) 000773 1786481376.982433 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x3d49c): Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d49c): Mirror 19: 0xa4a13b7a (stale) Mirror 21: 0xa4a13b7a (stale) Mirror 23: 0xa4a13b7a (stale) Mirror 24: 0xa4a13b7a (stale) Mirror 25: 0xa4a13b7a (stale) Mirror 26: 0xa4a13b7a (stale) Mirror 28: 0xa4a13b7a (stale) Mirror 29: 0xa4a13b7a (stale) Mirror 30: 0xa4a13b7a (stale) Mirror 31: 0xa4a13b7a (stale) Mirror 32: 0xa4a13b7a (stale) Mirror 33: 0xa4a13b7a (stale) Mirror 34: 0xa4a13b7a (stale) CRC-32 checksum value for chunk [0, 0x3d49c): Mirror 1: 0x90e03167 Mirror 35: 0x90e03167 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d49c: skipped 000774 1786481378.685652 mapread 0x00d397 thru 0x0140f6 (0x6d60 bytes) 000775 1786481378.744120 mapwrite 0x029344 thru 0x034bbc (0xb879 bytes) 000776 1786481379.022515 trunc from 0x03d49c to 0x002d18 (0x3a783 bytes) 000778 1786481379.066627 MIRROR_SPLIT mirror 11 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d19): Mirror 19: 0x45611a32 (stale) Mirror 21: 0x45611a32 (stale) Mirror 23: 0x45611a32 (stale) Mirror 24: 0x45611a32 (stale) Mirror 25: 0x45611a32 (stale) Mirror 26: 0x45611a32 (stale) Mirror 28: 0x45611a32 (stale) Mirror 29: 0x45611a32 (stale) Mirror 30: 0x45611a32 (stale) Mirror 31: 0x45611a32 (stale) Mirror 33: 0x45611a32 (stale) Mirror 34: 0x45611a32 (stale) Mirror 35: 0x45611a32 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d19): Mirror 19: 0x45611a32 (stale) Mirror 21: 0x45611a32 (stale) Mirror 23: 0x45611a32 (stale) Mirror 24: 0x45611a32 (stale) Mirror 25: 0x45611a32 (stale) Mirror 26: 0x45611a32 (stale) Mirror 28: 0x45611a32 (stale) Mirror 29: 0x45611a32 (stale) Mirror 30: 0x45611a32 (stale) Mirror 31: 0x45611a32 (stale) Mirror 33: 0x45611a32 (stale) Mirror 34: 0x45611a32 (stale) Mirror 35: 0x45611a32 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d19: skipped 000780 1786481379.688087 MIRROR_SPLIT mirror 6 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d19): Mirror 19: 0x45611a32 (stale) Mirror 21: 0x45611a32 (stale) Mirror 23: 0x45611a32 (stale) Mirror 24: 0x45611a32 (stale) Mirror 25: 0x45611a32 (stale) Mirror 28: 0x45611a32 (stale) Mirror 29: 0x45611a32 (stale) Mirror 30: 0x45611a32 (stale) Mirror 31: 0x45611a32 (stale) Mirror 33: 0x45611a32 (stale) Mirror 34: 0x45611a32 (stale) Mirror 35: 0x45611a32 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d19): Mirror 19: 0x45611a32 (stale) Mirror 21: 0x45611a32 (stale) Mirror 23: 0x45611a32 (stale) Mirror 24: 0x45611a32 (stale) Mirror 25: 0x45611a32 (stale) Mirror 28: 0x45611a32 (stale) Mirror 29: 0x45611a32 (stale) Mirror 30: 0x45611a32 (stale) Mirror 31: 0x45611a32 (stale) Mirror 33: 0x45611a32 (stale) Mirror 34: 0x45611a32 (stale) Mirror 35: 0x45611a32 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d19: skipped 000781 1786481380.219447 trunc from 0x002d19 to 0x01bed9 (0x191c1 bytes) 000784 1786481380.247231 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x1beda): Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1beda): Mirror 19: 0x9d5b87be (stale) Mirror 21: 0x9d5b87be (stale) Mirror 23: 0x9d5b87be (stale) Mirror 24: 0x9d5b87be (stale) Mirror 25: 0x9d5b87be (stale) Mirror 28: 0x9d5b87be (stale) Mirror 29: 0x9d5b87be (stale) Mirror 30: 0x9d5b87be (stale) Mirror 31: 0x9d5b87be (stale) Mirror 33: 0x9d5b87be (stale) Mirror 34: 0x9d5b87be (stale) Mirror 35: 0x9d5b87be (stale) CRC-32 checksum value for chunk [0, 0x1beda): Mirror 1: 0xebdb20bc Mirror 36: 0xebdb20bc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1beda: skipped 000785 1786481381.749189 read 0x005190 thru 0x014af1 (0xf962 bytes) 000786 1786481381.771293 write 0x016ce4 thru 0x023bf7 (0xcf14 bytes) 000787 1786481381.903963 mapread 0x004598 thru 0x00619c (0x1c05 bytes) 000788 1786481381.924408 read 0x005de9 thru 0x013299 (0xd4b1 bytes) 000790 1786481381.936875 mapwrite 0x01b1fa thru 0x01ffe4 (0x4deb bytes) 000791 1786481382.059015 mapread 0x01b374 thru 0x023bf7 (0x8884 bytes) 000795 1786481382.072313 mapwrite 0x0184de thru 0x0207c3 (0x82e6 bytes) 000796 1786481382.193064 mapwrite 0x0064ae thru 0x00fe4e (0x99a1 bytes) 000798 1786481382.276156 write 0x02ccfd thru 0x031353 (0x4657 bytes) 000800 1786481382.279128 mapwrite 0x03e8e9 thru 0x03ffff (0x1717 bytes) 000801 1786481382.387597 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x5d5937ef (stale) Mirror 21: 0x5d5937ef (stale) Mirror 23: 0x5d5937ef (stale) Mirror 24: 0x5d5937ef (stale) Mirror 25: 0x5d5937ef (stale) Mirror 28: 0x5d5937ef (stale) Mirror 29: 0x5d5937ef (stale) Mirror 30: 0x5d5937ef (stale) Mirror 31: 0x5d5937ef (stale) Mirror 33: 0x5d5937ef (stale) Mirror 34: 0x5d5937ef (stale) Mirror 35: 0x7ebc3d04 (stale) Mirror 36: 0xf3a4146c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x27c073ed Mirror 37: 0x27c073ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000802 1786481383.924006 mapread 0x03c81f thru 0x03ffff (0x37e1 bytes) 000803 1786481383.962617 MIRROR_SPLIT mirror 1 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x5d5937ef (stale) Mirror 23: 0x5d5937ef (stale) Mirror 24: 0x5d5937ef (stale) Mirror 25: 0x5d5937ef (stale) Mirror 28: 0x5d5937ef (stale) Mirror 29: 0x5d5937ef (stale) Mirror 30: 0x5d5937ef (stale) Mirror 31: 0x5d5937ef (stale) Mirror 33: 0x5d5937ef (stale) Mirror 34: 0x5d5937ef (stale) Mirror 35: 0x7ebc3d04 (stale) Mirror 36: 0xf3a4146c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x5d5937ef (stale) Mirror 23: 0x5d5937ef (stale) Mirror 24: 0x5d5937ef (stale) Mirror 25: 0x5d5937ef (stale) Mirror 28: 0x5d5937ef (stale) Mirror 29: 0x5d5937ef (stale) Mirror 30: 0x5d5937ef (stale) Mirror 31: 0x5d5937ef (stale) Mirror 33: 0x5d5937ef (stale) Mirror 34: 0x5d5937ef (stale) Mirror 35: 0x7ebc3d04 (stale) Mirror 36: 0xf3a4146c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x27c073ed Mirror 37: 0x27c073ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000806 1786481385.600217 trunc from 0x040000 to 0x01becd (0x24132 bytes) 000807 1786481385.784325 read 0x00964a thru 0x0107a2 (0x7159 bytes) truncating to largest ever: 0x3f211 000808 1786481385.798417 trunc from 0x01bece to 0x03f210 (0x23343 bytes) 000809 1786481385.847992 MIRROR_SPLIT mirror 4 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f211): Mirror 21: 0xf6ff376f (stale) Mirror 23: 0xf6ff376f (stale) Mirror 24: 0xf6ff376f (stale) Mirror 28: 0xf6ff376f (stale) Mirror 29: 0xf6ff376f (stale) Mirror 30: 0xf6ff376f (stale) Mirror 31: 0xf6ff376f (stale) Mirror 33: 0xf6ff376f (stale) Mirror 34: 0xf6ff376f (stale) Mirror 35: 0x938f7362 (stale) Mirror 36: 0x867acdad (stale) Mirror 37: 0x3a282e46 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f211): Mirror 21: 0xf6ff376f (stale) Mirror 23: 0xf6ff376f (stale) Mirror 24: 0xf6ff376f (stale) Mirror 28: 0xf6ff376f (stale) Mirror 29: 0xf6ff376f (stale) Mirror 30: 0xf6ff376f (stale) Mirror 31: 0xf6ff376f (stale) Mirror 33: 0xf6ff376f (stale) Mirror 34: 0xf6ff376f (stale) Mirror 35: 0x938f7362 (stale) Mirror 36: 0x867acdad (stale) Mirror 37: 0x3a282e46 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f211: skipped 000810 1786481386.775990 mapread 0x0030a5 thru 0x009315 (0x6271 bytes) 000812 1786481386.800704 read 0x031dcc thru 0x03954a (0x777f bytes) 000813 1786481386.809630 mapread 0x01b7c1 thru 0x027efc (0xc73c bytes) 000816 1786481386.833541 write 0x007b8b thru 0x00a19b (0x2611 bytes) 000818 1786481386.836903 write 0x023f24 thru 0x028c00 (0x4cdd bytes) 000819 1786481386.838945 write 0x01414a thru 0x01c025 (0x7edc bytes) 000820 1786481386.841231 write 0x00ebdd thru 0x01e3d2 (0xf7f6 bytes) 000822 1786481386.844906 mapread 0x0346e7 thru 0x0375f0 (0x2f0a bytes) 000823 1786481386.846153 read 0x03c8da thru 0x03d316 (0xa3d bytes) 000825 1786481386.859047 mapread 0x035156 thru 0x0361e7 (0x1092 bytes) 000827 1786481386.862807 read 0x026df3 thru 0x035608 (0xe816 bytes) 000828 1786481386.864061 mapread 0x034b21 thru 0x03cfae (0x848e bytes) 000829 1786481386.867507 mapread 0x020381 thru 0x02d220 (0xcea0 bytes) 000830 1786481386.869328 read 0x005a2c thru 0x0086d8 (0x2cad bytes) 000834 1786481386.873535 read 0x0208fb thru 0x02369f (0x2da5 bytes) 000835 1786481386.874844 trunc from 0x03f211 to 0x03e97c (0x894 bytes) 000836 1786481386.899895 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x3e97d): Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 37: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e97d): Mirror 21: 0x424c546 (stale) Mirror 23: 0x424c546 (stale) Mirror 24: 0x424c546 (stale) Mirror 28: 0x424c546 (stale) Mirror 29: 0x424c546 (stale) Mirror 30: 0x424c546 (stale) Mirror 31: 0x424c546 (stale) Mirror 33: 0x424c546 (stale) Mirror 34: 0x424c546 (stale) Mirror 35: 0x29b9c5dc (stale) Mirror 36: 0x6e1bf06 (stale) Mirror 37: 0x149d9b75 (stale) CRC-32 checksum value for chunk [0, 0x3e97d): Mirror 1: 0xec50a071 Mirror 38: 0xec50a071 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e97d: skipped 000837 1786481388.555961 mapread 0x0380de thru 0x03db49 (0x5a6c bytes) 000838 1786481388.592049 write 0x035fea thru 0x03ab4a (0x4b61 bytes) 000839 1786481388.714019 mapread 0x0373be thru 0x03e97c (0x75bf bytes) 000840 1786481388.728071 MIRROR_RESYNC 14 mirrors 000841 1786481391.621232 mapwrite 0x022bc6 thru 0x02d74d (0xab88 bytes) 000843 1786481391.915125 mapread 0x02870e thru 0x03144e (0x8d41 bytes) 000847 1786481391.925010 MIRROR_SPLIT mirror 4 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e97d): Mirror 21: 0x535cc86a (stale) Mirror 23: 0x535cc86a (stale) Mirror 24: 0x535cc86a (stale) Mirror 29: 0x535cc86a (stale) Mirror 30: 0x535cc86a (stale) Mirror 31: 0x535cc86a (stale) Mirror 33: 0x535cc86a (stale) Mirror 34: 0x535cc86a (stale) Mirror 35: 0x535cc86a (stale) Mirror 36: 0x535cc86a (stale) Mirror 37: 0x535cc86a (stale) Mirror 38: 0x535cc86a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e97d): Mirror 21: 0x535cc86a (stale) Mirror 23: 0x535cc86a (stale) Mirror 24: 0x535cc86a (stale) Mirror 29: 0x535cc86a (stale) Mirror 30: 0x535cc86a (stale) Mirror 31: 0x535cc86a (stale) Mirror 33: 0x535cc86a (stale) Mirror 34: 0x535cc86a (stale) Mirror 35: 0x535cc86a (stale) Mirror 36: 0x535cc86a (stale) Mirror 37: 0x535cc86a (stale) Mirror 38: 0x535cc86a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e97d: skipped 000848 1786481392.832375 write 0x025f0d thru 0x0293d6 (0x34ca bytes) 000849 1786481392.850214 MIRROR_SPLIT mirror 7 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e97d): Mirror 21: 0x535cc86a (stale) Mirror 23: 0x535cc86a (stale) Mirror 24: 0x535cc86a (stale) Mirror 29: 0x535cc86a (stale) Mirror 30: 0x535cc86a (stale) Mirror 31: 0x535cc86a (stale) Mirror 34: 0x535cc86a (stale) Mirror 35: 0x535cc86a (stale) Mirror 36: 0x535cc86a (stale) Mirror 37: 0x535cc86a (stale) Mirror 38: 0x535cc86a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e97d): Mirror 21: 0x535cc86a (stale) Mirror 23: 0x535cc86a (stale) Mirror 24: 0x535cc86a (stale) Mirror 29: 0x535cc86a (stale) Mirror 30: 0x535cc86a (stale) Mirror 31: 0x535cc86a (stale) Mirror 34: 0x535cc86a (stale) Mirror 35: 0x535cc86a (stale) Mirror 36: 0x535cc86a (stale) Mirror 37: 0x535cc86a (stale) Mirror 38: 0x535cc86a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e97d: skipped 000851 1786481393.655423 mapwrite 0x03b6a4 thru 0x03ffff (0x495c bytes) 000852 1786481393.765744 trunc from 0x040000 to 0x01b4a0 (0x24b5f bytes) 000853 1786481393.799714 trunc from 0x01b4a1 to 0x015f47 (0x5559 bytes) 000857 1786481393.820215 mapwrite 0x009003 thru 0x00f6ab (0x66a9 bytes) 000859 1786481393.876008 mapread 0x00b72b thru 0x0145a3 (0x8e79 bytes) 000860 1786481393.885754 read 0x00a248 thru 0x00bd9d (0x1b56 bytes) 000861 1786481393.891310 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15f48): Mirror 21: 0x29fba79b (stale) Mirror 23: 0x29fba79b (stale) Mirror 24: 0x29fba79b (stale) Mirror 29: 0x29fba79b (stale) Mirror 30: 0x29fba79b (stale) Mirror 31: 0x29fba79b (stale) Mirror 34: 0x29fba79b (stale) Mirror 35: 0x29fba79b (stale) Mirror 36: 0x29fba79b (stale) Mirror 38: 0x29fba79b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15f48): Mirror 21: 0x29fba79b (stale) Mirror 23: 0x29fba79b (stale) Mirror 24: 0x29fba79b (stale) Mirror 29: 0x29fba79b (stale) Mirror 30: 0x29fba79b (stale) Mirror 31: 0x29fba79b (stale) Mirror 34: 0x29fba79b (stale) Mirror 35: 0x29fba79b (stale) Mirror 36: 0x29fba79b (stale) Mirror 38: 0x29fba79b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15f48: skipped 000862 1786481394.759476 read 0x003bb1 thru 0x00648e (0x28de bytes) 000864 1786481394.777534 trunc from 0x015f48 to 0x030470 (0x1a529 bytes) 000865 1786481394.804014 trunc from 0x030471 to 0x015a8d (0x1a9e3 bytes) 000866 1786481394.836467 write 0x0046c6 thru 0x00d2e7 (0x8c22 bytes) 000867 1786481394.858149 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15a8e): Mirror 21: 0x493c04d3 (stale) Mirror 23: 0x493c04d3 (stale) Mirror 24: 0x493c04d3 (stale) Mirror 29: 0x493c04d3 (stale) Mirror 31: 0x493c04d3 (stale) Mirror 34: 0x493c04d3 (stale) Mirror 35: 0x493c04d3 (stale) Mirror 36: 0x493c04d3 (stale) Mirror 38: 0x493c04d3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15a8e): Mirror 21: 0x493c04d3 (stale) Mirror 23: 0x493c04d3 (stale) Mirror 24: 0x493c04d3 (stale) Mirror 29: 0x493c04d3 (stale) Mirror 31: 0x493c04d3 (stale) Mirror 34: 0x493c04d3 (stale) Mirror 35: 0x493c04d3 (stale) Mirror 36: 0x493c04d3 (stale) Mirror 38: 0x493c04d3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15a8e: skipped 000868 1786481395.470539 mapwrite 0x011c70 thru 0x0144fd (0x288e bytes) 000871 1786481395.543657 write 0x02b069 thru 0x02ef84 (0x3f1c bytes) 000872 1786481395.550044 MIRROR_RESYNC 10 mirrors 000874 1786481398.136246 mapread 0x023073 thru 0x02c41c (0x93aa bytes) 000876 1786481398.191006 write 0x013d99 thru 0x01c5b3 (0x881b bytes) 000879 1786481398.338244 read 0x0074b3 thru 0x01372d (0xc27b bytes) 000880 1786481398.359415 mapwrite 0x03cefb thru 0x03ffff (0x3105 bytes) 000882 1786481398.518495 write 0x028277 thru 0x030bf2 (0x897c bytes) 000884 1786481398.561974 write 0x01a476 thru 0x020a91 (0x661c bytes) 000886 1786481398.579444 trunc from 0x040000 to 0x02e63f (0x119c0 bytes) 000887 1786481398.629492 mapread 0x003208 thru 0x00a264 (0x705d bytes) 000889 1786481398.646619 mapwrite 0x023c3a thru 0x024c3a (0x1001 bytes) 000890 1786481398.762135 write 0x01dbda thru 0x021fe4 (0x440b bytes) 000891 1786481398.770089 write 0x025080 thru 0x02ef18 (0x9e99 bytes) 000894 1786481398.796647 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2ef19): Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 38: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ef19): Mirror 21: 0xd5a69a02 (stale) Mirror 23: 0xd5a69a02 (stale) Mirror 24: 0xd5a69a02 (stale) Mirror 29: 0xd5a69a02 (stale) Mirror 31: 0xd5a69a02 (stale) Mirror 34: 0xd5a69a02 (stale) Mirror 35: 0xd5a69a02 (stale) Mirror 36: 0xd5a69a02 (stale) Mirror 38: 0xd5a69a02 (stale) CRC-32 checksum value for chunk [0, 0x2ef19): Mirror 1: 0x3ccadf9a Mirror 39: 0x3ccadf9a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ef19: skipped 000896 1786481399.952059 trunc from 0x02ef19 to 0x010b04 (0x1e414 bytes) 000897 1786481400.061580 mapread 0x002e31 thru 0x0096c0 (0x6890 bytes) 000899 1786481400.083126 read 0x00a02c thru 0x010b04 (0x6ad9 bytes) 000900 1786481400.083801 read 0x0032b7 thru 0x0056a0 (0x23ea bytes) 000901 1786481400.085580 trunc from 0x010b05 to 0x0236ce (0x12bca bytes) 000902 1786481400.109733 mapread 0x00ac07 thru 0x0179af (0xcda9 bytes) 000903 1786481400.120233 mapread 0x01b3d0 thru 0x02038b (0x4fbc bytes) 000907 1786481400.124681 trunc from 0x0236cf to 0x0395aa (0x15edc bytes) 000908 1786481400.151858 write 0x003f88 thru 0x0095cf (0x5648 bytes) 000910 1786481400.155634 mapwrite 0x01b230 thru 0x0259f6 (0xa7c7 bytes) 000912 1786481400.262757 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x395ab): Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 38: (stale - read error) Mirror 39: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x395ab): Mirror 21: 0x8b2b42cd (stale) Mirror 23: 0x8b2b42cd (stale) Mirror 24: 0x8b2b42cd (stale) Mirror 29: 0x8b2b42cd (stale) Mirror 31: 0x8b2b42cd (stale) Mirror 34: 0x8b2b42cd (stale) Mirror 35: 0x8b2b42cd (stale) Mirror 36: 0x8b2b42cd (stale) Mirror 38: 0x8b2b42cd (stale) Mirror 39: 0x910b0d0f (stale) CRC-32 checksum value for chunk [0, 0x395ab): Mirror 1: 0x54f087ea Mirror 40: 0x54f087ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x395ab: skipped 000915 1786481401.631252 read 0x020ef9 thru 0x027df6 (0x6efe bytes) 000917 1786481401.667163 write 0x00ae2b thru 0x018bda (0xddb0 bytes) 000918 1786481401.807822 MIRROR_RESYNC 12 mirrors 000919 1786481404.368988 mapread 0x00bb3c thru 0x014f7d (0x9442 bytes) 000920 1786481404.389251 read 0x0108c6 thru 0x01d435 (0xcb70 bytes) 000921 1786481404.407361 trunc from 0x0395ab to 0x00f466 (0x2a144 bytes) 000923 1786481404.592585 trunc from 0x00f467 to 0x0015f7 (0xde6f bytes) 000924 1786481404.658170 mapread 0x0014ec thru 0x0015f7 (0x10c bytes) 000925 1786481404.670476 read 0x00044b thru 0x0015f7 (0x11ad bytes) 000926 1786481404.689689 mapread 0x000c62 thru 0x0015f7 (0x996 bytes) 000927 1786481404.691603 mapread 0x000515 thru 0x0015f7 (0x10e3 bytes) 000928 1786481404.723613 MIRROR_RESYNC 12 mirrors 000929 1786481406.830804 MIRROR_RESYNC 12 mirrors 000931 1786481407.302179 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) 000932 1786481407.764142 write 0x01141b thru 0x018933 (0x7519 bytes) 000934 1786481407.941800 read 0x00a014 thru 0x00e3b4 (0x43a1 bytes) 000937 1786481407.954764 trunc from 0x018934 to 0x0310df (0x187ac bytes) 000940 1786481408.026911 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x310e0): Mirror 23: 0x2134499a (stale) Mirror 24: 0x2134499a (stale) Mirror 31: 0x2134499a (stale) Mirror 34: 0x2134499a (stale) Mirror 35: 0x2134499a (stale) Mirror 36: 0x2134499a (stale) Mirror 38: 0x2134499a (stale) Mirror 39: 0x2134499a (stale) Mirror 40: 0x2134499a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x310e0): Mirror 23: 0x2134499a (stale) Mirror 24: 0x2134499a (stale) Mirror 31: 0x2134499a (stale) Mirror 34: 0x2134499a (stale) Mirror 35: 0x2134499a (stale) Mirror 36: 0x2134499a (stale) Mirror 38: 0x2134499a (stale) Mirror 39: 0x2134499a (stale) Mirror 40: 0x2134499a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x310e0: skipped 000941 1786481408.685102 mapread 0x01a132 thru 0x0217a4 (0x7673 bytes) 000943 1786481408.715698 mapwrite 0x00c14a thru 0x019d97 (0xdc4e bytes) 000945 1786481408.791853 read 0x008077 thru 0x00bd54 (0x3cde bytes) 000947 1786481408.795218 mapread 0x00e707 thru 0x00f756 (0x1050 bytes) 000948 1786481408.796652 write 0x0287ef thru 0x02aa9e (0x22b0 bytes) 000949 1786481408.806626 write 0x012ff9 thru 0x0199b4 (0x69bc bytes) 000952 1786481408.809142 write 0x0124cc thru 0x01c397 (0x9ecc bytes) 000953 1786481408.811343 write 0x02d0d1 thru 0x030fad (0x3edd bytes) 000954 1786481408.820173 trunc from 0x0310e0 to 0x01ec2a (0x124b5 bytes) 000955 1786481408.832929 mapwrite 0x037321 thru 0x03ffff (0x8cdf bytes) 000956 1786481408.925139 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: 0x62a44eee (stale) Mirror 24: 0x62a44eee (stale) Mirror 31: 0x62a44eee (stale) Mirror 34: 0x62a44eee (stale) Mirror 36: 0x62a44eee (stale) Mirror 38: 0x62a44eee (stale) Mirror 39: 0x62a44eee (stale) Mirror 40: 0x62a44eee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: 0x62a44eee (stale) Mirror 24: 0x62a44eee (stale) Mirror 31: 0x62a44eee (stale) Mirror 34: 0x62a44eee (stale) Mirror 36: 0x62a44eee (stale) Mirror 38: 0x62a44eee (stale) Mirror 39: 0x62a44eee (stale) Mirror 40: 0x62a44eee (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000957 1786481409.599469 mapwrite 0x00842c thru 0x012699 (0xa26e bytes) 000958 1786481409.685092 mapread 0x02e92c thru 0x0341e5 (0x58ba bytes) 000959 1786481409.696008 mapread 0x033439 thru 0x03ffff (0xcbc7 bytes) 000960 1786481409.706671 mapwrite 0x002bf3 thru 0x0083b6 (0x57c4 bytes) 000962 1786481409.760737 read 0x017409 thru 0x01fa4e (0x8646 bytes) 000964 1786481409.775657 read 0x02ee48 thru 0x0331a2 (0x435b bytes) 000965 1786481409.776627 mapwrite 0x03602e thru 0x03d367 (0x733a bytes) 000966 1786481409.823508 mapwrite 0x00d765 thru 0x018840 (0xb0dc bytes) 000968 1786481409.886090 mapread 0x020216 thru 0x02184d (0x1638 bytes) 000969 1786481409.888224 write 0x028ba5 thru 0x02a4da (0x1936 bytes) 000970 1786481409.889627 read 0x03721d thru 0x038065 (0xe49 bytes) 000971 1786481409.891183 write 0x0010ad thru 0x004b88 (0x3adc bytes) 000972 1786481409.892673 read 0x02dc0e thru 0x03600d (0x8400 bytes) 000973 1786481409.893724 write 0x03ea81 thru 0x03ffff (0x157f bytes) 000974 1786481409.899696 MIRROR_RESYNC 9 mirrors 000975 1786481412.299144 MIRROR_EXTEND to 10 mirrors 000976 1786481413.588461 trunc from 0x040000 to 0x03061e (0xf9e1 bytes) 000977 1786481413.783235 mapread 0x009fa5 thru 0x00c400 (0x245c bytes) 000978 1786481413.817054 mapread 0x02891c thru 0x03061e (0x7d03 bytes) 000979 1786481413.833347 trunc from 0x03061f to 0x01c047 (0x145d7 bytes) 000980 1786481413.866366 MIRROR_RESYNC 10 mirrors 000983 1786481416.433394 mapwrite 0x022c17 thru 0x031599 (0xe983 bytes) 000987 1786481416.604985 mapwrite 0x03dc2c thru 0x03ffff (0x23d4 bytes) 000991 1786481416.680425 trunc from 0x040000 to 0x028834 (0x177cb bytes) 000992 1786481416.696576 read 0x01b560 thru 0x027267 (0xbd08 bytes) 000993 1786481416.703506 mapwrite 0x01bf6a thru 0x0275de (0xb675 bytes) 000994 1786481416.759443 read 0x009d7d thru 0x013fe6 (0xa26a bytes) 000995 1786481416.775249 MIRROR_RESYNC 10 mirrors 000996 1786481418.343216 MIRROR_RESYNC 10 mirrors 000997 1786481419.095622 trunc from 0x028835 to 0x00b856 (0x1cfde bytes) 000998 1786481419.199434 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb857): Mirror 23: 0xa2d07a2a (stale) Mirror 24: 0xa2d07a2a (stale) Mirror 31: 0xa2d07a2a (stale) Mirror 34: 0xa2d07a2a (stale) Mirror 36: 0xa2d07a2a (stale) Mirror 38: 0xa2d07a2a (stale) Mirror 40: 0xa2d07a2a (stale) Mirror 41: 0xa2d07a2a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb857): Mirror 23: 0xa2d07a2a (stale) Mirror 24: 0xa2d07a2a (stale) Mirror 31: 0xa2d07a2a (stale) Mirror 34: 0xa2d07a2a (stale) Mirror 36: 0xa2d07a2a (stale) Mirror 38: 0xa2d07a2a (stale) Mirror 40: 0xa2d07a2a (stale) Mirror 41: 0xa2d07a2a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb857: skipped 000999 1786481419.920439 trunc from 0x00b857 to 0x0257df (0x19f89 bytes) 001000 1786481419.945593 mapwrite 0x00164f thru 0x0099fe (0x83b0 bytes) All operations completed A-OK! PASS 70a (190s) == sanity-flr test 71: check mirror extend parent fid ==== 16:50:26 (1786481426) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 16:50:30 (1786481430) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x306:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.332537 s, 6.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2eb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x306:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149019 s, 7.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2eb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x307:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x306:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2eb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x307:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x306:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ec:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 16:50:40 (1786481440) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 6834400 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5785568 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9045952 .. Extending file size to 6934112 .. Extending file size to 8471168 .. Extending file size to 8589504 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8616096 .. done Waiting 57519 57520 57523 57524 57525 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) 192.168.201.148@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (87s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 16:52:07 (1786481527) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 977152 .. Extending file size to 7350784 .. failed Extending file size to 2977984 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9196640 .. failed Extending file size to 6708608 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8289216 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8295072 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8324800 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8425344 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59042 59044 59045 59047 59048 done 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) 192.168.201.148@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (82s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 16:53:29 (1786481609) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.119985 s, 17.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.23082 s, 9.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x16afa77 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 16:53:51 (1786481631) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0905143 s, 23.2 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 16:53:58 (1786481638) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20c:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31d:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31d:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31e:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x301:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 16:54:07 (1786481647) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.21432 s, 9.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x302:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x303:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 16:54:16 (1786481656) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.223379 s, 9.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x320:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x321:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x304:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x320:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x321:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x304:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x305:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 16:54:26 (1786481666) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.269168 s, 7.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x322:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x306:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x322:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x306:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x307:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 16:54:59 (1786481699) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg148-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168543 s, 6.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x323:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x309:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x323:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x309:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } PASS 204d (59s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 16:55:58 (1786481758) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg148-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x325:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x326:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.255072 s, 8.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x325:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x326:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x325:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x326:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (62s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 16:57:00 (1786481820) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg148-server mds-ost sync done. Creating new pool test_204f oleg148-server: Pool lustre.test_204f created Adding targets to pool oleg148-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg148-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.276968 s, 7.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x327:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x327:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg148-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg148-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg148-server: Pool lustre.test_204f destroyed PASS 204f (79s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 16:58:19 (1786481899) PASS 205a (5s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 16:58:25 (1786481905) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 16:58:30 (1786481910) oleg148-server: Pool lustre.test_206 created Adding targets to pool oleg148-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg148-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg148-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg148-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg148-server: Pool lustre.test_206 destroyed PASS 206 (21s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 16:58:51 (1786481931) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852493 s, 12.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447933 s, 23.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xf70cb4c7 Mirror 3: 0xf70cb4c7 PASS 207 (7s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 16:58:58 (1786481938) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 16:59:01 (1786481941) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 16:59:04 (1786481944) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.110669 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x8a0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x770000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x770000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x770000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2350000 Read this (4 bytes): BBBB 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) PASS 209a (30s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 16:59:33 (1786481973) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0248974 s, 329 kB/s the very first read read with warm cache PASS 209b (5s) == sanity-flr test 210a: handle broken mirrored lovea ==== 16:59:38 (1786481978) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279179 s, 37.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x231 lmm_fid: [0x200000404:0x231:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x231 lmm_fid: [0x200000404:0x231:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x315:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834481 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-11 16:59:39.000000000 -0400 Modify: 2026-08-11 16:59:39.000000000 -0400 Change: 2026-08-11 16:59:40.000000000 -0400 Birth: 2026-08-11 16:59:40.000000000 -0400 PASS 210a (6s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 16:59:44 (1786481984) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 16:59:47 (1786481987) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.384278 s, 107 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0513371 s, 79.8 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 11 16:59 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x233:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 11 16:59 /mnt/lustre/f211.sanity-flr PASS 211 (6s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 16:59:53 (1786481993) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 16:59:59 (1786481999) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (7s) == sanity-flr test complete, duration 1715 sec =========== 17:00:06 (1786482006) === sanity-flr: start cleanup 17:00:07 (1786482007) === === sanity-flr: finish cleanup 17:00:13 (1786482013) ===