-----============= acceptance-small: sanity-flr ============----- Fri Aug 21 22:16:37 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg137-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 22:16:53 (1787365013) === oleg137-client.virtnet: executing check_config_client /mnt/lustre oleg137-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg137-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9c3d46c77800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c3d46c77800.idle_timeout=debug disable quota as required oleg137-server: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 22:17:29 (1787365049) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7658] == sanity-flr test 0a: lfs mirror create with -N option == 22:17:32 (1787365052) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 22:17:49 (1787365069) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 22:17:53 (1787365073) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 22:17:56 (1787365076) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 22:18:21 (1787365101) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 22:18:24 (1787365104) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 22:18:28 (1787365108) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9c3d46c77800.stats=clear osc.lustre-OST0001-osc-ffff9c3d46c77800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 22:18:36 (1787365116) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 22:18:46 (1787365126) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 22:18:53 (1787365133) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 22:18:56 (1787365136) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 22:19:03 (1787365143) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 22:19:11 (1787365151) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 22:19:17 (1787365157) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 22:19:20 (1787365160) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 22:19:28 (1787365168) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (9s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 22:19:41 (1787365181) PASS 7 (9s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 22:19:49 (1787365189) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.476699 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169043 s, 6.2 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 22:19:59 (1787365199) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.320082 s, 39.3 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 22:20:07 (1787365207) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (8s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 22:20:15 (1787365215) Stopping /mnt/lustre-ost1 (opts:) on oleg137-server oleg137-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c3d46c77800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c3d46c77800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg137-server: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg137-server: oleg137-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg137-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg137-server oleg137-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c3d46c77800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c3d46c77800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg137-server: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg137-server: oleg137-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg137-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (69s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 22:21:24 (1787365284) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.594286 s, 12.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg137-server oleg137-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c3d46c77800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c3d46c77800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg137-server: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg137-server: oleg137-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg137-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (43s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 22:22:08 (1787365328) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg137-server oleg137-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c3d46c77800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c3d46c77800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg137-server: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg137-server: oleg137-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg137-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg137-server oleg137-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c3d46c77800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c3d46c77800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg137-server: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg137-server: oleg137-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg137-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (78s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 22:23:26 (1787365406) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.26746 s, 10.0 MB/s -rw-r--r-- 1 root root 52428800 Aug 21 22:23 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.109s user 0m0.002s sys 0m0.187s Stopping /mnt/lustre-ost1 (opts:) on oleg137-server oleg137-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c3d46c77800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c3d46c77800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.062s user 0m0.000s sys 0m0.171s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg137-server: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg137-server: oleg137-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg137-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg137-server oleg137-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c3d46c77800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c3d46c77800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.427s user 0m0.002s sys 0m0.196s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg137-server: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg137-server: oleg137-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg137-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg137-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c3d46c77800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c3d46c77800.ost_server_uuid in FULL state after 0 sec PASS 33b (97s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 22:25:03 (1787365503) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 22:25:07 (1787365507) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 22:25:11 (1787365511) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 22:25:14 (1787365514) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 22:25:23 (1787365523) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.621588 s, 20.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 22:25:37 (1787365537) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473564 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487308 s, 21.5 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 22:25:45 (1787365545) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 22:25:55 (1787365555) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.001132 s, 905 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00129323 s, 792 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000913138 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00152769 s, 670 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 22:26:06 (1787365566) create test files with size 15748548 .. 4246c3ff6fef8edda8bdc76a31cd6b5b - 46f7219c0c5315974a9aeec4ba1c0423 - 21bca3851fa93261c7f5754de205079b - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg137-client.virtnet /mnt/lustre (opts:) Starting client: oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 22:26:29 (1787365589) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 22:26:43 (1787365603) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 22:26:50 (1787365610) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209198 s, 5.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417406 s, 25.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xe0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (7s) == sanity-flr test 41: lfs mirror resync check =========== 22:26:58 (1787365618) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131628 s, 15.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.484692 s, 8.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.239658 s, 8.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 22:27:12 (1787365632) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 22:27:15 (1787365635) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 22:27:19 (1787365639) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040236 s, 26.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 43c: read prefer uncompressed mirror == 22:27:28 (1787365648) llite.lustre-ffff9c3d44c94000.enable_compression=1 fail_loc=0x1425 fail_val=-1 PASS 43c (8s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 22:27:35 (1787365655) llite.lustre-ffff9c3d44c94000.enable_compression=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264659 s, 39.6 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374664 s, 28.0 MB/s **verify components PASS 43d (8s) == sanity-flr test 44a: lfs mirror split check =========== 22:27:44 (1787365664) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 22:27:47 (1787365667) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xea:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xea:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 22:27:57 (1787365677) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.303776 s, 34.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x1f:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg137-client.virtnet /mnt/lustre (opts:) Starting client: oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 22:28:08 (1787365688) llite.lustre-ffff9c3d480da800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.347924 s, 30.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg137-client.virtnet /mnt/lustre (opts:) Starting client: oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9c3d5060b000.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 22:28:23 (1787365703) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (40s) == sanity-flr test 46: Verify setstripe --copy option ==== 22:29:03 (1787365743) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (37s) == sanity-flr test 47: Verify mirror obj alloc =========== 22:29:40 (1787365780) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 22:29:44 (1787365784) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.233662 s, 13.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.187255 s, 16.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xa8e8c475 (nosync,stale) Mirror 2: 0xa8e8c475 (nosync,stale) ** verify mirror 2 doesn't change original checksum: aa0c16730354fd2f11cc42c928a1e95a - mirror 1 checksum: 34b6557aa3e33fd053ce8db29fd4654e - mirror 2 checksum: aa0c16730354fd2f11cc42c928a1e95a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-21T22:29:47-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-21T22:29:47-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: aa0c16730354fd2f11cc42c928a1e95a - mirror 1 checksum: 34b6557aa3e33fd053ce8db29fd4654e - mirror 2 checksum: 34b6557aa3e33fd053ce8db29fd4654e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-21T22:29:47-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-21T22:29:47-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 22:29:57 (1787365797) Failing mds1 on oleg137-server Stopping /mnt/lustre-mds1 (opts:) on oleg137-server 22:30:04 (1787365804) shut down facet: mds1 facet_host: oleg137-server facet_failover_host: oleg137-server Failover mds1 to oleg137-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg137-server: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-MDT0000 22:30:36 (1787365836) targets are mounted 22:30:36 (1787365836) facet_failover done oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443164 s, 23.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe1:0x0] } PASS 50A (60s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 22:30:57 (1787365857) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 22:31:01 (1787365861) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 22:31:04 (1787365864) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00719945 s, 2.3 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 22:31:12 (1787365872) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 22:31:16 (1787365876) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0122534 s, 334 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0881784 s, 46.5 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 22:31:25 (1787365885) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 22:31:48 (1787365908) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 22:32:17 (1787365937) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 22:32:40 (1787365960) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 22:33:05 (1787365985) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 22:33:14 (1787365994) mirror create pid 55581 mirror split pid 55582 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 55581 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 55582 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 22:34:22 (1787366062) Seed set to 6064 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x1809a 000001 1787366064.984682 trunc from 00000000 to 0x018099 (0x1809a bytes) 000005 1787366065.005183 write 0x0371f8 thru 0x03c38e (0x5197 bytes) 000006 1787366065.008395 mapread 0x02347e thru 0x02f590 (0xc113 bytes) 000007 1787366065.033093 read 0x022e8d thru 0x02e960 (0xbad4 bytes) 000013 1787366065.038886 mapwrite 0x00910f thru 0x0183fd (0xf2ef bytes) 000014 1787366065.130457 write 0x012741 thru 0x0167ca (0x408a bytes) 000015 1787366065.132047 mapread 0x013e44 thru 0x019662 (0x581f bytes) 000018 1787366065.135137 trunc from 0x03c38f to 0x0149ab (0x279e3 bytes) 000022 1787366065.161929 write 0x024a8b thru 0x032505 (0xda7b bytes) 000024 1787366065.164332 mapread 0x020343 thru 0x0252e1 (0x4f9f bytes) 000026 1787366065.172934 write 0x03600f thru 0x03f2e7 (0x92d9 bytes) 000029 1787366065.176045 read 0x0085f9 thru 0x0132f2 (0xacfa bytes) 000030 1787366065.176982 mapread 0x002c78 thru 0x009d04 (0x708d bytes) 000031 1787366065.178360 trunc from 0x03f2e8 to 0x0055d5 (0x39d12 bytes) 000034 1787366065.194442 mapwrite 0x0216bc thru 0x02283e (0x1183 bytes) 000035 1787366065.261368 mapwrite 0x006d94 thru 0x008e8a (0x20f7 bytes) 000042 1787366065.307506 trunc from 0x02283f to 0x00103f (0x217ff bytes) 000044 1787366065.334466 read 0x0005b8 thru 0x00103f (0xa88 bytes) 000045 1787366065.335316 trunc from 0x001040 to 0x00cfbf (0xbf80 bytes) 000046 1787366065.351819 mapwrite 0x001178 thru 0x00538d (0x4216 bytes) 000048 1787366065.411876 write 0x035213 thru 0x0354c1 (0x2af bytes) 000049 1787366065.413940 mapwrite 0x02fcd3 thru 0x03bd3c (0xc06a bytes) 000050 1787366065.495723 mapwrite 0x032f87 thru 0x03919f (0x6219 bytes) 000051 1787366065.541888 read 0x02406d thru 0x03357e (0xf512 bytes) 000052 1787366065.542614 write 0x023828 thru 0x02f3e5 (0xbbbe bytes) 000053 1787366065.543869 read 0x01859b thru 0x023ccb (0xb731 bytes) 000054 1787366065.544484 mapwrite 0x02cd12 thru 0x03a784 (0xda73 bytes) 000056 1787366065.605891 mapwrite 0x0176ae thru 0x0266b9 (0xf00c bytes) 000063 1787366065.666628 mapread 0x001afe thru 0x009d56 (0x8259 bytes) 000064 1787366065.668160 mapread 0x026b69 thru 0x02708d (0x525 bytes) 000066 1787366065.680360 MIRROR_EXTEND to 2 mirrors 000067 1787366066.539210 write 0x014be2 thru 0x0241ae (0xf5cd bytes) 000068 1787366066.615745 write 0x0046a7 thru 0x009cf9 (0x5653 bytes) 000070 1787366066.639993 mapread 0x01156e thru 0x01e655 (0xd0e8 bytes) 000071 1787366066.659962 mapread 0x018236 thru 0x0227e0 (0xa5ab bytes) 000072 1787366066.661057 mapread 0x01ba5a thru 0x028e22 (0xd3c9 bytes) 000073 1787366066.681442 mapread 0x032422 thru 0x03677e (0x435d bytes) 000074 1787366066.693034 write 0x037ced thru 0x03ffff (0x8313 bytes) 000076 1787366066.694996 read 0x02a8e2 thru 0x038da2 (0xe4c1 bytes) 000082 1787366066.699078 read 0x03fbe5 thru 0x03ffff (0x41b bytes) 000083 1787366066.706227 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) truncating to largest ever: 0x383b2 000084 1787366066.942515 trunc from 0x040000 to 0x0383b1 (0x7c4e bytes) 000087 1787366066.982114 MIRROR_EXTEND to 2 mirrors 000089 1787366067.710249 MIRROR_RESYNC 2 mirrors 000091 1787366068.061680 mapwrite 0x005823 thru 0x007479 (0x1c57 bytes) 000092 1787366068.218227 read 0x02b615 thru 0x034446 (0x8e32 bytes) 000093 1787366068.226807 write 0x02010e thru 0x024576 (0x4469 bytes) 000094 1787366068.237028 mapread 0x0020d3 thru 0x0085b9 (0x64e7 bytes) 000095 1787366068.238126 mapread 0x033f90 thru 0x0383b1 (0x4422 bytes) 000096 1787366068.247069 write 0x0064d0 thru 0x0159ec (0xf51d bytes) 000098 1787366068.254448 mapread 0x01e257 thru 0x023279 (0x5023 bytes) 000099 1787366068.263923 write 0x031173 thru 0x0311c5 (0x53 bytes) 000100 1787366068.264899 mapread 0x01061b thru 0x01cd7e (0xc764 bytes) 000102 1787366068.276559 MIRROR_RESYNC 2 mirrors 000103 1787366068.914455 mapwrite 0x01f1be thru 0x024d14 (0x5b57 bytes) 000106 1787366069.035805 trunc from 0x0383b2 to 0x03041e (0x7f93 bytes) 000107 1787366069.080093 mapread 0x00c062 thru 0x011cd8 (0x5c77 bytes) 000108 1787366069.090952 mapread 0x002e5f thru 0x005d29 (0x2ecb bytes) 000109 1787366069.098388 trunc from 0x03041f to 0x027547 (0x8ed7 bytes) 000114 1787366069.122215 trunc from 0x027548 to 0x003ccc (0x2387b bytes) 000116 1787366069.145117 mapwrite 0x03d154 thru 0x03ffff (0x2eac bytes) 000117 1787366069.235200 read 0x01945d thru 0x01c624 (0x31c8 bytes) 000122 1787366069.249248 read 0x008c73 thru 0x0146fd (0xba8b bytes) 000126 1787366069.276872 write 0x007f1a thru 0x0117a7 (0x988e bytes) 000127 1787366069.282378 trunc from 0x040000 to 0x01cacb (0x23534 bytes) 000130 1787366069.351482 write 0x01957c thru 0x0212b0 (0x7d35 bytes) 000131 1787366069.356365 read 0x01c0b6 thru 0x0212b0 (0x51fb bytes) 000135 1787366069.361105 mapwrite 0x0004d8 thru 0x00d632 (0xd15b bytes) 000137 1787366069.438818 MIRROR_RESYNC 2 mirrors 000138 1787366070.117877 mapwrite 0x00aab6 thru 0x012100 (0x764b bytes) 000139 1787366070.299993 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000140 1787366070.435320 write 0x03c94e thru 0x03ffff (0x36b2 bytes) 000141 1787366070.436861 read 0x00ed1b thru 0x01b5a3 (0xc889 bytes) 000142 1787366070.444957 write 0x0277f7 thru 0x02ec7b (0x7485 bytes) 000143 1787366070.456982 read 0x01b6d6 thru 0x01bc4f (0x57a bytes) 000144 1787366070.460443 MIRROR_EXTEND to 2 mirrors 000146 1787366071.685081 MIRROR_EXTEND to 3 mirrors 000149 1787366072.714752 write 0x0054e2 thru 0x011893 (0xc3b2 bytes) 000152 1787366072.853970 trunc from 0x040000 to 0x01b04e (0x24fb1 bytes) 000153 1787366072.910016 read 0x00af75 thru 0x00e2ed (0x3379 bytes) 000154 1787366072.911182 mapwrite 0x01516e thru 0x0183b0 (0x3243 bytes) 000155 1787366073.056831 trunc from 0x01b04f to 0x036b18 (0x1baca bytes) 000157 1787366073.083022 read 0x00601c thru 0x00e3e2 (0x83c7 bytes) 000158 1787366073.083871 read 0x01a3d4 thru 0x01a402 (0x2f bytes) 000163 1787366073.089729 write 0x02877c thru 0x02da01 (0x5286 bytes) 000165 1787366073.110747 trunc from 0x036b19 to 0x0140a3 (0x22a75 bytes) 000166 1787366073.133820 trunc from 0x0140a4 to 0x0090cf (0xafd4 bytes) 000167 1787366073.163873 MIRROR_RESYNC 3 mirrors 000169 1787366074.806926 write 0x02cc03 thru 0x03caf8 (0xfef6 bytes) 000170 1787366074.890972 write 0x03472c thru 0x03ffff (0xb8d4 bytes) 000171 1787366074.897985 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x67d95fd6 (stale) Mirror 3: 0x67d95fd6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe4eaf186 Mirror 4: 0xe4eaf186 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000172 1787366076.260140 mapwrite 0x03cf7d thru 0x03ffff (0x3083 bytes) 000176 1787366076.641489 mapwrite 0x017b19 thru 0x024003 (0xc4eb bytes) 000177 1787366076.811507 mapwrite 0x02305a thru 0x024b35 (0x1adc bytes) 000178 1787366076.925623 mapread 0x03abee thru 0x03ffff (0x5412 bytes) 000181 1787366076.942805 mapwrite 0x02c5ea thru 0x032d17 (0x672e bytes) truncating to largest ever: 0x3e8dc 000182 1787366077.087469 trunc from 0x040000 to 0x03e8db (0x1724 bytes) 000185 1787366077.149202 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e8dc): Mirror 2: 0x41d535cc (stale) Mirror 3: 0x41d535cc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e8dc): Mirror 2: 0x41d535cc (stale) Mirror 3: 0x41d535cc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e8dc: skipped 000186 1787366077.670951 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3e8dc): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e8dc): Mirror 2: 0x41d535cc (stale) Mirror 3: 0x41d535cc (stale) CRC-32 checksum value for chunk [0, 0x3e8dc): Mirror 1: 0x90dbd986 Mirror 4: 0x90dbd986 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e8dc: skipped 000188 1787366078.894618 write 0x01ee5d thru 0x02a754 (0xb8f8 bytes) 000190 1787366078.976219 mapwrite 0x001a8a thru 0x00d3d2 (0xb949 bytes) 000195 1787366079.099480 mapread 0x026c37 thru 0x02cf35 (0x62ff bytes) 000196 1787366079.132205 MIRROR_RESYNC 4 mirrors 000198 1787366080.494470 MIRROR_RESYNC 4 mirrors 000199 1787366080.917965 write 0x03f4af thru 0x03ffff (0xb51 bytes) 000200 1787366081.022149 read 0x013763 thru 0x01be12 (0x86b0 bytes) 000202 1787366081.067100 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3a5bbe85 (stale) Mirror 3: 0x3a5bbe85 (stale) Mirror 4: 0x3a5bbe85 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x27e4577f Mirror 5: 0x27e4577f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000203 1787366082.529516 write 0x01eec6 thru 0x029a67 (0xaba2 bytes) 000205 1787366082.634484 read 0x020fe1 thru 0x02df62 (0xcf82 bytes) 000206 1787366082.642387 read 0x00c476 thru 0x00e5e2 (0x216d bytes) 000207 1787366082.678101 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x3a5bbe85 (stale) Mirror 4: 0x3a5bbe85 (stale) Mirror 5: 0x27e4577f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x3a5bbe85 (stale) Mirror 4: 0x3a5bbe85 (stale) Mirror 5: 0x27e4577f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000209 1787366083.141307 mapwrite 0x002853 thru 0x00287b (0x29 bytes) 000212 1787366083.255935 write 0x024730 thru 0x02d800 (0x90d1 bytes) 000216 1787366083.291152 mapwrite 0x03a22a thru 0x03ffff (0x5dd6 bytes) 000217 1787366083.426288 mapwrite 0x01c512 thru 0x025e90 (0x997f bytes) 000218 1787366083.621846 mapread 0x02fd64 thru 0x031227 (0x14c4 bytes) 000219 1787366083.643500 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x3a5bbe85 (stale) Mirror 4: 0x3a5bbe85 (stale) Mirror 5: 0x27e4577f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa3c57aee Mirror 6: 0xa3c57aee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000220 1787366085.338737 write 0x02faf5 thru 0x0320b1 (0x25bd bytes) 000222 1787366085.463188 trunc from 0x040000 to 0x00bbf9 (0x34406 bytes) 000225 1787366085.502070 trunc from 0x00bbfa to 0x0022f4 (0x9905 bytes) 000227 1787366085.530679 mapread 0x0006ab thru 0x0022f4 (0x1c4a bytes) 000228 1787366085.550944 write 0x0058b7 thru 0x011350 (0xba9a bytes) 000229 1787366085.554147 read 0x00a039 thru 0x011350 (0x7318 bytes) 000230 1787366085.555085 read 0x00b1de thru 0x00e7e0 (0x3603 bytes) 000232 1787366085.556693 mapread 0x010140 thru 0x011350 (0x1211 bytes) 000235 1787366085.561804 mapwrite 0x0254e0 thru 0x025905 (0x426 bytes) 000236 1787366085.726245 read 0x01948e thru 0x020eea (0x7a5d bytes) 000238 1787366085.728246 read 0x00c374 thru 0x01347e (0x710b bytes) 000239 1787366085.730580 write 0x004247 thru 0x007310 (0x30ca bytes) 000243 1787366085.733125 write 0x00af45 thru 0x01aab0 (0xfb6c bytes) 000245 1787366085.750199 write 0x0294ac thru 0x038607 (0xf15c bytes) 000246 1787366085.771504 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x38608): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38608): Mirror 3: 0x91cbf781 (stale) Mirror 4: 0x91cbf781 (stale) Mirror 5: 0x91cbf781 (stale) Mirror 6: 0x8216e028 (stale) CRC-32 checksum value for chunk [0, 0x38608): Mirror 1: 0xb708611b Mirror 7: 0xb708611b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38608: skipped 000247 1787366087.436194 read 0x028283 thru 0x036963 (0xe6e1 bytes) 000248 1787366087.462664 mapread 0x032696 thru 0x038607 (0x5f72 bytes) 000249 1787366087.486093 MIRROR_RESYNC 6 mirrors 000250 1787366089.222776 write 0x02d759 thru 0x036993 (0x923b bytes) 000252 1787366089.388199 read 0x02c29c thru 0x038607 (0xc36c bytes) 000253 1787366089.396901 trunc from 0x038608 to 0x014ff8 (0x2360f bytes) 000254 1787366089.443984 write 0x0288c0 thru 0x02ffbd (0x76fe bytes) 000255 1787366089.445323 mapread 0x01a9de thru 0x01e4ca (0x3aed bytes) 000256 1787366089.474791 mapwrite 0x034d83 thru 0x03ffff (0xb27d bytes) 000258 1787366089.624033 MIRROR_RESYNC 6 mirrors 000259 1787366092.370494 mapwrite 0x010901 thru 0x0140d4 (0x37d4 bytes) 000260 1787366092.563038 write 0x026ef8 thru 0x036836 (0xf93f bytes) 000262 1787366092.584717 mapread 0x01b692 thru 0x020f85 (0x58f4 bytes) 000268 1787366092.601991 mapread 0x030d96 thru 0x036af6 (0x5d61 bytes) 000269 1787366092.609331 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x99084cc7 (stale) Mirror 5: 0x99084cc7 (stale) Mirror 6: 0x99084cc7 (stale) Mirror 7: 0x99084cc7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x99084cc7 (stale) Mirror 5: 0x99084cc7 (stale) Mirror 6: 0x99084cc7 (stale) Mirror 7: 0x99084cc7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000271 1787366093.241531 mapread 0x03acd4 thru 0x03c1b5 (0x14e2 bytes) 000273 1787366093.266108 read 0x0281de thru 0x031ee5 (0x9d08 bytes) 000274 1787366093.274429 trunc from 0x040000 to 0x011036 (0x2efc9 bytes) 000275 1787366093.292821 mapwrite 0x00c8e7 thru 0x01a9de (0xe0f8 bytes) 000276 1787366093.504716 trunc from 0x01a9df to 0x0042e1 (0x166fd bytes) 000277 1787366093.533955 mapwrite 0x016172 thru 0x01ecb1 (0x8b40 bytes) 000279 1787366093.755363 mapread 0x015b0c thru 0x01ecb1 (0x91a6 bytes) 000280 1787366093.757212 trunc from 0x01ecb2 to 0x02366d (0x49bc bytes) 000281 1787366093.808131 trunc from 0x02366e to 0x03a589 (0x16f1c bytes) 000282 1787366093.830728 mapwrite 0x03b1d6 thru 0x03ffff (0x4e2a bytes) 000284 1787366094.133332 mapread 0x0271c8 thru 0x034b1a (0xd953 bytes) 000285 1787366094.173554 mapwrite 0x004ad9 thru 0x004edd (0x405 bytes) 000286 1787366094.327953 write 0x038932 thru 0x03ffff (0x76ce bytes) 000288 1787366094.337466 MIRROR_RESYNC 5 mirrors 000289 1787366097.490755 write 0x00b6cb thru 0x012347 (0x6c7d bytes) 000290 1787366097.685499 read 0x021dcd thru 0x029efc (0x8130 bytes) 000291 1787366097.702156 mapwrite 0x039232 thru 0x03fc32 (0x6a01 bytes) 000293 1787366097.934389 trunc from 0x040000 to 0x00a1ec (0x35e13 bytes) 000294 1787366098.032138 trunc from 0x00a1ed to 0x039e7c (0x2fc90 bytes) 000296 1787366098.084534 mapwrite 0x03ff38 thru 0x03ffff (0xc8 bytes) 000301 1787366098.291495 mapread 0x01384d thru 0x01e3ff (0xabb3 bytes) 000302 1787366098.353724 write 0x028e5b thru 0x030c23 (0x7dc9 bytes) 000303 1787366098.396759 write 0x007126 thru 0x00c8fb (0x57d6 bytes) 000306 1787366098.432335 trunc from 0x040000 to 0x01c186 (0x23e79 bytes) 000307 1787366098.491459 write 0x02ee4f thru 0x033889 (0x4a3b bytes) 000308 1787366098.503390 read 0x01c56a thru 0x027fb0 (0xba47 bytes) 000309 1787366098.525162 trunc from 0x03388a to 0x01dcac (0x15bdd bytes) 000310 1787366098.570228 trunc from 0x01dcad to 0x03a294 (0x1c5e8 bytes) 000311 1787366098.601770 mapwrite 0x01b518 thru 0x01c3d2 (0xebb bytes) 000312 1787366098.643998 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a295): Mirror 4: 0x6978b929 (stale) Mirror 5: 0x6978b929 (stale) Mirror 7: 0x6978b929 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a295): Mirror 4: 0x6978b929 (stale) Mirror 5: 0x6978b929 (stale) Mirror 7: 0x6978b929 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a295: skipped 000314 1787366098.861981 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3a295): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a295): Mirror 4: 0x6978b929 (stale) Mirror 5: 0x6978b929 (stale) Mirror 7: 0x6978b929 (stale) CRC-32 checksum value for chunk [0, 0x3a295): Mirror 1: 0x84f02c84 Mirror 8: 0x84f02c84 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a295: skipped 000315 1787366100.724826 mapread 0x0181de thru 0x01c8cf (0x46f2 bytes) 000317 1787366100.753886 mapread 0x01df98 thru 0x022726 (0x478f bytes) 000321 1787366100.762850 MIRROR_RESYNC 5 mirrors 000324 1787366102.568768 MIRROR_RESYNC 5 mirrors 000325 1787366103.088201 mapwrite 0x00122d thru 0x008f64 (0x7d38 bytes) 000326 1787366103.341367 trunc from 0x03a295 to 0x00edde (0x2b4b6 bytes) 000327 1787366103.375932 trunc from 0x00eddf to 0x02f005 (0x20227 bytes) 000328 1787366103.414484 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2f006): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f006): Mirror 4: 0x71cf0ad5 (stale) Mirror 5: 0x71cf0ad5 (stale) Mirror 7: 0x71cf0ad5 (stale) Mirror 8: 0x71cf0ad5 (stale) CRC-32 checksum value for chunk [0, 0x2f006): Mirror 1: 0x62cb76c4 Mirror 9: 0x62cb76c4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f006: skipped 000329 1787366104.751627 read 0x0077bd thru 0x00dd5a (0x659e bytes) 000331 1787366104.781696 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f006): Mirror 4: 0x71cf0ad5 (stale) Mirror 5: 0x71cf0ad5 (stale) Mirror 7: 0x71cf0ad5 (stale) Mirror 8: 0x71cf0ad5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f006): Mirror 4: 0x71cf0ad5 (stale) Mirror 5: 0x71cf0ad5 (stale) Mirror 7: 0x71cf0ad5 (stale) Mirror 8: 0x71cf0ad5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f006: skipped 000334 1787366105.503492 write 0x007771 thru 0x0148f0 (0xd180 bytes) 000336 1787366105.578509 mapread 0x007dbf thru 0x00a32b (0x256d bytes) 000337 1787366105.580794 read 0x00e5fd thru 0x01cb1c (0xe520 bytes) 000338 1787366105.596599 mapwrite 0x0334a9 thru 0x036c57 (0x37af bytes) 000340 1787366105.749549 mapread 0x029263 thru 0x032724 (0x94c2 bytes) 000342 1787366105.766854 trunc from 0x036c58 to 0x005f6e (0x30ce9 bytes) 000345 1787366105.795743 read 0x004a9f thru 0x005f6e (0x14d0 bytes) 000346 1787366105.804352 read 0x004bd8 thru 0x005858 (0xc81 bytes) 000347 1787366105.812583 MIRROR_RESYNC 5 mirrors 000349 1787366107.489163 mapwrite 0x0372c0 thru 0x03fe2e (0x8b6f bytes) 000350 1787366107.756469 mapwrite 0x002442 thru 0x01213e (0xfcfd bytes) 000351 1787366107.851043 read 0x004db5 thru 0x00dc09 (0x8e55 bytes) 000353 1787366107.853877 write 0x00506e thru 0x009e5d (0x4df0 bytes) 000354 1787366107.856651 mapread 0x032fc7 thru 0x03697f (0x39b9 bytes) 000355 1787366107.868043 MIRROR_RESYNC 5 mirrors 000357 1787366109.358595 write 0x03fe3a thru 0x03ffff (0x1c6 bytes) 000358 1787366109.510997 mapwrite 0x03d1fc thru 0x03ffff (0x2e04 bytes) 000361 1787366109.576159 write 0x01e6fd thru 0x01fb95 (0x1499 bytes) 000362 1787366109.589248 mapwrite 0x0086d5 thru 0x012bc3 (0xa4ef bytes) 000363 1787366109.661546 trunc from 0x040000 to 0x012ae9 (0x2d516 bytes) 000364 1787366109.688673 mapread 0x00c028 thru 0x00dc30 (0x1c09 bytes) 000365 1787366109.690144 trunc from 0x012aea to 0x00c875 (0x6274 bytes) 000366 1787366109.707585 mapwrite 0x0221a0 thru 0x0278a2 (0x5703 bytes) 000367 1787366109.893661 mapread 0x01e193 thru 0x0278a2 (0x9710 bytes) 000368 1787366109.908038 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x278a3): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x278a3): Mirror 4: 0x17b39b68 (stale) Mirror 5: 0x17b39b68 (stale) Mirror 7: 0x17b39b68 (stale) Mirror 8: 0x17b39b68 (stale) CRC-32 checksum value for chunk [0, 0x278a3): Mirror 1: 0xceda9f9b Mirror 9: 0xceda9f9b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x278a3: skipped 000369 1787366111.231245 read 0x005ed4 thru 0x00d3bb (0x74e8 bytes) 000371 1787366111.244701 read 0x020997 thru 0x0278a2 (0x6f0c bytes) 000372 1787366111.256867 mapread 0x01884e thru 0x021288 (0x8a3b bytes) 000374 1787366111.264668 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x278a3): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x278a3): Mirror 4: 0x17b39b68 (stale) Mirror 5: 0x17b39b68 (stale) Mirror 7: 0x17b39b68 (stale) Mirror 8: 0x17b39b68 (stale) CRC-32 checksum value for chunk [0, 0x278a3): Mirror 1: 0xceda9f9b Mirror 9: 0xceda9f9b Mirror 10: 0xceda9f9b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x278a3: skipped 000378 1787366112.980096 trunc from 0x0278a3 to 0x019f98 (0xd90a bytes) 000380 1787366113.088608 mapwrite 0x016eee thru 0x025a6a (0xeb7d bytes) 000381 1787366113.306309 MIRROR_RESYNC 7 mirrors 000383 1787366115.484932 read 0x002e6b thru 0x0080fd (0x5293 bytes) 000385 1787366115.514560 MIRROR_EXTEND to 8 mirrors 000386 1787366116.535196 mapread 0x019942 thru 0x023c64 (0xa323 bytes) 000387 1787366116.559964 trunc from 0x025a6b to 0x017404 (0xe666 bytes) 000388 1787366116.723813 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x17405): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17405): Mirror 4: 0xca70ac4 (stale) Mirror 5: 0xca70ac4 (stale) Mirror 7: 0xca70ac4 (stale) Mirror 8: 0xca70ac4 (stale) Mirror 9: 0xca70ac4 (stale) Mirror 10: 0xca70ac4 (stale) Mirror 11: 0xca70ac4 (stale) CRC-32 checksum value for chunk [0, 0x17405): Mirror 1: 0xca70ac4 Mirror 12: 0xca70ac4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17405: skipped 000390 1787366117.610349 mapwrite 0x037694 thru 0x03e871 (0x71de bytes) 000391 1787366117.821160 write 0x00b0ee thru 0x00c976 (0x1889 bytes) 000392 1787366117.832883 mapwrite 0x02e1b4 thru 0x035566 (0x73b3 bytes) 000394 1787366117.907921 MIRROR_RESYNC 9 mirrors 000395 1787366120.342721 read 0x02c48f thru 0x02da1e (0x1590 bytes) 000396 1787366120.351450 mapwrite 0x03835f thru 0x03ffff (0x7ca1 bytes) 000398 1787366120.593332 read 0x00b3ae thru 0x01732a (0xbf7d bytes) 000399 1787366120.609421 write 0x013cf1 thru 0x01980d (0x5b1d bytes) 000400 1787366120.610610 mapwrite 0x038d0a thru 0x03ad3b (0x2032 bytes) 000401 1787366120.689884 read 0x0388d2 thru 0x03ffff (0x772e bytes) 000402 1787366120.699035 MIRROR_RESYNC 9 mirrors 000404 1787366123.246256 read 0x007613 thru 0x0081a9 (0xb97 bytes) 000406 1787366123.260066 mapread 0x035d20 thru 0x03f8f0 (0x9bd1 bytes) 000409 1787366123.302238 mapread 0x03afb7 thru 0x03ffff (0x5049 bytes) 000410 1787366123.303300 read 0x01c27d thru 0x02aa19 (0xe79d bytes) 000412 1787366123.349363 trunc from 0x040000 to 0x0387aa (0x7855 bytes) 000413 1787366123.596738 mapwrite 0x00f9df thru 0x010238 (0x85a bytes) 000414 1787366123.660876 write 0x020f81 thru 0x029fa7 (0x9027 bytes) 000417 1787366123.686508 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x387ab): Mirror 4: 0x636068e3 (stale) Mirror 5: 0x636068e3 (stale) Mirror 7: 0x636068e3 (stale) Mirror 8: 0x636068e3 (stale) Mirror 9: 0x636068e3 (stale) Mirror 10: 0x636068e3 (stale) Mirror 11: 0x636068e3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x387ab): Mirror 4: 0x636068e3 (stale) Mirror 5: 0x636068e3 (stale) Mirror 7: 0x636068e3 (stale) Mirror 8: 0x636068e3 (stale) Mirror 9: 0x636068e3 (stale) Mirror 10: 0x636068e3 (stale) Mirror 11: 0x636068e3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x387ab: skipped 000420 1787366124.313238 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x387ab): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x387ab): Mirror 4: 0x636068e3 (stale) Mirror 5: 0x636068e3 (stale) Mirror 7: 0x636068e3 (stale) Mirror 8: 0x636068e3 (stale) Mirror 9: 0x636068e3 (stale) Mirror 10: 0x636068e3 (stale) Mirror 11: 0x636068e3 (stale) CRC-32 checksum value for chunk [0, 0x387ab): Mirror 1: 0xc7d26b02 Mirror 12: 0xc7d26b02 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x387ab: skipped 000422 1787366125.653204 trunc from 0x0387ab to 0x02d609 (0xb1a1 bytes) 000423 1787366125.765542 mapread 0x008940 thru 0x012747 (0x9e08 bytes) 000424 1787366125.785551 mapwrite 0x010220 thru 0x014e6e (0x4c4f bytes) 000425 1787366125.849915 write 0x01aa85 thru 0x023a21 (0x8f9d bytes) 000426 1787366125.857066 mapwrite 0x03a3d6 thru 0x03d935 (0x3560 bytes) 000427 1787366125.945184 mapwrite 0x016aee thru 0x024d02 (0xe215 bytes) 000428 1787366126.002133 write 0x0398fd thru 0x03ffff (0x6703 bytes) 000429 1787366126.008889 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xde75bf1a (stale) Mirror 5: 0xde75bf1a (stale) Mirror 7: 0xde75bf1a (stale) Mirror 9: 0xde75bf1a (stale) Mirror 10: 0xde75bf1a (stale) Mirror 11: 0xde75bf1a (stale) Mirror 12: 0xa31f7845 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xde75bf1a (stale) Mirror 5: 0xde75bf1a (stale) Mirror 7: 0xde75bf1a (stale) Mirror 9: 0xde75bf1a (stale) Mirror 10: 0xde75bf1a (stale) Mirror 11: 0xde75bf1a (stale) Mirror 12: 0xa31f7845 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000430 1787366126.707981 mapread 0x01b7ef thru 0x01da2b (0x223d bytes) 000432 1787366126.758958 read 0x02cdc9 thru 0x034e01 (0x8039 bytes) 000433 1787366126.777051 read 0x03834a thru 0x03ffff (0x7cb6 bytes) 000436 1787366126.868061 trunc from 0x040000 to 0x03004f (0xffb0 bytes) 000440 1787366126.900195 mapread 0x0029e0 thru 0x0120aa (0xf6cb bytes) 000441 1787366126.908182 mapwrite 0x01f71d thru 0x02cf7c (0xd860 bytes) 000442 1787366127.008226 write 0x037f56 thru 0x038843 (0x8ee bytes) 000443 1787366127.009766 read 0x015e8f thru 0x01b73b (0x58ad bytes) 000444 1787366127.011298 mapread 0x02e7e2 thru 0x038843 (0xa062 bytes) 000447 1787366127.031292 read 0x00260c thru 0x00de88 (0xb87d bytes) 000450 1787366127.044090 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x38844): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38844): Mirror 4: 0xfd65137b (stale) Mirror 5: 0xfd65137b (stale) Mirror 7: 0xfd65137b (stale) Mirror 9: 0xfd65137b (stale) Mirror 10: 0xfd65137b (stale) Mirror 11: 0xfd65137b (stale) Mirror 12: 0x9ebf3fee (stale) CRC-32 checksum value for chunk [0, 0x38844): Mirror 1: 0x32e024d4 Mirror 13: 0x32e024d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38844: skipped 000451 1787366128.432504 mapread 0x00c1c8 thru 0x01ad0d (0xeb46 bytes) 000452 1787366128.475858 MIRROR_RESYNC 9 mirrors 000455 1787366130.357836 trunc from 0x038844 to 0x013b8c (0x24cb7 bytes) 000456 1787366130.504809 read 0x00db5b thru 0x013b8c (0x6032 bytes) 000458 1787366130.521842 read 0x012d38 thru 0x013b8c (0xe55 bytes) 000459 1787366130.522422 write 0x00dd3a thru 0x0193b7 (0xb67e bytes) 000462 1787366130.532227 trunc from 0x0193b8 to 0x028d78 (0xf9c1 bytes) 000465 1787366130.588232 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28d79): Mirror 4: 0x9180f889 (stale) Mirror 5: 0x9180f889 (stale) Mirror 7: 0x9180f889 (stale) Mirror 10: 0x9180f889 (stale) Mirror 11: 0x9180f889 (stale) Mirror 12: 0x9180f889 (stale) Mirror 13: 0x9180f889 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28d79): Mirror 4: 0x9180f889 (stale) Mirror 5: 0x9180f889 (stale) Mirror 7: 0x9180f889 (stale) Mirror 10: 0x9180f889 (stale) Mirror 11: 0x9180f889 (stale) Mirror 12: 0x9180f889 (stale) Mirror 13: 0x9180f889 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28d79: skipped 000466 1787366131.439419 write 0x015b21 thru 0x01968c (0x3b6c bytes) 000468 1787366131.472040 write 0x019646 thru 0x01c6d4 (0x308f bytes) 000471 1787366131.500100 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28d79): Mirror 5: 0x9180f889 (stale) Mirror 7: 0x9180f889 (stale) Mirror 10: 0x9180f889 (stale) Mirror 11: 0x9180f889 (stale) Mirror 12: 0x9180f889 (stale) Mirror 13: 0x9180f889 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28d79): Mirror 5: 0x9180f889 (stale) Mirror 7: 0x9180f889 (stale) Mirror 10: 0x9180f889 (stale) Mirror 11: 0x9180f889 (stale) Mirror 12: 0x9180f889 (stale) Mirror 13: 0x9180f889 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28d79: skipped 000475 1787366132.172218 trunc from 0x028d79 to 0x0380e5 (0xf36d bytes) 000476 1787366132.220175 mapread 0x001ddd thru 0x006fc2 (0x51e6 bytes) 000478 1787366132.257741 write 0x0150ee thru 0x018bf3 (0x3b06 bytes) 000479 1787366132.270616 read 0x02be84 thru 0x031b8a (0x5d07 bytes) 000480 1787366132.284073 read 0x024d78 thru 0x0297cf (0x4a58 bytes) 000482 1787366132.303779 write 0x020381 thru 0x02e63c (0xe2bc bytes) 000483 1787366132.310039 trunc from 0x0380e6 to 0x025280 (0x12e65 bytes) 000484 1787366132.346903 read 0x006df7 thru 0x00bda3 (0x4fad bytes) 000487 1787366132.355723 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x25281): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25281): Mirror 5: 0x658f3bbe (stale) Mirror 7: 0x658f3bbe (stale) Mirror 10: 0x658f3bbe (stale) Mirror 11: 0x658f3bbe (stale) Mirror 12: 0x658f3bbe (stale) Mirror 13: 0x658f3bbe (stale) CRC-32 checksum value for chunk [0, 0x25281): Mirror 1: 0xa545cd5c Mirror 14: 0xa545cd5c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25281: skipped 000490 1787366133.424998 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25281): Mirror 5: 0x658f3bbe (stale) Mirror 7: 0x658f3bbe (stale) Mirror 10: 0x658f3bbe (stale) Mirror 12: 0x658f3bbe (stale) Mirror 13: 0x658f3bbe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25281): Mirror 5: 0x658f3bbe (stale) Mirror 7: 0x658f3bbe (stale) Mirror 10: 0x658f3bbe (stale) Mirror 12: 0x658f3bbe (stale) Mirror 13: 0x658f3bbe (stale) CRC-32 checksum value for chunk [0, 0x25281): Mirror 1: 0xa545cd5c Mirror 14: 0xa545cd5c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25281: skipped 000491 1787366134.448036 mapread 0x010639 thru 0x02002f (0xf9f7 bytes) 000493 1787366134.526158 trunc from 0x025281 to 0x000862 (0x24a1e bytes) 000495 1787366134.793416 mapwrite 0x02ded7 thru 0x03340f (0x5539 bytes) 000497 1787366135.016806 trunc from 0x033410 to 0x0159ac (0x1da63 bytes) 000498 1787366135.050569 mapwrite 0x01a97e thru 0x01c82c (0x1eaf bytes) 000500 1787366135.270244 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c82d): Mirror 7: 0xf4cc24dc (stale) Mirror 10: 0xf4cc24dc (stale) Mirror 12: 0xf4cc24dc (stale) Mirror 13: 0xf4cc24dc (stale) Mirror 14: 0x766e7ab8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c82d): Mirror 7: 0xf4cc24dc (stale) Mirror 10: 0xf4cc24dc (stale) Mirror 12: 0xf4cc24dc (stale) Mirror 13: 0xf4cc24dc (stale) Mirror 14: 0x766e7ab8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c82d: skipped 000504 1787366136.147458 MIRROR_RESYNC 6 mirrors 000505 1787366139.226154 trunc from 0x01c82d to 0x02528f (0x8a63 bytes) 000506 1787366139.433826 MIRROR_RESYNC 6 mirrors 000507 1787366142.792627 mapwrite 0x024b3c thru 0x0250f3 (0x5b8 bytes) 000508 1787366143.030601 read 0x002e85 thru 0x0044cd (0x1649 bytes) 000511 1787366143.041539 read 0x023505 thru 0x0240a8 (0xba4 bytes) 000512 1787366143.044488 mapread 0x006c3f thru 0x00bbd9 (0x4f9b bytes) 000516 1787366143.073244 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25290): Mirror 10: 0xb95ede3d (stale) Mirror 12: 0xb95ede3d (stale) Mirror 13: 0xb95ede3d (stale) Mirror 14: 0xb95ede3d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25290): Mirror 10: 0xb95ede3d (stale) Mirror 12: 0xb95ede3d (stale) Mirror 13: 0xb95ede3d (stale) Mirror 14: 0xb95ede3d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25290: skipped 000518 1787366143.574220 trunc from 0x025290 to 0x03603f (0x10db0 bytes) 000519 1787366143.603789 MIRROR_RESYNC 5 mirrors 000520 1787366145.616546 mapwrite 0x010596 thru 0x0123f1 (0x1e5c bytes) 000521 1787366145.770616 mapwrite 0x00076d thru 0x0035d8 (0x2e6c bytes) 000522 1787366145.803913 write 0x00219b thru 0x010743 (0xe5a9 bytes) 000525 1787366145.806769 mapwrite 0x0234c0 thru 0x02877a (0x52bb bytes) 000526 1787366145.892345 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x36040): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36040): Mirror 10: 0xb6bffdd2 (stale) Mirror 12: 0xb6bffdd2 (stale) Mirror 13: 0xb6bffdd2 (stale) Mirror 14: 0xb6bffdd2 (stale) CRC-32 checksum value for chunk [0, 0x36040): Mirror 1: 0x1dc467ff Mirror 15: 0x1dc467ff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36040: skipped 000527 1787366147.555543 read 0x024a3a thru 0x034920 (0xfee7 bytes) 000529 1787366147.580629 write 0x012ac5 thru 0x02029f (0xd7db bytes) 000531 1787366147.789206 read 0x007354 thru 0x013220 (0xbecd bytes) 000532 1787366147.812171 mapread 0x00c2ee thru 0x019424 (0xd137 bytes) 000533 1787366147.822801 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x36040): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36040): Mirror 10: 0xb6bffdd2 (stale) Mirror 12: 0xb6bffdd2 (stale) Mirror 13: 0xb6bffdd2 (stale) Mirror 14: 0xb6bffdd2 (stale) Mirror 15: 0x1dc467ff (stale) CRC-32 checksum value for chunk [0, 0x36040): Mirror 1: 0xaa63c4fb Mirror 16: 0xaa63c4fb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36040: skipped 000534 1787366149.054846 read 0x012770 thru 0x021124 (0xe9b5 bytes) 000536 1787366149.069624 trunc from 0x036040 to 0x020ebe (0x15181 bytes) 000538 1787366149.159762 read 0x00f2ac thru 0x01ea55 (0xf7aa bytes) 000540 1787366149.174468 read 0x000ae7 thru 0x00c2fb (0xb815 bytes) 000544 1787366149.177756 write 0x03354e thru 0x03ffff (0xcab2 bytes) 000546 1787366149.180882 trunc from 0x040000 to 0x01d018 (0x22fe7 bytes) 000547 1787366149.210031 mapwrite 0x007ee6 thru 0x012cc5 (0xade0 bytes) 000549 1787366149.273073 MIRROR_RESYNC 7 mirrors 000550 1787366150.987363 read 0x001745 thru 0x005235 (0x3af1 bytes) 000551 1787366150.996902 trunc from 0x01d019 to 0x006d31 (0x162e7 bytes) 000552 1787366151.104211 MIRROR_RESYNC 7 mirrors 000554 1787366152.155080 read 0x000b6f thru 0x006d31 (0x61c3 bytes) 000555 1787366152.170761 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000556 1787366152.483426 write 0x01f679 thru 0x023211 (0x3b99 bytes) 000557 1787366152.583577 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x23212): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23212): Mirror 12: 0x7231a31a (stale) Mirror 13: 0x7231a31a (stale) Mirror 14: 0x7231a31a (stale) Mirror 15: 0x7231a31a (stale) Mirror 16: 0x7231a31a (stale) CRC-32 checksum value for chunk [0, 0x23212): Mirror 1: 0x481d78e Mirror 17: 0x481d78e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23212: skipped 000558 1787366153.658676 write 0x024a47 thru 0x02c7db (0x7d95 bytes) 000560 1787366153.752377 mapwrite 0x014766 thru 0x01abd3 (0x646e bytes) 000561 1787366153.863611 mapread 0x01b4d0 thru 0x028a5b (0xd58c bytes) 000563 1787366153.876140 mapread 0x02b87c thru 0x02c7db (0xf60 bytes) 000564 1787366153.877472 write 0x03b8cb thru 0x03b9b1 (0xe7 bytes) 000568 1787366153.880739 mapwrite 0x039012 thru 0x03ffff (0x6fee bytes) 000572 1787366153.988631 mapread 0x015435 thru 0x01d90f (0x84db bytes) 000573 1787366153.990814 trunc from 0x040000 to 0x03a58e (0x5a71 bytes) 000575 1787366154.019668 mapwrite 0x022741 thru 0x0255b9 (0x2e79 bytes) 000576 1787366154.072827 write 0x018394 thru 0x01d759 (0x53c6 bytes) 000578 1787366154.079706 read 0x02ae69 thru 0x033957 (0x8aef bytes) 000579 1787366154.099782 read 0x00ef6b thru 0x016b08 (0x7b9e bytes) 000580 1787366154.117611 write 0x02df61 thru 0x03d25a (0xf2fa bytes) 000582 1787366154.125610 MIRROR_RESYNC 7 mirrors 000583 1787366156.211111 mapread 0x03ba98 thru 0x03d25a (0x17c3 bytes) 000584 1787366156.231678 read 0x001ad6 thru 0x0036a5 (0x1bd0 bytes) 000585 1787366156.236746 trunc from 0x03d25b to 0x01fabc (0x1d79e bytes) 000586 1787366156.303404 read 0x0106ce thru 0x018d11 (0x8644 bytes) 000590 1787366156.312405 trunc from 0x01fabd to 0x0238e0 (0x3e24 bytes) 000591 1787366156.326732 read 0x010e8a thru 0x01a8fd (0x9a74 bytes) 000593 1787366156.327812 mapread 0x00c314 thru 0x01228e (0x5f7b bytes) 000594 1787366156.328691 mapread 0x0176c8 thru 0x01c68e (0x4fc7 bytes) 000595 1787366156.329665 mapwrite 0x022bb2 thru 0x02a917 (0x7d66 bytes) 000597 1787366156.415716 read 0x010213 thru 0x01c2c8 (0xc0b6 bytes) 000600 1787366156.425622 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a918): Mirror 13: 0xb9869b31 (stale) Mirror 14: 0xb9869b31 (stale) Mirror 15: 0xb9869b31 (stale) Mirror 16: 0xb9869b31 (stale) Mirror 17: 0xb9869b31 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a918): Mirror 13: 0xb9869b31 (stale) Mirror 14: 0xb9869b31 (stale) Mirror 15: 0xb9869b31 (stale) Mirror 16: 0xb9869b31 (stale) Mirror 17: 0xb9869b31 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a918: skipped 000602 1787366156.921812 write 0x01f7fb thru 0x02a4f3 (0xacf9 bytes) 000604 1787366156.947040 read 0x023a67 thru 0x02671d (0x2cb7 bytes) 000605 1787366156.955497 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2a918): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a918): Mirror 13: 0xb9869b31 (stale) Mirror 14: 0xb9869b31 (stale) Mirror 15: 0xb9869b31 (stale) Mirror 16: 0xb9869b31 (stale) Mirror 17: 0xb9869b31 (stale) CRC-32 checksum value for chunk [0, 0x2a918): Mirror 1: 0xba09201d Mirror 18: 0xba09201d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a918: skipped 000606 1787366158.166829 write 0x036a27 thru 0x03ffff (0x95d9 bytes) 000607 1787366158.268841 read 0x0229ba thru 0x028c0a (0x6251 bytes) 000608 1787366158.284806 read 0x0095f1 thru 0x009dff (0x80f bytes) 000611 1787366158.298089 read 0x03176d thru 0x03ba1b (0xa2af bytes) 000612 1787366158.299127 trunc from 0x040000 to 0x00df94 (0x3206b bytes) 000613 1787366158.347196 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xdf95): Mirror 13: 0xb936efe9 (stale) Mirror 14: 0xb936efe9 (stale) Mirror 16: 0xb936efe9 (stale) Mirror 17: 0xb936efe9 (stale) Mirror 18: 0xb936efe9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdf95): Mirror 13: 0xb936efe9 (stale) Mirror 14: 0xb936efe9 (stale) Mirror 16: 0xb936efe9 (stale) Mirror 17: 0xb936efe9 (stale) Mirror 18: 0xb936efe9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdf95: skipped 000614 1787366158.802127 mapread 0x00ccdf thru 0x00df94 (0x12b6 bytes) 000615 1787366158.824104 mapwrite 0x00c866 thru 0x018cea (0xc485 bytes) 000616 1787366158.933975 read 0x016fd6 thru 0x018cea (0x1d15 bytes) 000618 1787366158.936047 trunc from 0x018ceb to 0x010903 (0x83e7 bytes) 000619 1787366158.962321 read 0x00c1d6 thru 0x00db77 (0x19a2 bytes) 000620 1787366158.962950 write 0x02bc11 thru 0x0381ce (0xc5be bytes) 000621 1787366158.964598 write 0x024976 thru 0x031793 (0xce1e bytes) 000624 1787366158.983790 MIRROR_RESYNC 6 mirrors 000625 1787366161.024492 mapread 0x03807e thru 0x0381ce (0x151 bytes) 000627 1787366161.031701 mapwrite 0x039563 thru 0x03ffff (0x6a9d bytes) 000628 1787366161.248609 write 0x02a366 thru 0x032c20 (0x88bb bytes) 000629 1787366161.272108 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xb15cec96 (stale) Mirror 14: 0xb15cec96 (stale) Mirror 16: 0xb15cec96 (stale) Mirror 18: 0xb15cec96 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xb15cec96 (stale) Mirror 14: 0xb15cec96 (stale) Mirror 16: 0xb15cec96 (stale) Mirror 18: 0xb15cec96 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000631 1787366161.939242 read 0x0322b8 thru 0x03a75d (0x84a6 bytes) 000634 1787366161.949995 read 0x00cb4c thru 0x013693 (0x6b48 bytes) 000635 1787366161.959985 write 0x029e8b thru 0x02e6bc (0x4832 bytes) 000636 1787366161.961883 read 0x010a2f thru 0x0178f7 (0x6ec9 bytes) 000637 1787366161.962493 read 0x014fe1 thru 0x023567 (0xe587 bytes) 000641 1787366161.968566 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xb15cec96 (stale) Mirror 14: 0xb15cec96 (stale) Mirror 16: 0xb15cec96 (stale) Mirror 18: 0xb15cec96 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2e4a9b6d Mirror 19: 0x2e4a9b6d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000642 1787366163.442575 write 0x039628 thru 0x03ffff (0x69d8 bytes) 000645 1787366163.556228 write 0x013553 thru 0x01feae (0xc95c bytes) 000646 1787366163.569781 trunc from 0x040000 to 0x033e94 (0xc16b bytes) 000647 1787366163.596488 mapwrite 0x02ff4d thru 0x033c3e (0x3cf2 bytes) 000649 1787366163.773254 write 0x03463e thru 0x03ffff (0xb9c2 bytes) 000650 1787366163.779667 write 0x017fad thru 0x01cc3b (0x4c8f bytes) 000652 1787366163.792320 MIRROR_RESYNC 6 mirrors 000653 1787366165.226686 MIRROR_EXTEND to 7 mirrors 000655 1787366166.339951 trunc from 0x040000 to 0x035672 (0xa98d bytes) 000656 1787366166.467239 mapread 0x01d362 thru 0x020f8e (0x3c2d bytes) 000657 1787366166.491720 write 0x023e77 thru 0x02ded7 (0xa061 bytes) 000658 1787366166.505306 mapread 0x007440 thru 0x015fdd (0xeb9e bytes) 000659 1787366166.538521 mapwrite 0x006935 thru 0x00b450 (0x4b1c bytes) 000660 1787366166.591543 read 0x0326d3 thru 0x035672 (0x2fa0 bytes) 000662 1787366166.608654 read 0x021f2b thru 0x02e812 (0xc8e8 bytes) 000663 1787366166.619526 read 0x018a9b thru 0x018aec (0x52 bytes) 000666 1787366166.622565 write 0x00f884 thru 0x012554 (0x2cd1 bytes) 000667 1787366166.625212 read 0x034c73 thru 0x035672 (0xa00 bytes) 000668 1787366166.632031 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x35673): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35673): Mirror 13: 0x98c4ac0e (stale) Mirror 14: 0x98c4ac0e (stale) Mirror 16: 0x98c4ac0e (stale) Mirror 18: 0x98c4ac0e (stale) Mirror 19: 0x98c4ac0e (stale) Mirror 20: 0x98c4ac0e (stale) CRC-32 checksum value for chunk [0, 0x35673): Mirror 1: 0xbae1ef73 Mirror 21: 0xbae1ef73 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35673: skipped 000669 1787366167.754955 write 0x003748 thru 0x007dce (0x4687 bytes) 000670 1787366167.832362 read 0x0319d2 thru 0x035672 (0x3ca1 bytes) 000673 1787366167.844119 read 0x01a533 thru 0x01ead0 (0x459e bytes) 000675 1787366167.863937 write 0x013c0c thru 0x021630 (0xda25 bytes) 000678 1787366167.868773 trunc from 0x035673 to 0x027781 (0xdef1 bytes) 000679 1787366167.899622 mapread 0x020fa3 thru 0x021833 (0x891 bytes) 000680 1787366167.901745 read 0x00b991 thru 0x01163c (0x5cac bytes) 000682 1787366167.910097 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27782): Mirror 13: 0xc7ca18bb (stale) Mirror 14: 0xc7ca18bb (stale) Mirror 16: 0xc7ca18bb (stale) Mirror 18: 0xc7ca18bb (stale) Mirror 19: 0xc7ca18bb (stale) Mirror 20: 0xc7ca18bb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27782): Mirror 13: 0xc7ca18bb (stale) Mirror 14: 0xc7ca18bb (stale) Mirror 16: 0xc7ca18bb (stale) Mirror 18: 0xc7ca18bb (stale) Mirror 19: 0xc7ca18bb (stale) Mirror 20: 0xc7ca18bb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27782: skipped 000683 1787366168.400556 mapread 0x00cce2 thru 0x015590 (0x88af bytes) 000684 1787366168.426225 trunc from 0x027782 to 0x0044a4 (0x232dd bytes) 000685 1787366168.442820 trunc from 0x0044a5 to 0x00b32a (0x6e86 bytes) 000687 1787366168.462350 mapwrite 0x018179 thru 0x01eb7b (0x6a03 bytes) 000688 1787366168.568680 write 0x03b982 thru 0x03ffff (0x467e bytes) 000689 1787366168.573726 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xde04fb13 (stale) Mirror 14: 0xde04fb13 (stale) Mirror 16: 0xde04fb13 (stale) Mirror 18: 0xde04fb13 (stale) Mirror 20: 0xde04fb13 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xde04fb13 (stale) Mirror 14: 0xde04fb13 (stale) Mirror 16: 0xde04fb13 (stale) Mirror 18: 0xde04fb13 (stale) Mirror 20: 0xde04fb13 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000691 1787366168.993311 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xde04fb13 (stale) Mirror 14: 0xde04fb13 (stale) Mirror 16: 0xde04fb13 (stale) Mirror 18: 0xde04fb13 (stale) Mirror 20: 0xde04fb13 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3bb653db Mirror 21: 0x3bb653db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000693 1787366170.331412 trunc from 0x040000 to 0x0153b5 (0x2ac4a bytes) 000694 1787366170.453089 read 0x009ce8 thru 0x0153b5 (0xb6ce bytes) 000696 1787366170.478491 read 0x01153f thru 0x0153b5 (0x3e77 bytes) 000699 1787366170.481322 mapread 0x00a546 thru 0x0153b5 (0xae70 bytes) 000703 1787366170.485525 mapwrite 0x00ab13 thru 0x00f015 (0x4503 bytes) 000704 1787366170.583928 mapread 0x0138b0 thru 0x0153b5 (0x1b06 bytes) 000705 1787366170.584781 write 0x03de96 thru 0x03ffff (0x216a bytes) 000707 1787366170.586578 mapwrite 0x03247a thru 0x0399a4 (0x752b bytes) 000709 1787366170.684967 write 0x0307f7 thru 0x030b35 (0x33f bytes) 000710 1787366170.685876 read 0x013eec thru 0x01f9f7 (0xbb0c bytes) 000711 1787366170.686807 write 0x02417a thru 0x02b891 (0x7718 bytes) 000713 1787366170.706114 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xde04fb13 (stale) Mirror 14: 0xde04fb13 (stale) Mirror 16: 0xde04fb13 (stale) Mirror 18: 0xde04fb13 (stale) Mirror 20: 0xde04fb13 (stale) Mirror 21: 0x3bb653db (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5ba44ee3 Mirror 22: 0x5ba44ee3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000714 1787366171.906145 MIRROR_RESYNC 8 mirrors 000716 1787366173.963539 read 0x00a217 thru 0x011fa5 (0x7d8f bytes) 000717 1787366173.982017 MIRROR_RESYNC 8 mirrors 000718 1787366174.662192 mapread 0x030468 thru 0x0307cb (0x364 bytes) 000719 1787366174.677214 mapwrite 0x038e4c thru 0x03f69f (0x6854 bytes) 000721 1787366174.858195 MIRROR_RESYNC 8 mirrors 000722 1787366177.720189 mapwrite 0x021927 thru 0x022fdf (0x16b9 bytes) 000723 1787366177.928529 mapread 0x018ddc thru 0x01afae (0x21d3 bytes) 000725 1787366177.942003 write 0x03eb28 thru 0x03f6b7 (0xb90 bytes) 000726 1787366177.953309 mapread 0x03c954 thru 0x03ffff (0x36ac bytes) 000731 1787366177.968504 write 0x026a15 thru 0x029763 (0x2d4f bytes) 000732 1787366177.970034 read 0x0168f8 thru 0x019fc3 (0x36cc bytes) 000733 1787366177.971207 trunc from 0x040000 to 0x026b4f (0x194b0 bytes) 000736 1787366178.001919 read 0x01b157 thru 0x01c6aa (0x1554 bytes) 000737 1787366178.002883 mapwrite 0x01f7aa thru 0x02848b (0x8ce2 bytes) 000738 1787366178.081289 write 0x0122ef thru 0x01d987 (0xb699 bytes) 000739 1787366178.089139 MIRROR_RESYNC 8 mirrors 000740 1787366179.374366 write 0x00ba31 thru 0x014d27 (0x92f7 bytes) 000741 1787366179.498968 read 0x0075be thru 0x0083d1 (0xe14 bytes) 000742 1787366179.513457 mapwrite 0x038e7b thru 0x03ffff (0x7185 bytes) 000744 1787366179.790550 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xc1a526cb (stale) Mirror 14: 0xc1a526cb (stale) Mirror 16: 0xc1a526cb (stale) Mirror 20: 0xc1a526cb (stale) Mirror 21: 0xc1a526cb (stale) Mirror 22: 0xc1a526cb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xc1a526cb (stale) Mirror 14: 0xc1a526cb (stale) Mirror 16: 0xc1a526cb (stale) Mirror 20: 0xc1a526cb (stale) Mirror 21: 0xc1a526cb (stale) Mirror 22: 0xc1a526cb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000745 1787366180.456601 read 0x03b51e thru 0x03ffff (0x4ae2 bytes) 000747 1787366180.466323 mapread 0x0093e5 thru 0x0104a6 (0x70c2 bytes) 000748 1787366180.492945 write 0x03f4c3 thru 0x03ffff (0xb3d bytes) 000749 1787366180.498010 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xc1a526cb (stale) Mirror 14: 0xc1a526cb (stale) Mirror 16: 0xc1a526cb (stale) Mirror 20: 0xc1a526cb (stale) Mirror 22: 0xc1a526cb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xc1a526cb (stale) Mirror 14: 0xc1a526cb (stale) Mirror 16: 0xc1a526cb (stale) Mirror 20: 0xc1a526cb (stale) Mirror 22: 0xc1a526cb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000755 1787366180.889508 read 0x029f55 thru 0x038c48 (0xecf4 bytes) 000756 1787366180.899884 read 0x004caa thru 0x0074ef (0x2846 bytes) 000757 1787366180.914023 read 0x00c82b thru 0x016b73 (0xa349 bytes) 000758 1787366180.914791 trunc from 0x040000 to 0x0099d1 (0x3662e bytes) 000760 1787366180.937679 mapwrite 0x0107f9 thru 0x0124a3 (0x1cab bytes) 000762 1787366181.048174 mapread 0x00b93d thru 0x00e6b1 (0x2d75 bytes) 000764 1787366181.049731 mapread 0x005b7b thru 0x008c34 (0x30ba bytes) 000765 1787366181.050768 mapwrite 0x03c1d1 thru 0x03f8fc (0x372c bytes) 000767 1787366181.204053 trunc from 0x03f8fd to 0x0271ab (0x18751 bytes) 000768 1787366181.230801 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x271ac): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x271ac): Mirror 13: 0x7ff89148 (stale) Mirror 14: 0x7ff89148 (stale) Mirror 16: 0x7ff89148 (stale) Mirror 20: 0x7ff89148 (stale) Mirror 22: 0x7ff89148 (stale) CRC-32 checksum value for chunk [0, 0x271ac): Mirror 1: 0x83d698c2 Mirror 23: 0x83d698c2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x271ac: skipped 000769 1787366182.281663 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x271ac): Mirror 13: 0x7ff89148 (stale) Mirror 14: 0x7ff89148 (stale) Mirror 20: 0x7ff89148 (stale) Mirror 22: 0x7ff89148 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x271ac): Mirror 13: 0x7ff89148 (stale) Mirror 14: 0x7ff89148 (stale) Mirror 20: 0x7ff89148 (stale) Mirror 22: 0x7ff89148 (stale) CRC-32 checksum value for chunk [0, 0x271ac): Mirror 1: 0x83d698c2 Mirror 23: 0x83d698c2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x271ac: skipped 000771 1787366182.806488 write 0x02a1d2 thru 0x030803 (0x6632 bytes) 000778 1787366182.903957 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30804): Mirror 13: 0x8a3e8846 (stale) Mirror 20: 0x8a3e8846 (stale) Mirror 22: 0x8a3e8846 (stale) Mirror 23: 0xf6ab0bc3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30804): Mirror 13: 0x8a3e8846 (stale) Mirror 20: 0x8a3e8846 (stale) Mirror 22: 0x8a3e8846 (stale) Mirror 23: 0xf6ab0bc3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30804: skipped 000780 1787366183.333138 read 0x01aab2 thru 0x01f72b (0x4c7a bytes) 000781 1787366183.351510 MIRROR_RESYNC 5 mirrors 000782 1787366185.075401 trunc from 0x030804 to 0x004472 (0x2c391 bytes) 000783 1787366185.161082 mapread 0x0011d5 thru 0x004472 (0x329e bytes) 000785 1787366185.176421 write 0x024461 thru 0x02515f (0xcff bytes) 000789 1787366185.179412 write 0x00e03f thru 0x0182c4 (0xa286 bytes) 000790 1787366185.196634 MIRROR_RESYNC 5 mirrors 000791 1787366186.397736 mapwrite 0x0384eb thru 0x03ffff (0x7b15 bytes) 000792 1787366186.591017 mapwrite 0x01814d thru 0x01dc4d (0x5b01 bytes) 000794 1787366186.661105 write 0x005a48 thru 0x00985c (0x3e15 bytes) 000796 1787366186.674706 mapwrite 0x0126f3 thru 0x01dfe3 (0xb8f1 bytes) 000797 1787366186.730971 mapwrite 0x0322ec thru 0x03ffff (0xdd14 bytes) 000799 1787366186.801829 write 0x01b0a1 thru 0x02184d (0x67ad bytes) 000801 1787366186.810232 mapread 0x0373e4 thru 0x039f78 (0x2b95 bytes) 000803 1787366186.813179 mapwrite 0x00a904 thru 0x010113 (0x5810 bytes) 000804 1787366186.879898 trunc from 0x040000 to 0x0300b3 (0xff4c bytes) 000805 1787366186.904508 write 0x03f27f thru 0x03ffff (0xd81 bytes) 000806 1787366186.912103 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xd2718634 (stale) Mirror 20: 0xd2718634 (stale) Mirror 22: 0xd2718634 (stale) Mirror 23: 0xd2718634 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe79715a7 Mirror 24: 0xe79715a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000807 1787366188.209893 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0xd2718634 (stale) Mirror 22: 0xd2718634 (stale) Mirror 23: 0xd2718634 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0xd2718634 (stale) Mirror 22: 0xd2718634 (stale) Mirror 23: 0xd2718634 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe79715a7 Mirror 24: 0xe79715a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000810 1787366188.826298 mapwrite 0x008aa7 thru 0x009795 (0xcef bytes) 000811 1787366189.053762 write 0x038448 thru 0x03ffff (0x7bb8 bytes) 000812 1787366189.065254 read 0x02271e thru 0x031e37 (0xf71a bytes) 000813 1787366189.075649 mapread 0x036292 thru 0x03a023 (0x3d92 bytes) 000817 1787366189.088719 read 0x01e7cf thru 0x02022a (0x1a5c bytes) 000818 1787366189.105856 MIRROR_RESYNC 5 mirrors 000819 1787366190.583662 read 0x00edd4 thru 0x013aee (0x4d1b bytes) 000823 1787366190.598517 trunc from 0x040000 to 0x00dca4 (0x3235b bytes) 000824 1787366190.683645 write 0x02c1bf thru 0x031a0a (0x584c bytes) 000825 1787366190.685167 trunc from 0x031a0b to 0x002db0 (0x2ec5a bytes) 000826 1787366190.714751 mapwrite 0x01a82e thru 0x02785b (0xd02e bytes) 000828 1787366190.826254 read 0x01690d thru 0x018f25 (0x2619 bytes) 000829 1787366190.827166 mapread 0x02717b thru 0x02785b (0x6e1 bytes) 000832 1787366190.829774 write 0x03d517 thru 0x03ffff (0x2ae9 bytes) 000833 1787366190.831884 mapread 0x00bc92 thru 0x010191 (0x4500 bytes) 000834 1787366190.851797 trunc from 0x040000 to 0x0142f4 (0x2bd0b bytes) 000835 1787366190.869990 mapwrite 0x01a638 thru 0x021378 (0x6d41 bytes) 000836 1787366190.968887 trunc from 0x021379 to 0x0122f6 (0xf082 bytes) 000838 1787366190.983062 trunc from 0x0122f7 to 0x0204cf (0xe1d9 bytes) 000839 1787366191.000663 write 0x029cc3 thru 0x031990 (0x7cce bytes) 000841 1787366191.002719 trunc from 0x031991 to 0x032ed5 (0x1545 bytes) 000842 1787366191.025835 mapread 0x001b97 thru 0x0078ed (0x5d57 bytes) 000843 1787366191.031053 read 0x0145d3 thru 0x022214 (0xdc42 bytes) 000844 1787366191.046896 read 0x014a21 thru 0x01bcee (0x72ce bytes) 000846 1787366191.048215 trunc from 0x032ed6 to 0x03e407 (0xb532 bytes) 000848 1787366191.066532 mapread 0x0193ee thru 0x01ffd5 (0x6be8 bytes) 000849 1787366191.067617 read 0x02d253 thru 0x03bdd7 (0xeb85 bytes) 000851 1787366191.077135 trunc from 0x03e408 to 0x033b6d (0xa89a bytes) 000853 1787366191.098506 read 0x00c4a5 thru 0x0147f0 (0x834c bytes) 000854 1787366191.099107 mapwrite 0x01c304 thru 0x0205c1 (0x42be bytes) 000855 1787366191.147519 mapread 0x011232 thru 0x0209e0 (0xf7af bytes) 000856 1787366191.150194 mapwrite 0x01b992 thru 0x01d3b7 (0x1a26 bytes) 000858 1787366191.250851 trunc from 0x033b6e to 0x01828d (0x1b8e0 bytes) 000859 1787366191.271773 write 0x03d917 thru 0x03ffff (0x26e9 bytes) 000862 1787366191.279501 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0x19cf5fd (stale) Mirror 23: 0x19cf5fd (stale) Mirror 24: 0x19cf5fd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0x19cf5fd (stale) Mirror 23: 0x19cf5fd (stale) Mirror 24: 0x19cf5fd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000864 1787366191.672410 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: 0x19cf5fd (stale) Mirror 24: 0x19cf5fd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: 0x19cf5fd (stale) Mirror 24: 0x19cf5fd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000865 1787366192.044484 read 0x00a402 thru 0x011e4f (0x7a4e bytes) 000867 1787366192.064825 trunc from 0x040000 to 0x02469e (0x1b961 bytes) 000868 1787366192.092686 read 0x012de6 thru 0x017476 (0x4691 bytes) 000869 1787366192.124048 MIRROR_RESYNC 3 mirrors 000870 1787366193.264861 mapwrite 0x0311dd thru 0x0383d0 (0x71f4 bytes) 000872 1787366193.471700 trunc from 0x0383d1 to 0x0340db (0x42f5 bytes) 000874 1787366193.500356 mapwrite 0x01cb5e thru 0x024bfa (0x809d bytes) 000875 1787366193.565153 write 0x01246b thru 0x01bd64 (0x98fa bytes) 000876 1787366193.571084 write 0x02016d thru 0x02d955 (0xd7e9 bytes) 000877 1787366193.572845 read 0x02ee9c thru 0x0340db (0x5240 bytes) 000879 1787366193.575148 mapread 0x03058e thru 0x0340db (0x3b4e bytes) 000880 1787366193.582737 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x340dc): Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x340dc): Mirror 23: 0x7acd1bb3 (stale) Mirror 24: 0x7acd1bb3 (stale) CRC-32 checksum value for chunk [0, 0x340dc): Mirror 1: 0x8a9daff5 Mirror 25: 0x8a9daff5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x340dc: skipped 000881 1787366194.705031 mapwrite 0x037c6d thru 0x03e521 (0x68b5 bytes) 000882 1787366194.892157 read 0x025c75 thru 0x02cfd9 (0x7365 bytes) 000883 1787366194.911817 read 0x03b77d thru 0x03e521 (0x2da5 bytes) 000884 1787366194.912763 read 0x028ca3 thru 0x0365b9 (0xd917 bytes) 000885 1787366194.933239 MIRROR_RESYNC 4 mirrors 000886 1787366196.473531 read 0x01bcff thru 0x029e0e (0xe110 bytes) 000887 1787366196.487957 MIRROR_EXTEND to 5 mirrors 000888 1787366197.450320 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000889 1787366197.910012 mapwrite 0x037e6d thru 0x039339 (0x14cd bytes) 000891 1787366198.191993 read 0x02b268 thru 0x0341cd (0x8f66 bytes) 000892 1787366198.221138 MIRROR_RESYNC 4 mirrors 000893 1787366200.264710 mapwrite 0x00569a thru 0x011118 (0xba7f bytes) 000895 1787366200.446142 mapread 0x0009a3 thru 0x00cc46 (0xc2a4 bytes) 000896 1787366200.458565 write 0x01e6f9 thru 0x02a471 (0xbd79 bytes) 000901 1787366200.473707 trunc from 0x03e522 to 0x0143a7 (0x2a17a bytes) 000905 1787366200.509029 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x143a8): Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x143a8): Mirror 24: 0xa095ac1c (stale) Mirror 25: 0xa095ac1c (stale) Mirror 26: 0xa095ac1c (stale) CRC-32 checksum value for chunk [0, 0x143a8): Mirror 1: 0x72208cec Mirror 27: 0x72208cec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x143a8: skipped 000908 1787366201.748312 mapwrite 0x002ab2 thru 0x007cab (0x51fa bytes) 000909 1787366202.005741 mapwrite 0x00abce thru 0x015dd1 (0xb204 bytes) 000910 1787366202.100577 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15dd2): Mirror 24: 0x1c2204c6 (stale) Mirror 26: 0x1c2204c6 (stale) Mirror 27: 0x2fe943b1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15dd2): Mirror 24: 0x1c2204c6 (stale) Mirror 26: 0x1c2204c6 (stale) Mirror 27: 0x2fe943b1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15dd2: skipped 000911 1787366202.527375 write 0x01a294 thru 0x02836d (0xe0da bytes) 000912 1787366202.532944 mapwrite 0x030d85 thru 0x032e47 (0x20c3 bytes) 000913 1787366202.614774 read 0x01b635 thru 0x02a09a (0xea66 bytes) 000914 1787366202.621107 mapwrite 0x02ea36 thru 0x039104 (0xa6cf bytes) 000918 1787366202.705729 mapwrite 0x012c08 thru 0x01ca3b (0x9e34 bytes) 000921 1787366202.783082 mapread 0x016fc2 thru 0x021fce (0xb00d bytes) 000922 1787366202.790865 read 0x02540d thru 0x026a39 (0x162d bytes) 000924 1787366202.794021 mapwrite 0x031fa2 thru 0x03aaed (0x8b4c bytes) 000925 1787366202.882324 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3aaee): Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aaee): Mirror 24: 0xbb8e929f (stale) Mirror 26: 0xbb8e929f (stale) Mirror 27: 0x34c53b89 (stale) CRC-32 checksum value for chunk [0, 0x3aaee): Mirror 1: 0x85194b3e Mirror 28: 0x85194b3e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aaee: skipped 000928 1787366203.877539 read 0x0398ae thru 0x03aaed (0x1240 bytes) 000929 1787366203.883752 trunc from 0x03aaee to 0x0111c4 (0x29929 bytes) 000934 1787366203.979826 MIRROR_RESYNC 5 mirrors 000937 1787366205.054817 write 0x035aa6 thru 0x0391c7 (0x3722 bytes) 000939 1787366205.160464 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x391c8): Mirror 26: 0x10090b81 (stale) Mirror 27: 0x10090b81 (stale) Mirror 28: 0x10090b81 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x391c8): Mirror 26: 0x10090b81 (stale) Mirror 27: 0x10090b81 (stale) Mirror 28: 0x10090b81 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x391c8: skipped 000942 1787366205.657696 mapread 0x0004bd thru 0x008c82 (0x87c6 bytes) 000945 1787366205.706187 mapread 0x014c9a thru 0x020e1e (0xc185 bytes) 000948 1787366205.738830 trunc from 0x0391c8 to 0x01aa42 (0x1e785 bytes) 000949 1787366205.761657 trunc from 0x01aa43 to 0x016b64 (0x3ede bytes) 000953 1787366205.784610 mapwrite 0x02a10e thru 0x031307 (0x71fa bytes) 000955 1787366205.878008 read 0x024446 thru 0x02a36d (0x5f28 bytes) 000957 1787366205.879188 write 0x000b2c thru 0x0038be (0x2d93 bytes) 000961 1787366205.883913 trunc from 0x031308 to 0x00991e (0x279e9 bytes) 000962 1787366205.908377 mapread 0x0087ae thru 0x00991e (0x1171 bytes) 000963 1787366205.914077 MIRROR_RESYNC 4 mirrors truncating to largest ever: 0x3fb59 000964 1787366206.822852 trunc from 0x00991f to 0x03fb58 (0x3623a bytes) 000965 1787366206.906600 read 0x02c872 thru 0x02e6b2 (0x1e41 bytes) 000966 1787366206.917093 MIRROR_RESYNC 4 mirrors 000967 1787366208.196548 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000971 1787366208.765332 write 0x037b3e thru 0x03ffff (0x84c2 bytes) 000976 1787366208.905046 mapread 0x0284a8 thru 0x033edc (0xba35 bytes) 000979 1787366208.987492 mapread 0x0379b0 thru 0x03ffff (0x8650 bytes) 000980 1787366208.991661 mapread 0x00bc5a thru 0x00c638 (0x9df bytes) 000981 1787366209.014094 trunc from 0x040000 to 0x00bade (0x34521 bytes) 000982 1787366209.059116 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xbadf): Mirror 26: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbadf): Mirror 26: 0x2a2f9b23 (stale) Mirror 28: 0x2a2f9b23 (stale) CRC-32 checksum value for chunk [0, 0xbadf): Mirror 1: 0x2a2f9b23 Mirror 29: 0x2a2f9b23 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbadf: skipped 000983 1787366210.239028 trunc from 0x00badf to 0x003756 (0x8388 bytes) 000986 1787366210.367533 mapwrite 0x02c216 thru 0x02c9de (0x7c9 bytes) 000988 1787366210.447753 mapread 0x00ce13 thru 0x015083 (0x8271 bytes) 000989 1787366210.486009 mapwrite 0x03f4cb thru 0x03ffff (0xb35 bytes) 000990 1787366210.570311 read 0x000a76 thru 0x0019a8 (0xf33 bytes) 000991 1787366210.578681 read 0x010f58 thru 0x01df23 (0xcfcc bytes) 000995 1787366210.595616 trunc from 0x040000 to 0x02afee (0x15011 bytes) 000996 1787366210.620805 MIRROR_RESYNC 4 mirrors 000997 1787366212.197805 read 0x000d4d thru 0x0083af (0x7663 bytes) 000999 1787366212.226496 write 0x0271e9 thru 0x032321 (0xb139 bytes) 001000 1787366212.409038 mapwrite 0x02cbb7 thru 0x03328a (0x66d4 bytes) All operations completed A-OK! PASS 70a (156s) == sanity-flr test 71: check mirror extend parent fid ==== 22:36:58 (1787366218) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (5s) == sanity-flr test 72: check append on FLR file ========== 22:37:03 (1787366223) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2be:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.413221 s, 5.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2be:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219929 s, 4.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bf:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2be:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bf:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2be:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a4:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 22:37:14 (1787366234) Starting client: oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre2 Starting client: oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 2652352 .. Extending file size to 6420704 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8373024 .. Extending file size to 3586944 .. Extending file size to 5292416 .. Extending file size to 6331776 .. Extending file size to 6967264 .. Extending file size to 8383456 .. Extending file size to 7017664 .. Extending file size to 7604576 .. Extending file size to 8426528 .. Extending file size to 9252896 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8969920 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58694 58696 58697 58698 58700 done 192.168.201.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg137-client.virtnet /mnt/lustre2 (opts:) 192.168.201.137@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg137-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 22:38:37 (1787366317) Starting client: oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre2 Starting client: oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2581664 .. Extending file size to 2581664 .. done Extending file size to 5460352 .. Extending file size to 5460352 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 6232768 .. Extending file size to 6232768 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6558144 .. Extending file size to 6558144 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8013472 .. Extending file size to 8013472 .. Extending file size to 8677600 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8677600 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9264800 .. Extending file size to 9264800 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 60376 60378 60379 60380 60383 done 192.168.201.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg137-client.virtnet /mnt/lustre2 (opts:) 192.168.201.137@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg137-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 22:39:55 (1787366395) Starting client: oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0972915 s, 21.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.456279 s, 4.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0xccf882c2 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.201.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg137-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 22:40:16 (1787366416) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.155863 s, 13.5 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 22:40:24 (1787366424) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d0:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2db:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2db:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dc:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2bf:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 22:40:33 (1787366433) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.22137 s, 9.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 22:40:41 (1787366441) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.341152 s, 6.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 22:40:51 (1787366451) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.373812 s, 5.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 22:41:27 (1787366487) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg137-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178716 s, 5.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } PASS 204d (63s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 22:42:30 (1787366550) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg137-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.311135 s, 6.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (73s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 22:43:43 (1787366623) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg137-server mds-ost sync done. Creating new pool test_204f oleg137-server: Pool lustre.test_204f created Adding targets to pool oleg137-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg137-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.445885 s, 4.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg137-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg137-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg137-server: Pool lustre.test_204f destroyed PASS 204f (87s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 22:45:10 (1787366710) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 22:45:18 (1787366718) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 22:45:27 (1787366727) oleg137-server: Pool lustre.test_206 created Adding targets to pool oleg137-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg137-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg137-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg137-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg137-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 22:45:54 (1787366754) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989332 s, 10.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376629 s, 27.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x4c51f83e Mirror 3: 0x4c51f83e PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 22:46:07 (1787366767) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 22:46:12 (1787366772) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 22:46:15 (1787366775) Starting client: oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.245283 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1550000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1280000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1280000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1280000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1880000 Read this (4 bytes): BBBB 192.168.201.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg137-client.virtnet /mnt/lustre2 (opts:) PASS 209a (52s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 22:47:07 (1787366827) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0262207 s, 312 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 22:47:15 (1787366835) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583604 s, 18.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f5 lmm_fid: [0x200000404:0x1f5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f5 lmm_fid: [0x200000404:0x1f5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834421 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-21 22:47:17.000000000 -0400 Modify: 2026-08-21 22:47:17.000000000 -0400 Change: 2026-08-21 22:47:18.000000000 -0400 Birth: 2026-08-21 22:47:17.000000000 -0400 PASS 210a (12s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 22:47:27 (1787366847) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (5s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 22:47:33 (1787366853) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 1.07339 s, 38.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.118444 s, 34.6 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 21 22:47 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f7:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 21 22:47 /mnt/lustre/f211.sanity-flr PASS 211 (14s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 22:47:47 (1787366867) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 22:47:56 (1787366876) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (13s) == sanity-flr test complete, duration 1892 sec =========== 22:48:10 (1787366890) === sanity-flr: start cleanup 22:48:11 (1787366891) === === sanity-flr: finish cleanup 22:48:19 (1787366899) ===