-----============= acceptance-small: sanity-flr ============----- Sat Aug 8 02:21:30 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg448-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 02:21:54 (1786170114) === oleg448-client.virtnet: executing check_config_client /mnt/lustre oleg448-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg448-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff98c68d5af000.idle_timeout=debug osc.lustre-OST0001-osc-ffff98c68d5af000.idle_timeout=debug disable quota as required oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 02:22:35 (1786170155) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7656] == sanity-flr test 0a: lfs mirror create with -N option == 02:22:37 (1786170157) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 02:22:56 (1786170176) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 02:23:00 (1786170180) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 02:23:04 (1786170184) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 02:23:33 (1786170213) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (5s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 02:23:37 (1786170217) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 02:23:41 (1786170221) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff98c68d5af000.stats=clear osc.lustre-OST0001-osc-ffff98c68d5af000.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 02:23:51 (1786170231) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (16s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 02:24:07 (1786170247) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 02:24:17 (1786170257) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 02:24:22 (1786170262) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 02:24:31 (1786170271) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (12s) == sanity-flr test 2: create components from existing files ========================================================== 02:24:43 (1786170283) PASS 2 (10s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 02:24:53 (1786170293) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 02:24:58 (1786170298) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 02:25:06 (1786170306) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (10s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 02:25:21 (1786170321) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 02:25:29 (1786170329) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 1.49005 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105042 s, 10.0 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 02:25:42 (1786170342) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.178879 s, 17.6 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 02:25:51 (1786170351) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 02:26:04 (1786170364) Stopping /mnt/lustre-ost1 (opts:) on oleg448-server oleg448-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98c68d5af000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98c68d5af000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg448-server: oleg448-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg448-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg448-server oleg448-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98c68d5af000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98c68d5af000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg448-server: oleg448-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg448-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (82s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 02:27:26 (1786170446) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.433715 s, 12.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg448-server oleg448-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98c68d5af000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98c68d5af000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg448-server: oleg448-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg448-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (48s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 02:28:14 (1786170494) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg448-server oleg448-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98c68d5af000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98c68d5af000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg448-server: oleg448-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg448-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg448-server oleg448-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98c68d5af000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98c68d5af000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg448-server: oleg448-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg448-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (91s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 02:29:45 (1786170585) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.12733 s, 10.2 MB/s -rw-r--r-- 1 root root 52428800 Aug 8 02:29 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.141s user 0m0.000s sys 0m0.169s Stopping /mnt/lustre-ost1 (opts:) on oleg448-server oleg448-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98c68d5af000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98c68d5af000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.379s user 0m0.001s sys 0m0.182s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg448-server: oleg448-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg448-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg448-server oleg448-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98c68d5af000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98c68d5af000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.386s user 0m0.001s sys 0m0.165s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg448-server: oleg448-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg448-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg448-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98c68d5af000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98c68d5af000.ost_server_uuid in FULL state after 0 sec PASS 33b (106s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 02:31:32 (1786170692) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 02:31:36 (1786170696) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 02:31:40 (1786170700) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 02:31:45 (1786170705) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 02:31:54 (1786170714) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.505336 s, 24.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 02:32:07 (1786170727) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348104 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701315 s, 15.0 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 02:32:16 (1786170736) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 02:32:25 (1786170745) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000859088 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00299377 s, 342 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00169164 s, 605 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00171751 s, 596 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 02:32:36 (1786170756) create test files with size 15740499 .. cc713c53f9321375b12edf8ae4e502ce - 1f507b109919dd29bd836554106b5c46 - 430449e7c829d156e03240bd3dc07120 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg448-client.virtnet /mnt/lustre (opts:) Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 02:32:57 (1786170777) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 02:33:13 (1786170793) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 02:33:20 (1786170800) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194792 s, 5.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220852 s, 47.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xe0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 02:33:31 (1786170811) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.101884 s, 20.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.282168 s, 14.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.185007 s, 11.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 02:33:47 (1786170827) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 02:33:51 (1786170831) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 02:33:54 (1786170834) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223896 s, 46.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 02:34:04 (1786170844) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 02:34:08 (1786170848) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 02:34:20 (1786170860) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.368356 s, 28.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg448-client.virtnet /mnt/lustre (opts:) Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 02:34:33 (1786170873) llite.lustre-ffff98c685255800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.272111 s, 38.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg448-client.virtnet /mnt/lustre (opts:) Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff98c688a23000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 02:34:46 (1786170886) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (41s) == sanity-flr test 46: Verify setstripe --copy option ==== 02:35:27 (1786170927) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (37s) == sanity-flr test 47: Verify mirror obj alloc =========== 02:36:04 (1786170964) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 02:36:08 (1786170968) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.26601 s, 11.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.128727 s, 24.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xe8d85c4b (nosync,stale) Mirror 2: 0xe8d85c4b (nosync,stale) ** verify mirror 2 doesn't change original checksum: 8d4ca889ea1eb3b2a8590444e2ed0826 - mirror 1 checksum: 23b1f458b401552bd4a6ed9032d906e7 - mirror 2 checksum: 8d4ca889ea1eb3b2a8590444e2ed0826 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-08T02:36:11-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-08T02:36:11-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 8d4ca889ea1eb3b2a8590444e2ed0826 - mirror 1 checksum: 23b1f458b401552bd4a6ed9032d906e7 - mirror 2 checksum: 23b1f458b401552bd4a6ed9032d906e7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-08T02:36:11-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-08T02:36:11-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 02:36:23 (1786170983) Failing mds1 on oleg448-server Stopping /mnt/lustre-mds1 (opts:) on oleg448-server 02:36:30 (1786170990) shut down facet: mds1 facet_host: oleg448-server facet_failover_host: oleg448-server Failover mds1 to oleg448-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-MDT0000 02:37:06 (1786171026) targets are mounted 02:37:06 (1786171026) facet_failover done oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254989 s, 41.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } PASS 50A (66s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 02:37:28 (1786171048) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 02:37:32 (1786171052) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 02:37:36 (1786171056) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0130172 s, 1.3 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 02:37:44 (1786171064) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 02:37:48 (1786171068) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0108271 s, 378 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0520448 s, 78.7 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 02:37:57 (1786171077) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 02:38:20 (1786171100) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 02:38:49 (1786171129) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 02:39:15 (1786171155) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 02:39:41 (1786171181) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 02:39:49 (1786171189) mirror create pid 54345 mirror split pid 54346 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54345 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54346 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 02:40:59 (1786171259) Seed set to 1260 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1786171261.150629 write 0x00dcf7 thru 0x01636f (0x8679 bytes) 000002 1786171261.161524 mapwrite 0x0380de thru 0x039a0d (0x1930 bytes) 000005 1786171261.282654 mapwrite 0x009b93 thru 0x01289e (0x8d0c bytes) 000006 1786171261.344482 write 0x03a233 thru 0x03ffff (0x5dcd bytes) truncating to largest ever: 0xc9d0 000008 1786171261.347330 trunc from 0x040000 to 0x00c9cf (0x33630 bytes) 000011 1786171261.368688 mapwrite 0x0360af thru 0x03db79 (0x7acb bytes) truncating to largest ever: 0x1d194 000012 1786171261.447926 trunc from 0x03db7a to 0x01d193 (0x209e6 bytes) 000013 1786171261.465540 trunc from 0x01d194 to 0x01b771 (0x1a22 bytes) 000016 1786171261.490754 mapwrite 0x02a46b thru 0x02a789 (0x31f bytes) truncating to largest ever: 0x1ee27 000017 1786171261.566727 trunc from 0x02a78a to 0x01ee26 (0xb963 bytes) 000018 1786171261.596832 write 0x0339b0 thru 0x03a50f (0x6b60 bytes) truncating to largest ever: 0x29262 000019 1786171261.602361 trunc from 0x03a510 to 0x029261 (0x112ae bytes) 000021 1786171261.630271 read 0x012820 thru 0x0173fe (0x4bdf bytes) 000026 1786171261.646676 write 0x02d301 thru 0x036509 (0x9209 bytes) 000027 1786171261.650905 mapread 0x009f2d thru 0x0160ec (0xc1c0 bytes) 000029 1786171261.662937 mapread 0x0355cf thru 0x036509 (0xf3b bytes) 000030 1786171261.664537 mapwrite 0x012a40 thru 0x01d049 (0xa60a bytes) 000034 1786171261.725358 trunc from 0x03650a to 0x01f8d9 (0x16c30 bytes) 000035 1786171261.746225 trunc from 0x01f8da to 0x00b6df (0x141fa bytes) 000036 1786171261.784728 trunc from 0x00b6e0 to 0x003aba (0x7c25 bytes) 000038 1786171261.812971 mapwrite 0x00a285 thru 0x01117e (0x6efa bytes) 000040 1786171261.904753 read 0x00bf3b thru 0x01117e (0x5244 bytes) 000044 1786171261.908440 trunc from 0x01117f to 0x01935d (0x81df bytes) 000048 1786171261.930977 mapwrite 0x0132e7 thru 0x01f8fb (0xc615 bytes) 000053 1786171262.015889 trunc from 0x01f8fc to 0x00ca83 (0x12e78 bytes) 000054 1786171262.042519 MIRROR_EXTEND to 2 mirrors 000055 1786171262.701515 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000058 1786171262.870690 mapread 0x00402e thru 0x0047a9 (0x77c bytes) 000059 1786171262.880718 mapread 0x00233c thru 0x006892 (0x4557 bytes) 000062 1786171262.901922 write 0x02d042 thru 0x03cacc (0xfa8b bytes) 000063 1786171262.913420 read 0x02fb3d thru 0x03cacc (0xcf90 bytes) 000064 1786171262.914671 mapread 0x03c451 thru 0x03cacc (0x67c bytes) 000066 1786171262.917812 mapread 0x0000f9 thru 0x00898f (0x8897 bytes) 000068 1786171262.931009 mapread 0x00e9fa thru 0x00f955 (0xf5c bytes) 000069 1786171262.932457 write 0x03614f thru 0x03ffff (0x9eb1 bytes) 000070 1786171262.934455 write 0x01c97d thru 0x020184 (0x3808 bytes) 000074 1786171262.950397 MIRROR_EXTEND to 2 mirrors 000075 1786171263.673093 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000076 1786171263.829802 mapread 0x0392df thru 0x03ffff (0x6d21 bytes) 000077 1786171263.872549 mapread 0x02ce72 thru 0x03598e (0x8b1d bytes) 000078 1786171263.896069 read 0x0168a6 thru 0x018f8d (0x26e8 bytes) 000079 1786171263.908682 MIRROR_EXTEND to 2 mirrors 000080 1786171264.628524 trunc from 0x040000 to 0x00da59 (0x325a6 bytes) 000082 1786171264.742555 write 0x009ae9 thru 0x0174d6 (0xd9ee bytes) 000085 1786171264.754861 write 0x0062ff thru 0x0073ac (0x10ae bytes) 000088 1786171264.766816 read 0x0048b6 thru 0x009a6a (0x51b5 bytes) 000089 1786171264.773684 read 0x00d5f2 thru 0x0126c5 (0x50d4 bytes) 000093 1786171264.777381 mapwrite 0x011598 thru 0x01ace7 (0x9750 bytes) 000094 1786171264.895804 mapwrite 0x00ac8f thru 0x013e89 (0x91fb bytes) 000098 1786171264.966735 mapwrite 0x01a077 thru 0x026029 (0xbfb3 bytes) 000104 1786171265.065106 MIRROR_RESYNC 2 mirrors 000106 1786171265.718894 read 0x020143 thru 0x026029 (0x5ee7 bytes) 000107 1786171265.735927 trunc from 0x02602a to 0x00ff40 (0x160e9 bytes) 000108 1786171265.860214 mapread 0x00fd81 thru 0x00ff40 (0x1c0 bytes) 000109 1786171265.869526 mapread 0x00d998 thru 0x00ff40 (0x25a9 bytes) 000110 1786171265.890780 write 0x003816 thru 0x005fee (0x27d9 bytes) 000111 1786171265.907460 MIRROR_RESYNC 2 mirrors 000112 1786171266.533801 write 0x023fcb thru 0x0242ca (0x300 bytes) 000113 1786171266.587020 read 0x00ced3 thru 0x019c60 (0xcd8e bytes) 000114 1786171266.596928 trunc from 0x0242cb to 0x021429 (0x2ea1 bytes) 000115 1786171266.623863 write 0x03fc8a thru 0x03ffff (0x376 bytes) 000116 1786171266.630391 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7a1f18dd (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf58a98a2 Mirror 3: 0xf58a98a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000118 1786171267.852943 mapread 0x0344ec thru 0x03ffff (0xbb14 bytes) 000119 1786171267.888694 write 0x00e6ea thru 0x013320 (0x4c37 bytes) 000121 1786171268.092264 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7a1f18dd (stale) Mirror 3: 0xf58a98a2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3e0143cb Mirror 4: 0x3e0143cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000122 1786171269.249429 mapread 0x01fadb thru 0x02639e (0x68c4 bytes) truncating to largest ever: 0x3c644 000123 1786171269.289629 trunc from 0x040000 to 0x03c643 (0x39bc bytes) 000124 1786171269.363079 mapread 0x0226ae thru 0x02af46 (0x8899 bytes) 000126 1786171269.375846 mapread 0x0000e5 thru 0x00086c (0x788 bytes) 000127 1786171269.389151 mapwrite 0x00057a thru 0x010416 (0xfe9d bytes) 000129 1786171269.464026 mapwrite 0x001c00 thru 0x006ff5 (0x53f6 bytes) 000131 1786171269.516212 trunc from 0x03c644 to 0x02618a (0x164b9 bytes) 000134 1786171269.539472 read 0x01a52c thru 0x02618a (0xbc5f bytes) 000137 1786171269.546591 mapread 0x015c37 thru 0x018b4f (0x2f19 bytes) 000138 1786171269.548029 mapwrite 0x01bc1d thru 0x026505 (0xa8e9 bytes) 000139 1786171269.626711 trunc from 0x026506 to 0x027f5d (0x1a58 bytes) 000140 1786171269.659909 mapwrite 0x01bc2c thru 0x026c69 (0xb03e bytes) 000141 1786171269.757644 mapwrite 0x016a4e thru 0x023ee6 (0xd499 bytes) 000142 1786171269.876576 write 0x0145a2 thru 0x01c9c6 (0x8425 bytes) 000143 1786171269.882669 read 0x015dab thru 0x01af58 (0x51ae bytes) 000146 1786171269.887598 mapwrite 0x0203fd thru 0x021465 (0x1069 bytes) 000148 1786171269.951422 read 0x011f22 thru 0x016b16 (0x4bf5 bytes) 000149 1786171269.952110 mapread 0x002229 thru 0x0090b4 (0x6e8c bytes) 000150 1786171269.953867 mapread 0x017a14 thru 0x02015a (0x8747 bytes) 000152 1786171269.957342 mapread 0x01ee7f thru 0x027f5d (0x90df bytes) 000153 1786171269.965580 write 0x026fdb thru 0x02e5ea (0x7610 bytes) 000157 1786171269.972444 trunc from 0x02e5eb to 0x03bbd8 (0xd5ee bytes) 000158 1786171269.996313 mapwrite 0x00d95e thru 0x014ceb (0x738e bytes) 000160 1786171270.053319 write 0x0315cd thru 0x03c581 (0xafb5 bytes) 000161 1786171270.077524 MIRROR_RESYNC 4 mirrors 000162 1786171271.362778 read 0x00a1d9 thru 0x00f081 (0x4ea9 bytes) 000164 1786171271.376735 write 0x028e7f thru 0x02a101 (0x1283 bytes) truncating to largest ever: 0x3debb 000165 1786171271.448692 trunc from 0x03c582 to 0x03deba (0x1939 bytes) 000168 1786171271.474976 write 0x0096ae thru 0x00e22f (0x4b82 bytes) 000169 1786171271.485531 mapwrite 0x02b1c0 thru 0x031c9c (0x6add bytes) 000170 1786171271.580416 write 0x02404d thru 0x033342 (0xf2f6 bytes) 000171 1786171271.587754 write 0x016b09 thru 0x01b07e (0x4576 bytes) 000172 1786171271.603037 write 0x02eb2c thru 0x039701 (0xabd6 bytes) 000176 1786171271.619221 mapwrite 0x02ab4f thru 0x037952 (0xce04 bytes) 000177 1786171271.670649 mapread 0x004c94 thru 0x00703f (0x23ac bytes) 000179 1786171271.682742 trunc from 0x03debb to 0x02ee9f (0xf01b bytes) 000181 1786171271.717781 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2eea0): Mirror 3: 0x8da4a021 (stale) Mirror 4: 0x8da4a021 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2eea0): Mirror 3: 0x8da4a021 (stale) Mirror 4: 0x8da4a021 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2eea0: skipped 000182 1786171272.062862 trunc from 0x02eea0 to 0x00314e (0x2bd51 bytes) 000184 1786171272.122651 trunc from 0x00314f to 0x00765b (0x450d bytes) 000186 1786171272.161807 trunc from 0x00765c to 0x02541e (0x1ddc3 bytes) 000187 1786171272.189740 mapread 0x00cb56 thru 0x01a961 (0xde0c bytes) 000188 1786171272.259257 mapread 0x0034c6 thru 0x01259f (0xf0da bytes) 000192 1786171272.294485 trunc from 0x02541f to 0x035440 (0x10022 bytes) 000193 1786171272.355245 trunc from 0x035441 to 0x021857 (0x13be9 bytes) 000196 1786171272.388966 write 0x000ac1 thru 0x0011d2 (0x712 bytes) 000197 1786171272.409736 write 0x02ad6d thru 0x0353a8 (0xa63c bytes) 000198 1786171272.415946 trunc from 0x0353a9 to 0x00d627 (0x27d81 bytes) 000200 1786171272.451372 read 0x0060c1 thru 0x0091e4 (0x3124 bytes) 000201 1786171272.452517 write 0x02793f thru 0x02a9f5 (0x30b7 bytes) 000202 1786171272.455676 trunc from 0x02a9f6 to 0x02ca4a (0x2055 bytes) 000203 1786171272.482681 trunc from 0x02ca4b to 0x02168d (0xb3bd bytes) 000205 1786171272.506491 mapread 0x002d83 thru 0x0128dd (0xfb5b bytes) 000206 1786171272.518656 read 0x002bd4 thru 0x00d607 (0xaa34 bytes) 000207 1786171272.519528 mapread 0x002c5e thru 0x004441 (0x17e4 bytes) 000211 1786171272.523425 read 0x016d73 thru 0x01cd24 (0x5fb2 bytes) 000212 1786171272.533580 write 0x005300 thru 0x00f59f (0xa2a0 bytes) 000216 1786171272.539283 read 0x01409e thru 0x018e2e (0x4d91 bytes) 000217 1786171272.540111 mapread 0x01cea1 thru 0x01dfb5 (0x1115 bytes) 000218 1786171272.541493 mapwrite 0x039d67 thru 0x03ffff (0x6299 bytes) 000220 1786171272.624705 mapread 0x011e70 thru 0x01b11a (0x92ab bytes) 000221 1786171272.626828 mapwrite 0x012268 thru 0x01a69c (0x8435 bytes) 000222 1786171272.708704 mapwrite 0x0090a9 thru 0x0144c7 (0xb41f bytes) 000225 1786171272.788239 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x88a69f2 (stale) Mirror 4: 0x88a69f2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9cb60548 Mirror 5: 0x9cb60548 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000226 1786171273.791770 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x88a69f2 (stale) Mirror 4: 0x88a69f2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x88a69f2 (stale) Mirror 4: 0x88a69f2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000227 1786171274.123549 write 0x029fa5 thru 0x0336a4 (0x9700 bytes) 000228 1786171274.162593 write 0x0062f5 thru 0x015a80 (0xf78c bytes) 000229 1786171274.183411 mapwrite 0x013513 thru 0x014cb8 (0x17a6 bytes) 000231 1786171274.277851 MIRROR_RESYNC 3 mirrors 000232 1786171275.334482 write 0x033aff thru 0x038b3c (0x503e bytes) 000234 1786171275.415084 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x6e4382c3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x6e4382c3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000236 1786171275.814372 mapread 0x015453 thru 0x0191ce (0x3d7c bytes) 000238 1786171275.846097 trunc from 0x040000 to 0x004aa9 (0x3b556 bytes) 000242 1786171275.920096 trunc from 0x004aaa to 0x02dc33 (0x2918a bytes) 000245 1786171275.982516 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2dc34): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dc34): Mirror 4: 0xd516ac60 (stale) CRC-32 checksum value for chunk [0, 0x2dc34): Mirror 1: 0xd61c95de Mirror 5: 0xd61c95de lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dc34: skipped 000246 1786171277.696165 write 0x03f668 thru 0x03ffff (0x998 bytes) 000249 1786171277.775893 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x647a962b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x647a962b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000250 1786171278.095057 mapread 0x00fb61 thru 0x01cae2 (0xcf82 bytes) 000252 1786171278.129856 mapwrite 0x019d70 thru 0x01ac8f (0xf20 bytes) 000254 1786171278.175345 mapread 0x03f64b thru 0x03ffff (0x9b5 bytes) 000255 1786171278.188211 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000256 1786171278.339754 write 0x002fe8 thru 0x00783f (0x4858 bytes) 000258 1786171278.355450 trunc from 0x040000 to 0x01e5b4 (0x21a4b bytes) 000260 1786171278.381639 mapwrite 0x035e6f thru 0x03ffff (0xa191 bytes) 000262 1786171278.478734 mapread 0x01959b thru 0x027030 (0xda96 bytes) 000263 1786171278.499512 read 0x0038c4 thru 0x007735 (0x3e72 bytes) 000264 1786171278.501474 mapwrite 0x01f20b thru 0x02126e (0x2064 bytes) 000265 1786171278.550151 read 0x0118a9 thru 0x01bbf6 (0xa34e bytes) 000269 1786171278.562992 read 0x0101d8 thru 0x0159a6 (0x57cf bytes) 000270 1786171278.564460 read 0x0394c9 thru 0x03d8b3 (0x43eb bytes) 000271 1786171278.565857 write 0x000e72 thru 0x00cccb (0xbe5a bytes) 000273 1786171278.568462 mapread 0x00179a thru 0x00b822 (0xa089 bytes) 000276 1786171278.572709 trunc from 0x040000 to 0x03d111 (0x2eee bytes) 000277 1786171278.595704 mapwrite 0x03bf75 thru 0x03cafd (0xb89 bytes) 000278 1786171278.639044 mapwrite 0x010cbd thru 0x01f184 (0xe4c8 bytes) 000280 1786171278.685504 read 0x032d30 thru 0x035785 (0x2a56 bytes) 000281 1786171278.686600 mapwrite 0x03e740 thru 0x03ffff (0x18c0 bytes) 000282 1786171278.775943 mapread 0x030f58 thru 0x03a25d (0x9306 bytes) 000283 1786171278.777312 mapwrite 0x028b73 thru 0x035775 (0xcc03 bytes) 000285 1786171278.858483 mapread 0x027adf thru 0x0366be (0xebe0 bytes) 000286 1786171278.861328 write 0x006481 thru 0x015146 (0xecc6 bytes) 000290 1786171278.865279 mapread 0x023a8c thru 0x02f394 (0xb909 bytes) 000291 1786171278.867066 trunc from 0x040000 to 0x0024d6 (0x3db29 bytes) 000292 1786171278.885588 trunc from 0x0024d7 to 0x0113c3 (0xeeed bytes) 000293 1786171278.906450 trunc from 0x0113c4 to 0x011135 (0x28e bytes) 000294 1786171278.930971 write 0x01c612 thru 0x0233d4 (0x6dc3 bytes) 000295 1786171278.933681 read 0x007d04 thru 0x00e883 (0x6b80 bytes) 000296 1786171278.945048 mapwrite 0x038691 thru 0x03a6dc (0x204c bytes) 000297 1786171279.018106 mapread 0x01553e thru 0x01ecf1 (0x97b4 bytes) 000299 1786171279.021331 trunc from 0x03a6dd to 0x0102ae (0x2a42e bytes) 000300 1786171279.040689 read 0x009167 thru 0x00d709 (0x45a3 bytes) 000301 1786171279.041878 write 0x02210e thru 0x02585c (0x374f bytes) 000302 1786171279.044098 write 0x027b77 thru 0x03197b (0x9e05 bytes) 000303 1786171279.046735 trunc from 0x03197c to 0x0302af (0x16cc bytes) 000308 1786171279.069448 trunc from 0x0302b0 to 0x02299b (0xd914 bytes) 000314 1786171279.095697 mapread 0x0030b5 thru 0x00fb1e (0xca6a bytes) 000315 1786171279.102240 MIRROR_EXTEND to 2 mirrors 000316 1786171279.712398 MIRROR_RESYNC 2 mirrors 000317 1786171280.072301 mapwrite 0x039f66 thru 0x03ffff (0x609a bytes) 000318 1786171280.376451 mapread 0x039f0d thru 0x03ffff (0x60f3 bytes) 000319 1786171280.381082 trunc from 0x040000 to 0x03ad1d (0x52e2 bytes) 000321 1786171280.429045 mapwrite 0x01ef4b thru 0x02898c (0x9a42 bytes) 000322 1786171280.541982 mapread 0x01c429 thru 0x02a416 (0xdfee bytes) 000323 1786171280.545514 mapwrite 0x032d0a thru 0x03980e (0x6b05 bytes) 000326 1786171280.600847 write 0x00ab4f thru 0x018c93 (0xe145 bytes) 000330 1786171280.610892 mapwrite 0x020bac thru 0x02b899 (0xacee bytes) 000334 1786171280.664397 trunc from 0x03ad1e to 0x01acf2 (0x2002b bytes) 000336 1786171280.688238 trunc from 0x01acf3 to 0x020918 (0x5c26 bytes) 000341 1786171280.713935 mapwrite 0x021187 thru 0x02b08d (0x9f07 bytes) 000344 1786171280.810661 mapread 0x025d7f thru 0x02b08d (0x530f bytes) 000346 1786171280.812912 trunc from 0x02b08e to 0x02f776 (0x46e9 bytes) 000347 1786171280.831051 read 0x01fd35 thru 0x02bc63 (0xbf2f bytes) 000349 1786171280.836481 write 0x027a83 thru 0x036ba4 (0xf122 bytes) 000350 1786171280.839081 write 0x01d4df thru 0x0244c2 (0x6fe4 bytes) 000352 1786171280.842244 write 0x001f95 thru 0x00e694 (0xc700 bytes) 000354 1786171280.853717 mapread 0x035dd9 thru 0x036ba4 (0xdcc bytes) 000355 1786171280.861413 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36ba5): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36ba5): Mirror 2: 0x8d97e044 (stale) CRC-32 checksum value for chunk [0, 0x36ba5): Mirror 1: 0xa25126b7 Mirror 3: 0xa25126b7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36ba5: skipped 000357 1786171281.994889 mapwrite 0x000f30 thru 0x0040c1 (0x3192 bytes) 000361 1786171282.188085 mapread 0x028217 thru 0x033e3d (0xbc27 bytes) 000362 1786171282.206249 mapread 0x0123cf thru 0x01982b (0x745d bytes) 000363 1786171282.215297 mapwrite 0x021e18 thru 0x02f6e8 (0xd8d1 bytes) 000364 1786171282.280577 mapread 0x0325e5 thru 0x036ba4 (0x45c0 bytes) 000366 1786171282.283166 mapread 0x003014 thru 0x00fc9e (0xcc8b bytes) 000368 1786171282.291266 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x36ba5): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36ba5): Mirror 2: 0x8d97e044 (stale) Mirror 3: 0xa25126b7 (stale) CRC-32 checksum value for chunk [0, 0x36ba5): Mirror 1: 0x3d94a87e Mirror 4: 0x3d94a87e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36ba5: skipped 000369 1786171283.245747 mapwrite 0x015889 thru 0x01f589 (0x9d01 bytes) 000371 1786171283.498802 trunc from 0x036ba5 to 0x0194e2 (0x1d6c2 bytes) 000372 1786171283.549895 trunc from 0x0194e3 to 0x00e3b1 (0xb131 bytes) 000375 1786171283.571735 trunc from 0x00e3b2 to 0x038807 (0x2a456 bytes) 000377 1786171283.607963 trunc from 0x038808 to 0x00823e (0x305c9 bytes) 000378 1786171283.642558 mapread 0x000fac thru 0x00823e (0x7293 bytes) 000380 1786171283.664573 mapread 0x001644 thru 0x006700 (0x50bd bytes) 000382 1786171283.666884 mapread 0x001cfd thru 0x00734c (0x5650 bytes) 000383 1786171283.668298 mapwrite 0x014ee6 thru 0x01eef5 (0xa010 bytes) 000385 1786171283.774342 write 0x00eb35 thru 0x018cea (0xa1b6 bytes) 000386 1786171283.777324 read 0x010fa1 thru 0x01cb90 (0xbbf0 bytes) 000387 1786171283.783063 read 0x00e3d9 thru 0x01315a (0x4d82 bytes) 000389 1786171283.794561 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1eef6): Mirror 3: 0xa0f0f046 (stale) Mirror 4: 0xa62b4101 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1eef6): Mirror 3: 0xa0f0f046 (stale) Mirror 4: 0xa62b4101 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1eef6: skipped 000390 1786171284.220830 trunc from 0x01eef6 to 0x034562 (0x1566d bytes) 000391 1786171284.243816 mapwrite 0x02e8c4 thru 0x03a2d5 (0xba12 bytes) 000392 1786171284.360543 write 0x039db8 thru 0x03b5e9 (0x1832 bytes) 000393 1786171284.362109 mapread 0x03ac7b thru 0x03b5e9 (0x96f bytes) 000396 1786171284.369694 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b5ea): Mirror 3: 0x98554f2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b5ea): Mirror 3: 0x98554f2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b5ea: skipped 000397 1786171284.664935 mapwrite 0x01881f thru 0x024e26 (0xc608 bytes) 000399 1786171284.764992 write 0x01e20e thru 0x01f114 (0xf07 bytes) 000400 1786171284.770055 trunc from 0x03b5ea to 0x019a08 (0x21be1 bytes) 000401 1786171284.812370 write 0x019157 thru 0x026de8 (0xdc92 bytes) 000402 1786171284.815880 read 0x015777 thru 0x022740 (0xcfca bytes) 000405 1786171284.821016 read 0x0183de thru 0x01a4b3 (0x20d6 bytes) 000406 1786171284.823006 mapread 0x021489 thru 0x026cd1 (0x5849 bytes) 000410 1786171284.836776 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x26de9): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26de9): Mirror 3: 0x3c9ca67b (stale) CRC-32 checksum value for chunk [0, 0x26de9): Mirror 1: 0xccdc5461 Mirror 4: 0xccdc5461 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26de9: skipped 000412 1786171285.792590 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000415 1786171286.275674 MIRROR_RESYNC 2 mirrors 000416 1786171286.682994 trunc from 0x026de9 to 0x01c8f6 (0xa4f2 bytes) 000417 1786171286.811982 mapread 0x00cf61 thru 0x01aea8 (0xdf48 bytes) 000419 1786171286.872256 write 0x02440c thru 0x0333ff (0xeff4 bytes) 000420 1786171286.875282 mapwrite 0x02c7dd thru 0x02d160 (0x984 bytes) 000421 1786171286.942711 trunc from 0x033400 to 0x019f06 (0x194f9 bytes) 000422 1786171286.976102 trunc from 0x019f07 to 0x00da29 (0xc4dd bytes) 000423 1786171287.003240 read 0x008ebc thru 0x00da29 (0x4b6e bytes) 000424 1786171287.003975 write 0x00ad31 thru 0x0185ee (0xd8be bytes) 000425 1786171287.007037 read 0x014c47 thru 0x0185ee (0x39a8 bytes) 000427 1786171287.010053 trunc from 0x0185ef to 0x032c68 (0x1a67a bytes) 000429 1786171287.032836 trunc from 0x032c69 to 0x00bb06 (0x27162 bytes) 000431 1786171287.052785 trunc from 0x00bb07 to 0x02e7ac (0x22ca6 bytes) 000432 1786171287.077206 mapwrite 0x01b5dc thru 0x01ba4f (0x474 bytes) 000437 1786171287.151508 mapwrite 0x0044e5 thru 0x005732 (0x124e bytes) 000438 1786171287.197356 write 0x00bd82 thru 0x016879 (0xaaf8 bytes) 000439 1786171287.199213 trunc from 0x02e7ad to 0x01dff8 (0x107b4 bytes) 000440 1786171287.217002 write 0x01cd04 thru 0x022f39 (0x6236 bytes) 000442 1786171287.224832 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x22f3a): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22f3a): Mirror 4: 0x2d4df1e7 (stale) CRC-32 checksum value for chunk [0, 0x22f3a): Mirror 1: 0x58dec40c Mirror 5: 0x58dec40c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22f3a: skipped 000443 1786171288.048017 write 0x016d56 thru 0x0239a3 (0xcc4e bytes) 000448 1786171288.123296 trunc from 0x0239a4 to 0x03d438 (0x19a95 bytes) 000451 1786171288.148238 mapwrite 0x0033e5 thru 0x0123d4 (0xeff0 bytes) 000455 1786171288.228810 mapread 0x02745a thru 0x0291c3 (0x1d6a bytes) 000456 1786171288.239523 trunc from 0x03d439 to 0x039e97 (0x35a1 bytes) 000457 1786171288.259521 write 0x00f8f8 thru 0x01208c (0x2795 bytes) 000460 1786171288.262491 trunc from 0x039e98 to 0x020fe5 (0x18eb2 bytes) 000466 1786171288.287665 write 0x02ea57 thru 0x035e4e (0x73f8 bytes) 000467 1786171288.294266 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x35e4f): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35e4f): Mirror 4: 0xc1898882 (stale) Mirror 5: 0x1ac2d6eb (stale) CRC-32 checksum value for chunk [0, 0x35e4f): Mirror 1: 0x8edd6301 Mirror 6: 0x8edd6301 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35e4f: skipped 000468 1786171289.179532 mapwrite 0x00b33e thru 0x011d8c (0x6a4f bytes) 000469 1786171289.358972 write 0x0107b6 thru 0x01aa84 (0xa2cf bytes) 000470 1786171289.367810 write 0x0319a8 thru 0x03bac3 (0xa11c bytes) 000472 1786171289.374868 mapwrite 0x02ccc3 thru 0x030608 (0x3946 bytes) 000473 1786171289.427996 write 0x017c33 thru 0x01ac12 (0x2fe0 bytes) 000474 1786171289.429792 mapwrite 0x013fd6 thru 0x018554 (0x457f bytes) 000478 1786171289.477383 MIRROR_RESYNC 4 mirrors 000480 1786171290.321723 mapwrite 0x017ae6 thru 0x01c5db (0x4af6 bytes) 000481 1786171290.512725 MIRROR_RESYNC 4 mirrors 000482 1786171291.467097 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000485 1786171291.743879 mapread 0x0327f2 thru 0x03bac3 (0x92d2 bytes) 000486 1786171291.763502 trunc from 0x03bac4 to 0x006426 (0x3569d bytes) 000487 1786171291.848945 read 0x002791 thru 0x006426 (0x3c96 bytes) 000489 1786171291.859570 write 0x013e9a thru 0x015c4b (0x1db2 bytes) 000490 1786171291.865071 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15c4c): Mirror 6: 0x7dab2bbb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15c4c): Mirror 6: 0x7dab2bbb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15c4c: skipped 000491 1786171292.156740 read 0x005769 thru 0x0125e1 (0xce79 bytes) 000492 1786171292.172762 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x15c4c): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15c4c): Mirror 6: 0x7dab2bbb (stale) CRC-32 checksum value for chunk [0, 0x15c4c): Mirror 1: 0xd668b42f Mirror 7: 0xd668b42f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15c4c: skipped 000493 1786171293.115875 write 0x0234b2 thru 0x030b04 (0xd653 bytes) 000496 1786171293.242830 mapwrite 0x03137e thru 0x0373b2 (0x6035 bytes) 000499 1786171293.404487 MIRROR_RESYNC 3 mirrors 000500 1786171294.201951 trunc from 0x0373b3 to 0x003a57 (0x3395b bytes) 000501 1786171294.326359 write 0x01cecb thru 0x01f7f8 (0x292e bytes) 000503 1786171294.335826 MIRROR_RESYNC 3 mirrors 000505 1786171295.113894 write 0x022af7 thru 0x026a56 (0x3f60 bytes) truncating to largest ever: 0x3f4b0 000506 1786171295.182656 trunc from 0x026a57 to 0x03f4af (0x18a59 bytes) 000507 1786171295.206078 mapread 0x032c76 thru 0x032dd3 (0x15e bytes) 000509 1786171295.215897 write 0x020d88 thru 0x0228af (0x1b28 bytes) 000510 1786171295.222311 write 0x004118 thru 0x00c64c (0x8535 bytes) 000512 1786171295.235421 write 0x00ba8f thru 0x01af45 (0xf4b7 bytes) 000513 1786171295.242456 mapwrite 0x034df0 thru 0x036306 (0x1517 bytes) 000514 1786171295.327078 mapread 0x013e1f thru 0x01a0b6 (0x6298 bytes) 000515 1786171295.330099 read 0x0332f5 thru 0x03f4af (0xc1bb bytes) 000516 1786171295.341003 mapwrite 0x00f261 thru 0x01e419 (0xf1b9 bytes) 000517 1786171295.411770 read 0x037a4b thru 0x03f4af (0x7a65 bytes) 000518 1786171295.412970 mapread 0x038b27 thru 0x03f4af (0x6989 bytes) 000522 1786171295.417379 trunc from 0x03f4b0 to 0x006128 (0x39387 bytes) 000524 1786171295.443711 trunc from 0x006129 to 0x03cda8 (0x36c80 bytes) 000526 1786171295.472962 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3cda9): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cda9): Mirror 6: 0xbf5dd564 (stale) Mirror 7: 0xbf5dd564 (stale) CRC-32 checksum value for chunk [0, 0x3cda9): Mirror 1: 0x2b8635 Mirror 8: 0x2b8635 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cda9: skipped 000528 1786171296.434393 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3cda9): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cda9): Mirror 6: 0xbf5dd564 (stale) Mirror 7: 0xbf5dd564 (stale) CRC-32 checksum value for chunk [0, 0x3cda9): Mirror 1: 0x2b8635 Mirror 8: 0x2b8635 Mirror 9: 0x2b8635 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cda9: skipped 000530 1786171297.395273 mapread 0x027d51 thru 0x02ba84 (0x3d34 bytes) 000531 1786171297.422632 read 0x02728c thru 0x0304da (0x924f bytes) 000533 1786171297.423705 trunc from 0x03cda9 to 0x023ce4 (0x190c4 bytes) 000535 1786171297.541887 write 0x004010 thru 0x011fc4 (0xdfb5 bytes) 000536 1786171297.552764 read 0x0094c4 thru 0x00ff16 (0x6a53 bytes) 000537 1786171297.557465 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23ce5): Mirror 6: 0x46041401 (stale) Mirror 8: 0xc71613ee (stale) Mirror 9: 0xc71613ee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23ce5): Mirror 6: 0x46041401 (stale) Mirror 8: 0xc71613ee (stale) Mirror 9: 0xc71613ee (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23ce5: skipped 000538 1786171297.915637 mapwrite 0x03e2f4 thru 0x03ffff (0x1d0c bytes) 000541 1786171297.991887 read 0x004020 thru 0x006724 (0x2705 bytes) 000543 1786171297.998523 read 0x01aaab thru 0x02815f (0xd6b5 bytes) 000544 1786171298.015094 trunc from 0x040000 to 0x03d1f0 (0x2e0f bytes) 000546 1786171298.035820 mapwrite 0x002f50 thru 0x003db4 (0xe65 bytes) 000547 1786171298.081112 trunc from 0x03d1f1 to 0x03ca1d (0x7d3 bytes) 000548 1786171298.103343 write 0x022e61 thru 0x0287da (0x597a bytes) 000549 1786171298.104538 trunc from 0x03ca1e to 0x016198 (0x26885 bytes) 000550 1786171298.147678 MIRROR_RESYNC 4 mirrors 000551 1786171299.013780 trunc from 0x016199 to 0x01820c (0x2074 bytes) 000553 1786171299.136814 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1820d): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1820d): Mirror 6: 0xfb15a5f7 (stale) Mirror 8: 0xfb15a5f7 (stale) Mirror 9: 0xfb15a5f7 (stale) CRC-32 checksum value for chunk [0, 0x1820d): Mirror 1: 0xfb15a5f7 Mirror 10: 0xfb15a5f7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1820d: skipped 000554 1786171300.047095 trunc from 0x01820d to 0x00599d (0x1286f bytes) 000556 1786171300.135506 mapread 0x004cbc thru 0x00599d (0xce2 bytes) 000557 1786171300.145971 write 0x03b1e0 thru 0x03d898 (0x26b9 bytes) 000558 1786171300.147746 mapwrite 0x02a43d thru 0x032887 (0x844b bytes) 000559 1786171300.205110 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3d899): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d899): Mirror 6: 0xa7da6d1c (stale) Mirror 8: 0xa7da6d1c (stale) Mirror 9: 0xa7da6d1c (stale) Mirror 10: 0xa7da6d1c (stale) CRC-32 checksum value for chunk [0, 0x3d899): Mirror 1: 0xebf3e829 Mirror 11: 0xebf3e829 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d899: skipped 000560 1786171301.039762 trunc from 0x03d899 to 0x0371c0 (0x66d8 bytes) 000561 1786171301.164615 trunc from 0x0371c1 to 0x0252f5 (0x11ecb bytes) 000563 1786171301.186115 mapread 0x023bf8 thru 0x0252f5 (0x16fe bytes) 000564 1786171301.220243 MIRROR_RESYNC 6 mirrors 000565 1786171302.876813 MIRROR_EXTEND to 7 mirrors 000567 1786171303.831755 trunc from 0x0252f6 to 0x01f3fe (0x5ef7 bytes) 000572 1786171303.918824 mapread 0x00695c thru 0x01290a (0xbfaf bytes) 000573 1786171303.952756 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1f3ff): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f3ff): Mirror 6: 0xa328eb63 (stale) Mirror 8: 0xa328eb63 (stale) Mirror 9: 0xa328eb63 (stale) Mirror 10: 0xa328eb63 (stale) Mirror 11: 0xa328eb63 (stale) Mirror 12: 0xa328eb63 (stale) CRC-32 checksum value for chunk [0, 0x1f3ff): Mirror 1: 0xa328eb63 Mirror 13: 0xa328eb63 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f3ff: skipped 000574 1786171305.025082 read 0x01d78a thru 0x01f3fe (0x1c75 bytes) 000575 1786171305.038695 write 0x009e5a thru 0x00a6fe (0x8a5 bytes) 000576 1786171305.139691 trunc from 0x01f3ff to 0x006cff (0x186ff bytes) 000580 1786171305.170414 trunc from 0x006d00 to 0x03781a (0x30b1b bytes) 000582 1786171305.190505 trunc from 0x03781b to 0x00a384 (0x2d496 bytes) 000584 1786171305.214224 write 0x020674 thru 0x02e813 (0xe1a0 bytes) 000586 1786171305.217630 mapread 0x011f6e thru 0x01dc29 (0xbcbc bytes) 000590 1786171305.242073 mapwrite 0x00b472 thru 0x017ff7 (0xcb86 bytes) 000591 1786171305.317171 read 0x007417 thru 0x01578d (0xe377 bytes) 000593 1786171305.319245 write 0x02090a thru 0x03082f (0xff26 bytes) 000596 1786171305.323546 mapread 0x007401 thru 0x00aae3 (0x36e3 bytes) 000597 1786171305.324955 trunc from 0x030830 to 0x0312be (0xa8f bytes) 000599 1786171305.346497 write 0x006992 thru 0x011860 (0xaecf bytes) 000600 1786171305.348830 mapread 0x027fd4 thru 0x0312be (0x92eb bytes) 000603 1786171305.363002 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x312bf): Mirror 8: 0x452967ed (stale) Mirror 9: 0x452967ed (stale) Mirror 10: 0x452967ed (stale) Mirror 11: 0x452967ed (stale) Mirror 12: 0x452967ed (stale) Mirror 13: 0x452967ed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x312bf): Mirror 8: 0x452967ed (stale) Mirror 9: 0x452967ed (stale) Mirror 10: 0x452967ed (stale) Mirror 11: 0x452967ed (stale) Mirror 12: 0x452967ed (stale) Mirror 13: 0x452967ed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x312bf: skipped 000604 1786171305.788257 mapwrite 0x03f136 thru 0x03ffff (0xeca bytes) 000606 1786171305.860627 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x2413a37a (stale) Mirror 9: 0x2413a37a (stale) Mirror 10: 0x2413a37a (stale) Mirror 11: 0x2413a37a (stale) Mirror 12: 0x2413a37a (stale) Mirror 13: 0x2413a37a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd807a1d3 Mirror 14: 0xd807a1d3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000608 1786171306.741861 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x2413a37a (stale) Mirror 9: 0x2413a37a (stale) Mirror 11: 0x2413a37a (stale) Mirror 12: 0x2413a37a (stale) Mirror 13: 0x2413a37a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x2413a37a (stale) Mirror 9: 0x2413a37a (stale) Mirror 11: 0x2413a37a (stale) Mirror 12: 0x2413a37a (stale) Mirror 13: 0x2413a37a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd807a1d3 Mirror 14: 0xd807a1d3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000609 1786171307.470731 mapwrite 0x01c6af thru 0x026229 (0x9b7b bytes) 000610 1786171307.673787 mapread 0x000b97 thru 0x00ea08 (0xde72 bytes) 000612 1786171307.691884 write 0x03df0f thru 0x03ffff (0x20f1 bytes) 000614 1786171307.698748 mapread 0x0295fb thru 0x02ce68 (0x386e bytes) 000615 1786171307.700417 write 0x00437a thru 0x00b3ba (0x7041 bytes) 000616 1786171307.702337 read 0x0320b7 thru 0x0395d9 (0x7523 bytes) 000617 1786171307.712572 mapread 0x01bf18 thru 0x02810a (0xc1f3 bytes) 000618 1786171307.714946 mapread 0x02d0d0 thru 0x02e2fe (0x122f bytes) 000620 1786171307.716817 mapwrite 0x01951a thru 0x0229f5 (0x94dc bytes) 000621 1786171307.763605 write 0x00de80 thru 0x01aad6 (0xcc57 bytes) 000623 1786171307.767062 mapread 0x0049dc thru 0x00e28b (0x98b0 bytes) 000624 1786171307.768487 write 0x0295db thru 0x0325a5 (0x8fcb bytes) 000625 1786171307.770558 read 0x03ef02 thru 0x03ffff (0x10fe bytes) 000628 1786171307.773210 mapwrite 0x031fbc thru 0x03e884 (0xc8c9 bytes) 000630 1786171307.846820 write 0x03bb5c thru 0x03ffff (0x44a4 bytes) 000631 1786171307.850368 read 0x013561 thru 0x0136f5 (0x195 bytes) 000632 1786171307.852409 trunc from 0x040000 to 0x004788 (0x3b877 bytes) 000633 1786171307.890418 mapwrite 0x0059ca thru 0x007cdc (0x2313 bytes) 000637 1786171307.972723 MIRROR_RESYNC 7 mirrors 000640 1786171309.136395 MIRROR_RESYNC 7 mirrors 000641 1786171309.551732 mapread 0x0000a1 thru 0x00446f (0x43cf bytes) 000644 1786171309.578101 mapwrite 0x02e408 thru 0x03592b (0x7524 bytes) 000645 1786171309.739563 trunc from 0x03592c to 0x028c07 (0xcd24 bytes) 000647 1786171309.767190 trunc from 0x028c08 to 0x004bed (0x2401a bytes) 000649 1786171309.785775 write 0x019784 thru 0x01d0fb (0x3978 bytes) 000650 1786171309.787724 mapwrite 0x010be5 thru 0x01e041 (0xd45d bytes) 000651 1786171309.879852 write 0x020154 thru 0x02055a (0x407 bytes) 000654 1786171309.882871 mapwrite 0x01ee21 thru 0x029546 (0xa726 bytes) 000655 1786171309.981174 read 0x0150b2 thru 0x023860 (0xe7af bytes) 000656 1786171309.986201 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29547): Mirror 9: 0xf1c4327 (stale) Mirror 11: 0xf1c4327 (stale) Mirror 12: 0xf1c4327 (stale) Mirror 13: 0xf1c4327 (stale) Mirror 14: 0xf1c4327 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29547): Mirror 9: 0xf1c4327 (stale) Mirror 11: 0xf1c4327 (stale) Mirror 12: 0xf1c4327 (stale) Mirror 13: 0xf1c4327 (stale) Mirror 14: 0xf1c4327 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29547: skipped 000660 1786171310.416575 mapread 0x00b956 thru 0x00ca30 (0x10db bytes) 000663 1786171310.427934 read 0x0073db thru 0x0145c5 (0xd1eb bytes) 000664 1786171310.443592 mapwrite 0x039b4f thru 0x03a616 (0xac8 bytes) 000666 1786171310.517748 mapread 0x00f90a thru 0x018353 (0x8a4a bytes) 000667 1786171310.528666 write 0x0114fa thru 0x011eab (0x9b2 bytes) 000668 1786171310.530002 trunc from 0x03a617 to 0x039b9f (0xa77 bytes) 000669 1786171310.559484 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39ba0): Mirror 11: 0xe2d82611 (stale) Mirror 12: 0xe2d82611 (stale) Mirror 13: 0xe2d82611 (stale) Mirror 14: 0xe2d82611 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39ba0): Mirror 11: 0xe2d82611 (stale) Mirror 12: 0xe2d82611 (stale) Mirror 13: 0xe2d82611 (stale) Mirror 14: 0xe2d82611 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39ba0: skipped 000670 1786171310.914116 read 0x022b5a thru 0x02f0aa (0xc551 bytes) 000672 1786171310.937256 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x39ba0): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39ba0): Mirror 11: 0xe2d82611 (stale) Mirror 12: 0xe2d82611 (stale) Mirror 13: 0xe2d82611 (stale) Mirror 14: 0xe2d82611 (stale) CRC-32 checksum value for chunk [0, 0x39ba0): Mirror 1: 0x5c607e73 Mirror 15: 0x5c607e73 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39ba0: skipped 000673 1786171311.911560 read 0x028da9 thru 0x03551a (0xc772 bytes) 000675 1786171311.966816 read 0x0368fb thru 0x039b9f (0x32a5 bytes) 000677 1786171312.015060 mapwrite 0x02b2b2 thru 0x02f292 (0x3fe1 bytes) 000678 1786171312.293448 mapread 0x005ef7 thru 0x00c3a6 (0x64b0 bytes) 000680 1786171312.352528 mapread 0x008285 thru 0x011a55 (0x97d1 bytes) 000681 1786171312.381064 mapwrite 0x03f938 thru 0x03ffff (0x6c8 bytes) 000682 1786171312.460962 trunc from 0x040000 to 0x01dd6a (0x22295 bytes) 000683 1786171312.482703 read 0x00174d thru 0x00f8dc (0xe190 bytes) truncating to largest ever: 0x3fc4a 000684 1786171312.495440 trunc from 0x01dd6b to 0x03fc49 (0x21edf bytes) 000685 1786171312.516156 trunc from 0x03fc4a to 0x00a6e3 (0x35566 bytes) 000686 1786171312.540156 mapread 0x005a06 thru 0x007a63 (0x205e bytes) 000687 1786171312.541876 trunc from 0x00a6e4 to 0x00fc53 (0x5570 bytes) 000689 1786171312.566627 read 0x00a8a3 thru 0x00fc53 (0x53b1 bytes) 000691 1786171312.577024 trunc from 0x00fc54 to 0x0160b6 (0x6463 bytes) 000692 1786171312.596453 read 0x0128d3 thru 0x0160b6 (0x37e4 bytes) 000693 1786171312.612741 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x160b7): Mirror 12: 0x174ebedc (stale) Mirror 13: 0x174ebedc (stale) Mirror 14: 0x174ebedc (stale) Mirror 15: 0x71e0bdb3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x160b7): Mirror 12: 0x174ebedc (stale) Mirror 13: 0x174ebedc (stale) Mirror 14: 0x174ebedc (stale) Mirror 15: 0x71e0bdb3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x160b7: skipped 000695 1786171312.979967 mapwrite 0x0248d3 thru 0x02f498 (0xabc6 bytes) 000697 1786171313.146514 trunc from 0x02f499 to 0x035a55 (0x65bd bytes) 000698 1786171313.177300 write 0x00e305 thru 0x01d5e2 (0xf2de bytes) 000700 1786171313.194167 read 0x01f676 thru 0x02014d (0xad8 bytes) 000702 1786171313.201623 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x35a56): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35a56): Mirror 12: 0x1a0212a2 (stale) Mirror 13: 0x1a0212a2 (stale) Mirror 14: 0x1a0212a2 (stale) Mirror 15: 0x85b969bb (stale) CRC-32 checksum value for chunk [0, 0x35a56): Mirror 1: 0x23dde7a9 Mirror 16: 0x23dde7a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35a56: skipped 000703 1786171314.136818 read 0x0020fd thru 0x006252 (0x4156 bytes) 000705 1786171314.164850 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x35a56): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35a56): Mirror 12: 0x1a0212a2 (stale) Mirror 13: 0x1a0212a2 (stale) Mirror 14: 0x1a0212a2 (stale) Mirror 15: 0x85b969bb (stale) CRC-32 checksum value for chunk [0, 0x35a56): Mirror 1: 0x23dde7a9 Mirror 16: 0x23dde7a9 Mirror 17: 0x23dde7a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35a56: skipped 000709 1786171314.934093 trunc from 0x035a56 to 0x005188 (0x308cd bytes) 000711 1786171315.080413 MIRROR_RESYNC 7 mirrors 000714 1786171316.176615 trunc from 0x005189 to 0x028b75 (0x239ed bytes) 000715 1786171316.275673 trunc from 0x028b76 to 0x01382f (0x15346 bytes) 000716 1786171316.309991 read 0x00c0f5 thru 0x01382f (0x773b bytes) 000717 1786171316.325036 mapread 0x004a72 thru 0x010c0c (0xc19b bytes) 000718 1786171316.354715 write 0x00db17 thru 0x018ce3 (0xb1cd bytes) 000719 1786171316.358050 mapread 0x00d5c4 thru 0x015bc3 (0x8600 bytes) 000721 1786171316.362754 write 0x03e36b thru 0x03ffff (0x1c95 bytes) 000724 1786171316.369781 write 0x01e2c4 thru 0x0254c2 (0x71ff bytes) 000726 1786171316.384939 trunc from 0x040000 to 0x032cc3 (0xd33c bytes) 000727 1786171316.407955 write 0x033492 thru 0x036ee8 (0x3a57 bytes) 000728 1786171316.410473 read 0x00cf9e thru 0x015abb (0x8b1e bytes) 000729 1786171316.411627 mapread 0x01b08b thru 0x02221f (0x7195 bytes) 000730 1786171316.426101 write 0x029e3e thru 0x034d4e (0xaf11 bytes) 000731 1786171316.428102 trunc from 0x036ee9 to 0x030f60 (0x5f88 bytes) 000733 1786171316.453713 read 0x02272c thru 0x0296ce (0x6fa3 bytes) 000735 1786171316.456105 read 0x020652 thru 0x02637b (0x5d2a bytes) 000736 1786171316.458059 mapread 0x014f52 thru 0x01aa5a (0x5b09 bytes) 000737 1786171316.460282 mapwrite 0x00aebf thru 0x014a7e (0x9bc0 bytes) 000738 1786171316.515479 read 0x0118ee thru 0x01fd7c (0xe48f bytes) 000740 1786171316.517359 write 0x01d03e thru 0x02a37f (0xd342 bytes) 000741 1786171316.519963 read 0x00549a thru 0x00ea8e (0x95f5 bytes) 000743 1786171316.521465 mapread 0x01f622 thru 0x02cf2b (0xd90a bytes) 000744 1786171316.523409 mapwrite 0x027d87 thru 0x02c609 (0x4883 bytes) 000745 1786171316.583969 write 0x038c85 thru 0x03ffff (0x737b bytes) 000746 1786171316.590796 MIRROR_RESYNC 7 mirrors 000747 1786171317.745109 trunc from 0x040000 to 0x009782 (0x3687d bytes) 000748 1786171317.816654 write 0x00d47e thru 0x016cd9 (0x985c bytes) 000750 1786171317.824637 MIRROR_RESYNC 7 mirrors 000751 1786171318.857987 write 0x00d960 thru 0x01975d (0xbdfe bytes) 000752 1786171318.938389 trunc from 0x01975e to 0x0396e1 (0x1ff84 bytes) 000754 1786171318.963338 write 0x00b80f thru 0x01047a (0x4c6c bytes) 000755 1786171318.970442 read 0x00ad6c thru 0x01159f (0x6834 bytes) 000757 1786171318.977524 write 0x003d30 thru 0x00cbba (0x8e8b bytes) 000758 1786171318.985668 read 0x003beb thru 0x006f00 (0x3316 bytes) 000760 1786171318.987506 read 0x038af5 thru 0x0396e1 (0xbed bytes) 000761 1786171318.997473 mapwrite 0x02b260 thru 0x03a63f (0xf3e0 bytes) 000765 1786171319.091896 read 0x0003e3 thru 0x009fa3 (0x9bc1 bytes) 000766 1786171319.092907 write 0x02af95 thru 0x0337a6 (0x8812 bytes) 000768 1786171319.095982 trunc from 0x03a640 to 0x0067db (0x33e64 bytes) 000769 1786171319.116491 mapwrite 0x005d7b thru 0x006212 (0x498 bytes) 000771 1786171319.165635 trunc from 0x0067dc to 0x02c883 (0x260a8 bytes) 000773 1786171319.186544 read 0x0259e7 thru 0x02b01c (0x5636 bytes) 000774 1786171319.203963 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c884): Mirror 13: 0x1ebe1ba9 (stale) Mirror 14: 0x1ebe1ba9 (stale) Mirror 15: 0x1ebe1ba9 (stale) Mirror 16: 0x1ebe1ba9 (stale) Mirror 17: 0x1ebe1ba9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c884): Mirror 13: 0x1ebe1ba9 (stale) Mirror 14: 0x1ebe1ba9 (stale) Mirror 15: 0x1ebe1ba9 (stale) Mirror 16: 0x1ebe1ba9 (stale) Mirror 17: 0x1ebe1ba9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c884: skipped 000775 1786171319.638321 write 0x0284e8 thru 0x029ec3 (0x19dc bytes) 000776 1786171319.656875 write 0x006569 thru 0x0143ea (0xde82 bytes) 000777 1786171319.679433 trunc from 0x02c884 to 0x024ecf (0x79b4 bytes) 000778 1786171319.721820 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24ed0): Mirror 14: 0xcc649d10 (stale) Mirror 15: 0xcc649d10 (stale) Mirror 16: 0xcc649d10 (stale) Mirror 17: 0xcc649d10 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24ed0): Mirror 14: 0xcc649d10 (stale) Mirror 15: 0xcc649d10 (stale) Mirror 16: 0xcc649d10 (stale) Mirror 17: 0xcc649d10 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24ed0: skipped 000780 1786171320.197704 trunc from 0x024ed0 to 0x017a8d (0xd442 bytes) 000781 1786171320.225483 mapwrite 0x03725a thru 0x03b485 (0x422c bytes) 000782 1786171320.303282 read 0x02869e thru 0x029323 (0xc86 bytes) 000783 1786171320.314026 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b486): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b486): Mirror 14: 0x663a39a4 (stale) Mirror 15: 0x663a39a4 (stale) Mirror 16: 0x663a39a4 (stale) Mirror 17: 0x663a39a4 (stale) CRC-32 checksum value for chunk [0, 0x3b486): Mirror 1: 0x23d2206e Mirror 18: 0x23d2206e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b486: skipped 000785 1786171321.330580 mapread 0x0228b3 thru 0x02d44d (0xab9b bytes) 000786 1786171321.384894 MIRROR_RESYNC 6 mirrors 000791 1786171322.798529 mapread 0x01b6a9 thru 0x020490 (0x4de8 bytes) 000792 1786171322.822237 trunc from 0x03b486 to 0x013e04 (0x27681 bytes) 000795 1786171322.941252 write 0x0145fd thru 0x019eae (0x58b2 bytes) 000796 1786171322.948087 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19eaf): Mirror 15: 0xf5e2c4cb (stale) Mirror 16: 0xf5e2c4cb (stale) Mirror 17: 0xf5e2c4cb (stale) Mirror 18: 0xf5e2c4cb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19eaf): Mirror 15: 0xf5e2c4cb (stale) Mirror 16: 0xf5e2c4cb (stale) Mirror 17: 0xf5e2c4cb (stale) Mirror 18: 0xf5e2c4cb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19eaf: skipped 000800 1786171323.532790 trunc from 0x019eaf to 0x00d864 (0xc64a bytes) 000801 1786171323.567363 mapwrite 0x0331a1 thru 0x039101 (0x5f61 bytes) 000802 1786171323.689880 write 0x03012f thru 0x03c51a (0xc3ec bytes) 000803 1786171323.693502 mapwrite 0x028dc6 thru 0x0371b2 (0xe3ed bytes) 000804 1786171323.762802 MIRROR_RESYNC 5 mirrors 000807 1786171325.518364 mapread 0x039c3f thru 0x03c51a (0x28dc bytes) 000809 1786171325.549477 mapread 0x0162b8 thru 0x018eba (0x2c03 bytes) 000813 1786171325.569816 mapread 0x010c00 thru 0x01d094 (0xc495 bytes) 000815 1786171325.572298 trunc from 0x03c51b to 0x01a48c (0x2208e bytes) 000816 1786171325.697646 mapwrite 0x0164cb thru 0x02314e (0xcc84 bytes) 000819 1786171325.903681 mapread 0x008a83 thru 0x016b38 (0xe0b6 bytes) 000821 1786171325.953249 mapread 0x005165 thru 0x00f5dd (0xa479 bytes) 000823 1786171325.958526 write 0x0397ee thru 0x03ffff (0x6812 bytes) 000825 1786171325.968573 mapread 0x035713 thru 0x03ffff (0xa8ed bytes) 000826 1786171326.000261 MIRROR_RESYNC 5 mirrors 000828 1786171327.481766 trunc from 0x040000 to 0x016a9e (0x29561 bytes) 000829 1786171327.563869 write 0x0180ef thru 0x01e52f (0x6441 bytes) 000830 1786171327.566475 trunc from 0x01e530 to 0x026b53 (0x8624 bytes) 000831 1786171327.593492 write 0x028909 thru 0x0354ce (0xcbc6 bytes) 000832 1786171327.596422 read 0x016263 thru 0x020d76 (0xab14 bytes) 000833 1786171327.607197 read 0x0310cc thru 0x0354ce (0x4403 bytes) 000834 1786171327.608371 trunc from 0x0354cf to 0x003f0f (0x315bf bytes) 000835 1786171327.635517 mapread 0x000107 thru 0x003f0f (0x3e09 bytes) 000839 1786171327.647609 write 0x01afdd thru 0x01bc77 (0xc9b bytes) 000842 1786171327.651576 mapwrite 0x02760f thru 0x02bbcf (0x45c1 bytes) 000843 1786171327.765880 mapread 0x01e7f5 thru 0x02bbcf (0xd3db bytes) 000847 1786171327.771501 trunc from 0x02bbd0 to 0x034c1f (0x9050 bytes) 000850 1786171327.805890 mapwrite 0x008023 thru 0x011b01 (0x9adf bytes) 000851 1786171327.934012 trunc from 0x034c20 to 0x005e2c (0x2edf3 bytes) 000854 1786171327.983563 mapwrite 0x000170 thru 0x00e955 (0xe7e6 bytes) 000855 1786171328.241567 write 0x005e2f thru 0x01396e (0xdb40 bytes) 000860 1786171328.249901 mapwrite 0x0344db thru 0x03c8cb (0x83f1 bytes) 000862 1786171328.376081 write 0x02110d thru 0x024ec8 (0x3dbc bytes) 000863 1786171328.390340 write 0x02c6ef thru 0x033b5d (0x746f bytes) 000864 1786171328.392982 mapwrite 0x033bf2 thru 0x03b0e4 (0x74f3 bytes) 000866 1786171328.453020 mapwrite 0x025c16 thru 0x02e943 (0x8d2e bytes) 000867 1786171328.556721 trunc from 0x03c8cc to 0x02b8b2 (0x11019 bytes) 000868 1786171328.587975 mapwrite 0x028895 thru 0x031a8a (0x91f6 bytes) 000869 1786171328.716753 trunc from 0x031a8b to 0x01987f (0x1820b bytes) 000870 1786171328.741834 mapwrite 0x038762 thru 0x03acf2 (0x2591 bytes) 000871 1786171328.859505 read 0x01aed1 thru 0x02a3dd (0xf50d bytes) 000872 1786171328.877503 write 0x03a744 thru 0x03dbde (0x349b bytes) 000873 1786171328.880590 mapwrite 0x02c57b thru 0x037a73 (0xb4f9 bytes) 000875 1786171328.952008 mapwrite 0x02d124 thru 0x038960 (0xb83d bytes) 000876 1786171329.016366 write 0x00ec49 thru 0x0127cc (0x3b84 bytes) 000877 1786171329.018072 trunc from 0x03dbdf to 0x0229bf (0x1b21f bytes) 000878 1786171329.048870 read 0x00d0ab thru 0x013c0a (0x6b60 bytes) 000879 1786171329.050563 mapwrite 0x006a34 thru 0x015ea9 (0xf476 bytes) 000882 1786171329.124503 read 0x0006dd thru 0x0025e9 (0x1f0d bytes) 000883 1786171329.125707 trunc from 0x0229c0 to 0x00f3a0 (0x1361f bytes) 000884 1786171329.153587 write 0x020c0c thru 0x029019 (0x840e bytes) 000886 1786171329.157788 mapread 0x0125dc thru 0x014f35 (0x295a bytes) 000888 1786171329.170754 write 0x028704 thru 0x02a8ba (0x21b7 bytes) 000890 1786171329.173891 mapwrite 0x0336c4 thru 0x03ca20 (0x935d bytes) 000891 1786171329.315012 write 0x02501c thru 0x0307fa (0xb7df bytes) 000892 1786171329.331653 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3ca21): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ca21): Mirror 15: 0xff592fe6 (stale) Mirror 16: 0xff592fe6 (stale) Mirror 17: 0xff592fe6 (stale) Mirror 18: 0xff592fe6 (stale) CRC-32 checksum value for chunk [0, 0x3ca21): Mirror 1: 0xc54de2cc Mirror 19: 0xc54de2cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ca21: skipped 000893 1786171330.982564 trunc from 0x03ca21 to 0x02b874 (0x111ac bytes) 000894 1786171331.121564 write 0x03d699 thru 0x03ffff (0x2967 bytes) 000895 1786171331.126454 mapread 0x0167ff thru 0x01c6d4 (0x5ed6 bytes) 000896 1786171331.155732 trunc from 0x040000 to 0x010c60 (0x2f39f bytes) 000898 1786171331.195516 trunc from 0x010c61 to 0x03cb89 (0x2bf29 bytes) 000902 1786171331.229749 mapwrite 0x039ba7 thru 0x03c335 (0x278f bytes) 000904 1786171331.300943 write 0x002310 thru 0x00f2fa (0xcfeb bytes) 000905 1786171331.317737 mapwrite 0x035fd4 thru 0x03ffff (0xa02c bytes) 000906 1786171331.415965 mapread 0x01a830 thru 0x023723 (0x8ef4 bytes) 000907 1786171331.440650 write 0x00f0f6 thru 0x01a68e (0xb599 bytes) 000909 1786171331.443565 mapread 0x004911 thru 0x009d66 (0x5456 bytes) 000911 1786171331.452375 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xf56d3e05 (stale) Mirror 16: 0xf56d3e05 (stale) Mirror 17: 0xf56d3e05 (stale) Mirror 18: 0xf56d3e05 (stale) Mirror 19: 0x52494c4f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x552ecac6 Mirror 20: 0x552ecac6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000913 1786171333.331378 trunc from 0x040000 to 0x01e6ed (0x21912 bytes) 000914 1786171333.437624 read 0x018662 thru 0x01ce17 (0x47b6 bytes) 000915 1786171333.451346 mapread 0x00fc21 thru 0x010c9a (0x107a bytes) 000917 1786171333.463695 mapread 0x00140f thru 0x00ce76 (0xba68 bytes) 000918 1786171333.476262 trunc from 0x01e6ee to 0x039075 (0x1a988 bytes) 000919 1786171333.509581 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x39076): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39076): Mirror 15: 0xe075a4a5 (stale) Mirror 16: 0xe075a4a5 (stale) Mirror 17: 0xe075a4a5 (stale) Mirror 18: 0xe075a4a5 (stale) Mirror 19: 0xe1108d20 (stale) Mirror 20: 0x359d2771 (stale) CRC-32 checksum value for chunk [0, 0x39076): Mirror 1: 0x442f75e Mirror 21: 0x442f75e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39076: skipped 000923 1786171334.639911 mapwrite 0x028b12 thru 0x033d48 (0xb237 bytes) 000925 1786171334.843022 trunc from 0x039076 to 0x02aa23 (0xe652 bytes) 000927 1786171334.873226 read 0x004877 thru 0x0070d4 (0x285e bytes) 000928 1786171334.881811 mapread 0x012a6b thru 0x01f385 (0xc91b bytes) 000929 1786171334.898767 mapread 0x0078ac thru 0x00e375 (0x6aca bytes) 000930 1786171334.909940 write 0x006f68 thru 0x011c6b (0xad04 bytes) 000933 1786171334.917555 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2aa24): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aa24): Mirror 15: 0x1b0669df (stale) Mirror 16: 0x1b0669df (stale) Mirror 17: 0x1b0669df (stale) Mirror 18: 0x1b0669df (stale) Mirror 19: 0x261addce (stale) Mirror 20: 0x99579503 (stale) Mirror 21: 0x99579503 (stale) CRC-32 checksum value for chunk [0, 0x2aa24): Mirror 1: 0x9b8f11be Mirror 22: 0x9b8f11be lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aa24: skipped 000935 1786171336.069002 read 0x00f3f0 thru 0x0161f1 (0x6e02 bytes) 000936 1786171336.090875 mapread 0x00bbd0 thru 0x010511 (0x4942 bytes) 000938 1786171336.135980 trunc from 0x02aa24 to 0x02e3b3 (0x3990 bytes) 000939 1786171336.236483 read 0x00c9f6 thru 0x0152f2 (0x88fd bytes) 000940 1786171336.247924 mapread 0x02a66b thru 0x02ae87 (0x81d bytes) 000941 1786171336.249285 mapwrite 0x02873c thru 0x03098c (0x8251 bytes) 000942 1786171336.332612 mapwrite 0x006f0c thru 0x011eb2 (0xafa7 bytes) 000943 1786171336.380723 mapwrite 0x029c63 thru 0x02f71d (0x5abb bytes) 000944 1786171336.426334 mapwrite 0x00727c thru 0x00cc29 (0x59ae bytes) 000945 1786171336.472561 mapwrite 0x00257e thru 0x01144e (0xeed1 bytes) 000946 1786171336.524456 mapwrite 0x02322c thru 0x02881e (0x55f3 bytes) 000948 1786171336.570485 MIRROR_RESYNC 9 mirrors 000950 1786171337.906987 read 0x027040 thru 0x02740b (0x3cc bytes) 000952 1786171337.913059 read 0x013e2a thru 0x0230a3 (0xf27a bytes) 000953 1786171337.927604 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000954 1786171338.317628 trunc from 0x03098d to 0x01b09a (0x158f2 bytes) 000957 1786171338.407718 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b09b): Mirror 16: 0x5f5ef63 (stale) Mirror 17: 0x5f5ef63 (stale) Mirror 18: 0x5f5ef63 (stale) Mirror 19: 0x5f5ef63 (stale) Mirror 20: 0x5f5ef63 (stale) Mirror 21: 0x5f5ef63 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b09b): Mirror 16: 0x5f5ef63 (stale) Mirror 17: 0x5f5ef63 (stale) Mirror 18: 0x5f5ef63 (stale) Mirror 19: 0x5f5ef63 (stale) Mirror 20: 0x5f5ef63 (stale) Mirror 21: 0x5f5ef63 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b09b: skipped 000958 1786171338.850795 mapwrite 0x00315f thru 0x00f693 (0xc535 bytes) 000959 1786171338.930618 mapwrite 0x0205b9 thru 0x022590 (0x1fd8 bytes) 000963 1786171339.003920 mapwrite 0x01f4c1 thru 0x022ec1 (0x3a01 bytes) 000964 1786171339.061724 mapread 0x01b436 thru 0x022ec1 (0x7a8c bytes) 000966 1786171339.065373 mapwrite 0x029262 thru 0x02a056 (0xdf5 bytes) 000967 1786171339.133966 read 0x01e5d0 thru 0x02932d (0xad5e bytes) 000968 1786171339.135889 mapread 0x019663 thru 0x022b69 (0x9507 bytes) 000969 1786171339.138711 mapread 0x01d7c9 thru 0x0253de (0x7c16 bytes) 000970 1786171339.141052 mapread 0x022286 thru 0x02787d (0x55f8 bytes) 000973 1786171339.143788 read 0x013c66 thru 0x017623 (0x39be bytes) 000975 1786171339.154848 mapread 0x002cba thru 0x012688 (0xf9cf bytes) 000976 1786171339.157220 mapread 0x023f8e thru 0x025eaa (0x1f1d bytes) 000985 1786171339.164698 read 0x003f62 thru 0x008b9a (0x4c39 bytes) 000988 1786171339.166922 trunc from 0x02a057 to 0x02d959 (0x3903 bytes) 000989 1786171339.186372 trunc from 0x02d95a to 0x0314db (0x3b82 bytes) 000990 1786171339.206908 read 0x027dcc thru 0x0314db (0x9710 bytes) 000993 1786171339.219004 read 0x006219 thru 0x01220a (0xbff2 bytes) 000994 1786171339.220007 write 0x007e97 thru 0x00ee2a (0x6f94 bytes) 000998 1786171339.224894 write 0x025863 thru 0x034182 (0xe920 bytes) 000999 1786171339.227391 write 0x00d63b thru 0x0139f7 (0x63bd bytes) All operations completed A-OK! PASS 70a (87s) == sanity-flr test 71: check mirror extend parent fid ==== 02:42:25 (1786171345) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 02:42:29 (1786171349) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.326326 s, 6.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150992 s, 6.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b6:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b6:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 02:42:39 (1786171359) Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre2 Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 258720 .. Extending file size to 2579264 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 3461952 .. Extending file size to 8278464 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5944896 .. Extending file size to 7501632 .. Extending file size to 8588768 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7636352 .. Extending file size to 8582528 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8727712 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9043648 .. Waiting 57359 57361 57362 57363 57365 done 192.168.204.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg448-client.virtnet /mnt/lustre2 (opts:) 192.168.204.148@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg448-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 02:44:01 (1786171441) Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre2 Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3784928 .. Extending file size to 8197632 .. done Extending file size to 8343744 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5390720 .. Extending file size to 8564288 .. Extending file size to 7926560 .. Extending file size to 9089664 .. Extending file size to 9002176 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9095296 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 58721 58722 58723 58726 58727 192.168.204.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg448-client.virtnet /mnt/lustre2 (opts:) 192.168.204.148@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg448-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (82s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 02:45:24 (1786171524) Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0926961 s, 22.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.440276 s, 4.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x72a83156 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg448-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 02:45:47 (1786171547) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0921844 s, 22.7 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 02:45:57 (1786171557) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d1:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } PASS 203 (10s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 02:46:07 (1786171567) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.254063 s, 8.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (11s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 02:46:18 (1786171578) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.237306 s, 8.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2eb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2eb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ec:0x0] } PASS 204b (11s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 02:46:29 (1786171589) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.294969 s, 7.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } PASS 204c (38s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 02:47:07 (1786171627) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg448-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219995 s, 4.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } PASS 204d (63s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 02:48:10 (1786171690) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg448-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.295963 s, 7.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (69s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 02:49:19 (1786171759) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg448-server mds-ost sync done. Creating new pool test_204f oleg448-server: Pool lustre.test_204f created Adding targets to pool oleg448-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg448-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.635008 s, 3.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg448-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg448-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg448-server: Pool lustre.test_204f destroyed PASS 204f (87s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 02:50:47 (1786171847) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 02:50:56 (1786171856) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 02:51:06 (1786171866) oleg448-server: Pool lustre.test_206 created Adding targets to pool oleg448-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg448-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg448-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg448-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg448-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 02:51:36 (1786171896) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121066 s, 8.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101348 s, 10.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xec8370be Mirror 3: 0xec8370be PASS 207 (16s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 02:51:52 (1786171912) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (6s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 02:51:58 (1786171918) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 02:52:03 (1786171923) Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.177386 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xa60000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2400000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2400000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2400000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x790000 Read this (4 bytes): BBBB 192.168.204.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg448-client.virtnet /mnt/lustre2 (opts:) PASS 209a (44s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 02:52:47 (1786171967) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0175368 s, 467 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 02:52:56 (1786171976) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557082 s, 18.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f6 lmm_fid: [0x200000404:0x1f6:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f6 lmm_fid: [0x200000404:0x1f6:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834422 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-08 02:52:58.000000000 -0400 Modify: 2026-08-08 02:52:58.000000000 -0400 Change: 2026-08-08 02:52:59.000000000 -0400 Birth: 2026-08-08 02:52:57.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 02:53:05 (1786171985) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 02:53:10 (1786171990) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.60149 s, 68.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0460013 s, 89.0 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 8 02:53 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f8:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 8 02:53 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 02:53:20 (1786172000) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 02:53:29 (1786172009) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (13s) == sanity-flr test complete, duration 1930 sec =========== 02:53:42 (1786172022) === sanity-flr: start cleanup 02:53:44 (1786172024) === === sanity-flr: finish cleanup 02:53:51 (1786172031) ===