-----============= acceptance-small: sanity-flr ============----- Mon Aug 10 23:16:17 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg217-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 23:16:35 (1786418195) === oleg217-client.virtnet: executing check_config_client /mnt/lustre oleg217-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg217-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9042c7aa2800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9042c7aa2800.idle_timeout=debug disable quota as required oleg217-server: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 23:17:06 (1786418226) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7612] == sanity-flr test 0a: lfs mirror create with -N option == 23:17:08 (1786418228) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 23:17:22 (1786418242) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 23:17:26 (1786418246) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 23:17:29 (1786418249) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 23:17:57 (1786418277) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 23:18:01 (1786418281) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 23:18:04 (1786418284) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9042c7aa2800.stats=clear osc.lustre-OST0001-osc-ffff9042c7aa2800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 23:18:14 (1786418294) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (14s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 23:18:28 (1786418308) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 23:18:38 (1786418318) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 23:18:41 (1786418321) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 23:18:51 (1786418331) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 23:19:02 (1786418342) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 23:19:09 (1786418349) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 23:19:12 (1786418352) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 23:19:19 (1786418359) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 23:19:29 (1786418369) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 23:19:36 (1786418376) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.01229 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869677 s, 12.1 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 23:19:47 (1786418387) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.295624 s, 28.4 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 23:19:54 (1786418394) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 23:20:05 (1786418405) Stopping /mnt/lustre-ost1 (opts:) on oleg217-server oleg217-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9042c7aa2800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9042c7aa2800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg217-server: oleg217-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg217-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg217-server oleg217-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9042c7aa2800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9042c7aa2800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg217-server: oleg217-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg217-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (72s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 23:21:17 (1786418477) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.690943 s, 15.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg217-server oleg217-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9042c7aa2800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9042c7aa2800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg217-server: oleg217-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg217-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (45s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 23:22:02 (1786418522) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg217-server oleg217-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9042c7aa2800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9042c7aa2800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg217-server: oleg217-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg217-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg217-server oleg217-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9042c7aa2800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9042c7aa2800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg217-server: oleg217-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg217-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (83s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 23:23:25 (1786418605) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.11548 s, 12.7 MB/s -rw-r--r-- 1 root root 52428800 Aug 10 23:23 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.900s user 0m0.001s sys 0m0.160s Stopping /mnt/lustre-ost1 (opts:) on oleg217-server oleg217-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9042c7aa2800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9042c7aa2800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.292s user 0m0.001s sys 0m0.186s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg217-server: oleg217-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg217-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg217-server oleg217-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9042c7aa2800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9042c7aa2800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.316s user 0m0.001s sys 0m0.188s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg217-server: oleg217-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg217-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg217-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9042c7aa2800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9042c7aa2800.ost_server_uuid in FULL state after 0 sec PASS 33b (98s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 23:25:04 (1786418704) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 23:25:07 (1786418707) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 23:25:11 (1786418711) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 23:25:16 (1786418716) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 23:25:26 (1786418726) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.712027 s, 17.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 23:25:41 (1786418741) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606513 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313024 s, 33.5 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 23:25:49 (1786418749) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 23:25:57 (1786418757) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00112052 s, 914 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000803168 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000837487 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000842271 s, 1.2 MB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 23:26:06 (1786418766) create test files with size 15738419 .. 0e6b26616c24fee4065582b60c4c64fd - f98c071e8a928835a60bbc96ba65fe94 - 8fc0dc239f82ce6c85ff4140c618a749 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre PASS 37 (21s) == sanity-flr test 38: resync ============================ 23:26:27 (1786418787) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 23:26:42 (1786418802) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 23:26:48 (1786418808) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972359 s, 10.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382811 s, 27.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 23:26:58 (1786418818) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.109437 s, 19.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.437923 s, 9.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.306214 s, 6.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 23:27:16 (1786418836) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 23:27:20 (1786418840) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 23:27:23 (1786418843) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268205 s, 39.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 23:27:32 (1786418852) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 23:27:36 (1786418856) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 23:27:47 (1786418867) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.281035 s, 37.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 23:28:00 (1786418880) llite.lustre-ffff9042d02c5800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.230626 s, 45.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9042c5dec000.xattr_cache=1 PASS 44e (12s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 23:28:12 (1786418892) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (32s) == sanity-flr test 46: Verify setstripe --copy option ==== 23:28:44 (1786418924) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (31s) == sanity-flr test 47: Verify mirror obj alloc =========== 23:29:15 (1786418955) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 23:29:18 (1786418958) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.158845 s, 19.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.200333 s, 15.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xed05a867 (nosync,stale) Mirror 2: 0xed05a867 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 8ae85b0714eb09b5b78cf9c350a38cab - mirror 1 checksum: dc0dfbc644d3b28e15d5cb4d779e2d84 - mirror 2 checksum: 8ae85b0714eb09b5b78cf9c350a38cab - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-10T23:29:20-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-10T23:29:20-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 8ae85b0714eb09b5b78cf9c350a38cab - mirror 1 checksum: dc0dfbc644d3b28e15d5cb4d779e2d84 - mirror 2 checksum: dc0dfbc644d3b28e15d5cb4d779e2d84 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-10T23:29:20-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-10T23:29:20-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 23:29:31 (1786418971) Failing mds1 on oleg217-server Stopping /mnt/lustre-mds1 (opts:) on oleg217-server 23:29:38 (1786418978) shut down facet: mds1 facet_host: oleg217-server facet_failover_host: oleg217-server Failover mds1 to oleg217-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg217-server: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-MDT0000 23:30:01 (1786419001) targets are mounted 23:30:01 (1786419001) facet_failover done oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274594 s, 38.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } PASS 50A (49s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 23:30:20 (1786419020) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 23:30:25 (1786419025) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 23:30:29 (1786419029) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00585854 s, 2.8 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 23:30:37 (1786419037) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 23:30:40 (1786419040) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0149027 s, 275 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0593779 s, 69.0 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 23:30:48 (1786419048) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 23:31:11 (1786419071) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 23:31:40 (1786419100) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 23:32:06 (1786419126) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 23:32:30 (1786419150) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 23:32:38 (1786419158) mirror create pid 54293 mirror split pid 54294 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54293 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54294 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 23:33:45 (1786419225) Seed set to 9228 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1786419228.430955 MIRROR_EXTEND to 2 mirrors 000003 1786419228.976543 write 0x03cf22 thru 0x03ffff (0x30de bytes) 000005 1786419229.058372 mapwrite 0x036c16 thru 0x03b6c4 (0x4aaf bytes) truncating to largest ever: 0xccfd 000006 1786419229.246616 trunc from 0x040000 to 0x00ccfc (0x33303 bytes) 000008 1786419229.313748 read 0x000e85 thru 0x00aca9 (0x9e25 bytes) 000009 1786419229.329931 mapwrite 0x01fe52 thru 0x026c30 (0x6ddf bytes) 000011 1786419229.423109 mapread 0x013fc4 thru 0x0146b0 (0x6ed bytes) 000013 1786419229.441487 read 0x01d6f0 thru 0x026c30 (0x9541 bytes) 000015 1786419229.445524 write 0x035400 thru 0x03641b (0x101c bytes) 000016 1786419229.448484 mapread 0x003857 thru 0x0113a5 (0xdb4f bytes) 000017 1786419229.464530 MIRROR_RESYNC 2 mirrors 000018 1786419230.362326 mapread 0x01a3e1 thru 0x025295 (0xaeb5 bytes) 000019 1786419230.390937 write 0x011e92 thru 0x0178e4 (0x5a53 bytes) 000020 1786419230.488358 MIRROR_RESYNC 2 mirrors 000021 1786419231.269745 MIRROR_RESYNC 2 mirrors 000022 1786419231.695418 mapwrite 0x016871 thru 0x01a76f (0x3eff bytes) 000023 1786419231.933382 mapwrite 0x02e3e0 thru 0x039e41 (0xba62 bytes) 000024 1786419232.043726 write 0x00b1dc thru 0x013631 (0x8456 bytes) 000025 1786419232.062250 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x39e42): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39e42): Mirror 2: 0x85353551 (stale) CRC-32 checksum value for chunk [0, 0x39e42): Mirror 1: 0xb8da1a35 Mirror 3: 0xb8da1a35 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39e42: skipped 000028 1786419233.295893 mapread 0x0041cc thru 0x00f5e9 (0xb41e bytes) 000030 1786419233.344538 write 0x0173a7 thru 0x0228a2 (0xb4fc bytes) 000033 1786419233.451566 write 0x03307b thru 0x03a3b8 (0x733e bytes) 000034 1786419233.460444 write 0x02ce8a thru 0x03c828 (0xf99f bytes) 000035 1786419233.472791 read 0x03c5d9 thru 0x03c7fb (0x223 bytes) 000037 1786419233.483709 MIRROR_RESYNC 3 mirrors 000038 1786419234.499516 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) truncating to largest ever: 0xeb55 000039 1786419234.794743 trunc from 0x03c829 to 0x00eb54 (0x2dcd4 bytes) 000041 1786419234.871477 write 0x01b98a thru 0x026872 (0xaee9 bytes) 000043 1786419234.875534 write 0x01dfd2 thru 0x023271 (0x52a0 bytes) 000044 1786419234.877581 mapwrite 0x01d9b3 thru 0x02aac7 (0xd115 bytes) 000045 1786419234.977404 MIRROR_RESYNC 2 mirrors 000046 1786419235.677799 read 0x01008f thru 0x0175a3 (0x7515 bytes) 000048 1786419235.687037 write 0x01b206 thru 0x0282eb (0xd0e6 bytes) 000049 1786419235.767497 mapread 0x00ed1d thru 0x0118e5 (0x2bc9 bytes) 000050 1786419235.785981 write 0x007eae thru 0x010183 (0x82d6 bytes) 000051 1786419235.791521 mapread 0x014ea1 thru 0x0246ce (0xf82e bytes) 000052 1786419235.798955 mapread 0x01b496 thru 0x023cc7 (0x8832 bytes) 000053 1786419235.800845 mapread 0x01f9f9 thru 0x022286 (0x288e bytes) 000056 1786419235.805802 mapwrite 0x0354b1 thru 0x03ffff (0xab4f bytes) truncating to largest ever: 0x3563c 000057 1786419235.926214 trunc from 0x040000 to 0x03563b (0xa9c4 bytes) 000058 1786419235.942220 write 0x039f9b thru 0x03ffff (0x6065 bytes) 000059 1786419235.944460 mapwrite 0x039756 thru 0x03ca5c (0x3307 bytes) 000061 1786419235.987384 write 0x011d1a thru 0x01ba17 (0x9cfe bytes) 000062 1786419235.988752 mapread 0x016e92 thru 0x0206b7 (0x9826 bytes) 000063 1786419235.991251 write 0x03f450 thru 0x03ffff (0xbb0 bytes) 000064 1786419235.992948 trunc from 0x040000 to 0x0294ac (0x16b53 bytes) 000065 1786419236.013222 mapread 0x028615 thru 0x028932 (0x31e bytes) 000066 1786419236.018776 MIRROR_RESYNC 2 mirrors 000067 1786419236.543722 trunc from 0x0294ad to 0x00c2db (0x1d1d1 bytes) 000068 1786419236.634629 mapread 0x00be74 thru 0x00c2db (0x468 bytes) 000069 1786419236.645551 read 0x00808b thru 0x008fd8 (0xf4e bytes) 000070 1786419236.656738 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) 000071 1786419236.799739 write 0x026319 thru 0x034891 (0xe579 bytes) 000072 1786419236.803382 trunc from 0x034892 to 0x012bd3 (0x21cbe bytes) 000076 1786419236.821188 mapwrite 0x033f44 thru 0x0379ba (0x3a77 bytes) 000077 1786419236.928612 read 0x02c74b thru 0x0379ba (0xb270 bytes) 000081 1786419236.939441 read 0x006e92 thru 0x0093ac (0x251b bytes) 000083 1786419236.951375 mapwrite 0x003c43 thru 0x00d340 (0x96fe bytes) 000086 1786419237.020758 trunc from 0x0379bb to 0x024d74 (0x12c46 bytes) 000091 1786419237.061359 MIRROR_EXTEND to 2 mirrors 000092 1786419238.535894 mapwrite 0x01e85c thru 0x01f26c (0xa11 bytes) 000093 1786419238.792866 mapread 0x0010e3 thru 0x006500 (0x541e bytes) 000094 1786419238.813410 mapwrite 0x0121ba thru 0x017437 (0x527e bytes) 000097 1786419238.895983 mapwrite 0x00ca8a thru 0x015a14 (0x8f8b bytes) 000098 1786419238.984203 mapwrite 0x00cb39 thru 0x00edc7 (0x228f bytes) 000099 1786419239.115153 mapwrite 0x010140 thru 0x01e9f2 (0xe8b3 bytes) 000100 1786419239.227947 mapwrite 0x01a358 thru 0x025ecd (0xbb76 bytes) 000101 1786419239.383187 trunc from 0x025ece to 0x016f44 (0xef89 bytes) 000102 1786419239.424880 trunc from 0x016f45 to 0x01102d (0x5f17 bytes) 000106 1786419239.462212 mapwrite 0x03fb11 thru 0x03ffff (0x4ef bytes) 000111 1786419239.582267 write 0x00fa19 thru 0x0186b7 (0x8c9f bytes) 000113 1786419239.604725 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) 000115 1786419239.741928 trunc from 0x040000 to 0x00f0eb (0x30f14 bytes) 000116 1786419239.767402 MIRROR_EXTEND to 2 mirrors 000117 1786419240.374551 trunc from 0x00f0ec to 0x0063ab (0x8d40 bytes) 000118 1786419240.450693 MIRROR_RESYNC 2 mirrors 000119 1786419241.221714 write 0x012f4f thru 0x01abe9 (0x7c9b bytes) 000120 1786419241.403015 mapread 0x016f93 thru 0x01abe9 (0x3c57 bytes) 000122 1786419241.409763 write 0x011caf thru 0x01a354 (0x86a6 bytes) 000123 1786419241.448396 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1abea): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1abea): Mirror 2: 0x2ea9bb96 (stale) CRC-32 checksum value for chunk [0, 0x1abea): Mirror 1: 0x81fa4c15 Mirror 3: 0x81fa4c15 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1abea: skipped 000124 1786419242.803246 trunc from 0x01abea to 0x026339 (0xb750 bytes) 000126 1786419242.918473 mapread 0x0091ec thru 0x00a83a (0x164f bytes) 000127 1786419242.929642 trunc from 0x02633a to 0x03104a (0xad11 bytes) 000128 1786419242.954158 MIRROR_RESYNC 3 mirrors 000129 1786419243.919564 read 0x008961 thru 0x013488 (0xab28 bytes) 000131 1786419243.929658 read 0x0139a0 thru 0x0216a5 (0xdd06 bytes) 000132 1786419243.940158 mapwrite 0x002fec thru 0x009053 (0x6068 bytes) 000135 1786419244.127560 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3104b): Mirror 2: 0x353dca2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3104b): Mirror 2: 0x353dca2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3104b: skipped 000136 1786419244.553391 trunc from 0x03104b to 0x012f1a (0x1e130 bytes) 000137 1786419244.572801 mapwrite 0x002c9f thru 0x005695 (0x29f7 bytes) 000138 1786419244.640109 read 0x00b89f thru 0x00fd80 (0x44e2 bytes) 000139 1786419244.640853 read 0x00995d thru 0x00f013 (0x56b7 bytes) 000140 1786419244.641493 mapwrite 0x0177ca thru 0x01f7b6 (0x7fed bytes) 000141 1786419244.717793 mapwrite 0x01b8f6 thru 0x025ed7 (0xa5e2 bytes) 000142 1786419244.795375 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) 000145 1786419244.943513 MIRROR_EXTEND to 2 mirrors 000149 1786419245.953788 mapwrite 0x013ac8 thru 0x014a00 (0xf39 bytes) 000153 1786419246.127317 MIRROR_RESYNC 2 mirrors 000154 1786419246.869228 read 0x00e5a7 thru 0x01688c (0x82e6 bytes) 000155 1786419246.878617 read 0x017dd8 thru 0x022b32 (0xad5b bytes) 000156 1786419246.892279 MIRROR_RESYNC 2 mirrors 000157 1786419247.301496 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) 000160 1786419247.472083 write 0x02006a thru 0x02f322 (0xf2b9 bytes) 000162 1786419247.501196 trunc from 0x02f323 to 0x02c334 (0x2fee bytes) 000163 1786419247.518805 mapwrite 0x02c002 thru 0x03ba63 (0xfa62 bytes) 000164 1786419247.610272 mapwrite 0x028b36 thru 0x02ce22 (0x42ed bytes) 000165 1786419247.647430 read 0x0339e5 thru 0x03ba63 (0x807f bytes) 000167 1786419247.649479 read 0x015f51 thru 0x019d9f (0x3e4f bytes) 000170 1786419247.661384 mapwrite 0x011c57 thru 0x019a25 (0x7dcf bytes) 000171 1786419247.708130 read 0x01c711 thru 0x023679 (0x6f69 bytes) 000172 1786419247.708912 mapwrite 0x00833b thru 0x013b2e (0xb7f4 bytes) 000174 1786419247.758435 write 0x03b61b thru 0x03ffff (0x49e5 bytes) 000175 1786419247.759641 mapread 0x00bc1b thru 0x00e95e (0x2d44 bytes) 000179 1786419247.781142 write 0x03dd74 thru 0x03ffff (0x228c bytes) 000180 1786419247.782583 mapwrite 0x02d1c4 thru 0x039afd (0xc93a bytes) 000181 1786419247.837026 read 0x022b57 thru 0x02ffac (0xd456 bytes) 000184 1786419247.838928 mapwrite 0x00b725 thru 0x0124e6 (0x6dc2 bytes) 000186 1786419247.887521 mapwrite 0x01ef21 thru 0x02d2f6 (0xe3d6 bytes) 000187 1786419247.949170 MIRROR_EXTEND to 2 mirrors 000188 1786419248.653916 MIRROR_EXTEND to 3 mirrors 000191 1786419249.427524 write 0x030897 thru 0x03c85c (0xbfc6 bytes) 000193 1786419249.547059 read 0x003b27 thru 0x00be40 (0x831a bytes) 000194 1786419249.559678 read 0x03ce5f thru 0x03ffff (0x31a1 bytes) 000198 1786419249.583510 write 0x014019 thru 0x022a20 (0xea08 bytes) 000199 1786419249.588475 write 0x027194 thru 0x0354c6 (0xe333 bytes) 000200 1786419249.594515 mapread 0x0167c3 thru 0x01714d (0x98b bytes) 000201 1786419249.605530 MIRROR_RESYNC 3 mirrors 000202 1786419250.532332 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) 000203 1786419250.819456 mapwrite 0x02c03b thru 0x02d4c3 (0x1489 bytes) 000206 1786419251.040173 mapread 0x019dd9 thru 0x01e176 (0x439e bytes) 000207 1786419251.058377 mapwrite 0x0322f7 thru 0x035e41 (0x3b4b bytes) 000211 1786419251.110690 read 0x032a9d thru 0x03f855 (0xcdb9 bytes) 000212 1786419251.130904 trunc from 0x040000 to 0x02288a (0x1d775 bytes) 000213 1786419251.166456 trunc from 0x02288b to 0x003305 (0x1f585 bytes) 000214 1786419251.189186 mapread 0x00203c thru 0x003305 (0x12ca bytes) 000219 1786419251.203805 mapread 0x001266 thru 0x003305 (0x20a0 bytes) 000220 1786419251.215322 trunc from 0x003306 to 0x01f31d (0x1c018 bytes) 000221 1786419251.251964 write 0x02ef81 thru 0x03d2ed (0xe36d bytes) 000222 1786419251.263752 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) 000223 1786419251.425308 write 0x024848 thru 0x02a286 (0x5a3f bytes) 000224 1786419251.455400 mapwrite 0x0209f2 thru 0x02a6f5 (0x9d04 bytes) 000225 1786419251.562922 write 0x02c390 thru 0x03c1c1 (0xfe32 bytes) 000227 1786419251.579143 mapwrite 0x01d5c7 thru 0x01d71e (0x158 bytes) 000228 1786419251.663009 write 0x016985 thru 0x017f0c (0x1588 bytes) 000229 1786419251.680564 mapwrite 0x00cfda thru 0x013455 (0x647c bytes) truncating to largest ever: 0x37c53 000231 1786419251.794139 trunc from 0x03d2ee to 0x037c52 (0x569b bytes) 000232 1786419251.848563 read 0x01217a thru 0x020f33 (0xedba bytes) 000233 1786419251.877415 read 0x000b06 thru 0x00e854 (0xdd4f bytes) 000235 1786419251.910460 trunc from 0x037c53 to 0x003322 (0x34930 bytes) 000237 1786419251.939141 mapwrite 0x03a683 thru 0x03ffff (0x597d bytes) 000238 1786419252.082779 write 0x036ca3 thru 0x037cb9 (0x1017 bytes) 000239 1786419252.098873 mapread 0x0175c8 thru 0x024113 (0xcb4c bytes) 000242 1786419252.175983 mapread 0x022130 thru 0x02de84 (0xbd55 bytes) 000243 1786419252.217468 mapread 0x013e87 thru 0x0213c3 (0xd53d bytes) 000245 1786419252.259339 MIRROR_EXTEND to 2 mirrors 000246 1786419253.072233 trunc from 0x040000 to 0x01c800 (0x237ff bytes) 000248 1786419253.268225 trunc from 0x01c801 to 0x034350 (0x17b50 bytes) 000249 1786419253.293933 mapread 0x01c154 thru 0x021215 (0x50c2 bytes) 000250 1786419253.323779 trunc from 0x034351 to 0x027f6b (0xc3e5 bytes) 000251 1786419253.349653 read 0x0176ce thru 0x026fb9 (0xf8ec bytes) 000252 1786419253.351563 write 0x030477 thru 0x03ffff (0xfb89 bytes) 000254 1786419253.358531 read 0x02f468 thru 0x03e8e0 (0xf479 bytes) 000255 1786419253.376381 read 0x005aa0 thru 0x014ed7 (0xf438 bytes) 000257 1786419253.378321 mapread 0x03085c thru 0x032758 (0x1efd bytes) 000258 1786419253.383607 write 0x01b621 thru 0x026448 (0xae28 bytes) 000260 1786419253.396593 MIRROR_RESYNC 2 mirrors 000262 1786419254.190746 MIRROR_EXTEND to 3 mirrors 000264 1786419255.376747 write 0x033f09 thru 0x03ffff (0xc0f7 bytes) 000265 1786419255.470959 mapread 0x01f8ca thru 0x029b0e (0xa245 bytes) 000266 1786419255.508433 read 0x01b35b thru 0x025427 (0xa0cd bytes) 000267 1786419255.509356 mapwrite 0x015a44 thru 0x019f10 (0x44cd bytes) 000268 1786419255.592794 trunc from 0x040000 to 0x02b7f0 (0x1480f bytes) 000269 1786419255.632843 write 0x00a052 thru 0x00b12d (0x10dc bytes) 000271 1786419255.644644 mapread 0x004c85 thru 0x0145a3 (0xf91f bytes) 000273 1786419255.667580 write 0x02298e thru 0x023f03 (0x1576 bytes) 000274 1786419255.669195 mapwrite 0x02dca7 thru 0x038bdf (0xaf39 bytes) 000275 1786419255.831836 mapread 0x025dcc thru 0x02b791 (0x59c6 bytes) 000276 1786419255.834681 trunc from 0x038be0 to 0x015759 (0x23486 bytes) 000277 1786419255.895115 read 0x011778 thru 0x015759 (0x3fe2 bytes) 000278 1786419255.896304 read 0x00ea4a thru 0x015759 (0x6d10 bytes) 000280 1786419255.907610 mapread 0x012271 thru 0x015759 (0x34e9 bytes) 000284 1786419255.916227 read 0x008c62 thru 0x00e864 (0x5c03 bytes) 000285 1786419255.935698 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1575a): Mirror 3: 0xa465ee0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1575a): Mirror 3: 0xa465ee0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1575a: skipped 000291 1786419256.409557 mapwrite 0x002757 thru 0x010790 (0xe03a bytes) 000292 1786419256.567750 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) 000294 1786419256.752428 mapwrite 0x038400 thru 0x03ffff (0x7c00 bytes) 000295 1786419256.896465 trunc from 0x040000 to 0x00e8b8 (0x31747 bytes) 000298 1786419256.924708 mapwrite 0x022a0f thru 0x024330 (0x1922 bytes) 000302 1786419257.021991 mapread 0x020762 thru 0x024330 (0x3bcf bytes) 000304 1786419257.032422 mapread 0x0224b6 thru 0x024330 (0x1e7b bytes) 000307 1786419257.039649 write 0x0333e6 thru 0x03ffff (0xcc1a bytes) 000308 1786419257.046468 mapwrite 0x01a8cd thru 0x025937 (0xb06b bytes) 000310 1786419257.157976 mapread 0x0291c7 thru 0x0343c6 (0xb200 bytes) 000311 1786419257.173007 trunc from 0x040000 to 0x02ad2b (0x152d4 bytes) 000312 1786419257.194668 read 0x005873 thru 0x01505a (0xf7e8 bytes) 000314 1786419257.212687 read 0x00208c thru 0x0023bf (0x334 bytes) 000315 1786419257.222209 write 0x0299bf thru 0x0323a1 (0x89e3 bytes) 000317 1786419257.224277 write 0x03e034 thru 0x03ffff (0x1fcc bytes) 000318 1786419257.226626 mapread 0x025aa8 thru 0x028dca (0x3323 bytes) 000319 1786419257.228705 mapread 0x0103f1 thru 0x01cf6c (0xcb7c bytes) 000320 1786419257.235402 mapread 0x0029c1 thru 0x00be0f (0x944f bytes) 000321 1786419257.237588 mapread 0x01b09c thru 0x025cc6 (0xac2b bytes) 000324 1786419257.247770 mapread 0x002082 thru 0x002a42 (0x9c1 bytes) 000326 1786419257.250777 mapwrite 0x01c84e thru 0x029dbc (0xd56f bytes) 000328 1786419257.317278 mapwrite 0x01be74 thru 0x01dd0d (0x1e9a bytes) 000329 1786419257.367551 write 0x02518a thru 0x032caa (0xdb21 bytes) 000330 1786419257.370741 mapwrite 0x0128ab thru 0x01a4bd (0x7c13 bytes) 000331 1786419257.446630 trunc from 0x040000 to 0x029f41 (0x160be bytes) 000332 1786419257.462670 mapread 0x017f7d thru 0x01f2cc (0x7350 bytes) 000334 1786419257.467030 read 0x01eda2 thru 0x02051e (0x177d bytes) 000337 1786419257.470772 write 0x00f4fc thru 0x015f88 (0x6a8d bytes) 000339 1786419257.473606 write 0x00769d thru 0x011f24 (0xa888 bytes) 000340 1786419257.476078 mapread 0x010010 thru 0x0184ed (0x84de bytes) 000345 1786419257.481640 write 0x01b527 thru 0x01e71a (0x31f4 bytes) 000350 1786419257.488072 write 0x00f8ea thru 0x01351d (0x3c34 bytes) 000352 1786419257.498741 MIRROR_EXTEND to 2 mirrors 000354 1786419258.178429 mapwrite 0x014d2d thru 0x01fb85 (0xae59 bytes) 000355 1786419258.438396 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) 000356 1786419258.649794 read 0x0068cb thru 0x007031 (0x767 bytes) 000357 1786419258.659522 write 0x03351f thru 0x038092 (0x4b74 bytes) 000358 1786419258.660878 write 0x004879 thru 0x00e662 (0x9dea bytes) 000361 1786419258.680803 mapread 0x01e413 thru 0x0218a0 (0x348e bytes) 000362 1786419258.707874 mapread 0x01cc82 thru 0x026bce (0x9f4d bytes) 000365 1786419258.716710 write 0x01d502 thru 0x02b205 (0xdd04 bytes) 000367 1786419258.727681 trunc from 0x038093 to 0x02bbb6 (0xc4dc bytes) 000369 1786419258.774263 mapread 0x01fad4 thru 0x0248d7 (0x4e04 bytes) 000370 1786419258.775539 read 0x01bbef thru 0x028b72 (0xcf84 bytes) 000373 1786419258.780103 read 0x01e9ce thru 0x020a1c (0x204f bytes) 000377 1786419258.783362 mapwrite 0x02d628 thru 0x02e145 (0xb1e bytes) 000381 1786419258.874209 trunc from 0x02e146 to 0x023a5b (0xa6ea bytes) 000382 1786419258.920566 trunc from 0x023a5c to 0x02aabd (0x7062 bytes) 000383 1786419258.952187 mapread 0x01f6af thru 0x024a67 (0x53b9 bytes) 000384 1786419258.961542 read 0x004016 thru 0x005a93 (0x1a7e bytes) 000385 1786419258.962387 mapwrite 0x001761 thru 0x00c9be (0xb25e bytes) 000386 1786419259.034700 mapread 0x026c4a thru 0x02aabd (0x3e74 bytes) truncating to largest ever: 0x394fb 000392 1786419259.058483 trunc from 0x02aabe to 0x0394fa (0xea3d bytes) 000398 1786419259.089304 read 0x0301ef thru 0x03268e (0x24a0 bytes) 000399 1786419259.097214 write 0x011dce thru 0x01738c (0x55bf bytes) 000400 1786419259.098710 write 0x01b2b3 thru 0x021726 (0x6474 bytes) 000401 1786419259.100961 write 0x03c3fd thru 0x03c44a (0x4e bytes) 000406 1786419259.104998 write 0x0305af thru 0x03943c (0x8e8e bytes) 000408 1786419259.107672 mapread 0x01e53d thru 0x02836f (0x9e33 bytes) 000409 1786419259.110086 trunc from 0x03c44b to 0x000fe6 (0x3b464 bytes) 000412 1786419259.164681 MIRROR_EXTEND to 2 mirrors 000414 1786419259.775499 mapread 0x000006 thru 0x000fe6 (0xfe1 bytes) 000416 1786419259.790346 write 0x009e28 thru 0x019de5 (0xffbe bytes) 000418 1786419259.900183 read 0x013673 thru 0x016296 (0x2c24 bytes) 000419 1786419259.902119 mapread 0x013eec thru 0x019de5 (0x5efa bytes) 000421 1786419259.909276 read 0x008f31 thru 0x0161eb (0xd2bb bytes) 000422 1786419259.921191 write 0x012e1e thru 0x01b492 (0x8675 bytes) 000424 1786419259.927307 read 0x0110ce thru 0x01852b (0x745e bytes) 000425 1786419259.928394 mapwrite 0x03e2bb thru 0x03ffff (0x1d45 bytes) 000426 1786419260.063027 write 0x01c394 thru 0x01e8e3 (0x2550 bytes) 000427 1786419260.065501 read 0x02fa1a thru 0x03cced (0xd2d4 bytes) 000428 1786419260.066873 write 0x031e43 thru 0x03ffff (0xe1bd bytes) 000429 1786419260.069697 mapwrite 0x0033d0 thru 0x00846a (0x509b bytes) 000431 1786419260.128495 mapread 0x001c01 thru 0x00298d (0xd8d bytes) 000433 1786419260.131607 trunc from 0x040000 to 0x019a69 (0x26596 bytes) 000435 1786419260.151535 trunc from 0x019a6a to 0x038e17 (0x1f3ae bytes) 000436 1786419260.176320 mapread 0x006dc5 thru 0x00da4d (0x6c89 bytes) 000438 1786419260.179730 read 0x016c0a thru 0x01d449 (0x6840 bytes) 000439 1786419260.191949 read 0x02c6f8 thru 0x032292 (0x5b9b bytes) 000440 1786419260.193129 mapwrite 0x00dffc thru 0x015292 (0x7297 bytes) 000441 1786419260.264077 write 0x007f81 thru 0x00cddd (0x4e5d bytes) 000442 1786419260.269402 mapwrite 0x00d95c thru 0x01b27c (0xd921 bytes) 000443 1786419260.325144 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x38e18): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38e18): Mirror 2: 0xf6d5fcfa (stale) CRC-32 checksum value for chunk [0, 0x38e18): Mirror 1: 0x852dfa62 Mirror 3: 0x852dfa62 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38e18: skipped 000444 1786419261.494322 write 0x02e85a thru 0x0341da (0x5981 bytes) 000445 1786419261.603671 mapwrite 0x00fb88 thru 0x0185e2 (0x8a5b bytes) 000446 1786419261.739800 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38e18): Mirror 3: 0x852dfa62 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38e18): Mirror 3: 0x852dfa62 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38e18: skipped 000447 1786419262.032169 mapwrite 0x0350c5 thru 0x036406 (0x1342 bytes) 000449 1786419262.093061 write 0x02a763 thru 0x03766f (0xcf0d bytes) 000450 1786419262.107264 trunc from 0x038e18 to 0x034d10 (0x4107 bytes) 000453 1786419262.141236 read 0x0283a6 thru 0x034d10 (0xc96b bytes) 000454 1786419262.148783 write 0x0195ed thru 0x02868b (0xf09f bytes) 000456 1786419262.157512 write 0x02138b thru 0x0215dc (0x252 bytes) 000457 1786419262.158431 read 0x02aa08 thru 0x03441a (0x9a13 bytes) 000459 1786419262.160451 mapwrite 0x011d3c thru 0x017d38 (0x5ffd bytes) 000463 1786419262.224898 mapwrite 0x02bb31 thru 0x035696 (0x9b66 bytes) 000466 1786419262.294580 mapwrite 0x011c27 thru 0x01c0b1 (0xa48b bytes) 000467 1786419262.355333 MIRROR_RESYNC 2 mirrors 000469 1786419263.169774 mapread 0x014433 thru 0x01f3aa (0xaf78 bytes) 000470 1786419263.195182 write 0x0338ba thru 0x03d4d5 (0x9c1c bytes) 000471 1786419263.338808 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) 000473 1786419263.621609 mapread 0x006040 thru 0x007aab (0x1a6c bytes) 000474 1786419263.649133 mapread 0x00e1eb thru 0x01a0a5 (0xbebb bytes) 000476 1786419263.684549 mapread 0x0120a1 thru 0x01bddd (0x9d3d bytes) 000477 1786419263.686296 mapwrite 0x01beb1 thru 0x029101 (0xd251 bytes) 000478 1786419263.807404 read 0x029819 thru 0x0354b5 (0xbc9d bytes) 000480 1786419263.840345 read 0x01d927 thru 0x021f2e (0x4608 bytes) 000482 1786419263.841630 read 0x0240d1 thru 0x033877 (0xf7a7 bytes) 000483 1786419263.849954 mapread 0x005f89 thru 0x00d98c (0x7a04 bytes) 000484 1786419263.872879 trunc from 0x03d4d6 to 0x020ae1 (0x1c9f4 bytes) 000485 1786419263.954634 mapread 0x013a5d thru 0x01af78 (0x751c bytes) 000487 1786419263.958697 read 0x01b714 thru 0x01cf21 (0x180e bytes) 000489 1786419263.964664 mapread 0x013b23 thru 0x020ae1 (0xcfbf bytes) 000490 1786419263.967195 read 0x007c7c thru 0x007ec3 (0x248 bytes) 000491 1786419263.967914 mapwrite 0x02abac thru 0x036c32 (0xc087 bytes) 000492 1786419264.101078 write 0x03d78c thru 0x03dd6a (0x5df bytes) 000493 1786419264.105040 trunc from 0x03dd6b to 0x02b7b3 (0x125b7 bytes) 000494 1786419264.130488 MIRROR_EXTEND to 2 mirrors 000499 1786419265.227574 write 0x027896 thru 0x02887f (0xfea bytes) 000500 1786419265.377599 trunc from 0x02b7b4 to 0x01c762 (0xf051 bytes) 000501 1786419265.412156 MIRROR_RESYNC 2 mirrors 000504 1786419265.999696 mapread 0x0186cd thru 0x0190cc (0xa00 bytes) 000506 1786419266.011724 mapwrite 0x00ec55 thru 0x019761 (0xab0d bytes) 000507 1786419266.187379 mapwrite 0x0220bb thru 0x0269a9 (0x48ef bytes) 000508 1786419266.276692 read 0x013906 thru 0x015a63 (0x215e bytes) 000509 1786419266.282173 MIRROR_RESYNC 2 mirrors 000512 1786419266.936436 trunc from 0x0269aa to 0x02c820 (0x5e77 bytes) 000515 1786419267.112372 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) 000516 1786419267.257496 mapwrite 0x006143 thru 0x0121b5 (0xc073 bytes) 000517 1786419267.336613 write 0x016bee thru 0x025502 (0xe915 bytes) 000518 1786419267.344119 trunc from 0x02c821 to 0x0089ae (0x23e72 bytes) 000520 1786419267.362904 write 0x023086 thru 0x029d54 (0x6ccf bytes) 000524 1786419267.367901 read 0x017411 thru 0x01a264 (0x2e54 bytes) 000525 1786419267.373455 read 0x01d7f6 thru 0x022945 (0x5150 bytes) 000526 1786419267.383284 mapwrite 0x001a67 thru 0x0029c1 (0xf5b bytes) 000528 1786419267.450158 mapread 0x011527 thru 0x01a0af (0x8b89 bytes) 000529 1786419267.451429 trunc from 0x029d55 to 0x0314b3 (0x775f bytes) 000530 1786419267.469256 write 0x02e3e3 thru 0x038fea (0xac08 bytes) 000533 1786419267.480436 read 0x019c01 thru 0x027119 (0xd519 bytes) 000536 1786419267.483293 trunc from 0x038feb to 0x02a526 (0xeac4 bytes) 000537 1786419267.506025 trunc from 0x02a527 to 0x01c3cc (0xe15a bytes) 000538 1786419267.522521 trunc from 0x01c3cd to 0x029859 (0xd48d bytes) 000540 1786419267.545305 trunc from 0x02985a to 0x010e26 (0x18a33 bytes) 000541 1786419267.565644 trunc from 0x010e27 to 0x036cd1 (0x25eab bytes) 000545 1786419267.592648 trunc from 0x036cd2 to 0x00d5fc (0x296d5 bytes) 000548 1786419267.617618 mapwrite 0x00c6e5 thru 0x01177d (0x5099 bytes) 000549 1786419267.682468 trunc from 0x01177e to 0x0210eb (0xf96e bytes) 000551 1786419267.698447 mapwrite 0x021693 thru 0x02d2f1 (0xbc5f bytes) 000553 1786419267.765628 mapread 0x01aac1 thru 0x028f9e (0xe4de bytes) 000554 1786419267.767618 trunc from 0x02d2f2 to 0x0251d4 (0x811d bytes) 000555 1786419267.785508 write 0x02c2a7 thru 0x02cb89 (0x8e3 bytes) 000557 1786419267.789192 write 0x026c10 thru 0x031891 (0xac82 bytes) 000561 1786419267.796548 mapread 0x01917e thru 0x028efd (0xfd80 bytes) 000566 1786419267.802317 trunc from 0x031892 to 0x015573 (0x1c31e bytes) 000567 1786419267.819666 write 0x02e0f3 thru 0x033f8d (0x5e9b bytes) 000569 1786419267.824321 mapwrite 0x01aa99 thru 0x01cb89 (0x20f1 bytes) 000570 1786419267.888234 read 0x02d2fe thru 0x0336cc (0x63cf bytes) 000572 1786419267.897952 write 0x02f0e7 thru 0x03bdef (0xcd09 bytes) 000577 1786419267.905108 read 0x027ab6 thru 0x033d17 (0xc262 bytes) 000578 1786419267.906227 mapwrite 0x014fa1 thru 0x024762 (0xf7c2 bytes) 000579 1786419267.959778 mapread 0x00d4d8 thru 0x019a95 (0xc5be bytes) 000583 1786419267.965369 mapwrite 0x02c589 thru 0x0362b0 (0x9d28 bytes) 000587 1786419268.031646 read 0x01acda thru 0x022a27 (0x7d4e bytes) 000588 1786419268.039742 write 0x028929 thru 0x02dfce (0x56a6 bytes) 000590 1786419268.046145 trunc from 0x03bdf0 to 0x0158f4 (0x264fb bytes) 000592 1786419268.071894 mapread 0x004f7f thru 0x00b2d9 (0x635b bytes) 000594 1786419268.077162 write 0x01f968 thru 0x0278d9 (0x7f72 bytes) 000596 1786419268.082190 mapwrite 0x004536 thru 0x00a6f7 (0x61c2 bytes) 000597 1786419268.143471 write 0x01646c thru 0x01b73a (0x52cf bytes) 000598 1786419268.154309 write 0x012cf0 thru 0x0208ec (0xdbfd bytes) 000599 1786419268.155765 mapread 0x0171e9 thru 0x01a781 (0x3599 bytes) 000601 1786419268.161276 mapwrite 0x0234e0 thru 0x02f9a5 (0xc4c6 bytes) 000602 1786419268.254747 write 0x01fbe0 thru 0x028a07 (0x8e28 bytes) 000604 1786419268.259156 trunc from 0x02f9a6 to 0x01e6fb (0x112aa bytes) 000608 1786419268.287485 mapwrite 0x020eee thru 0x0242fb (0x340e bytes) 000614 1786419268.392268 mapread 0x01eedb thru 0x0242fb (0x5421 bytes) 000616 1786419268.400918 read 0x017678 thru 0x01a71c (0x30a5 bytes) 000617 1786419268.402198 trunc from 0x0242fc to 0x0043cd (0x1ff2e bytes) 000620 1786419268.432528 MIRROR_EXTEND to 2 mirrors 000622 1786419269.054390 write 0x02ff14 thru 0x034787 (0x4874 bytes) 000624 1786419269.196499 read 0x023053 thru 0x030a92 (0xda40 bytes) 000625 1786419269.209341 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) 000629 1786419269.444285 read 0x0116c6 thru 0x01cc56 (0xb591 bytes) 000631 1786419269.454276 read 0x018e3b thru 0x022dd3 (0x9f99 bytes) 000632 1786419269.463616 mapwrite 0x03ff97 thru 0x03ffff (0x69 bytes) 000633 1786419269.524042 mapread 0x021bbc thru 0x031136 (0xf57b bytes) 000635 1786419269.532866 MIRROR_EXTEND to 2 mirrors 000636 1786419270.468699 MIRROR_EXTEND to 3 mirrors 000637 1786419271.798161 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) 000638 1786419272.171072 mapread 0x00627a thru 0x01018c (0x9f13 bytes) 000639 1786419272.202605 trunc from 0x040000 to 0x01565c (0x2a9a3 bytes) 000640 1786419272.322651 mapwrite 0x015215 thru 0x016b2f (0x191b bytes) 000643 1786419272.405535 read 0x00654e thru 0x011abb (0xb56e bytes) 000644 1786419272.437689 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) 000645 1786419272.597215 mapwrite 0x01dc98 thru 0x028652 (0xa9bb bytes) 000648 1786419272.768479 mapread 0x009594 thru 0x011caa (0x8717 bytes) 000649 1786419272.780882 read 0x01eafd thru 0x028652 (0x9b56 bytes) 000650 1786419272.781871 trunc from 0x028653 to 0x0074ee (0x21164 bytes) 000652 1786419272.805363 trunc from 0x0074ef to 0x01250c (0xb01e bytes) 000654 1786419272.827580 trunc from 0x01250d to 0x00b787 (0x6d85 bytes) 000655 1786419272.847159 mapread 0x005189 thru 0x006638 (0x14b0 bytes) 000656 1786419272.851345 MIRROR_EXTEND to 2 mirrors 000657 1786419273.873224 trunc from 0x00b788 to 0x0024ea (0x929d bytes) 000659 1786419274.055635 MIRROR_RESYNC 2 mirrors 000660 1786419274.621342 mapread 0x0014ca thru 0x0024ea (0x1021 bytes) 000662 1786419274.638237 write 0x00f948 thru 0x011d4b (0x2404 bytes) 000666 1786419274.716683 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x11d4c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11d4c): Mirror 2: 0xf76bb99b (stale) CRC-32 checksum value for chunk [0, 0x11d4c): Mirror 1: 0xd73dd003 Mirror 3: 0xd73dd003 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11d4c: skipped 000667 1786419275.757332 read 0x001db5 thru 0x011004 (0xf250 bytes) 000669 1786419275.783255 mapwrite 0x0109a1 thru 0x017863 (0x6ec3 bytes) 000671 1786419275.930481 mapread 0x00f76f thru 0x0173f5 (0x7c87 bytes) 000672 1786419275.935777 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x17864): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17864): Mirror 2: 0xdd58fa4a (stale) Mirror 3: 0x2d565c69 (stale) CRC-32 checksum value for chunk [0, 0x17864): Mirror 1: 0xdb92e1d0 Mirror 4: 0xdb92e1d0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17864: skipped 000681 1786419276.856048 trunc from 0x017864 to 0x02ae20 (0x135bd bytes) 000682 1786419276.977422 read 0x0190ae thru 0x01f338 (0x628b bytes) 000683 1786419276.989106 mapread 0x018528 thru 0x01f4af (0x6f88 bytes) 000684 1786419276.997677 write 0x016f7a thru 0x01c104 (0x518b bytes) 000685 1786419277.005117 write 0x021a1c thru 0x0253d3 (0x39b8 bytes) 000687 1786419277.026326 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2ae21): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ae21): Mirror 2: 0xbd2f95df (stale) Mirror 3: 0x765139d0 (stale) Mirror 4: 0x1a60834 (stale) CRC-32 checksum value for chunk [0, 0x2ae21): Mirror 1: 0x1c1bab1e Mirror 5: 0x1c1bab1e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ae21: skipped 000689 1786419278.406145 trunc from 0x02ae21 to 0x019943 (0x114dd bytes) 000690 1786419278.598585 write 0x02d4ef thru 0x035613 (0x8125 bytes) 000692 1786419278.604041 mapwrite 0x0224eb thru 0x0250a8 (0x2bbe bytes) 000694 1786419278.775413 MIRROR_RESYNC 5 mirrors 000695 1786419281.528763 read 0x01b3ca thru 0x027a23 (0xc65a bytes) 000696 1786419281.566200 write 0x02f06a thru 0x03a3ee (0xb385 bytes) 000697 1786419281.660129 write 0x0014b6 thru 0x00aa60 (0x95ab bytes) 000698 1786419281.671492 trunc from 0x03a3ef to 0x009dd1 (0x3061d bytes) 000699 1786419281.697197 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9dd2): Mirror 2: 0x49a232e7 (stale) Mirror 3: 0x49a232e7 (stale) Mirror 5: 0x49a232e7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9dd2): Mirror 2: 0x49a232e7 (stale) Mirror 3: 0x49a232e7 (stale) Mirror 5: 0x49a232e7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9dd2: skipped 000703 1786419282.134398 mapread 0x0072de thru 0x009dd1 (0x2af4 bytes) 000705 1786419282.160111 mapwrite 0x039b22 thru 0x03ffff (0x64de bytes) 000706 1786419282.223291 trunc from 0x040000 to 0x014aa3 (0x2b55c bytes) 000708 1786419282.243474 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x14aa4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14aa4): Mirror 2: 0x7e9b47f1 (stale) Mirror 3: 0x7e9b47f1 (stale) Mirror 5: 0x7e9b47f1 (stale) CRC-32 checksum value for chunk [0, 0x14aa4): Mirror 1: 0x663a23ba Mirror 6: 0x663a23ba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14aa4: skipped 000709 1786419283.686523 trunc from 0x014aa4 to 0x0047a9 (0x102fa bytes) 000710 1786419283.801554 write 0x013969 thru 0x01765c (0x3cf4 bytes) 000711 1786419283.811023 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1765d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1765d): Mirror 2: 0xc9cb16c (stale) Mirror 3: 0xc9cb16c (stale) Mirror 5: 0xc9cb16c (stale) Mirror 6: 0xc20e463 (stale) CRC-32 checksum value for chunk [0, 0x1765d): Mirror 1: 0x22e8232 Mirror 7: 0x22e8232 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1765d: skipped 000712 1786419285.188690 write 0x03fa62 thru 0x03ffff (0x59e bytes) 000713 1786419285.290967 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x2f9633cc (stale) Mirror 3: 0x2f9633cc (stale) Mirror 5: 0x2f9633cc (stale) Mirror 6: 0xf8e6c593 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x2f9633cc (stale) Mirror 3: 0x2f9633cc (stale) Mirror 5: 0x2f9633cc (stale) Mirror 6: 0xf8e6c593 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000715 1786419285.715861 mapread 0x01d7ea thru 0x022d48 (0x555f bytes) 000716 1786419285.763317 trunc from 0x040000 to 0x024d32 (0x1b2cd bytes) 000717 1786419285.790717 mapread 0x003abc thru 0x00d9a0 (0x9ee5 bytes) 000719 1786419285.806784 mapread 0x02110d thru 0x024d32 (0x3c26 bytes) 000720 1786419285.808447 mapwrite 0x009580 thru 0x016c7a (0xd6fb bytes) 000721 1786419285.885950 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x24d33): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24d33): Mirror 2: 0x70157e93 (stale) Mirror 3: 0x70157e93 (stale) Mirror 5: 0x70157e93 (stale) Mirror 6: 0xdbd0b05b (stale) CRC-32 checksum value for chunk [0, 0x24d33): Mirror 1: 0x79be9eb2 Mirror 7: 0x79be9eb2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24d33: skipped 000722 1786419287.231043 write 0x03f77e thru 0x03ffff (0x882 bytes) 000724 1786419287.386561 MIRROR_RESYNC 6 mirrors 000726 1786419289.729382 mapwrite 0x033812 thru 0x0342cc (0xabb bytes) 000727 1786419289.930481 MIRROR_RESYNC 6 mirrors truncating to largest ever: 0x3c064 000730 1786419292.351119 trunc from 0x040000 to 0x03c063 (0x3f9c bytes) 000731 1786419292.447384 mapread 0x005339 thru 0x01244a (0xd112 bytes) 000732 1786419292.479183 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c064): Mirror 2: 0x37bb6579 (stale) Mirror 3: 0x37bb6579 (stale) Mirror 5: 0x37bb6579 (stale) Mirror 6: 0x37bb6579 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c064): Mirror 2: 0x37bb6579 (stale) Mirror 3: 0x37bb6579 (stale) Mirror 5: 0x37bb6579 (stale) Mirror 6: 0x37bb6579 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c064: skipped 000734 1786419292.875958 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c064): Mirror 2: 0x37bb6579 (stale) Mirror 3: 0x37bb6579 (stale) Mirror 5: 0x37bb6579 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c064): Mirror 2: 0x37bb6579 (stale) Mirror 3: 0x37bb6579 (stale) Mirror 5: 0x37bb6579 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c064: skipped 000735 1786419293.252925 trunc from 0x03c064 to 0x00dfed (0x2e076 bytes) 000736 1786419293.267415 write 0x009ace thru 0x00b2d7 (0x180a bytes) 000737 1786419293.277249 read 0x00177d thru 0x00dfed (0xc871 bytes) 000738 1786419293.297317 MIRROR_RESYNC 4 mirrors 000739 1786419294.997010 read 0x00cb38 thru 0x00dfed (0x14b6 bytes) 000740 1786419295.030008 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) 000741 1786419295.394744 trunc from 0x00dfee to 0x021bab (0x13bbe bytes) 000742 1786419295.469204 read 0x006af9 thru 0x00780b (0xd13 bytes) 000743 1786419295.474660 trunc from 0x021bac to 0x0309cc (0xee21 bytes) 000744 1786419295.496704 read 0x029ec5 thru 0x0309cc (0x6b08 bytes) 000746 1786419295.510365 trunc from 0x0309cd to 0x0353bf (0x49f3 bytes) 000747 1786419295.530444 read 0x01217b thru 0x0156e2 (0x3568 bytes) 000749 1786419295.532705 trunc from 0x0353c0 to 0x01b408 (0x19fb7 bytes) 000752 1786419295.550610 trunc from 0x01b409 to 0x0225f3 (0x71eb bytes) 000753 1786419295.571399 write 0x03d82f thru 0x03ffff (0x27d1 bytes) 000755 1786419295.574389 mapread 0x025bce thru 0x02dda3 (0x81d6 bytes) 000756 1786419295.584204 read 0x004395 thru 0x00e630 (0xa29c bytes) 000757 1786419295.585034 read 0x02dcac thru 0x032557 (0x48ac bytes) 000762 1786419295.589283 write 0x001b76 thru 0x003377 (0x1802 bytes) 000763 1786419295.591017 mapread 0x0289a0 thru 0x02c013 (0x3674 bytes) 000764 1786419295.592589 read 0x0353ff thru 0x03cf2b (0x7b2d bytes) 000765 1786419295.593354 write 0x011dd3 thru 0x016e0e (0x503c bytes) 000769 1786419295.599984 MIRROR_RESYNC 3 mirrors 000772 1786419296.564000 mapread 0x02990e thru 0x033fe3 (0xa6d6 bytes) 000773 1786419296.588607 mapread 0x00d846 thru 0x0105ae (0x2d69 bytes) 000774 1786419296.600137 MIRROR_EXTEND to 4 mirrors 000776 1786419297.423147 write 0x01730d thru 0x01b069 (0x3d5d bytes) 000778 1786419297.503192 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2e36b7cf (stale) Mirror 5: 0x2e36b7cf (stale) Mirror 6: 0x2e36b7cf (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x139abb96 Mirror 7: 0x139abb96 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000779 1786419298.627176 read 0x033662 thru 0x03ffff (0xc99e bytes) 000780 1786419298.650360 MIRROR_RESYNC 5 mirrors 000782 1786419299.836673 mapread 0x009a42 thru 0x00f499 (0x5a58 bytes) 000783 1786419299.855961 trunc from 0x040000 to 0x027fe8 (0x18017 bytes) 000784 1786419299.936061 mapread 0x01a97b thru 0x027fe8 (0xd66e bytes) 000786 1786419299.956676 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27fe9): Mirror 3: 0x8fea2d88 (stale) Mirror 6: 0x8fea2d88 (stale) Mirror 7: 0x8fea2d88 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27fe9): Mirror 3: 0x8fea2d88 (stale) Mirror 6: 0x8fea2d88 (stale) Mirror 7: 0x8fea2d88 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27fe9: skipped 000787 1786419300.473890 read 0x0048ac thru 0x0100c5 (0xb81a bytes) 000789 1786419300.483561 write 0x03f83e thru 0x03f8ed (0xb0 bytes) 000791 1786419300.486151 read 0x018620 thru 0x0213ae (0x8d8f bytes) 000794 1786419300.497420 mapread 0x00e942 thru 0x01693a (0x7ff9 bytes) 000797 1786419300.501297 trunc from 0x03f8ee to 0x000603 (0x3f2ea bytes) 000800 1786419300.539655 trunc from 0x000604 to 0x001b0c (0x1509 bytes) 000803 1786419300.570330 mapread 0x0006c5 thru 0x001b0c (0x1448 bytes) 000804 1786419300.580988 mapread 0x000c20 thru 0x0019c8 (0xda9 bytes) 000805 1786419300.582371 read 0x000962 thru 0x001b0c (0x11ab bytes) 000806 1786419300.588506 MIRROR_RESYNC 4 mirrors 000808 1786419301.314137 mapread 0x001971 thru 0x001b0c (0x19c bytes) 000809 1786419301.318976 read 0x000a26 thru 0x001b0c (0x10e7 bytes) 000811 1786419301.323405 read 0x000d76 thru 0x001b0c (0xd97 bytes) 000812 1786419301.326969 MIRROR_EXTEND to 5 mirrors 000814 1786419302.102809 trunc from 0x001b0d to 0x03b4ed (0x399e1 bytes) 000815 1786419302.210360 mapread 0x009b28 thru 0x017203 (0xd6dc bytes) 000816 1786419302.241214 trunc from 0x03b4ee to 0x013acd (0x27a20 bytes) 000817 1786419302.275989 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13ace): Mirror 3: 0xa4b8558c (stale) Mirror 7: 0xa4b8558c (stale) Mirror 8: 0xa4b8558c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13ace): Mirror 3: 0xa4b8558c (stale) Mirror 7: 0xa4b8558c (stale) Mirror 8: 0xa4b8558c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13ace: skipped 000819 1786419302.601460 mapwrite 0x032fd6 thru 0x0367cd (0x37f8 bytes) 000821 1786419302.747601 read 0x00e128 thru 0x014172 (0x604b bytes) 000822 1786419302.756234 mapread 0x006d11 thru 0x00e7bc (0x7aac bytes) 000823 1786419302.766952 write 0x008a7a thru 0x011db9 (0x9340 bytes) 000824 1786419302.769652 mapread 0x01191a thru 0x020f51 (0xf638 bytes) 000825 1786419302.790636 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x367ce): Mirror 7: 0xd8dc3a8c (stale) Mirror 8: 0xd8dc3a8c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x367ce): Mirror 7: 0xd8dc3a8c (stale) Mirror 8: 0xd8dc3a8c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x367ce: skipped 000826 1786419303.146868 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x367ce): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x367ce): Mirror 7: 0xd8dc3a8c (stale) Mirror 8: 0xd8dc3a8c (stale) CRC-32 checksum value for chunk [0, 0x367ce): Mirror 1: 0xdfb5a6d1 Mirror 9: 0xdfb5a6d1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x367ce: skipped 000827 1786419304.332070 mapwrite 0x03795d thru 0x03ffff (0x86a3 bytes) 000828 1786419304.604229 MIRROR_RESYNC 4 mirrors 000829 1786419306.338926 mapread 0x014542 thru 0x016f88 (0x2a47 bytes) 000830 1786419306.357352 trunc from 0x040000 to 0x002abf (0x3d540 bytes) 000832 1786419306.428083 mapwrite 0x032316 thru 0x0324db (0x1c6 bytes) 000833 1786419306.496319 mapwrite 0x03b4ed thru 0x03ffff (0x4b13 bytes) 000835 1786419306.576576 mapwrite 0x0326ed thru 0x0343ec (0x1d00 bytes) 000836 1786419306.620441 mapread 0x038944 thru 0x03ffff (0x76bc bytes) 000837 1786419306.622306 read 0x01c3f3 thru 0x0249da (0x85e8 bytes) 000838 1786419306.630196 trunc from 0x040000 to 0x02a8f3 (0x1570c bytes) 000839 1786419306.657026 write 0x02dd5d thru 0x033366 (0x560a bytes) 000840 1786419306.662143 MIRROR_RESYNC 4 mirrors 000843 1786419307.743614 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) 000845 1786419308.064179 write 0x01cb22 thru 0x01ea16 (0x1ef5 bytes) 000850 1786419308.179902 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x33367): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33367): Mirror 7: 0xdfc586d4 (stale) Mirror 8: 0xdfc586d4 (stale) CRC-32 checksum value for chunk [0, 0x33367): Mirror 1: 0xfa29c2e8 Mirror 9: 0xfa29c2e8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33367: skipped 000851 1786419309.251691 mapwrite 0x0350ed thru 0x039551 (0x4465 bytes) 000852 1786419309.425399 mapwrite 0x0043ac thru 0x00c28e (0x7ee3 bytes) 000857 1786419309.531594 read 0x01bc2c thru 0x027310 (0xb6e5 bytes) 000858 1786419309.542950 read 0x02d570 thru 0x032b06 (0x5597 bytes) 000859 1786419309.558420 MIRROR_RESYNC 4 mirrors 000861 1786419310.940483 MIRROR_EXTEND to 5 mirrors 000862 1786419311.647684 write 0x02e10c thru 0x031872 (0x3767 bytes) 000864 1786419311.783488 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39552): Mirror 8: 0x260b959e (stale) Mirror 9: 0x260b959e (stale) Mirror 10: 0x260b959e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39552): Mirror 8: 0x260b959e (stale) Mirror 9: 0x260b959e (stale) Mirror 10: 0x260b959e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39552: skipped 000865 1786419312.321800 mapread 0x003565 thru 0x00daad (0xa549 bytes) 000866 1786419312.340168 read 0x006411 thru 0x007952 (0x1542 bytes) 000867 1786419312.341195 read 0x031aef thru 0x039551 (0x7a63 bytes) 000870 1786419312.353216 write 0x03764b thru 0x03ffff (0x89b5 bytes) 000872 1786419312.356174 write 0x005e19 thru 0x013545 (0xd72d bytes) 000874 1786419312.359323 trunc from 0x040000 to 0x024ff2 (0x1b00d bytes) 000875 1786419312.376892 mapwrite 0x0261c0 thru 0x027cc6 (0x1b07 bytes) 000877 1786419312.452247 mapwrite 0x02eae6 thru 0x03ccd4 (0xe1ef bytes) 000878 1786419312.625468 read 0x017f90 thru 0x01cee8 (0x4f59 bytes) 000879 1786419312.634090 write 0x019eed thru 0x0203c8 (0x64dc bytes) 000880 1786419312.638759 write 0x0183be thru 0x0247bc (0xc3ff bytes) 000882 1786419312.647840 trunc from 0x03ccd5 to 0x025c79 (0x1705b bytes) 000883 1786419312.682563 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x25c7a): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25c7a): Mirror 8: 0x8374bb42 (stale) Mirror 9: 0x8374bb42 (stale) Mirror 10: 0x8374bb42 (stale) CRC-32 checksum value for chunk [0, 0x25c7a): Mirror 1: 0x8a178af3 Mirror 11: 0x8a178af3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25c7a: skipped 000884 1786419314.027895 mapread 0x021812 thru 0x023596 (0x1d85 bytes) 000886 1786419314.112696 trunc from 0x025c7a to 0x02bf17 (0x629e bytes) 000887 1786419314.277180 mapwrite 0x031df3 thru 0x03d536 (0xb744 bytes) 000888 1786419314.377290 MIRROR_RESYNC 5 mirrors 000889 1786419316.969271 mapwrite 0x0336ac thru 0x03a82d (0x7182 bytes) 000890 1786419317.248482 mapwrite 0x03b4e7 thru 0x03ffff (0x4b19 bytes) 000893 1786419317.332729 trunc from 0x040000 to 0x000057 (0x3ffa8 bytes) 000895 1786419317.351482 write 0x0127cc thru 0x016ede (0x4713 bytes) 000896 1786419317.354176 trunc from 0x016edf to 0x01e797 (0x78b9 bytes) 000897 1786419317.374712 mapread 0x005a6a thru 0x011acf (0xc066 bytes) 000898 1786419317.397712 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1e798): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e798): Mirror 8: 0x473cd184 (stale) Mirror 9: 0x473cd184 (stale) Mirror 10: 0x473cd184 (stale) Mirror 11: 0x473cd184 (stale) CRC-32 checksum value for chunk [0, 0x1e798): Mirror 1: 0xb3ec9413 Mirror 12: 0xb3ec9413 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e798: skipped 000899 1786419318.792961 read 0x00d9a1 thru 0x019bf8 (0xc258 bytes) 000900 1786419318.810649 mapwrite 0x013592 thru 0x021907 (0xe376 bytes) 000901 1786419318.966375 read 0x007e3e thru 0x00a52c (0x26ef bytes) 000903 1786419318.974564 read 0x017798 thru 0x021907 (0xa170 bytes) 000905 1786419318.978130 write 0x02f820 thru 0x03afd8 (0xb7b9 bytes) 000907 1786419318.981693 mapwrite 0x0253fe thru 0x02eb03 (0x9706 bytes) 000908 1786419319.086756 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3afd9): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3afd9): Mirror 8: 0x85463139 (stale) Mirror 9: 0x85463139 (stale) Mirror 10: 0x85463139 (stale) Mirror 11: 0x85463139 (stale) Mirror 12: 0x1136ae8a (stale) CRC-32 checksum value for chunk [0, 0x3afd9): Mirror 1: 0x1184632d Mirror 13: 0x1184632d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3afd9: skipped 000911 1786419320.168780 write 0x011c50 thru 0x01377e (0x1b2f bytes) 000912 1786419320.249533 mapwrite 0x021700 thru 0x02d179 (0xba7a bytes) 000913 1786419320.327127 write 0x00e8ef thru 0x01bfe1 (0xd6f3 bytes) 000914 1786419320.344218 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3afd9): Mirror 8: 0x85463139 (stale) Mirror 9: 0x85463139 (stale) Mirror 11: 0x85463139 (stale) Mirror 12: 0x1136ae8a (stale) Mirror 13: 0x1184632d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3afd9): Mirror 8: 0x85463139 (stale) Mirror 9: 0x85463139 (stale) Mirror 11: 0x85463139 (stale) Mirror 12: 0x1136ae8a (stale) Mirror 13: 0x1184632d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3afd9: skipped 000915 1786419320.992626 read 0x02cf48 thru 0x0364b1 (0x956a bytes) 000916 1786419321.025126 write 0x035fcf thru 0x03ee29 (0x8e5b bytes) 000917 1786419321.034140 read 0x00dd20 thru 0x01c5c2 (0xe8a3 bytes) 000918 1786419321.054400 mapwrite 0x023a9a thru 0x02fca2 (0xc209 bytes) 000920 1786419321.171659 mapwrite 0x014819 thru 0x017765 (0x2f4d bytes) 000921 1786419321.268410 MIRROR_RESYNC 6 mirrors 000922 1786419323.063468 write 0x0183ca thru 0x01cfce (0x4c05 bytes) 000923 1786419323.165336 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ee2a): Mirror 8: 0x918e629b (stale) Mirror 11: 0x918e629b (stale) Mirror 12: 0x918e629b (stale) Mirror 13: 0x918e629b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ee2a): Mirror 8: 0x918e629b (stale) Mirror 11: 0x918e629b (stale) Mirror 12: 0x918e629b (stale) Mirror 13: 0x918e629b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ee2a: skipped 000924 1786419323.686505 MIRROR_RESYNC 5 mirrors 000925 1786419325.317788 read 0x039842 thru 0x03ee29 (0x55e8 bytes) 000927 1786419325.331403 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) 000929 1786419325.652412 trunc from 0x03ee2a to 0x00c967 (0x324c2 bytes) 000930 1786419325.722081 mapwrite 0x02cdb9 thru 0x0322c5 (0x550d bytes) 000931 1786419325.819968 mapwrite 0x03929b thru 0x03a2d8 (0x103e bytes) 000932 1786419325.962850 write 0x0177fd thru 0x026236 (0xea3a bytes) 000936 1786419325.988243 trunc from 0x03a2d9 to 0x009454 (0x30e84 bytes) 000939 1786419326.013431 mapwrite 0x01ce0a thru 0x022c3c (0x5e33 bytes) 000940 1786419326.170125 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x22c3d): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22c3d): Mirror 8: 0x17cc8c3a (stale) Mirror 11: 0x17cc8c3a (stale) Mirror 13: 0x17cc8c3a (stale) CRC-32 checksum value for chunk [0, 0x22c3d): Mirror 1: 0xbd86815b Mirror 14: 0xbd86815b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22c3d: skipped 000941 1786419327.414541 mapwrite 0x03a5ea thru 0x03ffff (0x5a16 bytes) 000944 1786419327.682067 mapread 0x0038f0 thru 0x008ea3 (0x55b4 bytes) 000945 1786419327.698713 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x1bb916a4 (stale) Mirror 13: 0x1bb916a4 (stale) Mirror 14: 0xb11d8762 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x1bb916a4 (stale) Mirror 13: 0x1bb916a4 (stale) Mirror 14: 0xb11d8762 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000946 1786419328.135092 write 0x01f32f thru 0x02be67 (0xcb39 bytes) 000950 1786419328.146418 mapread 0x0105a8 thru 0x017168 (0x6bc1 bytes) 000951 1786419328.168728 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x1bb916a4 (stale) Mirror 13: 0x1bb916a4 (stale) Mirror 14: 0xb11d8762 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1fb6c645 Mirror 15: 0x1fb6c645 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000959 1786419329.388081 mapread 0x00d5f6 thru 0x014a4d (0x7458 bytes) 000960 1786419329.448225 MIRROR_RESYNC 5 mirrors 000961 1786419330.734666 write 0x0100ff thru 0x010d94 (0xc96 bytes) 000964 1786419330.821492 trunc from 0x040000 to 0x02a79a (0x15865 bytes) 000965 1786419330.872724 write 0x00212c thru 0x00fc07 (0xdadc bytes) 000967 1786419330.885742 trunc from 0x02a79b to 0x007adb (0x22cbf bytes) 000968 1786419330.911137 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x7adc): Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7adc): Mirror 11: 0x68c76368 (stale) Mirror 13: 0x68c76368 (stale) Mirror 14: 0x68c76368 (stale) Mirror 15: 0x68c76368 (stale) CRC-32 checksum value for chunk [0, 0x7adc): Mirror 1: 0xfde24daf Mirror 16: 0xfde24daf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7adc: skipped 000969 1786419331.658282 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x7adc): Mirror 11: 0x68c76368 (stale) Mirror 13: 0x68c76368 (stale) Mirror 15: 0x68c76368 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7adc): Mirror 11: 0x68c76368 (stale) Mirror 13: 0x68c76368 (stale) Mirror 15: 0x68c76368 (stale) CRC-32 checksum value for chunk [0, 0x7adc): Mirror 1: 0xfde24daf Mirror 16: 0xfde24daf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7adc: skipped 000971 1786419332.273960 trunc from 0x007adc to 0x02095b (0x18e80 bytes) 000974 1786419332.364970 mapread 0x01673b thru 0x018d61 (0x2627 bytes) 000975 1786419332.388962 read 0x01cbac thru 0x02095b (0x3db0 bytes) 000977 1786419332.393113 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2095c): Mirror 13: 0x8506985d (stale) Mirror 15: 0x8506985d (stale) Mirror 16: 0x235930f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2095c): Mirror 13: 0x8506985d (stale) Mirror 15: 0x8506985d (stale) Mirror 16: 0x235930f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2095c: skipped 000979 1786419332.875716 mapread 0x013d70 thru 0x01d059 (0x92ea bytes) 000981 1786419332.895937 read 0x003546 thru 0x00baac (0x8567 bytes) 000986 1786419332.916803 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2095c): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2095c): Mirror 13: 0x8506985d (stale) Mirror 15: 0x8506985d (stale) Mirror 16: 0x235930f1 (stale) CRC-32 checksum value for chunk [0, 0x2095c): Mirror 1: 0x235930f1 Mirror 17: 0x235930f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2095c: skipped 000987 1786419333.935333 mapread 0x005307 thru 0x01138c (0xc086 bytes) 000988 1786419333.969591 mapread 0x007289 thru 0x012036 (0xadae bytes) 000989 1786419333.983611 write 0x02db4a thru 0x0381d4 (0xa68b bytes) 000990 1786419334.066764 read 0x021a28 thru 0x026a3a (0x5013 bytes) 000992 1786419334.082665 read 0x00cca8 thru 0x00ce04 (0x15d bytes) 000993 1786419334.100938 read 0x001a61 thru 0x008885 (0x6e25 bytes) 000996 1786419334.102748 mapread 0x0319b4 thru 0x034784 (0x2dd1 bytes) 000997 1786419334.106887 trunc from 0x0381d5 to 0x00b7a8 (0x2ca2c bytes) 000998 1786419334.133764 mapwrite 0x00f5a1 thru 0x0122e1 (0x2d41 bytes) 000999 1786419334.211793 mapread 0x00303d thru 0x0031ca (0x18e bytes) 001000 1786419334.213055 read 0x00ef09 thru 0x00f1b3 (0x2ab bytes) All operations completed A-OK! PASS 70a (114s) == sanity-flr test 71: check mirror extend parent fid ==== 23:35:39 (1786419339) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 23:35:43 (1786419343) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.18494 s, 11.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377089 s, 2.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f2:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f2:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 23:35:53 (1786419353) Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4671040 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8081056 .. failed Extending file size to 4699968 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8350304 .. Extending file size to 5691776 .. Extending file size to 8690656 .. Extending file size to 6584928 .. done Extending file size to 9061984 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57411 57413 57414 57415 57416 done 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) 192.168.202.117@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (81s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:37:14 (1786419434) Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7272192 .. Extending file size to 8967872 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8089056 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8440704 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8694016 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9159264 .. Waiting 58717 58719 58720 58721 58722 done 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) 192.168.202.117@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 23:38:34 (1786419514) Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.152064 s, 13.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.273025 s, 7.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xe01863d8 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 23:38:55 (1786419535) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0839218 s, 25.0 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 23:39:02 (1786419542) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21f:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x309:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 23:39:10 (1786419550) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.255716 s, 8.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x325:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x326:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 23:39:18 (1786419558) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.15719 s, 13.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x327:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x327:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x328:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 23:39:26 (1786419566) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.066815 s, 31.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x329:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x329:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32a:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 23:40:00 (1786419600) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg217-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184059 s, 5.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30f:0x0] } PASS 204d (60s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 23:41:01 (1786419661) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg217-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.419401 s, 5.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x310:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x310:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (57s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 23:41:58 (1786419718) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg217-server mds-ost sync done. Creating new pool test_204f oleg217-server: Pool lustre.test_204f created Adding targets to pool oleg217-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg217-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.443151 s, 4.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x311:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x311:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg217-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg217-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg217-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 23:43:19 (1786419799) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 23:43:28 (1786419808) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 23:43:36 (1786419816) oleg217-server: Pool lustre.test_206 created Adding targets to pool oleg217-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg217-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg217-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg217-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg217-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 23:44:04 (1786419844) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791032 s, 13.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541201 s, 19.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x49265fe1 Mirror 3: 0x49265fe1 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 23:44:16 (1786419856) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 23:44:19 (1786419859) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 23:44:23 (1786419863) Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.110758 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1e20000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1420000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1420000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1420000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1040000 Read this (4 bytes): BBBB 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 23:45:03 (1786419903) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0330494 s, 248 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 23:45:11 (1786419911) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519878 s, 20.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x244 lmm_fid: [0x200000404:0x244:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x244 lmm_fid: [0x200000404:0x244:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x31b:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834500 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-10 23:45:13.000000000 -0400 Modify: 2026-08-10 23:45:13.000000000 -0400 Change: 2026-08-10 23:45:15.000000000 -0400 Birth: 2026-08-10 23:45:12.000000000 -0400 PASS 210a (12s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 23:45:22 (1786419922) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 23:45:26 (1786419926) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.739228 s, 55.4 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.087752 s, 46.7 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 10 23:45 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x246:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 10 23:45 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 23:45:35 (1786419935) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 23:45:44 (1786419944) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (14s) == sanity-flr test complete, duration 1780 sec =========== 23:45:58 (1786419958) === sanity-flr: start cleanup 23:46:00 (1786419960) === === sanity-flr: finish cleanup 23:46:07 (1786419967) ===