-----============= acceptance-small: sanity-flr ============----- Fri Jun 12 09:53:58 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg339-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 09:54:18 (1781272458) === oleg339-client.virtnet: executing check_config_client /mnt/lustre oleg339-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg339-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9aaac2aad000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9aaac2aad000.idle_timeout=debug disable quota as required oleg339-server: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 09:54:52 (1781272492) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7623] == sanity-flr test 0a: lfs mirror create with -N option == 09:54:55 (1781272495) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:55:12 (1781272512) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:55:15 (1781272515) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:55:18 (1781272518) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:55:45 (1781272545) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:55:48 (1781272548) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 09:55:50 (1781272550) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9aaac2aad000.stats=clear osc.lustre-OST0001-osc-ffff9aaac2aad000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:55:57 (1781272557) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:56:07 (1781272567) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:56:14 (1781272574) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 09:56:17 (1781272577) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:56:24 (1781272584) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 09:56:33 (1781272593) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:56:39 (1781272599) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:56:43 (1781272603) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:56:50 (1781272610) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:57:02 (1781272622) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:57:09 (1781272629) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.862732 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157055 s, 6.7 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:57:20 (1781272640) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 0.392663 s, 45.4 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 09:57:29 (1781272649) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:57:39 (1781272659) Stopping /mnt/lustre-ost1 (opts:) on oleg339-server oleg339-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9aaac2aad000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9aaac2aad000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg339-server: oleg339-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg339-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg339-server oleg339-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9aaac2aad000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9aaac2aad000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg339-server: oleg339-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg339-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (73s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:58:52 (1781272732) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.179799 s, 17.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg339-server oleg339-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9aaac2aad000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9aaac2aad000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg339-server: oleg339-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg339-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (41s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:59:33 (1781272773) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg339-server oleg339-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9aaac2aad000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9aaac2aad000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg339-server: oleg339-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg339-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg339-server oleg339-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9aaac2aad000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9aaac2aad000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg339-server: oleg339-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg339-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (78s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:00:51 (1781272851) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.89234 s, 8.9 MB/s -rw-r--r-- 1 root root 52428800 Jun 12 10:00 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.053s user 0m0.001s sys 0m0.179s Stopping /mnt/lustre-ost1 (opts:) on oleg339-server oleg339-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9aaac2aad000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9aaac2aad000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.468s user 0m0.000s sys 0m0.183s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg339-server: oleg339-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg339-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg339-server oleg339-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9aaac2aad000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9aaac2aad000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.709s user 0m0.002s sys 0m0.188s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg339-server: oleg339-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg339-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg339-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9aaac2aad000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9aaac2aad000.ost_server_uuid in FULL state after 0 sec PASS 33b (100s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:02:31 (1781272951) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:02:35 (1781272955) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:02:38 (1781272958) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 10:02:41 (1781272961) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 10:02:49 (1781272969) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.704398 s, 17.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:03:03 (1781272983) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032229 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261111 s, 40.2 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:03:11 (1781272991) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:03:18 (1781272998) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00084248 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00123905 s, 826 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000790507 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000863956 s, 1.2 MB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 10:03:29 (1781273009) create test files with size 15750280 .. 7ff03b2c68686956eecec0a0bef770c4 - 8940e619e528a67a49197b0ed07cd184 - 9c9717a93b41eda57fa5ad0681e7ad16 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg339-client.virtnet /mnt/lustre (opts:) Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 10:03:53 (1781273033) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:04:07 (1781273047) PASS 39 (5s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:04:13 (1781273053) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986522 s, 10.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xca:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xc8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358316 s, 29.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xc9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xca:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (7s) == sanity-flr test 41: lfs mirror resync check =========== 10:04:20 (1781273060) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.150924 s, 13.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.532618 s, 7.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.257642 s, 8.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 10:04:38 (1781273078) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 10:04:41 (1781273081) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:04:44 (1781273084) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320725 s, 32.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 10:04:52 (1781273092) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:04:55 (1781273095) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd1:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd3:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:05:05 (1781273105) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.471011 s, 22.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg339-client.virtnet /mnt/lustre (opts:) Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:05:17 (1781273117) llite.lustre-ffff9aaad048b800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.283573 s, 37.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg339-client.virtnet /mnt/lustre (opts:) Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9aaad0751000.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:05:31 (1781273131) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (37s) == sanity-flr test 46: Verify setstripe --copy option ==== 10:06:08 (1781273168) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 10:06:47 (1781273207) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 10:06:49 (1781273209) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.231007 s, 13.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.153779 s, 20.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xde94a43e (nosync,stale) Mirror 2: 0xde94a43e (nosync,stale) ** verify mirror 2 doesn't change original checksum: 2110b7f5e5c87a013f970c849aad4e55 - mirror 1 checksum: 78a791f4dc995cbf3c5a1afc76ea420f - mirror 2 checksum: 2110b7f5e5c87a013f970c849aad4e55 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-12T10:06:52-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-12T10:06:52-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 2110b7f5e5c87a013f970c849aad4e55 - mirror 1 checksum: 78a791f4dc995cbf3c5a1afc76ea420f - mirror 2 checksum: 78a791f4dc995cbf3c5a1afc76ea420f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-12T10:06:52-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-12T10:06:52-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:07:01 (1781273221) Failing mds1 on oleg339-server Stopping /mnt/lustre-mds1 (opts:) on oleg339-server 10:07:08 (1781273228) shut down facet: mds1 facet_host: oleg339-server facet_failover_host: oleg339-server Failover mds1 to oleg339-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 10:07:29 (1781273249) targets are mounted 10:07:29 (1781273249) facet_failover done oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305163 s, 34.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xea:0x0] } PASS 50A (44s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 10:07:45 (1781273265) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 10:07:49 (1781273269) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:07:52 (1781273272) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0160364 s, 1.0 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 10:08:00 (1781273280) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:08:03 (1781273283) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:08:07 (1781273287) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (30s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:08:37 (1781273317) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 10:09:07 (1781273347) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 10:09:32 (1781273372) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 10:09:57 (1781273397) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 10:10:04 (1781273404) mirror create pid 53770 mirror split pid 53772 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5019: 53770 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5019: 53772 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 10:11:11 (1781273471) Seed set to 3473 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000004 1781273473.757998 mapwrite 0x037dd8 thru 0x03ffff (0x8228 bytes) 000006 1781273473.892808 mapread 0x02153f thru 0x022a08 (0x14ca bytes) 000007 1781273473.911001 mapwrite 0x03d217 thru 0x03ffff (0x2de9 bytes) 000008 1781273473.970111 mapwrite 0x031bd2 thru 0x0359d4 (0x3e03 bytes) 000010 1781273474.065446 MIRROR_EXTEND to 2 mirrors 000011 1781273474.949156 mapwrite 0x01870c thru 0x01bdd4 (0x36c9 bytes) 000012 1781273475.229438 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) 000019 1781273475.435121 read 0x00721c thru 0x015305 (0xe0ea bytes) truncating to largest ever: 0x1648f 000020 1781273475.452682 trunc from 0x040000 to 0x01648e (0x29b71 bytes) 000024 1781273475.485988 mapread 0x001c3f thru 0x008522 (0x68e4 bytes) truncating to largest ever: 0x19ef0 000026 1781273475.520051 trunc from 0x01648f to 0x019eef (0x3a61 bytes) truncating to largest ever: 0x1bb7c 000028 1781273475.557427 trunc from 0x019ef0 to 0x01bb7b (0x1c8c bytes) 000029 1781273475.598777 write 0x022d24 thru 0x024b9e (0x1e7b bytes) 000034 1781273475.607873 mapwrite 0x025414 thru 0x02fbec (0xa7d9 bytes) 000035 1781273475.761758 mapwrite 0x0374e2 thru 0x03ffff (0x8b1e bytes) 000036 1781273475.898028 read 0x03c85c thru 0x03ffff (0x37a4 bytes) 000039 1781273475.899841 read 0x01eaca thru 0x02a1dd (0xb714 bytes) 000040 1781273475.901743 read 0x032a7a thru 0x03b692 (0x8c19 bytes) 000044 1781273475.905729 write 0x0351c2 thru 0x038f88 (0x3dc7 bytes) 000045 1781273475.907257 trunc from 0x040000 to 0x005dd9 (0x3a226 bytes) 000047 1781273475.932759 mapwrite 0x027ea7 thru 0x02b4b5 (0x360f bytes) 000048 1781273476.055252 write 0x02c910 thru 0x0327a7 (0x5e98 bytes) 000049 1781273476.067906 MIRROR_EXTEND to 2 mirrors 000052 1781273476.908095 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) 000059 1781273477.089251 mapwrite 0x01c964 thru 0x027701 (0xad9e bytes) 000060 1781273477.198631 mapwrite 0x017873 thru 0x01d6e3 (0x5e71 bytes) 000061 1781273477.248787 read 0x007768 thru 0x00fed4 (0x876d bytes) 000062 1781273477.258084 mapread 0x02b05d thru 0x0327a7 (0x774b bytes) 000063 1781273477.265687 mapwrite 0x004017 thru 0x00f075 (0xb05f bytes) 000065 1781273477.371825 write 0x029c72 thru 0x03268b (0x8a1a bytes) 000067 1781273477.377187 write 0x018d9f thru 0x026f67 (0xe1c9 bytes) 000068 1781273477.384878 read 0x014b38 thru 0x017d7b (0x3244 bytes) 000069 1781273477.401281 read 0x01673c thru 0x023c4d (0xd512 bytes) 000070 1781273477.402058 read 0x028820 thru 0x0327a7 (0x9f88 bytes) truncating to largest ever: 0x38ad3 000074 1781273477.409846 trunc from 0x0327a8 to 0x038ad2 (0x632b bytes) 000075 1781273477.434875 mapwrite 0x013840 thru 0x014224 (0x9e5 bytes) 000076 1781273477.521969 MIRROR_EXTEND to 2 mirrors 000077 1781273478.410785 mapread 0x025747 thru 0x03290f (0xd1c9 bytes) 000078 1781273478.460969 MIRROR_EXTEND to 3 mirrors 000080 1781273479.216838 mapread 0x005ca3 thru 0x008c85 (0x2fe3 bytes) 000083 1781273479.308076 write 0x017466 thru 0x02742c (0xffc7 bytes) 000084 1781273479.397595 read 0x02a6d7 thru 0x02ea58 (0x4382 bytes) 000085 1781273479.409798 mapwrite 0x026ca6 thru 0x02e9f4 (0x7d4f bytes) 000086 1781273479.470946 read 0x01a74a thru 0x0260a8 (0xb95f bytes) 000087 1781273479.471711 mapwrite 0x026460 thru 0x028f69 (0x2b0a bytes) 000088 1781273479.528871 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38ad3): Mirror 2: 0x49ab203a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38ad3): Mirror 2: 0x49ab203a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38ad3: skipped 000090 1781273479.951467 trunc from 0x038ad3 to 0x017c66 (0x20e6c bytes) 000091 1781273479.984177 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x17c67): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17c67): Mirror 2: 0x24b75d5 (stale) CRC-32 checksum value for chunk [0, 0x17c67): Mirror 1: 0x2ac9ec0c Mirror 3: 0x2ac9ec0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17c67: skipped 000094 1781273480.912925 MIRROR_RESYNC 3 mirrors 000096 1781273481.771236 mapread 0x00e878 thru 0x016949 (0x80d2 bytes) 000097 1781273481.794273 mapread 0x00e235 thru 0x017c66 (0x9a32 bytes) 000098 1781273481.796101 read 0x015198 thru 0x017c66 (0x2acf bytes) 000099 1781273481.796822 write 0x0026fc thru 0x00724a (0x4b4f bytes) 000100 1781273481.893150 write 0x027f97 thru 0x031fb3 (0xa01d bytes) 000101 1781273481.896192 mapread 0x024dc5 thru 0x02e365 (0x95a1 bytes) 000102 1781273481.907243 mapread 0x02e969 thru 0x031fb3 (0x364b bytes) 000105 1781273481.911014 mapwrite 0x03e2a7 thru 0x03ffff (0x1d59 bytes) 000106 1781273482.001015 mapwrite 0x03c260 thru 0x03cf21 (0xcc2 bytes) 000107 1781273482.044016 mapwrite 0x030dc6 thru 0x03ffff (0xf23a bytes) 000108 1781273482.130965 mapread 0x000903 thru 0x00b40d (0xab0b bytes) 000109 1781273482.152964 mapwrite 0x00d358 thru 0x00e591 (0x123a bytes) 000110 1781273482.207427 mapwrite 0x0096b9 thru 0x014065 (0xa9ad bytes) 000113 1781273482.350962 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xfcc595dc (stale) Mirror 3: 0xfcc595dc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x71c5c44a Mirror 4: 0x71c5c44a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000116 1781273483.480070 write 0x0167b7 thru 0x020fb4 (0xa7fe bytes) 000117 1781273483.660759 trunc from 0x040000 to 0x031d94 (0xe26b bytes) 000118 1781273483.783518 mapwrite 0x0102c6 thru 0x01cdea (0xcb25 bytes) 000119 1781273483.924045 mapwrite 0x015fad thru 0x01b59c (0x55f0 bytes) 000120 1781273484.009510 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31d95): Mirror 3: 0x9171beab (stale) Mirror 4: 0xa7d4d066 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31d95): Mirror 3: 0x9171beab (stale) Mirror 4: 0xa7d4d066 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31d95: skipped 000121 1781273484.571416 write 0x032fa6 thru 0x039b7d (0x6bd8 bytes) 000122 1781273484.572652 trunc from 0x039b7e to 0x02256d (0x17610 bytes) 000124 1781273484.590718 write 0x00d6b0 thru 0x017496 (0x9de7 bytes) 000125 1781273484.622131 read 0x005c8f thru 0x00aaf8 (0x4e6a bytes) 000127 1781273484.643662 read 0x0028de thru 0x01153b (0xec5e bytes) 000129 1781273484.659014 mapwrite 0x0160c0 thru 0x021754 (0xb695 bytes) 000130 1781273484.798282 trunc from 0x02256e to 0x02b6c9 (0x915c bytes) 000131 1781273484.847478 mapread 0x024fc2 thru 0x02b6c9 (0x6708 bytes) 000133 1781273484.879010 write 0x009781 thru 0x013ce5 (0xa565 bytes) 000134 1781273484.884054 mapread 0x01b30f thru 0x01bd9a (0xa8c bytes) 000135 1781273484.889006 read 0x01a92b thru 0x0297f7 (0xeecd bytes) 000136 1781273484.894899 write 0x02367a thru 0x027c21 (0x45a8 bytes) 000138 1781273484.896837 trunc from 0x02b6ca to 0x020f09 (0xa7c0 bytes) 000139 1781273484.934730 write 0x02a1c8 thru 0x039cf2 (0xfb2b bytes) 000143 1781273484.945068 mapread 0x01ae9c thru 0x01b806 (0x96b bytes) 000144 1781273484.953604 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39cf3): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39cf3): Mirror 3: 0xd3c3f72a (stale) Mirror 4: 0xc033af14 (stale) CRC-32 checksum value for chunk [0, 0x39cf3): Mirror 1: 0x4f43a91b Mirror 5: 0x4f43a91b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39cf3: skipped 000145 1781273486.206246 write 0x00096c thru 0x005e1d (0x54b2 bytes) 000148 1781273486.372119 mapread 0x02e352 thru 0x039cf2 (0xb9a1 bytes) 000150 1781273486.409699 mapwrite 0x011519 thru 0x01e998 (0xd480 bytes) 000151 1781273486.534430 MIRROR_RESYNC 4 mirrors 000153 1781273487.942236 write 0x010563 thru 0x01df1d (0xd9bb bytes) 000157 1781273488.093040 mapread 0x02e649 thru 0x02e79c (0x154 bytes) 000159 1781273488.112432 mapread 0x00a5d6 thru 0x017b59 (0xd584 bytes) 000161 1781273488.133995 write 0x01d631 thru 0x01de30 (0x800 bytes) 000167 1781273488.153972 trunc from 0x039cf3 to 0x02772b (0x125c7 bytes) 000169 1781273488.187751 write 0x037b2e thru 0x03ffff (0x84d2 bytes) 000170 1781273488.189234 write 0x007a97 thru 0x00bfb1 (0x451b bytes) 000171 1781273488.198036 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8d077815 (stale) Mirror 5: 0x8d077815 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8d077815 (stale) Mirror 5: 0x8d077815 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000172 1781273488.700546 read 0x02c173 thru 0x036b5d (0xa9eb bytes) 000175 1781273488.757314 read 0x03c197 thru 0x03ffff (0x3e69 bytes) 000176 1781273488.784323 trunc from 0x040000 to 0x011e7a (0x2e185 bytes) 000178 1781273488.829770 write 0x03c542 thru 0x03ffff (0x3abe bytes) 000180 1781273488.866937 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x8d077815 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x8d077815 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000181 1781273489.481689 read 0x03ee0b thru 0x03ffff (0x11f5 bytes) 000183 1781273489.489041 mapwrite 0x0062eb thru 0x0156a7 (0xf3bd bytes) 000184 1781273489.629243 write 0x024d7e thru 0x0288be (0x3b41 bytes) 000186 1781273489.655690 mapwrite 0x00e4aa thru 0x00f789 (0x12e0 bytes) 000188 1781273489.775127 write 0x02fe60 thru 0x037100 (0x72a1 bytes) 000190 1781273489.831975 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x8d077815 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7fc9b225 Mirror 6: 0x7fc9b225 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000191 1781273491.063180 read 0x01ec9e thru 0x0202f1 (0x1654 bytes) 000192 1781273491.104704 mapread 0x00e4d3 thru 0x017524 (0x9052 bytes) 000194 1781273491.219524 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) 000195 1781273491.669270 MIRROR_RESYNC 2 mirrors 000196 1781273492.196454 write 0x0003f9 thru 0x0017d9 (0x13e1 bytes) 000197 1781273492.414588 write 0x00da29 thru 0x0123e8 (0x49c0 bytes) 000198 1781273492.427064 mapwrite 0x0038f3 thru 0x00dcfa (0xa408 bytes) 000199 1781273492.599783 trunc from 0x040000 to 0x011d4d (0x2e2b2 bytes) 000202 1781273492.633760 mapwrite 0x03db4b thru 0x03ffff (0x24b5 bytes) 000204 1781273492.767098 write 0x0092ff thru 0x01646c (0xd16e bytes) 000206 1781273492.778716 read 0x02469c thru 0x02c0b7 (0x7a1c bytes) 000209 1781273492.798904 read 0x01f31a thru 0x024abd (0x57a4 bytes) 000210 1781273492.814825 mapwrite 0x036911 thru 0x036c02 (0x2f2 bytes) 000212 1781273492.894676 mapread 0x01d913 thru 0x029667 (0xbd55 bytes) 000213 1781273492.897790 mapwrite 0x022f2f thru 0x024e39 (0x1f0b bytes) 000214 1781273492.946906 read 0x01bdc9 thru 0x0239e8 (0x7c20 bytes) 000216 1781273492.948204 read 0x032667 thru 0x033145 (0xadf bytes) 000217 1781273492.951566 read 0x0072ca thru 0x00a64d (0x3384 bytes) 000219 1781273492.954141 read 0x01894f thru 0x0197b7 (0xe69 bytes) 000220 1781273492.955622 mapread 0x0256b9 thru 0x02721d (0x1b65 bytes) 000221 1781273492.957095 read 0x02a21b thru 0x02d83a (0x3620 bytes) 000225 1781273492.959541 mapread 0x02b5ef thru 0x0368d9 (0xb2eb bytes) 000230 1781273492.966662 MIRROR_RESYNC 2 mirrors 000231 1781273493.784244 read 0x028735 thru 0x034a12 (0xc2de bytes) 000232 1781273493.798092 read 0x017ed3 thru 0x0273b9 (0xf4e7 bytes) 000234 1781273493.824576 MIRROR_EXTEND to 3 mirrors 000236 1781273494.702682 trunc from 0x040000 to 0x005942 (0x3a6bd bytes) 000237 1781273494.778773 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x5943): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5943): Mirror 6: 0x15c5bdb7 (stale) Mirror 7: 0x15c5bdb7 (stale) CRC-32 checksum value for chunk [0, 0x5943): Mirror 1: 0x15c5bdb7 Mirror 8: 0x15c5bdb7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5943: skipped 000238 1781273495.603375 read 0x003932 thru 0x005942 (0x2011 bytes) 000240 1781273495.636075 trunc from 0x005943 to 0x022cff (0x1d3bd bytes) 000241 1781273495.752007 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x22d00): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22d00): Mirror 6: 0xd523f142 (stale) Mirror 7: 0xd523f142 (stale) Mirror 8: 0xbe542827 (stale) CRC-32 checksum value for chunk [0, 0x22d00): Mirror 1: 0xbe542827 Mirror 9: 0xbe542827 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22d00: skipped 000242 1781273496.781053 mapwrite 0x02be3e thru 0x0312e4 (0x54a7 bytes) 000243 1781273496.986206 mapread 0x029303 thru 0x0312e4 (0x7fe2 bytes) 000244 1781273496.987873 trunc from 0x0312e5 to 0x005e9e (0x2b446 bytes) 000245 1781273497.034208 write 0x00512c thru 0x013316 (0xe1eb bytes) 000247 1781273497.042034 mapwrite 0x01f05f thru 0x024ade (0x5a80 bytes) 000249 1781273497.131120 mapread 0x0231db thru 0x024ade (0x1904 bytes) 000250 1781273497.134129 read 0x01e9fa thru 0x0231fd (0x4804 bytes) 000251 1781273497.135307 read 0x021fd6 thru 0x024ade (0x2b09 bytes) 000253 1781273497.141287 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24adf): Mirror 6: 0x36a9d5c0 (stale) Mirror 7: 0x36a9d5c0 (stale) Mirror 8: 0x5e20a87a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24adf): Mirror 6: 0x36a9d5c0 (stale) Mirror 7: 0x36a9d5c0 (stale) Mirror 8: 0x5e20a87a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24adf: skipped 000254 1781273497.546541 read 0x00993a thru 0x014fa9 (0xb670 bytes) 000258 1781273497.560884 mapwrite 0x02f456 thru 0x03df71 (0xeb1c bytes) 000264 1781273497.671966 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3df72): Mirror 7: 0x43681a5d (stale) Mirror 8: 0x2d3a8c42 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3df72): Mirror 7: 0x43681a5d (stale) Mirror 8: 0x2d3a8c42 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3df72: skipped 000265 1781273498.111729 trunc from 0x03df72 to 0x02e3bb (0xfbb6 bytes) 000266 1781273498.138784 mapwrite 0x00fe10 thru 0x01706c (0x725d bytes) 000269 1781273498.266859 read 0x00f562 thru 0x00f983 (0x422 bytes) 000271 1781273498.268149 mapread 0x006134 thru 0x00ad58 (0x4c25 bytes) 000273 1781273498.288475 mapwrite 0x00ba99 thru 0x01b56c (0xfad4 bytes) 000276 1781273498.352704 MIRROR_RESYNC 3 mirrors 000277 1781273499.171148 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) 000281 1781273499.499225 read 0x00f27a thru 0x016a0c (0x7793 bytes) 000282 1781273499.513126 trunc from 0x02e3bc to 0x001964 (0x2ca57 bytes) 000283 1781273499.701985 mapread 0x0000ef thru 0x0007fd (0x70f bytes) 000284 1781273499.712749 mapwrite 0x01d541 thru 0x02b67a (0xe13a bytes) 000285 1781273499.941696 write 0x02f3ac thru 0x02f942 (0x597 bytes) 000286 1781273499.946958 MIRROR_RESYNC 2 mirrors 000289 1781273500.798129 MIRROR_RESYNC 2 mirrors 000290 1781273501.230581 MIRROR_RESYNC 2 mirrors 000291 1781273501.625012 mapwrite 0x038d07 thru 0x03e411 (0x570b bytes) 000293 1781273501.950239 mapread 0x03ac93 thru 0x03e411 (0x377f bytes) 000296 1781273501.955469 trunc from 0x03e412 to 0x02802c (0x163e5 bytes) 000298 1781273501.985679 trunc from 0x02802d to 0x013b23 (0x14509 bytes) 000300 1781273502.048330 MIRROR_RESYNC 2 mirrors 000301 1781273502.782176 write 0x000898 thru 0x001d48 (0x14b1 bytes) 000304 1781273502.860634 read 0x012694 thru 0x013b23 (0x1490 bytes) 000305 1781273502.866020 mapwrite 0x01ba42 thru 0x02b0b5 (0xf674 bytes) 000306 1781273503.043693 mapread 0x008e8c thru 0x00d459 (0x45ce bytes) 000307 1781273503.069330 mapwrite 0x025325 thru 0x02f28c (0x9f68 bytes) 000308 1781273503.172953 read 0x00e762 thru 0x014c36 (0x64d5 bytes) 000310 1781273503.183418 trunc from 0x02f28d to 0x02f065 (0x227 bytes) 000312 1781273503.206892 mapwrite 0x02881a thru 0x02bc83 (0x346a bytes) 000313 1781273503.250528 read 0x00787c thru 0x00d3b3 (0x5b38 bytes) 000314 1781273503.251224 write 0x0182c8 thru 0x024134 (0xbe6d bytes) 000315 1781273503.252565 mapread 0x00c29d thru 0x01b9aa (0xf70e bytes) 000317 1781273503.258021 read 0x02da3c thru 0x02f065 (0x162a bytes) 000318 1781273503.259061 trunc from 0x02f066 to 0x003470 (0x2bbf5 bytes) 000319 1781273503.280931 mapread 0x000886 thru 0x003470 (0x2beb bytes) 000324 1781273503.286050 write 0x009b58 thru 0x00a7da (0xc83 bytes) 000326 1781273503.293680 MIRROR_RESYNC 2 mirrors 000327 1781273503.965148 read 0x0074ac thru 0x008e72 (0x19c7 bytes) 000328 1781273503.973405 write 0x01d9a0 thru 0x020c2e (0x328f bytes) 000332 1781273504.081096 write 0x014914 thru 0x01df22 (0x960f bytes) 000334 1781273504.088382 mapwrite 0x02615a thru 0x034341 (0xe1e8 bytes) 000335 1781273504.255577 MIRROR_RESYNC 2 mirrors 000336 1781273504.764200 read 0x0009f5 thru 0x00246c (0x1a78 bytes) 000339 1781273504.772625 trunc from 0x034342 to 0x035fe7 (0x1ca6 bytes) 000340 1781273504.917192 write 0x01d2bd thru 0x027d6e (0xaab2 bytes) 000341 1781273504.958499 MIRROR_RESYNC 2 mirrors 000342 1781273505.742268 mapwrite 0x01cfbd thru 0x02997a (0xc9be bytes) 000343 1781273505.934893 read 0x02ed58 thru 0x031fd2 (0x327b bytes) 000345 1781273505.946714 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x35fe8): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35fe8): Mirror 8: 0x6d349915 (stale) CRC-32 checksum value for chunk [0, 0x35fe8): Mirror 1: 0x6c71a94f Mirror 9: 0x6c71a94f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35fe8: skipped 000346 1781273506.940051 write 0x00a942 thru 0x013594 (0x8c53 bytes) 000347 1781273507.099058 write 0x0230e6 thru 0x03058e (0xd4a9 bytes) 000348 1781273507.119753 read 0x029bbd thru 0x0345c5 (0xaa09 bytes) 000349 1781273507.138647 write 0x03f27b thru 0x03ffff (0xd85 bytes) 000350 1781273507.152249 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x86f1a1c0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x86f1a1c0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000351 1781273507.535181 trunc from 0x040000 to 0x034ea6 (0xb159 bytes) 000353 1781273507.566761 write 0x03f15c thru 0x03ffff (0xea4 bytes) 000355 1781273507.568651 trunc from 0x040000 to 0x025232 (0x1adcd bytes) 000358 1781273507.599493 mapread 0x00dfff thru 0x01d4f3 (0xf4f5 bytes) 000360 1781273507.632937 mapread 0x01c138 thru 0x025232 (0x90fb bytes) 000361 1781273507.648530 MIRROR_RESYNC 2 mirrors 000362 1781273508.567727 mapread 0x00ce77 thru 0x017658 (0xa7e2 bytes) 000363 1781273508.638329 MIRROR_RESYNC 2 mirrors 000364 1781273509.002034 mapread 0x01cce0 thru 0x01d57b (0x89c bytes) 000365 1781273509.002827 write 0x025a95 thru 0x026235 (0x7a1 bytes) 000366 1781273509.092433 write 0x03560e thru 0x03d980 (0x8373 bytes) 000367 1781273509.093934 trunc from 0x03d981 to 0x02b61c (0x12364 bytes) 000369 1781273509.134719 mapread 0x01505e thru 0x01f7a1 (0xa744 bytes) 000370 1781273509.193977 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2b61d): Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b61d): Mirror 9: 0x885114f0 (stale) CRC-32 checksum value for chunk [0, 0x2b61d): Mirror 1: 0x4d8adbb5 Mirror 10: 0x4d8adbb5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b61d: skipped 000371 1781273510.130600 write 0x03f31c thru 0x03ffff (0xce4 bytes) 000372 1781273510.239845 write 0x013fbc thru 0x021ad8 (0xdb1d bytes) 000373 1781273510.261903 read 0x0299eb thru 0x0377d9 (0xddef bytes) 000374 1781273510.276847 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x3b4b795c (stale) Mirror 10: 0x19e78889 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd983e3f6 Mirror 11: 0xd983e3f6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000375 1781273511.129003 mapwrite 0x03913a thru 0x03ffff (0x6ec6 bytes) 000377 1781273511.477767 mapread 0x03c424 thru 0x03ffff (0x3bdc bytes) 000379 1781273511.485754 mapread 0x034bae thru 0x03ffff (0xb452 bytes) 000381 1781273511.497657 trunc from 0x040000 to 0x02a441 (0x15bbe bytes) 000383 1781273511.567318 read 0x007921 thru 0x00d7c4 (0x5ea4 bytes) 000385 1781273511.599482 write 0x0317c8 thru 0x0353ae (0x3be7 bytes) 000386 1781273511.619625 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x353af): Mirror 9: 0x71e52cc5 (stale) Mirror 10: 0xdf34604c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x353af): Mirror 9: 0x71e52cc5 (stale) Mirror 10: 0xdf34604c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x353af: skipped 000387 1781273512.160160 mapwrite 0x02a10d thru 0x031a74 (0x7968 bytes) 000388 1781273512.331802 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x353af): Mirror 10: 0xdf34604c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x353af): Mirror 10: 0xdf34604c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x353af: skipped 000389 1781273512.679425 read 0x015201 thru 0x017345 (0x2145 bytes) 000390 1781273512.691404 mapwrite 0x014f13 thru 0x01fe61 (0xaf4f bytes) 000392 1781273512.781990 mapread 0x0175b2 thru 0x01b736 (0x4185 bytes) 000394 1781273512.784929 mapwrite 0x02a2eb thru 0x035aae (0xb7c4 bytes) 000396 1781273512.876630 mapread 0x032832 thru 0x035aae (0x327d bytes) 000398 1781273512.884608 MIRROR_RESYNC 2 mirrors 000401 1781273513.743033 write 0x00af4a thru 0x017470 (0xc527 bytes) 000402 1781273513.849796 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) 000403 1781273514.043289 read 0x01dc48 thru 0x02a81a (0xcbd3 bytes) 000405 1781273514.059430 trunc from 0x035aaf to 0x0287e0 (0xd2ce bytes) 000407 1781273514.078143 mapwrite 0x00c94e thru 0x00f6f7 (0x2daa bytes) 000408 1781273514.153012 read 0x01ba0d thru 0x0274be (0xbab2 bytes) 000411 1781273514.170659 MIRROR_EXTEND to 2 mirrors 000412 1781273515.159598 read 0x01bc76 thru 0x020eff (0x528a bytes) 000413 1781273515.178675 mapwrite 0x024b13 thru 0x02c30c (0x77fa bytes) 000414 1781273515.489335 write 0x007ac9 thru 0x00eb72 (0x70aa bytes) 000416 1781273515.500124 write 0x01058c thru 0x01c100 (0xbb75 bytes) 000417 1781273515.515146 read 0x012083 thru 0x018012 (0x5f90 bytes) 000418 1781273515.516572 trunc from 0x02c30d to 0x034656 (0x834a bytes) 000420 1781273515.558203 mapread 0x009aae thru 0x016688 (0xcbdb bytes) 000423 1781273515.592910 MIRROR_RESYNC 2 mirrors 000424 1781273516.176789 mapwrite 0x03c250 thru 0x03ffff (0x3db0 bytes) 000425 1781273516.408536 write 0x022809 thru 0x02b267 (0x8a5f bytes) 000428 1781273516.445227 MIRROR_RESYNC 2 mirrors 000433 1781273517.257916 MIRROR_EXTEND to 3 mirrors 000436 1781273518.027627 mapread 0x00f9e6 thru 0x018361 (0x897c bytes) 000439 1781273518.094702 read 0x011d43 thru 0x020b04 (0xedc2 bytes) 000440 1781273518.110627 trunc from 0x040000 to 0x0307fd (0xf802 bytes) 000442 1781273518.236068 trunc from 0x0307fe to 0x01d5aa (0x13253 bytes) 000444 1781273518.291731 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d5ab): Mirror 3: 0x4f0074da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d5ab): Mirror 3: 0x4f0074da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d5ab: skipped 000446 1781273518.710977 mapread 0x007757 thru 0x010094 (0x893e bytes) 000448 1781273518.761951 read 0x0165bb thru 0x01d5aa (0x6ff0 bytes) 000450 1781273518.779465 write 0x03e33d thru 0x03ffff (0x1cc3 bytes) 000451 1781273518.788674 MIRROR_RESYNC 2 mirrors 000453 1781273519.803170 write 0x004820 thru 0x009546 (0x4d27 bytes) 000456 1781273519.937840 mapread 0x01ff00 thru 0x0222f6 (0x23f7 bytes) 000458 1781273519.989927 read 0x019958 thru 0x01dc71 (0x431a bytes) 000459 1781273520.011011 write 0x005653 thru 0x00ed77 (0x9725 bytes) 000460 1781273520.024479 trunc from 0x040000 to 0x02b1b6 (0x14e49 bytes) 000461 1781273520.051193 mapread 0x0224ea thru 0x02b1b6 (0x8ccd bytes) 000463 1781273520.059290 mapread 0x010be6 thru 0x01bcbd (0xb0d8 bytes) 000466 1781273520.081165 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2b1b7): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b1b7): Mirror 3: 0xf90a2a92 (stale) CRC-32 checksum value for chunk [0, 0x2b1b7): Mirror 1: 0xda91c505 Mirror 4: 0xda91c505 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b1b7: skipped truncating to largest ever: 0x3aa12 000467 1781273521.089855 trunc from 0x02b1b7 to 0x03aa11 (0xf85b bytes) 000468 1781273521.207185 mapread 0x03470b thru 0x03aa11 (0x6307 bytes) 000470 1781273521.227723 write 0x039266 thru 0x03eac3 (0x585e bytes) 000473 1781273521.235738 mapread 0x035819 thru 0x03eac3 (0x92ab bytes) 000475 1781273521.239142 mapwrite 0x03c6a8 thru 0x03ffff (0x3958 bytes) 000476 1781273521.346216 write 0x033031 thru 0x03b3f5 (0x83c5 bytes) 000477 1781273521.361395 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfe98e491 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfe98e491 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000478 1781273521.745287 MIRROR_RESYNC 2 mirrors 000481 1781273522.455636 mapwrite 0x009f34 thru 0x012e9b (0x8f68 bytes) 000482 1781273522.672809 write 0x01e6d9 thru 0x02a238 (0xbb60 bytes) 000483 1781273522.689072 write 0x0151c5 thru 0x01ba9e (0x68da bytes) 000484 1781273522.712404 mapread 0x02fac0 thru 0x0323a7 (0x28e8 bytes) 000485 1781273522.744003 mapread 0x016d0f thru 0x01ea1f (0x7d11 bytes) 000486 1781273522.752765 trunc from 0x040000 to 0x007b42 (0x384bd bytes) 000487 1781273522.794291 write 0x03ba66 thru 0x03ffff (0x459a bytes) 000488 1781273522.795904 write 0x00d510 thru 0x0130ae (0x5b9f bytes) 000490 1781273522.828332 mapread 0x01042c thru 0x01f387 (0xef5c bytes) 000491 1781273522.850536 mapread 0x033d75 thru 0x03dd6c (0x9ff8 bytes) 000492 1781273522.868942 write 0x02be9c thru 0x02cf78 (0x10dd bytes) 000495 1781273522.876041 mapwrite 0x008e20 thru 0x012f43 (0xa124 bytes) 000497 1781273522.962012 mapread 0x036f29 thru 0x03c66c (0x5744 bytes) truncating to largest ever: 0x3af94 000498 1781273522.967140 trunc from 0x040000 to 0x03af93 (0x506c bytes) 000500 1781273523.009781 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3af94): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3af94): Mirror 3: 0xf4fea0ac (stale) CRC-32 checksum value for chunk [0, 0x3af94): Mirror 1: 0xb14cc4f6 Mirror 4: 0xb14cc4f6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3af94: skipped 000501 1781273524.237503 trunc from 0x03af94 to 0x01f027 (0x1bf6c bytes) 000502 1781273524.380013 trunc from 0x01f028 to 0x016a33 (0x85f4 bytes) 000506 1781273524.452726 MIRROR_RESYNC 3 mirrors 000509 1781273525.388222 MIRROR_EXTEND to 4 mirrors 000510 1781273526.229146 mapwrite 0x032520 thru 0x03aa71 (0x8552 bytes) 000512 1781273526.393835 trunc from 0x03aa72 to 0x035a67 (0x500a bytes) 000516 1781273526.446191 write 0x028b6e thru 0x028f86 (0x419 bytes) 000518 1781273526.478824 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35a68): Mirror 3: 0x5402e95a (stale) Mirror 5: 0x5402e95a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35a68): Mirror 3: 0x5402e95a (stale) Mirror 5: 0x5402e95a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35a68: skipped 000519 1781273526.849922 mapwrite 0x00d445 thru 0x01a618 (0xd1d4 bytes) 000521 1781273526.947466 write 0x02eb69 thru 0x03c14f (0xd5e7 bytes) 000522 1781273526.954653 mapwrite 0x02b658 thru 0x0361b7 (0xab60 bytes) 000523 1781273527.053249 mapwrite 0x00834f thru 0x00f1e9 (0x6e9b bytes) 000524 1781273527.173133 read 0x0013d9 thru 0x00eddd (0xda05 bytes) 000525 1781273527.196237 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3c150): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c150): Mirror 3: 0x595e9f70 (stale) Mirror 5: 0x595e9f70 (stale) CRC-32 checksum value for chunk [0, 0x3c150): Mirror 1: 0xe86fa1f1 Mirror 6: 0xe86fa1f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c150: skipped 000526 1781273528.158213 write 0x00ad93 thru 0x014d86 (0x9ff4 bytes) 000528 1781273528.297529 MIRROR_RESYNC 4 mirrors 000531 1781273529.666134 write 0x01fe5f thru 0x02bdf0 (0xbf92 bytes) 000532 1781273529.729089 read 0x015f6a thru 0x023500 (0xd597 bytes) 000534 1781273529.739579 trunc from 0x03c150 to 0x028dec (0x13363 bytes) 000535 1781273529.763909 mapwrite 0x00db4e thru 0x01d2ff (0xf7b2 bytes) 000537 1781273529.868774 write 0x00e862 thru 0x019415 (0xabb4 bytes) 000540 1781273529.889582 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x28ded): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28ded): Mirror 3: 0x968abcb6 (stale) Mirror 5: 0x968abcb6 (stale) Mirror 6: 0x968abcb6 (stale) CRC-32 checksum value for chunk [0, 0x28ded): Mirror 1: 0xc4bc4d38 Mirror 7: 0xc4bc4d38 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28ded: skipped 000544 1781273530.995519 mapwrite 0x027d49 thru 0x03312b (0xb3e3 bytes) 000545 1781273531.236250 mapread 0x0212bf thru 0x025252 (0x3f94 bytes) 000546 1781273531.254806 trunc from 0x03312c to 0x010d62 (0x223c9 bytes) 000547 1781273531.293651 mapwrite 0x004c76 thru 0x00b718 (0x6aa3 bytes) 000548 1781273531.371662 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x10d63): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10d63): Mirror 3: 0x74119489 (stale) Mirror 5: 0x74119489 (stale) Mirror 6: 0x74119489 (stale) Mirror 7: 0x329ddaa4 (stale) CRC-32 checksum value for chunk [0, 0x10d63): Mirror 1: 0x5761c003 Mirror 8: 0x5761c003 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10d63: skipped 000549 1781273532.385838 write 0x0302a4 thru 0x0385ac (0x8309 bytes) 000550 1781273532.471165 mapwrite 0x01b15a thru 0x026959 (0xb800 bytes) 000554 1781273532.559774 mapwrite 0x03ca33 thru 0x03ffff (0x35cd bytes) 000556 1781273532.657309 mapwrite 0x01a6f7 thru 0x027be7 (0xd4f1 bytes) 000557 1781273532.704119 mapread 0x00f9e7 thru 0x019240 (0x985a bytes) 000559 1781273532.720566 mapread 0x02a80a thru 0x02ca49 (0x2240 bytes) 000562 1781273532.724412 mapread 0x00211d thru 0x010355 (0xe239 bytes) 000563 1781273532.743884 read 0x01eae4 thru 0x02cb39 (0xe056 bytes) 000565 1781273532.746944 trunc from 0x040000 to 0x02a704 (0x158fb bytes) 000566 1781273532.773789 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a705): Mirror 5: 0x35584cc6 (stale) Mirror 6: 0x35584cc6 (stale) Mirror 7: 0x5f060531 (stale) Mirror 8: 0x85c0f443 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a705): Mirror 5: 0x35584cc6 (stale) Mirror 6: 0x35584cc6 (stale) Mirror 7: 0x5f060531 (stale) Mirror 8: 0x85c0f443 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a705: skipped 000567 1781273533.242659 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a705): Mirror 5: 0x35584cc6 (stale) Mirror 6: 0x35584cc6 (stale) Mirror 7: 0x5f060531 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a705): Mirror 5: 0x35584cc6 (stale) Mirror 6: 0x35584cc6 (stale) Mirror 7: 0x5f060531 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a705: skipped 000568 1781273533.632073 mapwrite 0x033ef9 thru 0x03ffff (0xc107 bytes) 000570 1781273533.757656 read 0x01c612 thru 0x021cdc (0x56cb bytes) 000571 1781273533.773909 write 0x022640 thru 0x0294fa (0x6ebb bytes) 000574 1781273533.797211 mapwrite 0x00a1b6 thru 0x0121f7 (0x8042 bytes) 000577 1781273533.886096 mapwrite 0x02ea10 thru 0x03d788 (0xed79 bytes) 000580 1781273533.931320 write 0x033c26 thru 0x03913c (0x5517 bytes) 000581 1781273533.937214 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x7dce7098 (stale) Mirror 6: 0x7dce7098 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x7dce7098 (stale) Mirror 6: 0x7dce7098 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000583 1781273534.356118 read 0x022d15 thru 0x0237cc (0xab8 bytes) 000585 1781273534.365037 write 0x004c24 thru 0x005863 (0xc40 bytes) 000586 1781273534.393156 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x7dce7098 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x7dce7098 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000588 1781273534.769461 mapread 0x001813 thru 0x0034e4 (0x1cd2 bytes) 000593 1781273534.810554 trunc from 0x040000 to 0x024b4d (0x1b4b2 bytes) 000594 1781273534.839030 write 0x0086db thru 0x013eeb (0xb811 bytes) 000595 1781273534.858041 write 0x0252df thru 0x02dc7b (0x899d bytes) 000598 1781273534.872562 read 0x00effb thru 0x012218 (0x321e bytes) 000599 1781273534.876943 MIRROR_RESYNC 2 mirrors 000603 1781273535.708709 mapwrite 0x036a4b thru 0x03c902 (0x5eb8 bytes) 000605 1781273535.934827 mapwrite 0x02a81c thru 0x030397 (0x5b7c bytes) 000606 1781273536.010745 mapwrite 0x02fdc2 thru 0x03734b (0x758a bytes) 000607 1781273536.072258 mapread 0x035461 thru 0x03c902 (0x74a2 bytes) 000610 1781273536.075680 trunc from 0x03c903 to 0x002b12 (0x39df0 bytes) 000612 1781273536.106049 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) 000613 1781273536.307176 read 0x002379 thru 0x002b12 (0x79a bytes) 000614 1781273536.322634 mapread 0x000f34 thru 0x002b12 (0x1bdf bytes) 000615 1781273536.348152 mapread 0x001950 thru 0x002b12 (0x11c3 bytes) 000616 1781273536.349438 mapread 0x000e58 thru 0x002b12 (0x1cbb bytes) 000619 1781273536.354407 mapwrite 0x0325f7 thru 0x034b96 (0x25a0 bytes) 000620 1781273536.462426 mapwrite 0x008e0d thru 0x01795c (0xeb50 bytes) 000621 1781273536.602232 mapwrite 0x012e98 thru 0x017bf5 (0x4d5e bytes) 000623 1781273536.653196 mapwrite 0x0339d1 thru 0x035df8 (0x2428 bytes) 000624 1781273536.731976 read 0x01d3b1 thru 0x02c796 (0xf3e6 bytes) 000625 1781273536.742961 write 0x01c904 thru 0x023e6e (0x756b bytes) 000627 1781273536.748335 mapwrite 0x016594 thru 0x019bb8 (0x3625 bytes) 000628 1781273536.803109 read 0x02819b thru 0x035df8 (0xdc5e bytes) 000636 1781273536.813259 mapwrite 0x000ff3 thru 0x009439 (0x8447 bytes) 000644 1781273536.875456 read 0x001ae7 thru 0x005c8b (0x41a5 bytes) 000646 1781273536.878065 read 0x01290b thru 0x01a485 (0x7b7b bytes) 000647 1781273536.879469 write 0x0089dd thru 0x009e0c (0x1430 bytes) 000648 1781273536.880821 read 0x02d17c thru 0x035df8 (0x8c7d bytes) 000651 1781273536.882775 trunc from 0x035df9 to 0x034b65 (0x1293 bytes) 000652 1781273536.917235 trunc from 0x034b66 to 0x005903 (0x2f262 bytes) 000655 1781273536.944324 mapwrite 0x0078e9 thru 0x0100d5 (0x87ed bytes) 000658 1781273537.060213 mapread 0x003cdc thru 0x0100d5 (0xc3fa bytes) 000659 1781273537.067070 MIRROR_EXTEND to 2 mirrors 000660 1781273537.843755 trunc from 0x0100d6 to 0x0032f2 (0xcde3 bytes) 000662 1781273537.978995 mapwrite 0x018a7e thru 0x01e931 (0x5eb4 bytes) 000663 1781273538.110893 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1e932): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e932): Mirror 2: 0xc128f65f (stale) CRC-32 checksum value for chunk [0, 0x1e932): Mirror 1: 0x856f6ebd Mirror 3: 0x856f6ebd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e932: skipped 000666 1781273539.079316 mapread 0x00ed11 thru 0x014ef9 (0x61e9 bytes) 000668 1781273539.116305 mapread 0x0113ac thru 0x015c46 (0x489b bytes) 000671 1781273539.133423 mapread 0x013912 thru 0x01e931 (0xb020 bytes) 000674 1781273539.170284 write 0x01a587 thru 0x028c6b (0xe6e5 bytes) 000675 1781273539.284824 read 0x00f01f thru 0x01339b (0x437d bytes) 000676 1781273539.299938 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x28c6c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28c6c): Mirror 2: 0xbe8f0739 (stale) Mirror 3: 0xe4c1ca1a (stale) CRC-32 checksum value for chunk [0, 0x28c6c): Mirror 1: 0xb360bc19 Mirror 4: 0xb360bc19 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28c6c: skipped 000678 1781273540.370459 mapread 0x003983 thru 0x00a565 (0x6be3 bytes) 000682 1781273540.410662 trunc from 0x028c6c to 0x0181b9 (0x10ab2 bytes) 000684 1781273540.518398 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x181ba): Mirror 2: 0xd577b5db (stale) Mirror 3: 0x2ed4f563 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x181ba): Mirror 2: 0xd577b5db (stale) Mirror 3: 0x2ed4f563 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x181ba: skipped 000686 1781273540.956808 mapwrite 0x0286d9 thru 0x029674 (0xf9c bytes) 000687 1781273541.066765 MIRROR_RESYNC 3 mirrors 000688 1781273542.165555 mapwrite 0x014f74 thru 0x0239e0 (0xea6d bytes) 000690 1781273542.431623 mapread 0x00fc5b thru 0x016990 (0x6d36 bytes) 000693 1781273542.436557 mapwrite 0x02fcc6 thru 0x033992 (0x3ccd bytes) 000694 1781273542.567633 mapread 0x025eb9 thru 0x03283a (0xc982 bytes) 000696 1781273542.583371 mapread 0x001c84 thru 0x00d5aa (0xb927 bytes) 000697 1781273542.617049 read 0x029e55 thru 0x02b716 (0x18c2 bytes) 000698 1781273542.617877 read 0x033543 thru 0x033992 (0x450 bytes) 000699 1781273542.621153 mapread 0x019f7d thru 0x01d7b1 (0x3835 bytes) 000700 1781273542.627476 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33993): Mirror 3: 0xb27340ea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33993): Mirror 3: 0xb27340ea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33993: skipped 000701 1781273543.040571 mapread 0x005d5c thru 0x009d3b (0x3fe0 bytes) 000702 1781273543.072724 mapwrite 0x004cde thru 0x00acc6 (0x5fe9 bytes) 000704 1781273543.158262 read 0x00246b thru 0x009610 (0x71a6 bytes) 000705 1781273543.161685 mapwrite 0x00f102 thru 0x016f74 (0x7e73 bytes) 000707 1781273543.224740 mapread 0x00e31a thru 0x019857 (0xb53e bytes) 000708 1781273543.226952 write 0x00f692 thru 0x01c67b (0xcfea bytes) 000709 1781273543.240450 read 0x020e7d thru 0x020f11 (0x95 bytes) 000712 1781273543.298297 mapwrite 0x00c157 thru 0x012f07 (0x6db1 bytes) 000713 1781273543.419312 trunc from 0x033993 to 0x0340bd (0x72b bytes) 000714 1781273543.443689 trunc from 0x0340be to 0x032411 (0x1cac bytes) 000716 1781273543.474693 mapread 0x02137b thru 0x028ad4 (0x775a bytes) 000717 1781273543.478699 mapread 0x00a637 thru 0x00b815 (0x11df bytes) 000719 1781273543.484810 mapread 0x01a6cd thru 0x01b980 (0x12b4 bytes) 000720 1781273543.490124 read 0x0317d6 thru 0x032411 (0xc3c bytes) 000721 1781273543.491697 write 0x01b45a thru 0x01cd8c (0x1933 bytes) 000722 1781273543.494059 mapread 0x025e23 thru 0x032411 (0xc5ef bytes) 000723 1781273543.496910 mapread 0x01b324 thru 0x021fd5 (0x6cb2 bytes) 000724 1781273543.498528 mapwrite 0x0311cd thru 0x037587 (0x63bb bytes) 000727 1781273543.670835 mapread 0x00d7fe thru 0x01cae1 (0xf2e4 bytes) 000728 1781273543.677501 trunc from 0x037588 to 0x008364 (0x2f223 bytes) 000729 1781273543.712153 mapread 0x00178e thru 0x008364 (0x6bd7 bytes) 000731 1781273543.720095 mapread 0x0034b9 thru 0x007a85 (0x45cd bytes) 000732 1781273543.721669 mapwrite 0x006654 thru 0x0142e9 (0xdc96 bytes) 000733 1781273543.933680 write 0x00f6a9 thru 0x011c43 (0x259b bytes) 000734 1781273543.939598 read 0x00a98c thru 0x00cef5 (0x256a bytes) 000735 1781273543.945246 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x142ea): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x142ea): Mirror 3: 0xfaf94c9f (stale) CRC-32 checksum value for chunk [0, 0x142ea): Mirror 1: 0xddbde785 Mirror 4: 0xddbde785 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x142ea: skipped 000736 1781273545.038872 write 0x028014 thru 0x037f21 (0xff0e bytes) 000738 1781273545.158492 mapwrite 0x02ef11 thru 0x0330f1 (0x41e1 bytes) 000741 1781273545.298135 write 0x00729f thru 0x01496e (0xd6d0 bytes) 000743 1781273545.320838 trunc from 0x037f22 to 0x00b917 (0x2c60a bytes) 000748 1781273545.380058 mapread 0x002236 thru 0x009e73 (0x7c3e bytes) 000750 1781273545.404905 write 0x01bde7 thru 0x01c36b (0x585 bytes) 000751 1781273545.406683 mapread 0x01a13c thru 0x01a61b (0x4e0 bytes) 000753 1781273545.415194 mapwrite 0x00c1ae thru 0x014bf4 (0x8a47 bytes) 000754 1781273545.462225 read 0x005612 thru 0x011f55 (0xc944 bytes) 000755 1781273545.463393 mapread 0x00f1b5 thru 0x01c36b (0xd1b7 bytes) 000756 1781273545.466140 write 0x01e334 thru 0x025426 (0x70f3 bytes) 000758 1781273545.468772 mapwrite 0x02cb41 thru 0x0324b2 (0x5972 bytes) 000759 1781273545.555664 MIRROR_RESYNC 3 mirrors 000761 1781273546.384088 write 0x03109c thru 0x03647f (0x53e4 bytes) 000763 1781273546.446027 mapread 0x016280 thru 0x024c2c (0xe9ad bytes) 000765 1781273546.476555 read 0x00284d thru 0x010532 (0xdce6 bytes) 000766 1781273546.490494 mapread 0x00c83d thru 0x017f5d (0xb721 bytes) 000767 1781273546.496755 MIRROR_RESYNC 3 mirrors 000771 1781273547.368147 trunc from 0x036480 to 0x01f499 (0x16fe6 bytes) 000773 1781273547.464985 mapread 0x0197a3 thru 0x01f499 (0x5cf7 bytes) 000774 1781273547.487375 write 0x035f33 thru 0x03b3f5 (0x54c3 bytes) 000775 1781273547.488794 mapwrite 0x032cb2 thru 0x03453a (0x1889 bytes) 000776 1781273547.537533 write 0x020f85 thru 0x024e8b (0x3f07 bytes) 000777 1781273547.549277 mapread 0x0215d3 thru 0x0217c2 (0x1f0 bytes) 000779 1781273547.552305 read 0x0229d0 thru 0x02d153 (0xa784 bytes) 000780 1781273547.562141 read 0x025ea5 thru 0x0327a7 (0xc903 bytes) 000781 1781273547.563372 trunc from 0x03b3f6 to 0x0195aa (0x21e4b bytes) 000782 1781273547.600778 write 0x018046 thru 0x027574 (0xf52f bytes) 000783 1781273547.609648 write 0x02fb15 thru 0x03733c (0x7828 bytes) 000784 1781273547.617259 mapwrite 0x014946 thru 0x017f1f (0x35da bytes) 000785 1781273547.676204 mapwrite 0x02b448 thru 0x03771e (0xc2d7 bytes) 000786 1781273547.811198 mapread 0x013f19 thru 0x01be0b (0x7ef3 bytes) 000787 1781273547.814455 write 0x014e5f thru 0x015011 (0x1b3 bytes) 000788 1781273547.816508 mapwrite 0x03b012 thru 0x03ffff (0x4fee bytes) 000789 1781273547.909443 read 0x039883 thru 0x03b9de (0x215c bytes) 000792 1781273547.916164 mapread 0x03c8f6 thru 0x03db74 (0x127f bytes) 000794 1781273547.919278 write 0x035ac8 thru 0x03c28e (0x67c7 bytes) 000795 1781273547.922211 mapread 0x026f47 thru 0x027abe (0xb78 bytes) 000796 1781273547.924266 mapread 0x02e952 thru 0x03d7bf (0xee6e bytes) 000797 1781273547.926659 write 0x03a81a thru 0x03ffff (0x57e6 bytes) 000798 1781273547.929215 trunc from 0x040000 to 0x0174a7 (0x28b58 bytes) 000803 1781273547.955479 mapread 0x014b4b thru 0x0174a7 (0x295d bytes) 000804 1781273547.957711 mapwrite 0x03ff65 thru 0x03ffff (0x9b bytes) 000805 1781273548.046199 write 0x03a666 thru 0x03ffff (0x599a bytes) 000808 1781273548.055439 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6b331b12 (stale) Mirror 4: 0x6b331b12 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7f783cac Mirror 5: 0x7f783cac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000809 1781273549.233663 write 0x02a805 thru 0x039943 (0xf13f bytes) 000812 1781273549.343071 trunc from 0x040000 to 0x038596 (0x7a69 bytes) 000815 1781273549.378283 mapwrite 0x03fa80 thru 0x03ffff (0x580 bytes) 000819 1781273549.480200 mapwrite 0x010be9 thru 0x018c32 (0x804a bytes) 000820 1781273549.558197 mapread 0x00d8db thru 0x013d36 (0x645c bytes) 000821 1781273549.562875 MIRROR_RESYNC 4 mirrors 000822 1781273550.643659 mapread 0x013139 thru 0x0228bf (0xf787 bytes) 000823 1781273550.695391 write 0x005b86 thru 0x00a697 (0x4b12 bytes) 000824 1781273550.803048 read 0x0373a1 thru 0x03ffff (0x8c5f bytes) 000825 1781273550.827543 mapwrite 0x01e7c7 thru 0x02e6d2 (0xff0c bytes) 000827 1781273551.018426 MIRROR_RESYNC 4 mirrors 000828 1781273552.262547 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) 000829 1781273552.591967 mapwrite 0x02fa83 thru 0x035c99 (0x6217 bytes) 000830 1781273552.750648 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6252b226 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6252b226 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000831 1781273553.155446 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) 000832 1781273553.349144 mapwrite 0x013f47 thru 0x017777 (0x3831 bytes) 000835 1781273553.465157 read 0x032a0a thru 0x03e656 (0xbc4d bytes) 000836 1781273553.480980 mapread 0x0356e1 thru 0x036cc2 (0x15e2 bytes) 000837 1781273553.486400 trunc from 0x040000 to 0x003a39 (0x3c5c6 bytes) 000838 1781273553.529523 MIRROR_EXTEND to 2 mirrors 000839 1781273554.150252 read 0x0030a3 thru 0x003a39 (0x997 bytes) 000842 1781273554.182848 MIRROR_RESYNC 2 mirrors 000845 1781273554.630089 mapread 0x002111 thru 0x003a39 (0x1929 bytes) 000846 1781273554.639371 read 0x00040f thru 0x003a39 (0x362b bytes) 000847 1781273554.645626 trunc from 0x003a3a to 0x00eefa (0xb4c1 bytes) 000848 1781273554.785092 trunc from 0x00eefb to 0x02af12 (0x1c018 bytes) 000850 1781273554.818404 write 0x038c28 thru 0x03ffff (0x73d8 bytes) 000853 1781273554.825863 mapread 0x02a3b7 thru 0x02a7f6 (0x440 bytes) 000856 1781273554.840298 mapwrite 0x006c15 thru 0x00e83a (0x7c26 bytes) 000859 1781273554.964088 read 0x00d1b4 thru 0x0170d0 (0x9f1d bytes) 000861 1781273554.999183 write 0x01e626 thru 0x02a8d0 (0xc2ab bytes) 000865 1781273555.004401 mapwrite 0x030e91 thru 0x030f8f (0xff bytes) 000866 1781273555.062550 write 0x031fe9 thru 0x03627b (0x4293 bytes) 000868 1781273555.076975 trunc from 0x040000 to 0x020bbf (0x1f440 bytes) 000869 1781273555.116995 trunc from 0x020bc0 to 0x024d4d (0x418e bytes) 000871 1781273555.154055 trunc from 0x024d4e to 0x016b0b (0xe242 bytes) 000872 1781273555.190654 read 0x0048e1 thru 0x00f493 (0xabb3 bytes) 000873 1781273555.191810 mapwrite 0x014b20 thru 0x01a8b3 (0x5d94 bytes) 000874 1781273555.303524 mapwrite 0x009bbf thru 0x00c21d (0x265f bytes) 000876 1781273555.348240 mapwrite 0x026700 thru 0x02b922 (0x5223 bytes) 000878 1781273555.496394 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2b923): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b923): Mirror 2: 0x32516314 (stale) CRC-32 checksum value for chunk [0, 0x2b923): Mirror 1: 0x7e5720b7 Mirror 3: 0x7e5720b7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b923: skipped 000880 1781273556.463089 mapwrite 0x0280e1 thru 0x030b13 (0x8a33 bytes) 000881 1781273556.638559 mapread 0x0279e2 thru 0x030b13 (0x9132 bytes) 000882 1781273556.639922 mapwrite 0x010534 thru 0x0198aa (0x9377 bytes) 000883 1781273556.714456 mapwrite 0x0244ab thru 0x029fe4 (0x5b3a bytes) 000887 1781273556.783720 read 0x00c062 thru 0x0174bc (0xb45b bytes) 000892 1781273556.790655 write 0x02251b thru 0x030c8d (0xe773 bytes) 000893 1781273556.805840 mapwrite 0x01c183 thru 0x01fc46 (0x3ac4 bytes) 000894 1781273556.887430 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x30c8e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30c8e): Mirror 2: 0x3c04f0d2 (stale) Mirror 3: 0x48fcdc87 (stale) CRC-32 checksum value for chunk [0, 0x30c8e): Mirror 1: 0xde12e9f9 Mirror 4: 0xde12e9f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30c8e: skipped 000895 1781273557.804419 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30c8e): Mirror 3: 0x48fcdc87 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30c8e): Mirror 3: 0x48fcdc87 (stale) CRC-32 checksum value for chunk [0, 0x30c8e): Mirror 1: 0xde12e9f9 Mirror 4: 0xde12e9f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30c8e: skipped 000898 1781273558.758525 mapwrite 0x0274b6 thru 0x029afa (0x2645 bytes) 000899 1781273558.968277 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x30c8e): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30c8e): Mirror 3: 0x48fcdc87 (stale) Mirror 4: 0xde12e9f9 (stale) CRC-32 checksum value for chunk [0, 0x30c8e): Mirror 1: 0xba9988ae Mirror 5: 0xba9988ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30c8e: skipped 000900 1781273560.172196 mapwrite 0x001052 thru 0x00b2b2 (0xa261 bytes) 000903 1781273560.485503 read 0x00abf1 thru 0x01a86c (0xfc7c bytes) 000904 1781273560.500442 trunc from 0x030c8e to 0x016cbf (0x19fce bytes) 000907 1781273560.547573 read 0x004ff6 thru 0x008413 (0x341e bytes) 000908 1781273560.548619 mapwrite 0x030134 thru 0x035480 (0x534d bytes) 000912 1781273560.702928 MIRROR_RESYNC 4 mirrors 000913 1781273562.056726 mapwrite 0x02b525 thru 0x033f07 (0x89e3 bytes) 000915 1781273562.288237 read 0x00d0ff thru 0x00f3c0 (0x22c2 bytes) 000916 1781273562.299595 mapread 0x007580 thru 0x00864a (0x10cb bytes) 000918 1781273562.319921 read 0x026b82 thru 0x02b0da (0x4559 bytes) 000919 1781273562.320778 mapread 0x001d51 thru 0x0081b6 (0x6466 bytes) 000920 1781273562.324896 trunc from 0x035481 to 0x00a84b (0x2ac35 bytes) 000921 1781273562.347551 write 0x016b58 thru 0x01ac85 (0x412e bytes) 000923 1781273562.351246 mapread 0x019abd thru 0x01ac85 (0x11c9 bytes) 000925 1781273562.355161 trunc from 0x01ac86 to 0x00cc37 (0xe04e bytes) 000927 1781273562.390865 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xcc38): Mirror 3: 0x454f2bb6 (stale) Mirror 5: 0x454f2bb6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xcc38): Mirror 3: 0x454f2bb6 (stale) Mirror 5: 0x454f2bb6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xcc38: skipped 000929 1781273562.756971 mapwrite 0x03fce9 thru 0x03ffff (0x317 bytes) 000934 1781273562.834904 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9514c330 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9514c330 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000935 1781273563.174122 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9514c330 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf60bcf95 Mirror 4: 0xf60bcf95 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000937 1781273564.304021 mapread 0x01e067 thru 0x020409 (0x23a3 bytes) 000938 1781273564.352183 mapread 0x03848f thru 0x03ffff (0x7b71 bytes) 000939 1781273564.399644 MIRROR_RESYNC 3 mirrors 000941 1781273565.177195 mapwrite 0x03c39d thru 0x03d2d5 (0xf39 bytes) 000942 1781273565.400088 read 0x03c7b7 thru 0x03ffff (0x3849 bytes) 000944 1781273565.405544 mapread 0x0088ac thru 0x00c926 (0x407b bytes) 000947 1781273565.428016 trunc from 0x040000 to 0x00dd08 (0x322f7 bytes) 000949 1781273565.455905 mapwrite 0x03a84f thru 0x03ffff (0x57b1 bytes) 000951 1781273565.556096 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf60bcf95 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf60bcf95 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped truncating to largest ever: 0x3c413 000952 1781273565.986433 trunc from 0x040000 to 0x03c412 (0x3bed bytes) 000954 1781273566.019049 write 0x03bc1c thru 0x03ffff (0x43e4 bytes) 000955 1781273566.028532 read 0x02b980 thru 0x02d3a4 (0x1a25 bytes) 000956 1781273566.049036 read 0x01b722 thru 0x01d517 (0x1df6 bytes) 000957 1781273566.095163 trunc from 0x040000 to 0x022ce8 (0x1d317 bytes) 000958 1781273566.128301 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x22ce9): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22ce9): Mirror 4: 0xfedc1620 (stale) CRC-32 checksum value for chunk [0, 0x22ce9): Mirror 1: 0xfedc1620 Mirror 5: 0xfedc1620 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22ce9: skipped 000964 1781273566.815525 mapread 0x004c42 thru 0x014ba5 (0xff64 bytes) 000967 1781273566.900818 read 0x0037f6 thru 0x00510f (0x191a bytes) 000969 1781273566.911583 write 0x017e02 thru 0x023551 (0xb750 bytes) 000970 1781273567.021484 mapread 0x00bc29 thru 0x00dc40 (0x2018 bytes) 000971 1781273567.041278 read 0x00dbd6 thru 0x0155f6 (0x7a21 bytes) 000972 1781273567.054587 mapread 0x014db0 thru 0x023551 (0xe7a2 bytes) 000973 1781273567.063115 trunc from 0x023552 to 0x008b23 (0x1aa2e bytes) 000975 1781273567.202527 mapwrite 0x01a49b thru 0x026309 (0xbe6f bytes) 000976 1781273567.473030 write 0x01d11e thru 0x02012f (0x3012 bytes) 000980 1781273567.490203 read 0x001b36 thru 0x00cd81 (0xb24c bytes) 000981 1781273567.518632 read 0x02595a thru 0x026309 (0x9b0 bytes) 000982 1781273567.519263 read 0x00a6cb thru 0x01984b (0xf181 bytes) 000985 1781273567.523811 mapread 0x007c78 thru 0x013101 (0xb48a bytes) 000986 1781273567.533534 MIRROR_RESYNC 3 mirrors 000987 1781273568.539945 trunc from 0x02630a to 0x000204 (0x26105 bytes) 000989 1781273568.643877 trunc from 0x000205 to 0x029bc9 (0x299c5 bytes) 000991 1781273568.696554 MIRROR_RESYNC 3 mirrors 000992 1781273569.553922 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17b:0x0]: Invalid argument (22) 000993 1781273569.987898 mapread 0x020f5a thru 0x021f23 (0xfca bytes) 000994 1781273570.008270 write 0x01b71e thru 0x01dad1 (0x23b4 bytes) 000995 1781273570.113275 write 0x026ab1 thru 0x02fe96 (0x93e6 bytes) 000996 1781273570.122607 write 0x02e1d4 thru 0x033050 (0x4e7d bytes) 000997 1781273570.125103 write 0x021175 thru 0x024d27 (0x3bb3 bytes) 000998 1781273570.137596 write 0x02b3bb thru 0x03206b (0x6cb1 bytes) 000999 1781273570.138950 mapwrite 0x02130d thru 0x028997 (0x768b bytes) 001000 1781273570.249787 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x33051): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33051): Mirror 5: 0x99d62a9d (stale) CRC-32 checksum value for chunk [0, 0x33051): Mirror 1: 0xe3a02a49 Mirror 6: 0xe3a02a49 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33051: skipped All operations completed A-OK! PASS 70a (105s) == sanity-flr test 71: check mirror extend parent fid ==== 10:12:56 (1781273576) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 10:12:59 (1781273579) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.237571 s, 8.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174603 s, 6.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a5:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 10:13:08 (1781273588) Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre2 Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2020608 .. Extending file size to 3480960 .. done Extending file size to 4772352 .. Extending file size to 6878624 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 6454880 .. Extending file size to 6687072 .. Extending file size to 8288704 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8599040 .. Extending file size to 9359680 .. Extending file size to 8815872 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8895712 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9087744 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 56861 56862 56864 56865 56868 done 192.168.203.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg339-client.virtnet /mnt/lustre2 (opts:) 192.168.203.139@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg339-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (85s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 10:14:34 (1781273674) Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre2 Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1670656 .. Extending file size to 3450432 .. Extending file size to 4094560 .. Extending file size to 7706240 .. Extending file size to 7923040 .. Extending file size to 8608736 .. Extending file size to 8183392 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8803680 .. Extending file size to 8342784 .. Extending file size to 8705824 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9222688 .. Extending file size to 8850400 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9166336 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 58175 58176 58177 58178 58181 192.168.203.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg339-client.virtnet /mnt/lustre2 (opts:) 192.168.203.139@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg339-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (82s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 10:15:56 (1781273756) Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123901 s, 16.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.432489 s, 4.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x52926f82 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg339-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 10:16:16 (1781273776) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0799898 s, 26.2 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 10:16:24 (1781273784) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ce:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bf:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2ba:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 10:16:32 (1781273792) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.261205 s, 8.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 10:16:40 (1781273800) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.228411 s, 9.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2be:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 10:16:49 (1781273809) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.356372 s, 5.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bf:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bf:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c0:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 10:17:26 (1781273846) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg339-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122974 s, 8.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } PASS 204d (62s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 10:18:27 (1781273907) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg339-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.385765 s, 5.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (63s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 10:19:30 (1781273970) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg339-server mds-ost sync done. Creating new pool test_204f oleg339-server: Pool lustre.test_204f created Adding targets to pool oleg339-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg339-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.166788 s, 12.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg339-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg339-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg339-server: Pool lustre.test_204f destroyed PASS 204f (79s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 10:20:49 (1781274049) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 10:20:56 (1781274056) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 10:21:04 (1781274064) oleg339-server: Pool lustre.test_206 created Adding targets to pool oleg339-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg339-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg339-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg339-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg339-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 10:21:31 (1781274091) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058263 s, 18.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308546 s, 34.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x1dcba485 Mirror 3: 0x1dcba485 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 10:21:42 (1781274102) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 10:21:47 (1781274107) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 10:21:51 (1781274111) Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.105156 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xc30000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2170000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2170000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2170000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x9d0000 Read this (4 bytes): BBBB 192.168.203.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg339-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 10:22:30 (1781274150) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0461041 s, 178 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 10:22:38 (1781274158) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371744 s, 28.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f3 lmm_fid: [0x200000404:0x1f3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f3 lmm_fid: [0x200000404:0x1f3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834419 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-12 10:22:40.000000000 -0400 Modify: 2026-06-12 10:22:40.000000000 -0400 Change: 2026-06-12 10:22:42.000000000 -0400 Birth: 2026-06-12 10:22:40.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 10:22:47 (1781274167) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 10:22:51 (1781274171) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.614393 s, 66.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0870639 s, 47.0 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 12 10:22 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f5:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 12 10:22 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 10:23:00 (1781274180) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 10:23:09 (1781274189) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1758 sec =========== 10:23:18 (1781274198) === sanity-flr: start cleanup 10:23:20 (1781274200) === === sanity-flr: finish cleanup 10:23:27 (1781274207) ===