-----============= acceptance-small: sanity-flr ============----- Tue Aug 18 13:54:21 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg405-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 13:54:42 (1787075682) === oleg405-client.virtnet: executing check_config_client /mnt/lustre oleg405-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg405-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff940d87b95800.idle_timeout=debug osc.lustre-OST0001-osc-ffff940d87b95800.idle_timeout=debug disable quota as required oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 13:55:23 (1787075723) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7631] == sanity-flr test 0a: lfs mirror create with -N option == 13:55:27 (1787075727) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 13:55:48 (1787075748) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 13:55:52 (1787075752) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 13:55:56 (1787075756) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 13:56:24 (1787075784) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 13:56:27 (1787075787) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 13:56:30 (1787075790) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff940d87b95800.stats=clear osc.lustre-OST0001-osc-ffff940d87b95800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 13:56:38 (1787075798) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 13:56:49 (1787075809) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 13:57:00 (1787075820) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (5s) == sanity-flr test 0m: mirror extend with -N= option ===== 13:57:05 (1787075825) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 13:57:14 (1787075834) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 13:57:25 (1787075845) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 13:57:34 (1787075854) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 13:57:37 (1787075857) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 13:57:45 (1787075865) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 13:57:56 (1787075876) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 13:58:03 (1787075883) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.711402 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124813 s, 8.4 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 13:58:12 (1787075892) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.292137 s, 17.9 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 13:58:20 (1787075900) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 13:58:29 (1787075909) Stopping /mnt/lustre-ost1 (opts:) on oleg405-server oleg405-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff940d87b95800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff940d87b95800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg405-server: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg405-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg405-server oleg405-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff940d87b95800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff940d87b95800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg405-server: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg405-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (77s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 13:59:46 (1787075986) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.390556 s, 18.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg405-server oleg405-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff940d87b95800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff940d87b95800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg405-server: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg405-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (40s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 14:00:26 (1787076026) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg405-server oleg405-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff940d87b95800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff940d87b95800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg405-server: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg405-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg405-server oleg405-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff940d87b95800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff940d87b95800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg405-server: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg405-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (77s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 14:01:43 (1787076103) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.16555 s, 10.1 MB/s -rw-r--r-- 1 root root 52428800 Aug 18 14:01 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.013s user 0m0.002s sys 0m0.180s Stopping /mnt/lustre-ost1 (opts:) on oleg405-server oleg405-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff940d87b95800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff940d87b95800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.171s user 0m0.000s sys 0m0.202s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg405-server: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg405-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg405-server oleg405-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff940d87b95800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff940d87b95800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.914s user 0m0.001s sys 0m0.201s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg405-server: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg405-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg405-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff940d87b95800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff940d87b95800.ost_server_uuid in FULL state after 0 sec PASS 33b (84s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 14:03:07 (1787076187) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 14:03:11 (1787076191) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 14:03:13 (1787076193) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 14:03:17 (1787076197) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 14:03:24 (1787076204) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.5773 s, 21.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 14:03:37 (1787076217) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408043 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267273 s, 39.2 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 14:03:46 (1787076226) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 14:03:54 (1787076234) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00113657 s, 901 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00450216 s, 227 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00108035 s, 948 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00218706 s, 468 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 14:04:05 (1787076245) create test files with size 15739429 .. dfefdb1a5c482680beffd2cc29549bc6 - 8cab5e789b475d93a5f73006bfa153e2 - 3417562cd9840ac1a785ede899abbfb4 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg405-client.virtnet /mnt/lustre (opts:) Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 14:04:28 (1787076268) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (19s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 14:04:47 (1787076287) PASS 39 (9s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 14:04:56 (1787076296) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950918 s, 11.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637315 s, 16.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 14:05:08 (1787076308) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.145794 s, 14.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.397633 s, 10.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.222365 s, 9.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 14:05:25 (1787076325) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 14:05:29 (1787076329) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 14:05:32 (1787076332) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362218 s, 28.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 14:05:40 (1787076340) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 14:05:45 (1787076345) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 14:05:56 (1787076356) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.264133 s, 39.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg405-client.virtnet /mnt/lustre (opts:) Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 14:06:07 (1787076367) llite.lustre-ffff940d90b5a800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.423215 s, 24.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg405-client.virtnet /mnt/lustre (opts:) Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff940d862eb000.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 14:06:23 (1787076383) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (49s) == sanity-flr test 46: Verify setstripe --copy option ==== 14:07:12 (1787076432) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 14:07:50 (1787076470) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 14:07:53 (1787076473) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.224277 s, 14.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.155558 s, 20.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xabc95f25 (nosync,stale) Mirror 2: 0xabc95f25 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 202b5e38e8ac3b599c57ea646ee13598 - mirror 1 checksum: 9f1df1e777311fb0f496be2807f4fb10 - mirror 2 checksum: 202b5e38e8ac3b599c57ea646ee13598 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-18T14:07:57-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-18T14:07:57-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 202b5e38e8ac3b599c57ea646ee13598 - mirror 1 checksum: 9f1df1e777311fb0f496be2807f4fb10 - mirror 2 checksum: 9f1df1e777311fb0f496be2807f4fb10 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-18T14:07:57-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-18T14:07:57-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 14:08:06 (1787076486) Failing mds1 on oleg405-server Stopping /mnt/lustre-mds1 (opts:) on oleg405-server 14:08:13 (1787076493) shut down facet: mds1 facet_host: oleg405-server facet_failover_host: oleg405-server Failover mds1 to oleg405-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-MDT0000 14:08:44 (1787076524) targets are mounted 14:08:44 (1787076524) facet_failover done oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429451 s, 24.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } PASS 50A (55s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 14:09:02 (1787076542) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 14:09:05 (1787076545) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 14:09:09 (1787076549) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0138598 s, 1.2 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 14:09:17 (1787076557) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 14:09:21 (1787076561) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0246994 s, 166 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0804308 s, 50.9 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 14:09:30 (1787076570) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 14:09:55 (1787076595) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 14:10:24 (1787076624) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 14:10:49 (1787076649) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 14:11:14 (1787076674) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 14:11:22 (1787076682) mirror create pid 54336 mirror split pid 54337 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54336 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54337 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 14:12:31 (1787076751) Seed set to 6753 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x37038 000001 1787076753.370268 trunc from 00000000 to 0x037037 (0x37038 bytes) 000003 1787076753.390786 mapwrite 0x0290db thru 0x037fae (0xeed4 bytes) 000004 1787076753.602283 write 0x00c37a thru 0x00cbad (0x834 bytes) truncating to largest ever: 0x38488 000006 1787076753.616015 trunc from 0x037faf to 0x038487 (0x4d9 bytes) 000008 1787076753.653174 MIRROR_EXTEND to 2 mirrors 000010 1787076754.917246 read 0x0285f3 thru 0x02f369 (0x6d77 bytes) 000012 1787076754.946639 mapread 0x037e73 thru 0x038487 (0x615 bytes) 000013 1787076754.976781 MIRROR_EXTEND to 3 mirrors 000014 1787076756.349362 mapwrite 0x02dbbb thru 0x037557 (0x999d bytes) 000015 1787076756.597911 trunc from 0x038488 to 0x026ca9 (0x117de bytes) 000016 1787076756.653913 mapread 0x022685 thru 0x026ca9 (0x4625 bytes) 000017 1787076756.710402 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26caa): Mirror 3: 0xb2350b6f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26caa): Mirror 3: 0xb2350b6f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26caa: skipped 000018 1787076757.123282 mapwrite 0x011685 thru 0x013142 (0x1abe bytes) 000019 1787076757.201547 write 0x00c810 thru 0x00e39b (0x1b8c bytes) 000020 1787076757.208416 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) 000022 1787076757.376924 mapread 0x024509 thru 0x026ca9 (0x27a1 bytes) 000023 1787076757.406430 write 0x0241f1 thru 0x028f0a (0x4d1a bytes) 000025 1787076757.410253 write 0x021be6 thru 0x031345 (0xf760 bytes) 000026 1787076757.413539 write 0x010993 thru 0x01da67 (0xd0d5 bytes) 000029 1787076757.439563 read 0x02db61 thru 0x031345 (0x37e5 bytes) 000031 1787076757.443732 trunc from 0x031346 to 0x018479 (0x18ecc bytes) 000033 1787076757.474885 mapwrite 0x03c762 thru 0x03ffff (0x389e bytes) 000036 1787076757.600946 trunc from 0x040000 to 0x003680 (0x3c97f bytes) 000038 1787076757.641621 read 0x002ea0 thru 0x003680 (0x7e1 bytes) 000039 1787076757.653428 write 0x001234 thru 0x009f9b (0x8d68 bytes) 000040 1787076757.657855 write 0x0044eb thru 0x00c504 (0x801a bytes) 000042 1787076757.667858 write 0x00374d thru 0x00c0b3 (0x8967 bytes) 000043 1787076757.682707 MIRROR_EXTEND to 2 mirrors 000044 1787076758.479742 read 0x008edf thru 0x00c504 (0x3626 bytes) 000048 1787076758.493303 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) 000049 1787076758.684536 write 0x017804 thru 0x01d1b0 (0x59ad bytes) 000050 1787076758.700904 write 0x0115ab thru 0x012cf0 (0x1746 bytes) 000051 1787076758.715869 trunc from 0x01d1b1 to 0x00ace7 (0x124c9 bytes) 000054 1787076758.744308 mapwrite 0x00282c thru 0x010c17 (0xe3ec bytes) 000055 1787076758.936005 mapwrite 0x0338e6 thru 0x03ffff (0xc71a bytes) 000056 1787076759.085403 mapread 0x0376d7 thru 0x03ffff (0x8929 bytes) 000057 1787076759.095050 MIRROR_EXTEND to 2 mirrors 000058 1787076760.048137 mapread 0x038066 thru 0x03b137 (0x30d2 bytes) 000060 1787076760.093060 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) 000061 1787076760.311476 mapwrite 0x0369b1 thru 0x039794 (0x2de4 bytes) 000062 1787076760.497474 read 0x03f6d6 thru 0x03ffff (0x92a bytes) 000063 1787076760.498993 mapread 0x024af8 thru 0x0258f5 (0xdfe bytes) 000066 1787076760.517281 trunc from 0x040000 to 0x03591b (0xa6e4 bytes) truncating to largest ever: 0x398ba 000067 1787076760.578456 trunc from 0x03591c to 0x0398b9 (0x3f9e bytes) 000068 1787076760.606014 read 0x00f92e thru 0x011023 (0x16f6 bytes) 000072 1787076760.629865 read 0x0028e5 thru 0x00d022 (0xa73e bytes) 000074 1787076760.632920 write 0x02ec9b thru 0x038040 (0x93a6 bytes) 000075 1787076760.634927 mapwrite 0x027b62 thru 0x02c058 (0x44f7 bytes) 000077 1787076760.697488 mapwrite 0x02b537 thru 0x02e866 (0x3330 bytes) 000078 1787076760.757109 mapwrite 0x033b64 thru 0x03678a (0x2c27 bytes) 000079 1787076760.797110 mapread 0x0076fc thru 0x013084 (0xb989 bytes) 000081 1787076760.800335 read 0x03817b thru 0x0398b9 (0x173f bytes) 000086 1787076760.805937 mapwrite 0x026da7 thru 0x02cf3a (0x6194 bytes) 000087 1787076760.853564 mapread 0x0350d1 thru 0x0398b9 (0x47e9 bytes) 000088 1787076760.856677 read 0x028608 thru 0x02f32b (0x6d24 bytes) 000089 1787076760.857451 read 0x016a4a thru 0x024202 (0xd7b9 bytes) 000090 1787076760.861631 mapwrite 0x00c81f thru 0x0189d1 (0xc1b3 bytes) 000092 1787076760.914799 mapread 0x00a463 thru 0x010d43 (0x68e1 bytes) 000094 1787076760.918132 read 0x01c0b5 thru 0x024d3b (0x8c87 bytes) 000095 1787076760.919145 mapread 0x02add3 thru 0x0362ff (0xb52d bytes) 000096 1787076760.921440 write 0x01ace8 thru 0x02540a (0xa723 bytes) 000097 1787076760.923115 mapwrite 0x000146 thru 0x007368 (0x7223 bytes) 000098 1787076760.969993 mapwrite 0x027993 thru 0x02e490 (0x6afe bytes) 000099 1787076761.015167 mapwrite 0x00c067 thru 0x00c400 (0x39a bytes) 000101 1787076761.061104 write 0x01802f thru 0x01fea7 (0x7e79 bytes) 000104 1787076761.064279 write 0x031faa thru 0x0383e8 (0x643f bytes) 000105 1787076761.067138 mapwrite 0x00d8fa thru 0x00f031 (0x1738 bytes) 000106 1787076761.132010 write 0x02f1fc thru 0x02f78d (0x592 bytes) 000109 1787076761.134732 write 0x01082b thru 0x0110df (0x8b5 bytes) 000110 1787076761.137260 write 0x001438 thru 0x009f2f (0x8af8 bytes) 000111 1787076761.139466 read 0x032caa thru 0x0398b9 (0x6c10 bytes) 000112 1787076761.148564 MIRROR_EXTEND to 2 mirrors 000113 1787076761.969729 read 0x01d5f7 thru 0x02bd9a (0xe7a4 bytes) 000116 1787076761.994886 MIRROR_RESYNC 2 mirrors truncating to largest ever: 0x3de18 000117 1787076762.500603 trunc from 0x0398ba to 0x03de17 (0x455e bytes) 000118 1787076762.642384 mapwrite 0x007642 thru 0x0131a6 (0xbb65 bytes) 000119 1787076762.725702 MIRROR_RESYNC 2 mirrors 000120 1787076763.617045 mapread 0x010416 thru 0x01e4b0 (0xe09b bytes) 000121 1787076763.648697 trunc from 0x03de18 to 0x02f332 (0xeae5 bytes) 000122 1787076763.745648 mapread 0x01bf85 thru 0x02a157 (0xe1d3 bytes) 000123 1787076763.780426 read 0x000ede thru 0x00ac4b (0x9d6e bytes) 000126 1787076763.793340 write 0x020526 thru 0x02cb06 (0xc5e1 bytes) 000127 1787076763.795063 mapread 0x025d9a thru 0x02f332 (0x9599 bytes) 000128 1787076763.806787 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) 000130 1787076764.019538 write 0x028c42 thru 0x032634 (0x99f3 bytes) 000131 1787076764.032953 trunc from 0x032635 to 0x0101e9 (0x2244b bytes) 000132 1787076764.061680 write 0x007f78 thru 0x0109c7 (0x8a50 bytes) 000133 1787076764.073923 write 0x0304f9 thru 0x03cd17 (0xc81f bytes) 000135 1787076764.078868 write 0x0212b6 thru 0x02e376 (0xd0c1 bytes) 000136 1787076764.089620 trunc from 0x03cd18 to 0x03504d (0x7cca bytes) 000137 1787076764.107000 write 0x01599f thru 0x017e5d (0x24bf bytes) 000140 1787076764.118735 mapwrite 0x039659 thru 0x039971 (0x319 bytes) 000141 1787076764.189681 mapwrite 0x0243f2 thru 0x027019 (0x2c28 bytes) 000142 1787076764.238832 mapread 0x01f78e thru 0x02b688 (0xbefb bytes) 000144 1787076764.255172 read 0x02b482 thru 0x0312fa (0x5e79 bytes) 000145 1787076764.261213 write 0x000c02 thru 0x00f20a (0xe609 bytes) 000148 1787076764.272918 read 0x027660 thru 0x02f967 (0x8308 bytes) 000149 1787076764.273763 mapread 0x007910 thru 0x0135af (0xbca0 bytes) 000151 1787076764.285879 read 0x02822a thru 0x037de1 (0xfbb8 bytes) 000153 1787076764.289041 trunc from 0x039972 to 0x01e742 (0x1b22f bytes) 000155 1787076764.311948 trunc from 0x01e743 to 0x028800 (0xa0be bytes) 000156 1787076764.329314 mapwrite 0x022afb thru 0x02b73a (0x8c40 bytes) 000157 1787076764.406542 read 0x012ae8 thru 0x0201d6 (0xd6ef bytes) 000158 1787076764.407397 mapread 0x0056a7 thru 0x0095ed (0x3f47 bytes) 000159 1787076764.409433 read 0x02a9f5 thru 0x02b73a (0xd46 bytes) 000160 1787076764.410537 mapwrite 0x0027c3 thru 0x00cb53 (0xa391 bytes) 000161 1787076764.454396 write 0x03797c thru 0x03cfbd (0x5642 bytes) 000164 1787076764.457775 read 0x01a7fd thru 0x0268a8 (0xc0ac bytes) 000166 1787076764.459162 read 0x019b48 thru 0x020b82 (0x703b bytes) 000168 1787076764.461497 read 0x005570 thru 0x011749 (0xc1da bytes) 000171 1787076764.464033 mapread 0x035ea7 thru 0x03962c (0x3786 bytes) 000173 1787076764.483856 trunc from 0x03cfbe to 0x00a7f3 (0x327ca bytes) 000174 1787076764.504570 trunc from 0x00a7f4 to 0x03caf3 (0x32300 bytes) 000176 1787076764.523502 mapread 0x03a6cb thru 0x03caf3 (0x2429 bytes) 000178 1787076764.556179 write 0x0195ed thru 0x01c521 (0x2f35 bytes) 000180 1787076764.568464 write 0x000207 thru 0x0003f7 (0x1f1 bytes) 000182 1787076764.570356 read 0x0074da thru 0x014f13 (0xda3a bytes) 000184 1787076764.590865 write 0x01b049 thru 0x02a88c (0xf844 bytes) 000186 1787076764.610945 MIRROR_EXTEND to 2 mirrors 000187 1787076765.827834 mapread 0x025e5e thru 0x028917 (0x2aba bytes) 000188 1787076765.874969 trunc from 0x03caf4 to 0x02a595 (0x1255e bytes) 000190 1787076766.037049 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2a596): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a596): Mirror 2: 0x2aee5e90 (stale) CRC-32 checksum value for chunk [0, 0x2a596): Mirror 1: 0x2aee5e90 Mirror 3: 0x2aee5e90 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a596: skipped 000192 1787076767.395928 trunc from 0x02a596 to 0x0387fc (0xe267 bytes) 000193 1787076767.502585 read 0x01150c thru 0x012694 (0x1189 bytes) 000195 1787076767.511252 write 0x021b79 thru 0x026ca7 (0x512f bytes) 000200 1787076767.546864 read 0x01dba1 thru 0x01f85a (0x1cba bytes) 000202 1787076767.586096 mapread 0x02bfb4 thru 0x032b2f (0x6b7c bytes) 000204 1787076767.588582 trunc from 0x0387fd to 0x02d319 (0xb4e3 bytes) 000205 1787076767.612413 trunc from 0x02d31a to 0x022576 (0xada3 bytes) 000206 1787076767.635552 trunc from 0x022577 to 0x00bf1d (0x16659 bytes) 000207 1787076767.677477 trunc from 0x00bf1e to 0x00c09f (0x182 bytes) 000208 1787076767.703912 trunc from 0x00c0a0 to 0x00cd2e (0xc8f bytes) 000209 1787076767.726894 write 0x020a47 thru 0x022444 (0x19fe bytes) 000210 1787076767.728720 mapwrite 0x01ec76 thru 0x021c6c (0x2ff7 bytes) 000214 1787076767.820327 mapwrite 0x03f712 thru 0x03ffff (0x8ee bytes) 000215 1787076767.982668 write 0x03e400 thru 0x03fbf7 (0x17f8 bytes) 000216 1787076767.987513 MIRROR_RESYNC 3 mirrors 000218 1787076769.294668 mapwrite 0x008e6e thru 0x010a83 (0x7c16 bytes) 000220 1787076769.547225 write 0x02da2f thru 0x03124a (0x381c bytes) 000224 1787076769.567207 mapread 0x013846 thru 0x0156cc (0x1e87 bytes) 000228 1787076769.590184 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xc8aa36a1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xc8aa36a1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000229 1787076769.963984 mapread 0x03a7f6 thru 0x03ffff (0x580a bytes) 000231 1787076769.990569 write 0x03169c thru 0x03ffff (0xe964 bytes) 000232 1787076770.008476 trunc from 0x040000 to 0x01edff (0x21200 bytes) 000233 1787076770.054772 trunc from 0x01ee00 to 0x03ce58 (0x1e059 bytes) 000235 1787076770.106120 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ce59): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ce59): Mirror 3: 0x612e3b5c (stale) CRC-32 checksum value for chunk [0, 0x3ce59): Mirror 1: 0xb6c1d496 Mirror 4: 0xb6c1d496 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ce59: skipped 000236 1787076771.393082 read 0x0153a2 thru 0x01ab5f (0x57be bytes) 000237 1787076771.413581 read 0x024c5e thru 0x02e4d3 (0x9876 bytes) 000239 1787076771.449659 write 0x017dee thru 0x019c1e (0x1e31 bytes) 000241 1787076771.563571 mapwrite 0x005bc7 thru 0x0066ce (0xb08 bytes) 000244 1787076771.627274 mapread 0x0300fc thru 0x03a84f (0xa754 bytes) 000249 1787076771.694230 mapread 0x0071cd thru 0x009707 (0x253b bytes) 000250 1787076771.716283 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ce59): Mirror 4: 0xb6c1d496 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ce59): Mirror 4: 0xb6c1d496 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ce59: skipped 000251 1787076772.069804 mapread 0x01281c thru 0x013d89 (0x156e bytes) 000252 1787076772.102322 write 0x03159f thru 0x03a28c (0x8cee bytes) 000253 1787076772.134943 mapwrite 0x01960e thru 0x01af8f (0x1982 bytes) 000257 1787076772.253703 mapwrite 0x003004 thru 0x0116f7 (0xe6f4 bytes) 000259 1787076772.320627 trunc from 0x03ce59 to 0x01a129 (0x22d2f bytes) 000262 1787076772.361554 trunc from 0x01a12a to 0x02d363 (0x1323a bytes) 000263 1787076772.426838 mapread 0x0034ca thru 0x0065f5 (0x312c bytes) 000266 1787076772.443309 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2d364): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d364): Mirror 4: 0x62a7765b (stale) CRC-32 checksum value for chunk [0, 0x2d364): Mirror 1: 0x44f24ee2 Mirror 5: 0x44f24ee2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d364: skipped 000268 1787076773.848497 mapwrite 0x0183fa thru 0x0236f0 (0xb2f7 bytes) 000269 1787076774.189182 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d364): Mirror 4: 0x62a7765b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d364): Mirror 4: 0x62a7765b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d364: skipped 000272 1787076774.602374 read 0x01c545 thru 0x02454c (0x8008 bytes) 000273 1787076774.629455 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2d364): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d364): Mirror 4: 0x62a7765b (stale) CRC-32 checksum value for chunk [0, 0x2d364): Mirror 1: 0xb51d44a7 Mirror 5: 0xb51d44a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d364: skipped 000274 1787076775.770632 mapwrite 0x001226 thru 0x010345 (0xf120 bytes) 000276 1787076776.163569 mapwrite 0x009c9f thru 0x016d6c (0xd0ce bytes) 000277 1787076776.233960 mapread 0x018bbd thru 0x0215f0 (0x8a34 bytes) 000278 1787076776.272387 MIRROR_RESYNC 3 mirrors 000282 1787076777.410847 trunc from 0x02d364 to 0x031cfb (0x4998 bytes) 000283 1787076777.483826 trunc from 0x031cfc to 0x019b12 (0x181e9 bytes) 000285 1787076777.507513 mapwrite 0x00848e thru 0x00c75c (0x42cf bytes) 000287 1787076777.592513 MIRROR_RESYNC 3 mirrors 000288 1787076778.465880 MIRROR_EXTEND to 4 mirrors 000290 1787076779.335513 write 0x01fab4 thru 0x020a7a (0xfc7 bytes) 000291 1787076779.450889 mapread 0x005f36 thru 0x012183 (0xc24e bytes) 000293 1787076779.510237 mapread 0x00b33b thru 0x00eb3b (0x3801 bytes) 000294 1787076779.519302 mapwrite 0x016c24 thru 0x01b5ee (0x49cb bytes) 000295 1787076779.611290 mapwrite 0x02d750 thru 0x034107 (0x69b8 bytes) 000297 1787076779.747201 read 0x001c5b thru 0x006887 (0x4c2d bytes) 000298 1787076779.751283 read 0x003907 thru 0x013814 (0xff0e bytes) 000299 1787076779.752370 mapwrite 0x009bba thru 0x019287 (0xf6ce bytes) 000302 1787076779.849514 trunc from 0x034108 to 0x0349e6 (0x8df bytes) 000303 1787076779.884242 MIRROR_RESYNC 4 mirrors 000304 1787076780.842055 write 0x00c871 thru 0x011a9f (0x522f bytes) 000305 1787076780.916485 read 0x011c42 thru 0x01aefa (0x92b9 bytes) 000307 1787076780.932943 read 0x02e262 thru 0x031677 (0x3416 bytes) 000309 1787076780.951055 mapread 0x02ee50 thru 0x0349e6 (0x5b97 bytes) 000311 1787076780.954512 mapwrite 0x00da6b thru 0x011be1 (0x4177 bytes) 000313 1787076781.012441 write 0x016ae1 thru 0x02246b (0xb98b bytes) 000315 1787076781.015879 write 0x02d663 thru 0x037e10 (0xa7ae bytes) 000316 1787076781.017991 mapwrite 0x016f1d thru 0x01c5c4 (0x56a8 bytes) 000319 1787076781.087787 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x37e11): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37e11): Mirror 4: 0x78a67092 (stale) Mirror 5: 0x78a67092 (stale) Mirror 6: 0x78a67092 (stale) CRC-32 checksum value for chunk [0, 0x37e11): Mirror 1: 0xc570b9fa Mirror 7: 0xc570b9fa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37e11: skipped 000321 1787076782.025091 mapwrite 0x004d0e thru 0x00bfe5 (0x72d8 bytes) 000322 1787076782.325400 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37e11): Mirror 4: 0x78a67092 (stale) Mirror 6: 0x78a67092 (stale) Mirror 7: 0xc570b9fa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37e11): Mirror 4: 0x78a67092 (stale) Mirror 6: 0x78a67092 (stale) Mirror 7: 0xc570b9fa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37e11: skipped 000323 1787076782.785147 trunc from 0x037e11 to 0x013148 (0x24cc8 bytes) 000324 1787076782.842494 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x13149): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13149): Mirror 4: 0xfe751ad8 (stale) Mirror 6: 0xfe751ad8 (stale) Mirror 7: 0x92527274 (stale) CRC-32 checksum value for chunk [0, 0x13149): Mirror 1: 0x49c15067 Mirror 8: 0x49c15067 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13149: skipped 000325 1787076784.008318 mapwrite 0x00d87c thru 0x01c9c5 (0xf14a bytes) 000326 1787076784.185202 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1c9c6): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c9c6): Mirror 4: 0xf778bb63 (stale) Mirror 6: 0xf778bb63 (stale) Mirror 7: 0xe5b3da72 (stale) Mirror 8: 0x1580b162 (stale) CRC-32 checksum value for chunk [0, 0x1c9c6): Mirror 1: 0x66e44542 Mirror 9: 0x66e44542 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c9c6: skipped 000327 1787076785.209292 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c9c6): Mirror 4: 0xf778bb63 (stale) Mirror 7: 0xe5b3da72 (stale) Mirror 8: 0x1580b162 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c9c6): Mirror 4: 0xf778bb63 (stale) Mirror 7: 0xe5b3da72 (stale) Mirror 8: 0x1580b162 (stale) CRC-32 checksum value for chunk [0, 0x1c9c6): Mirror 1: 0x66e44542 Mirror 9: 0x66e44542 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c9c6: skipped 000328 1787076785.901229 trunc from 0x01c9c6 to 0x0086a8 (0x1431d bytes) 000329 1787076786.029761 trunc from 0x0086a9 to 0x027a28 (0x1f380 bytes) 000330 1787076786.061451 trunc from 0x027a29 to 0x01a94f (0xd0d9 bytes) 000331 1787076786.098822 write 0x02a1a3 thru 0x02a22b (0x89 bytes) 000332 1787076786.101408 mapread 0x021250 thru 0x027e58 (0x6c09 bytes) 000333 1787076786.126955 trunc from 0x02a22c to 0x022114 (0x8117 bytes) 000334 1787076786.149969 write 0x03d2ab thru 0x03ffff (0x2d55 bytes) 000335 1787076786.155580 read 0x02b426 thru 0x033370 (0x7f4b bytes) 000336 1787076786.167819 mapwrite 0x0182ef thru 0x027faa (0xfcbc bytes) 000337 1787076786.287107 mapwrite 0x015472 thru 0x01b311 (0x5ea0 bytes) 000339 1787076786.375948 trunc from 0x040000 to 0x017258 (0x28da7 bytes) 000340 1787076786.404294 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x17259): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17259): Mirror 4: 0x3fcf9d86 (stale) Mirror 7: 0xf78bd2af (stale) Mirror 8: 0x4ad8daa6 (stale) Mirror 9: 0x6f8a1c24 (stale) CRC-32 checksum value for chunk [0, 0x17259): Mirror 1: 0x969bf8c2 Mirror 10: 0x969bf8c2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17259: skipped 000341 1787076787.412103 mapread 0x012462 thru 0x017258 (0x4df7 bytes) 000342 1787076787.443945 trunc from 0x017259 to 0x026e05 (0xfbad bytes) 000343 1787076787.579810 read 0x00a5e2 thru 0x010b34 (0x6553 bytes) 000345 1787076787.595403 MIRROR_RESYNC 6 mirrors 000347 1787076789.356211 mapwrite 0x02965c thru 0x02b81a (0x21bf bytes) 000348 1787076789.600263 mapwrite 0x01f322 thru 0x021d0c (0x29eb bytes) 000349 1787076789.679860 trunc from 0x02b81b to 0x03b847 (0x1002d bytes) 000352 1787076789.726788 read 0x00ccd3 thru 0x010d20 (0x404e bytes) 000353 1787076789.741372 write 0x010143 thru 0x016260 (0x611e bytes) 000354 1787076789.755370 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3b848): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b848): Mirror 4: 0x8a3028f2 (stale) Mirror 7: 0x8a3028f2 (stale) Mirror 8: 0x8a3028f2 (stale) Mirror 9: 0x8a3028f2 (stale) Mirror 10: 0x8a3028f2 (stale) CRC-32 checksum value for chunk [0, 0x3b848): Mirror 1: 0x8fc5864d Mirror 11: 0x8fc5864d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b848: skipped 000356 1787076791.322480 write 0x000231 thru 0x006ac1 (0x6891 bytes) 000359 1787076791.431160 read 0x01f743 thru 0x029533 (0x9df1 bytes) 000360 1787076791.445151 mapread 0x0288c3 thru 0x0327d6 (0x9f14 bytes) 000361 1787076791.467736 mapwrite 0x01f1f1 thru 0x01f7ea (0x5fa bytes) 000364 1787076791.509811 read 0x037471 thru 0x03b847 (0x43d7 bytes) 000367 1787076791.535865 trunc from 0x03b848 to 0x00514c (0x366fb bytes) 000368 1787076791.563046 read 0x004f28 thru 0x00514c (0x225 bytes) 000369 1787076791.565386 mapread 0x000768 thru 0x00514c (0x49e5 bytes) 000371 1787076791.567799 mapread 0x004666 thru 0x00514c (0xae7 bytes) 000372 1787076791.568899 trunc from 0x00514d to 0x011d01 (0xcbb5 bytes) 000373 1787076791.598164 mapwrite 0x0098a0 thru 0x00fe7d (0x65de bytes) 000375 1787076791.680069 write 0x00d99d thru 0x014187 (0x67eb bytes) 000378 1787076791.684277 mapread 0x00c0d0 thru 0x014187 (0x80b8 bytes) 000379 1787076791.692402 MIRROR_RESYNC 7 mirrors 000381 1787076793.111313 MIRROR_EXTEND to 8 mirrors 000384 1787076794.014956 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) 000385 1787076794.677448 mapwrite 0x02f93a thru 0x03f1f2 (0xf8b9 bytes) 000386 1787076794.865077 read 0x003cfd thru 0x00c5ff (0x8903 bytes) 000387 1787076794.880695 write 0x0346e5 thru 0x038ada (0x43f6 bytes) 000388 1787076794.882220 read 0x0076b8 thru 0x013ae1 (0xc42a bytes) 000389 1787076794.894766 mapwrite 0x034912 thru 0x03ffff (0xb6ee bytes) 000391 1787076795.007828 mapwrite 0x01de65 thru 0x026036 (0x81d2 bytes) 000393 1787076795.125573 trunc from 0x040000 to 0x0128bd (0x2d742 bytes) 000396 1787076795.185933 write 0x03abc9 thru 0x03ffff (0x5437 bytes) 000398 1787076795.194864 trunc from 0x040000 to 0x00a53a (0x35ac5 bytes) 000401 1787076795.250797 mapwrite 0x02a062 thru 0x03224e (0x81ed bytes) 000405 1787076795.526480 mapread 0x027bfe thru 0x03224e (0xa651 bytes) 000406 1787076795.574005 mapread 0x02baf1 thru 0x03224e (0x675e bytes) 000407 1787076795.579940 mapread 0x003154 thru 0x00b4d0 (0x837d bytes) 000408 1787076795.601271 mapwrite 0x018aaf thru 0x01cc9f (0x41f1 bytes) 000410 1787076795.771642 trunc from 0x03224f to 0x00b04c (0x27202 bytes) 000411 1787076795.810281 mapread 0x005550 thru 0x00aaa0 (0x5551 bytes) 000412 1787076795.812936 write 0x034a11 thru 0x03d91e (0x8f0e bytes) 000413 1787076795.822040 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3d91f): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d91f): Mirror 4: 0xadaacc8a (stale) Mirror 7: 0xadaacc8a (stale) Mirror 9: 0xadaacc8a (stale) Mirror 10: 0xadaacc8a (stale) Mirror 11: 0xadaacc8a (stale) Mirror 12: 0xadaacc8a (stale) CRC-32 checksum value for chunk [0, 0x3d91f): Mirror 1: 0x98a3e15a Mirror 13: 0x98a3e15a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d91f: skipped 000414 1787076797.282739 mapwrite 0x01fbda thru 0x02d2cc (0xd6f3 bytes) 000415 1787076797.746810 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3d91f): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d91f): Mirror 4: 0xadaacc8a (stale) Mirror 7: 0xadaacc8a (stale) Mirror 9: 0xadaacc8a (stale) Mirror 10: 0xadaacc8a (stale) Mirror 11: 0xadaacc8a (stale) Mirror 12: 0xadaacc8a (stale) Mirror 13: 0x98a3e15a (stale) CRC-32 checksum value for chunk [0, 0x3d91f): Mirror 1: 0x56c315b5 Mirror 14: 0x56c315b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d91f: skipped 000416 1787076799.038029 write 0x011970 thru 0x0199da (0x806b bytes) 000418 1787076799.140080 mapwrite 0x038964 thru 0x03aac2 (0x215f bytes) 000420 1787076799.280499 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3d91f): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d91f): Mirror 4: 0xadaacc8a (stale) Mirror 7: 0xadaacc8a (stale) Mirror 9: 0xadaacc8a (stale) Mirror 10: 0xadaacc8a (stale) Mirror 11: 0xadaacc8a (stale) Mirror 12: 0xadaacc8a (stale) Mirror 13: 0x98a3e15a (stale) Mirror 14: 0x56c315b5 (stale) CRC-32 checksum value for chunk [0, 0x3d91f): Mirror 1: 0x2967c8d4 Mirror 15: 0x2967c8d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d91f: skipped 000421 1787076800.536666 mapread 0x00e6db thru 0x01822e (0x9b54 bytes) 000422 1787076800.579442 mapread 0x01439a thru 0x019ce3 (0x594a bytes) 000424 1787076800.606652 trunc from 0x03d91f to 0x037178 (0x67a6 bytes) 000425 1787076800.792905 write 0x013aa1 thru 0x01ad13 (0x7273 bytes) 000428 1787076800.812085 trunc from 0x037179 to 0x027b0a (0xf66e bytes) 000429 1787076800.843850 write 0x010673 thru 0x019dd6 (0x9764 bytes) 000431 1787076800.856894 trunc from 0x027b0b to 0x036773 (0xec69 bytes) 000433 1787076800.892905 write 0x0235a9 thru 0x024337 (0xd8f bytes) 000434 1787076800.912414 read 0x02671d thru 0x02b36e (0x4c52 bytes) 000436 1787076800.929398 read 0x02a3e9 thru 0x02aae1 (0x6f9 bytes) 000437 1787076800.930066 mapread 0x034aba thru 0x036773 (0x1cba bytes) 000439 1787076800.964588 mapwrite 0x003ea9 thru 0x0103a4 (0xc4fc bytes) 000441 1787076801.071146 read 0x00b9f7 thru 0x019495 (0xda9f bytes) 000442 1787076801.073060 mapread 0x0193c3 thru 0x026eb9 (0xdaf7 bytes) 000443 1787076801.079969 read 0x02d64c thru 0x034c11 (0x75c6 bytes) 000445 1787076801.081172 mapread 0x01dcb7 thru 0x0263ca (0x8714 bytes) 000447 1787076801.094811 mapwrite 0x020cc3 thru 0x021742 (0xa80 bytes) 000448 1787076801.227997 mapread 0x018778 thru 0x0225ca (0x9e53 bytes) 000449 1787076801.229277 read 0x00e8c4 thru 0x01b267 (0xc9a4 bytes) 000450 1787076801.231792 trunc from 0x036774 to 0x0175ab (0x1f1c8 bytes) 000451 1787076801.265685 MIRROR_RESYNC 10 mirrors 000452 1787076803.725300 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) 000453 1787076804.386194 mapread 0x015d93 thru 0x0175ab (0x1819 bytes) 000458 1787076804.418134 mapwrite 0x00f840 thru 0x01d51b (0xdcdc bytes) 000460 1787076804.635208 mapread 0x00431a thru 0x00f119 (0xae00 bytes) 000461 1787076804.657700 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1d51c): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d51c): Mirror 4: 0x4ccf931b (stale) Mirror 7: 0x4ccf931b (stale) Mirror 9: 0x4ccf931b (stale) Mirror 10: 0x4ccf931b (stale) Mirror 12: 0x4ccf931b (stale) Mirror 13: 0x4ccf931b (stale) Mirror 14: 0x4ccf931b (stale) Mirror 15: 0x4ccf931b (stale) CRC-32 checksum value for chunk [0, 0x1d51c): Mirror 1: 0xc77d4def Mirror 16: 0xc77d4def lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d51c: skipped 000463 1787076806.212916 write 0x034edd thru 0x03d8c9 (0x89ed bytes) 000465 1787076806.337030 mapwrite 0x029de5 thru 0x02a12d (0x349 bytes) 000467 1787076806.452212 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3d8ca): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d8ca): Mirror 4: 0xba28340 (stale) Mirror 7: 0xba28340 (stale) Mirror 9: 0xba28340 (stale) Mirror 10: 0xba28340 (stale) Mirror 12: 0xba28340 (stale) Mirror 13: 0xba28340 (stale) Mirror 14: 0xba28340 (stale) Mirror 15: 0xba28340 (stale) Mirror 16: 0x2965641a (stale) CRC-32 checksum value for chunk [0, 0x3d8ca): Mirror 1: 0x19aea3de Mirror 17: 0x19aea3de lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d8ca: skipped 000468 1787076807.503536 mapread 0x0318f6 thru 0x03b081 (0x978c bytes) 000473 1787076807.592166 trunc from 0x03d8ca to 0x004973 (0x38f56 bytes) 000474 1787076807.758402 mapwrite 0x034603 thru 0x03ffff (0xb9fd bytes) 000476 1787076807.902112 mapread 0x03bb24 thru 0x03ffff (0x44dc bytes) 000478 1787076807.911959 MIRROR_RESYNC 11 mirrors 000479 1787076811.405046 trunc from 0x040000 to 0x01f0b5 (0x20f4a bytes) 000480 1787076811.511753 read 0x003b6f thru 0x006060 (0x24f2 bytes) 000481 1787076811.524731 read 0x010b11 thru 0x017878 (0x6d68 bytes) 000482 1787076811.538918 read 0x014cf8 thru 0x01f0b5 (0xa3be bytes) 000483 1787076811.540193 mapwrite 0x0225fb thru 0x0292dd (0x6ce3 bytes) 000485 1787076811.655889 write 0x031e92 thru 0x03f5e6 (0xd755 bytes) 000486 1787076811.659704 write 0x002cbe thru 0x00e8a9 (0xbbec bytes) 000489 1787076811.664045 write 0x01a4a2 thru 0x01e8ff (0x445e bytes) 000490 1787076811.665869 read 0x03a26b thru 0x03f5e6 (0x537c bytes) 000492 1787076811.668057 read 0x00e0ca thru 0x00f4b5 (0x13ec bytes) 000493 1787076811.669019 trunc from 0x03f5e7 to 0x02100c (0x1e5da bytes) 000498 1787076811.694076 read 0x019f7b thru 0x020db9 (0x6e3f bytes) 000499 1787076811.701042 MIRROR_RESYNC 11 mirrors 000502 1787076813.799119 mapwrite 0x01fd7b thru 0x02f51c (0xf7a2 bytes) 000503 1787076814.032643 mapwrite 0x00047c thru 0x00cfa5 (0xcb2a bytes) 000504 1787076814.145325 MIRROR_RESYNC 11 mirrors 000506 1787076816.674926 MIRROR_EXTEND to 12 mirrors 000507 1787076817.954847 MIRROR_SPLIT mirror 9 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) 000509 1787076818.464838 mapwrite 0x02f729 thru 0x033553 (0x3e2b bytes) 000513 1787076818.743522 write 0x003380 thru 0x010e41 (0xdac2 bytes) 000517 1787076818.757573 write 0x02e491 thru 0x03bff5 (0xdb65 bytes) 000518 1787076818.760549 trunc from 0x03bff6 to 0x03d854 (0x185f bytes) 000519 1787076818.780903 write 0x0302fe thru 0x03cd62 (0xca65 bytes) 000523 1787076818.801915 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d855): Mirror 4: 0xca48434a (stale) Mirror 7: 0xca48434a (stale) Mirror 9: 0xca48434a (stale) Mirror 10: 0xca48434a (stale) Mirror 12: 0xca48434a (stale) Mirror 13: 0xca48434a (stale) Mirror 14: 0xca48434a (stale) Mirror 17: 0xca48434a (stale) Mirror 18: 0xca48434a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d855): Mirror 4: 0xca48434a (stale) Mirror 7: 0xca48434a (stale) Mirror 9: 0xca48434a (stale) Mirror 10: 0xca48434a (stale) Mirror 12: 0xca48434a (stale) Mirror 13: 0xca48434a (stale) Mirror 14: 0xca48434a (stale) Mirror 17: 0xca48434a (stale) Mirror 18: 0xca48434a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d855: skipped 000526 1787076819.770997 mapread 0x019231 thru 0x02386d (0xa63d bytes) 000528 1787076819.818777 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3d855): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d855): Mirror 4: 0xca48434a (stale) Mirror 7: 0xca48434a (stale) Mirror 9: 0xca48434a (stale) Mirror 10: 0xca48434a (stale) Mirror 12: 0xca48434a (stale) Mirror 13: 0xca48434a (stale) Mirror 14: 0xca48434a (stale) Mirror 17: 0xca48434a (stale) Mirror 18: 0xca48434a (stale) CRC-32 checksum value for chunk [0, 0x3d855): Mirror 1: 0xb0d6452c Mirror 19: 0xb0d6452c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d855: skipped 000531 1787076821.441811 trunc from 0x03d855 to 0x01c839 (0x2101b bytes) 000534 1787076821.599522 read 0x0183c3 thru 0x01c839 (0x4477 bytes) 000535 1787076821.613115 mapwrite 0x030e82 thru 0x03fdf8 (0xef77 bytes) 000538 1787076821.816799 write 0x03f464 thru 0x03ffff (0xb9c bytes) 000541 1787076821.827052 trunc from 0x040000 to 0x027ad6 (0x18529 bytes) 000548 1787076821.870021 mapwrite 0x00f27d thru 0x01c6b3 (0xd437 bytes) 000549 1787076821.952874 mapwrite 0x00e3e9 thru 0x01894e (0xa566 bytes) 000550 1787076822.010717 read 0x017074 thru 0x024eb3 (0xde40 bytes) 000552 1787076822.026685 read 0x003ad8 thru 0x0040d9 (0x602 bytes) 000553 1787076822.027500 read 0x00989f thru 0x00ddda (0x453c bytes) 000557 1787076822.042767 MIRROR_RESYNC 11 mirrors 000558 1787076824.854553 write 0x00c9c2 thru 0x01861c (0xbc5b bytes) 000559 1787076824.956314 trunc from 0x027ad7 to 0x013782 (0x14354 bytes) 000560 1787076825.004158 mapread 0x011c9f thru 0x013782 (0x1ae4 bytes) 000561 1787076825.005447 mapwrite 0x001818 thru 0x0079c1 (0x61aa bytes) 000563 1787076825.196443 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x13783): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13783): Mirror 4: 0x7ed568ad (stale) Mirror 7: 0x7ed568ad (stale) Mirror 9: 0x7ed568ad (stale) Mirror 10: 0x7ed568ad (stale) Mirror 12: 0x7ed568ad (stale) Mirror 13: 0x7ed568ad (stale) Mirror 14: 0x7ed568ad (stale) Mirror 17: 0x7ed568ad (stale) Mirror 18: 0x7ed568ad (stale) Mirror 19: 0x7ed568ad (stale) CRC-32 checksum value for chunk [0, 0x13783): Mirror 1: 0xba6aead Mirror 20: 0xba6aead lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13783: skipped 000564 1787076826.619963 read 0x013379 thru 0x013782 (0x40a bytes) 000568 1787076826.650445 write 0x00b400 thru 0x0147c9 (0x93ca bytes) 000569 1787076826.763629 trunc from 0x0147ca to 0x02131b (0xcb52 bytes) 000571 1787076826.832422 mapread 0x0132d9 thru 0x021095 (0xddbd bytes) 000573 1787076826.863127 trunc from 0x02131c to 0x0311a8 (0xfe8d bytes) 000574 1787076826.882508 write 0x0265f9 thru 0x027eab (0x18b3 bytes) 000575 1787076826.903667 write 0x0232fe thru 0x029e0a (0x6b0d bytes) 000576 1787076826.918075 read 0x000bc9 thru 0x009c04 (0x903c bytes) 000577 1787076826.928126 read 0x002d55 thru 0x011812 (0xeabe bytes) 000578 1787076826.928937 mapwrite 0x02c0d4 thru 0x0387e2 (0xc70f bytes) 000579 1787076827.102184 write 0x00c411 thru 0x014303 (0x7ef3 bytes) 000582 1787076827.126712 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x387e3): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x387e3): Mirror 4: 0x9d61f584 (stale) Mirror 7: 0x9d61f584 (stale) Mirror 9: 0x9d61f584 (stale) Mirror 10: 0x9d61f584 (stale) Mirror 12: 0x9d61f584 (stale) Mirror 13: 0x9d61f584 (stale) Mirror 14: 0x9d61f584 (stale) Mirror 17: 0x9d61f584 (stale) Mirror 18: 0x9d61f584 (stale) Mirror 19: 0x9d61f584 (stale) Mirror 20: 0x62cbf2ad (stale) CRC-32 checksum value for chunk [0, 0x387e3): Mirror 1: 0xa6739ef2 Mirror 21: 0xa6739ef2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x387e3: skipped 000586 1787076828.987820 mapread 0x026b0c thru 0x035935 (0xee2a bytes) 000588 1787076829.046341 read 0x0329ed thru 0x037f5c (0x5570 bytes) 000589 1787076829.055169 trunc from 0x0387e3 to 0x0114d9 (0x27309 bytes) 000590 1787076829.181801 read 0x0026cc thru 0x002e2c (0x761 bytes) 000594 1787076829.197832 read 0x010302 thru 0x0114d9 (0x11d8 bytes) 000595 1787076829.199045 mapwrite 0x006f91 thru 0x010775 (0x97e5 bytes) 000597 1787076829.280915 read 0x0054e4 thru 0x00ce77 (0x7994 bytes) 000599 1787076829.285214 mapwrite 0x02350c thru 0x030773 (0xd268 bytes) 000600 1787076829.492866 read 0x012f59 thru 0x018112 (0x51ba bytes) 000602 1787076829.499919 write 0x010301 thru 0x014a34 (0x4734 bytes) 000603 1787076829.501357 trunc from 0x030774 to 0x00446e (0x2c305 bytes) 000604 1787076829.533402 mapwrite 0x015318 thru 0x018c7c (0x3965 bytes) 000605 1787076829.654969 write 0x01a156 thru 0x0222cc (0x8177 bytes) 000610 1787076829.663042 write 0x01f939 thru 0x02a0f0 (0xa7b8 bytes) 000611 1787076829.666115 trunc from 0x02a0f1 to 0x01ade3 (0xf30d bytes) 000614 1787076829.695903 read 0x0159fa thru 0x017f5e (0x2565 bytes) 000615 1787076829.698743 write 0x028d3d thru 0x034154 (0xb418 bytes) 000619 1787076829.707368 trunc from 0x034155 to 0x013ac7 (0x2068d bytes) 000620 1787076829.727964 read 0x00e70f thru 0x013ac7 (0x53b9 bytes) 000621 1787076829.729488 trunc from 0x013ac8 to 0x02078c (0xccc5 bytes) 000622 1787076829.753479 mapread 0x00bc2a thru 0x017f7a (0xc351 bytes) 000623 1787076829.776472 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x2078d): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2078d): Mirror 4: 0xe1c0463d (stale) Mirror 7: 0xe1c0463d (stale) Mirror 9: 0xe1c0463d (stale) Mirror 10: 0xe1c0463d (stale) Mirror 12: 0xe1c0463d (stale) Mirror 13: 0xe1c0463d (stale) Mirror 14: 0xe1c0463d (stale) Mirror 17: 0xe1c0463d (stale) Mirror 18: 0xe1c0463d (stale) Mirror 19: 0xe1c0463d (stale) Mirror 20: 0xe32e5bb9 (stale) Mirror 21: 0x78cda4ae (stale) CRC-32 checksum value for chunk [0, 0x2078d): Mirror 1: 0x1b39e845 Mirror 22: 0x1b39e845 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2078d: skipped 000628 1787076831.700375 mapwrite 0x009ec5 thru 0x0103e8 (0x6524 bytes) 000629 1787076831.973236 read 0x009d9d thru 0x0181fd (0xe461 bytes) 000636 1787076831.998680 trunc from 0x02078d to 0x00874e (0x1803e bytes) 000637 1787076832.064166 MIRROR_RESYNC 14 mirrors 000638 1787076834.430787 mapread 0x005079 thru 0x007a8d (0x2a15 bytes) 000639 1787076834.454842 trunc from 0x00874f to 0x01fe57 (0x17709 bytes) 000641 1787076834.596413 read 0x01bb0d thru 0x01fe57 (0x434b bytes) 000642 1787076834.625415 trunc from 0x01fe58 to 0x005ac3 (0x1a394 bytes) 000644 1787076834.677434 MIRROR_RESYNC 14 mirrors 000646 1787076836.702198 trunc from 0x005ac4 to 0x002539 (0x358a bytes) 000647 1787076836.807808 read 0x001ad7 thru 0x002539 (0xa63 bytes) 000648 1787076836.813883 mapwrite 0x015e4a thru 0x01d87e (0x7a35 bytes) 000649 1787076837.073803 write 0x024453 thru 0x033aee (0xf69c bytes) 000650 1787076837.077545 mapwrite 0x002653 thru 0x0118bf (0xf26d bytes) 000653 1787076837.213675 mapwrite 0x03104d thru 0x03ace6 (0x9c9a bytes) 000654 1787076837.353501 trunc from 0x03ace7 to 0x03cbf6 (0x1f10 bytes) 000655 1787076837.399540 write 0x03d3d2 thru 0x03ffff (0x2c2e bytes) 000656 1787076837.403531 read 0x00a5c0 thru 0x00da1b (0x345c bytes) 000657 1787076837.412054 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x729e70d5 (stale) Mirror 7: 0x729e70d5 (stale) Mirror 9: 0x729e70d5 (stale) Mirror 10: 0x729e70d5 (stale) Mirror 12: 0x729e70d5 (stale) Mirror 13: 0x729e70d5 (stale) Mirror 14: 0x729e70d5 (stale) Mirror 17: 0x729e70d5 (stale) Mirror 18: 0x729e70d5 (stale) Mirror 19: 0x729e70d5 (stale) Mirror 20: 0x729e70d5 (stale) Mirror 21: 0x729e70d5 (stale) Mirror 22: 0x729e70d5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd3167c82 Mirror 23: 0xd3167c82 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000658 1787076838.875128 trunc from 0x040000 to 0x023a39 (0x1c5c6 bytes) 000659 1787076838.971582 mapwrite 0x00f5ee thru 0x018220 (0x8c33 bytes) 000661 1787076839.061874 mapread 0x002b75 thru 0x006eca (0x4356 bytes) 000663 1787076839.076419 trunc from 0x023a3a to 0x001edc (0x21b5d bytes) 000665 1787076839.122967 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x1edd): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1edd): Mirror 4: 0x3590513c (stale) Mirror 7: 0x3590513c (stale) Mirror 9: 0x3590513c (stale) Mirror 10: 0x3590513c (stale) Mirror 12: 0x3590513c (stale) Mirror 13: 0x3590513c (stale) Mirror 14: 0x3590513c (stale) Mirror 17: 0x3590513c (stale) Mirror 18: 0x3590513c (stale) Mirror 19: 0x3590513c (stale) Mirror 20: 0x3590513c (stale) Mirror 21: 0x3590513c (stale) Mirror 22: 0x3590513c (stale) Mirror 23: 0x3590513c (stale) CRC-32 checksum value for chunk [0, 0x1edd): Mirror 1: 0x3590513c Mirror 24: 0x3590513c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1edd: skipped 000667 1787076840.314105 read 0x000083 thru 0x000dc0 (0xd3e bytes) 000668 1787076840.332466 write 0x02cbdc thru 0x038eb2 (0xc2d7 bytes) 000672 1787076840.467077 trunc from 0x038eb3 to 0x00d58e (0x2b924 bytes) 000674 1787076840.505234 trunc from 0x00d58f to 0x000308 (0xd286 bytes) 000675 1787076840.527646 read 0x00008d thru 0x000308 (0x27c bytes) 000677 1787076840.537181 trunc from 0x000309 to 0x021665 (0x2135d bytes) 000678 1787076840.559909 write 0x016e90 thru 0x01c8a3 (0x5a14 bytes) 000679 1787076840.575779 read 0x0215c8 thru 0x021665 (0x9e bytes) 000680 1787076840.582275 MIRROR_RESYNC 16 mirrors 000682 1787076844.923999 write 0x01f7f9 thru 0x02264a (0x2e52 bytes) 000683 1787076845.110685 mapread 0x006e66 thru 0x00b2d5 (0x4470 bytes) 000686 1787076845.151393 read 0x01d6e0 thru 0x02264a (0x4f6b bytes) 000687 1787076845.169061 trunc from 0x02264b to 0x037983 (0x15339 bytes) 000689 1787076845.244628 read 0x014a56 thru 0x022acf (0xe07a bytes) 000691 1787076845.275647 mapread 0x02cc7c thru 0x037983 (0xad08 bytes) 000692 1787076845.283377 read 0x024ac3 thru 0x02aad0 (0x600e bytes) 000693 1787076845.286172 trunc from 0x037984 to 0x0176dd (0x202a6 bytes) 000696 1787076845.326957 mapread 0x00a182 thru 0x00e653 (0x44d2 bytes) 000698 1787076845.335144 mapwrite 0x02a6f9 thru 0x0375bb (0xcec3 bytes) 000699 1787076845.433374 mapwrite 0x032229 thru 0x03cc83 (0xaa5b bytes) 000701 1787076845.513466 read 0x029242 thru 0x02a9fa (0x17b9 bytes) 000702 1787076845.515207 mapwrite 0x01f456 thru 0x026e11 (0x79bc bytes) 000703 1787076845.571927 mapread 0x007ca6 thru 0x00b1eb (0x3546 bytes) 000704 1787076845.575306 trunc from 0x03cc84 to 0x02aeb6 (0x11dcd bytes) 000705 1787076845.599839 mapread 0x0114c6 thru 0x02053c (0xf077 bytes) 000707 1787076845.605496 trunc from 0x02aeb7 to 0x0160e6 (0x14dd0 bytes) 000708 1787076845.633710 MIRROR_RESYNC 16 mirrors 000710 1787076848.257292 write 0x00593e thru 0x007a42 (0x2105 bytes) 000711 1787076848.417512 mapread 0x00a8ee thru 0x01246f (0x7b82 bytes) 000715 1787076848.466687 read 0x00703b thru 0x011f9e (0xaf64 bytes) 000716 1787076848.478946 MIRROR_RESYNC 16 mirrors 000718 1787076851.405491 mapwrite 0x006fd0 thru 0x00ead0 (0x7b01 bytes) 000719 1787076851.698378 mapwrite 0x038264 thru 0x03bbfd (0x399a bytes) 000720 1787076851.815407 MIRROR_SPLIT mirror 1 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bbfe): Mirror 7: 0x3e78d884 (stale) Mirror 9: 0x3e78d884 (stale) Mirror 10: 0x3e78d884 (stale) Mirror 12: 0x3e78d884 (stale) Mirror 13: 0x3e78d884 (stale) Mirror 14: 0x3e78d884 (stale) Mirror 17: 0x3e78d884 (stale) Mirror 18: 0x3e78d884 (stale) Mirror 19: 0x3e78d884 (stale) Mirror 20: 0x3e78d884 (stale) Mirror 21: 0x3e78d884 (stale) Mirror 22: 0x3e78d884 (stale) Mirror 23: 0x3e78d884 (stale) Mirror 24: 0x3e78d884 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bbfe): Mirror 7: 0x3e78d884 (stale) Mirror 9: 0x3e78d884 (stale) Mirror 10: 0x3e78d884 (stale) Mirror 12: 0x3e78d884 (stale) Mirror 13: 0x3e78d884 (stale) Mirror 14: 0x3e78d884 (stale) Mirror 17: 0x3e78d884 (stale) Mirror 18: 0x3e78d884 (stale) Mirror 19: 0x3e78d884 (stale) Mirror 20: 0x3e78d884 (stale) Mirror 21: 0x3e78d884 (stale) Mirror 22: 0x3e78d884 (stale) Mirror 23: 0x3e78d884 (stale) Mirror 24: 0x3e78d884 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bbfe: skipped 000721 1787076852.781083 MIRROR_RESYNC 15 mirrors 000722 1787076856.256232 MIRROR_RESYNC 15 mirrors 000724 1787076857.236174 trunc from 0x03bbfe to 0x01979f (0x2245e bytes) 000726 1787076857.343174 mapwrite 0x03e880 thru 0x03ffff (0x1780 bytes) 000728 1787076857.516426 MIRROR_SPLIT mirror 1 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x188ddb4e (stale) Mirror 10: 0x188ddb4e (stale) Mirror 12: 0x188ddb4e (stale) Mirror 13: 0x188ddb4e (stale) Mirror 14: 0x188ddb4e (stale) Mirror 17: 0x188ddb4e (stale) Mirror 18: 0x188ddb4e (stale) Mirror 19: 0x188ddb4e (stale) Mirror 20: 0x188ddb4e (stale) Mirror 21: 0x188ddb4e (stale) Mirror 22: 0x188ddb4e (stale) Mirror 23: 0x188ddb4e (stale) Mirror 24: 0x188ddb4e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x188ddb4e (stale) Mirror 10: 0x188ddb4e (stale) Mirror 12: 0x188ddb4e (stale) Mirror 13: 0x188ddb4e (stale) Mirror 14: 0x188ddb4e (stale) Mirror 17: 0x188ddb4e (stale) Mirror 18: 0x188ddb4e (stale) Mirror 19: 0x188ddb4e (stale) Mirror 20: 0x188ddb4e (stale) Mirror 21: 0x188ddb4e (stale) Mirror 22: 0x188ddb4e (stale) Mirror 23: 0x188ddb4e (stale) Mirror 24: 0x188ddb4e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000729 1787076858.564176 mapread 0x00250e thru 0x00b08e (0x8b81 bytes) 000730 1787076858.592112 MIRROR_SPLIT mirror 8 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x188ddb4e (stale) Mirror 10: 0x188ddb4e (stale) Mirror 12: 0x188ddb4e (stale) Mirror 13: 0x188ddb4e (stale) Mirror 14: 0x188ddb4e (stale) Mirror 17: 0x188ddb4e (stale) Mirror 18: 0x188ddb4e (stale) Mirror 20: 0x188ddb4e (stale) Mirror 21: 0x188ddb4e (stale) Mirror 22: 0x188ddb4e (stale) Mirror 23: 0x188ddb4e (stale) Mirror 24: 0x188ddb4e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x188ddb4e (stale) Mirror 10: 0x188ddb4e (stale) Mirror 12: 0x188ddb4e (stale) Mirror 13: 0x188ddb4e (stale) Mirror 14: 0x188ddb4e (stale) Mirror 17: 0x188ddb4e (stale) Mirror 18: 0x188ddb4e (stale) Mirror 20: 0x188ddb4e (stale) Mirror 21: 0x188ddb4e (stale) Mirror 22: 0x188ddb4e (stale) Mirror 23: 0x188ddb4e (stale) Mirror 24: 0x188ddb4e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000731 1787076859.636463 write 0x01d43a thru 0x02a348 (0xcf0f bytes) 000734 1787076859.661762 mapread 0x0305b0 thru 0x0362e0 (0x5d31 bytes) 000736 1787076859.696832 mapwrite 0x018528 thru 0x025353 (0xce2c bytes) 000739 1787076859.794848 read 0x020c4a thru 0x025893 (0x4c4a bytes) 000741 1787076859.796958 trunc from 0x040000 to 0x014d8a (0x2b275 bytes) 000742 1787076859.823436 trunc from 0x014d8b to 0x01d448 (0x86be bytes) 000748 1787076859.861083 write 0x02b807 thru 0x03b6d3 (0xfecd bytes) 000749 1787076859.865281 mapread 0x00cbc8 thru 0x018dc8 (0xc201 bytes) 000751 1787076859.895171 mapwrite 0x020318 thru 0x029ed7 (0x9bc0 bytes) 000752 1787076859.968852 write 0x00db46 thru 0x01474d (0x6c08 bytes) 000753 1787076859.974856 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x3b6d4): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b6d4): Mirror 9: 0x26f61995 (stale) Mirror 10: 0x26f61995 (stale) Mirror 12: 0x26f61995 (stale) Mirror 13: 0x26f61995 (stale) Mirror 14: 0x26f61995 (stale) Mirror 17: 0x26f61995 (stale) Mirror 18: 0x26f61995 (stale) Mirror 20: 0x26f61995 (stale) Mirror 21: 0x26f61995 (stale) Mirror 22: 0x26f61995 (stale) Mirror 23: 0x26f61995 (stale) Mirror 24: 0x26f61995 (stale) CRC-32 checksum value for chunk [0, 0x3b6d4): Mirror 1: 0x185e5ca5 Mirror 25: 0x185e5ca5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b6d4: skipped 000755 1787076861.308215 mapread 0x03157f thru 0x034812 (0x3294 bytes) 000756 1787076861.361982 MIRROR_RESYNC 14 mirrors 000757 1787076863.989721 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) 000760 1787076864.606203 MIRROR_RESYNC 13 mirrors 000761 1787076865.646174 read 0x00b9d9 thru 0x016f2f (0xb557 bytes) 000762 1787076865.669001 read 0x02e5c8 thru 0x02f7de (0x1217 bytes) 000764 1787076865.684398 write 0x033a71 thru 0x037735 (0x3cc5 bytes) 000766 1787076865.821351 mapwrite 0x00be0b thru 0x010d61 (0x4f57 bytes) 000768 1787076865.975005 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x3b6d4): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b6d4): Mirror 10: 0x185e5ca5 (stale) Mirror 12: 0x185e5ca5 (stale) Mirror 13: 0x185e5ca5 (stale) Mirror 14: 0x185e5ca5 (stale) Mirror 17: 0x185e5ca5 (stale) Mirror 18: 0x185e5ca5 (stale) Mirror 20: 0x185e5ca5 (stale) Mirror 21: 0x185e5ca5 (stale) Mirror 22: 0x185e5ca5 (stale) Mirror 23: 0x185e5ca5 (stale) Mirror 24: 0x185e5ca5 (stale) Mirror 25: 0x185e5ca5 (stale) CRC-32 checksum value for chunk [0, 0x3b6d4): Mirror 1: 0x7bac83b3 Mirror 26: 0x7bac83b3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b6d4: skipped 000769 1787076867.281139 mapwrite 0x03cfd2 thru 0x03ffff (0x302e bytes) 000770 1787076867.448850 MIRROR_SPLIT mirror 4 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xf3b71a92 (stale) Mirror 12: 0xf3b71a92 (stale) Mirror 13: 0xf3b71a92 (stale) Mirror 17: 0xf3b71a92 (stale) Mirror 18: 0xf3b71a92 (stale) Mirror 20: 0xf3b71a92 (stale) Mirror 21: 0xf3b71a92 (stale) Mirror 22: 0xf3b71a92 (stale) Mirror 23: 0xf3b71a92 (stale) Mirror 24: 0xf3b71a92 (stale) Mirror 25: 0xf3b71a92 (stale) Mirror 26: 0xe67f5910 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xf3b71a92 (stale) Mirror 12: 0xf3b71a92 (stale) Mirror 13: 0xf3b71a92 (stale) Mirror 17: 0xf3b71a92 (stale) Mirror 18: 0xf3b71a92 (stale) Mirror 20: 0xf3b71a92 (stale) Mirror 21: 0xf3b71a92 (stale) Mirror 22: 0xf3b71a92 (stale) Mirror 23: 0xf3b71a92 (stale) Mirror 24: 0xf3b71a92 (stale) Mirror 25: 0xf3b71a92 (stale) Mirror 26: 0xe67f5910 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000772 1787076868.592358 MIRROR_SPLIT mirror 2 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xf3b71a92 (stale) Mirror 13: 0xf3b71a92 (stale) Mirror 17: 0xf3b71a92 (stale) Mirror 18: 0xf3b71a92 (stale) Mirror 20: 0xf3b71a92 (stale) Mirror 21: 0xf3b71a92 (stale) Mirror 22: 0xf3b71a92 (stale) Mirror 23: 0xf3b71a92 (stale) Mirror 24: 0xf3b71a92 (stale) Mirror 25: 0xf3b71a92 (stale) Mirror 26: 0xe67f5910 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xf3b71a92 (stale) Mirror 13: 0xf3b71a92 (stale) Mirror 17: 0xf3b71a92 (stale) Mirror 18: 0xf3b71a92 (stale) Mirror 20: 0xf3b71a92 (stale) Mirror 21: 0xf3b71a92 (stale) Mirror 22: 0xf3b71a92 (stale) Mirror 23: 0xf3b71a92 (stale) Mirror 24: 0xf3b71a92 (stale) Mirror 25: 0xf3b71a92 (stale) Mirror 26: 0xe67f5910 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000773 1787076869.286491 mapwrite 0x02049c thru 0x023f75 (0x3ada bytes) 000774 1787076869.366011 read 0x015f45 thru 0x01cdf0 (0x6eac bytes) 000776 1787076869.386441 read 0x031100 thru 0x038e53 (0x7d54 bytes) 000777 1787076869.438767 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xf3b71a92 (stale) Mirror 13: 0xf3b71a92 (stale) Mirror 17: 0xf3b71a92 (stale) Mirror 18: 0xf3b71a92 (stale) Mirror 20: 0xf3b71a92 (stale) Mirror 21: 0xf3b71a92 (stale) Mirror 22: 0xf3b71a92 (stale) Mirror 23: 0xf3b71a92 (stale) Mirror 24: 0xf3b71a92 (stale) Mirror 25: 0xf3b71a92 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xf3b71a92 (stale) Mirror 13: 0xf3b71a92 (stale) Mirror 17: 0xf3b71a92 (stale) Mirror 18: 0xf3b71a92 (stale) Mirror 20: 0xf3b71a92 (stale) Mirror 21: 0xf3b71a92 (stale) Mirror 22: 0xf3b71a92 (stale) Mirror 23: 0xf3b71a92 (stale) Mirror 24: 0xf3b71a92 (stale) Mirror 25: 0xf3b71a92 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000780 1787076870.276340 read 0x02b461 thru 0x034141 (0x8ce1 bytes) 000782 1787076870.289318 write 0x0117fa thru 0x01787d (0x6084 bytes) 000783 1787076870.302966 trunc from 0x040000 to 0x037e3a (0x81c5 bytes) 000785 1787076870.331556 mapwrite 0x013a68 thru 0x021146 (0xd6df bytes) 000787 1787076870.455757 write 0x0105d4 thru 0x012680 (0x20ad bytes) 000788 1787076870.464109 mapwrite 0x01fcb1 thru 0x029d13 (0xa063 bytes) 000789 1787076870.599386 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37e3b): Mirror 10: 0xbfa7b54a (stale) Mirror 13: 0xbfa7b54a (stale) Mirror 17: 0xbfa7b54a (stale) Mirror 18: 0xbfa7b54a (stale) Mirror 20: 0xbfa7b54a (stale) Mirror 21: 0xbfa7b54a (stale) Mirror 22: 0xbfa7b54a (stale) Mirror 23: 0xbfa7b54a (stale) Mirror 25: 0xbfa7b54a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37e3b): Mirror 10: 0xbfa7b54a (stale) Mirror 13: 0xbfa7b54a (stale) Mirror 17: 0xbfa7b54a (stale) Mirror 18: 0xbfa7b54a (stale) Mirror 20: 0xbfa7b54a (stale) Mirror 21: 0xbfa7b54a (stale) Mirror 22: 0xbfa7b54a (stale) Mirror 23: 0xbfa7b54a (stale) Mirror 25: 0xbfa7b54a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37e3b: skipped 000790 1787076871.499326 mapread 0x0237f7 thru 0x032866 (0xf070 bytes) 000791 1787076871.540562 write 0x00a944 thru 0x01a032 (0xf6ef bytes) 000792 1787076871.555818 trunc from 0x037e3b to 0x006871 (0x315c9 bytes) 000793 1787076871.588514 mapread 0x0052e5 thru 0x006871 (0x158d bytes) 000798 1787076871.606765 mapwrite 0x027883 thru 0x02e894 (0x7012 bytes) 000799 1787076871.767887 read 0x015df5 thru 0x01d019 (0x7225 bytes) 000804 1787076871.809230 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2e895): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e895): Mirror 10: 0x7cd5aa77 (stale) Mirror 13: 0x7cd5aa77 (stale) Mirror 17: 0x7cd5aa77 (stale) Mirror 18: 0x7cd5aa77 (stale) Mirror 20: 0x7cd5aa77 (stale) Mirror 21: 0x7cd5aa77 (stale) Mirror 22: 0x7cd5aa77 (stale) Mirror 23: 0x7cd5aa77 (stale) Mirror 25: 0x7cd5aa77 (stale) CRC-32 checksum value for chunk [0, 0x2e895): Mirror 1: 0x7c3fb41e Mirror 26: 0x7c3fb41e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e895: skipped 000805 1787076873.278486 trunc from 0x02e895 to 0x029751 (0x5143 bytes) 000809 1787076873.368652 mapwrite 0x007aea thru 0x00e1b0 (0x66c7 bytes) 000812 1787076873.514502 write 0x03a6be thru 0x03c898 (0x21db bytes) 000813 1787076873.515813 trunc from 0x03c899 to 0x003813 (0x39085 bytes) 000814 1787076873.558251 read 0x000958 thru 0x003813 (0x2ebc bytes) 000816 1787076873.571071 mapread 0x0021ad thru 0x003813 (0x1667 bytes) 000819 1787076873.574649 trunc from 0x003814 to 0x025ec4 (0x226b1 bytes) 000820 1787076873.596250 read 0x004664 thru 0x0060ff (0x1a9c bytes) 000822 1787076873.635847 mapread 0x00adaf thru 0x00d7b7 (0x2a09 bytes) 000823 1787076873.636952 write 0x0080e0 thru 0x014785 (0xc6a6 bytes) 000825 1787076873.643549 write 0x03072d thru 0x03b325 (0xabf9 bytes) 000826 1787076873.646936 mapwrite 0x0101a8 thru 0x016a15 (0x686e bytes) 000828 1787076873.773622 write 0x0094cd thru 0x00d8cf (0x4403 bytes) 000832 1787076873.780394 mapread 0x00ee76 thru 0x01a3c2 (0xb54d bytes) 000833 1787076873.784820 read 0x034d7c thru 0x03b325 (0x65aa bytes) 000836 1787076873.794950 read 0x02fbf0 thru 0x03091e (0xd2f bytes) 000838 1787076873.811430 write 0x034ee2 thru 0x03ffff (0xb11e bytes) 000839 1787076873.814126 mapread 0x001084 thru 0x009ba2 (0x8b1f bytes) 000840 1787076873.815353 write 0x0178a1 thru 0x01e75c (0x6ebc bytes) 000842 1787076873.821182 mapwrite 0x027ef8 thru 0x02bacb (0x3bd4 bytes) 000843 1787076873.982818 mapwrite 0x013f40 thru 0x01bf28 (0x7fe9 bytes) 000844 1787076874.101838 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xf3b71a92 (stale) Mirror 17: 0xf3b71a92 (stale) Mirror 18: 0xf3b71a92 (stale) Mirror 20: 0xf3b71a92 (stale) Mirror 21: 0xf3b71a92 (stale) Mirror 22: 0xf3b71a92 (stale) Mirror 23: 0xf3b71a92 (stale) Mirror 25: 0xf3b71a92 (stale) Mirror 26: 0x1aacdd64 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xf3b71a92 (stale) Mirror 17: 0xf3b71a92 (stale) Mirror 18: 0xf3b71a92 (stale) Mirror 20: 0xf3b71a92 (stale) Mirror 21: 0xf3b71a92 (stale) Mirror 22: 0xf3b71a92 (stale) Mirror 23: 0xf3b71a92 (stale) Mirror 25: 0xf3b71a92 (stale) Mirror 26: 0x1aacdd64 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000845 1787076874.946043 mapwrite 0x0368ec thru 0x03726a (0x97f bytes) 000847 1787076875.006890 mapwrite 0x013489 thru 0x020796 (0xd30e bytes) 000848 1787076875.092754 mapread 0x0086e1 thru 0x0140c6 (0xb9e6 bytes) 000849 1787076875.104675 trunc from 0x040000 to 0x01aa01 (0x255fe bytes) 000850 1787076875.124674 trunc from 0x01aa02 to 0x01414e (0x68b3 bytes) 000851 1787076875.148900 mapread 0x0140b3 thru 0x01414e (0x9c bytes) 000852 1787076875.156350 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1414f): Mirror 13: 0x5570e61c (stale) Mirror 17: 0x5570e61c (stale) Mirror 18: 0x5570e61c (stale) Mirror 20: 0x5570e61c (stale) Mirror 21: 0x5570e61c (stale) Mirror 22: 0x5570e61c (stale) Mirror 25: 0x5570e61c (stale) Mirror 26: 0xa697d40b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1414f): Mirror 13: 0x5570e61c (stale) Mirror 17: 0x5570e61c (stale) Mirror 18: 0x5570e61c (stale) Mirror 20: 0x5570e61c (stale) Mirror 21: 0x5570e61c (stale) Mirror 22: 0x5570e61c (stale) Mirror 25: 0x5570e61c (stale) Mirror 26: 0xa697d40b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1414f: skipped 000854 1787076875.719911 mapread 0x001b88 thru 0x01050d (0xe986 bytes) 000855 1787076875.749200 mapread 0x00d0bb thru 0x01414e (0x7094 bytes) 000856 1787076875.757265 read 0x013902 thru 0x01414e (0x84d bytes) 000857 1787076875.763668 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1414f): Mirror 13: 0x5570e61c (stale) Mirror 18: 0x5570e61c (stale) Mirror 20: 0x5570e61c (stale) Mirror 21: 0x5570e61c (stale) Mirror 22: 0x5570e61c (stale) Mirror 25: 0x5570e61c (stale) Mirror 26: 0xa697d40b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1414f): Mirror 13: 0x5570e61c (stale) Mirror 18: 0x5570e61c (stale) Mirror 20: 0x5570e61c (stale) Mirror 21: 0x5570e61c (stale) Mirror 22: 0x5570e61c (stale) Mirror 25: 0x5570e61c (stale) Mirror 26: 0xa697d40b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1414f: skipped 000858 1787076876.356667 write 0x021300 thru 0x022735 (0x1436 bytes) 000863 1787076876.368495 trunc from 0x022736 to 0x006118 (0x1c61d bytes) 000865 1787076876.416233 mapread 0x0032a5 thru 0x006118 (0x2e74 bytes) 000866 1787076876.449816 mapread 0x0007ef thru 0x001fc3 (0x17d5 bytes) 000870 1787076876.465551 write 0x011fd9 thru 0x018b97 (0x6bbf bytes) 000871 1787076876.479310 MIRROR_RESYNC 8 mirrors 000872 1787076878.167637 mapread 0x0183ae thru 0x018b97 (0x7ea bytes) 000873 1787076878.173747 mapwrite 0x005de5 thru 0x013af7 (0xdd13 bytes) 000875 1787076878.548520 trunc from 0x018b98 to 0x000177 (0x18a20 bytes) 000876 1787076878.588045 mapread 0x000056 thru 0x000177 (0x122 bytes) 000877 1787076878.595810 mapread 0x00009d thru 0x000177 (0xdb bytes) 000878 1787076878.597266 write 0x020f51 thru 0x023f72 (0x3022 bytes) 000880 1787076878.602528 mapread 0x01e7b6 thru 0x023f72 (0x57bd bytes) 000881 1787076878.615272 mapwrite 0x03f7b3 thru 0x03ffff (0x84d bytes) 000884 1787076878.734450 mapwrite 0x028399 thru 0x033b21 (0xb789 bytes) 000885 1787076878.823007 mapread 0x034581 thru 0x03ffff (0xba7f bytes) 000886 1787076878.825382 trunc from 0x040000 to 0x030966 (0xf699 bytes) 000887 1787076878.849549 MIRROR_RESYNC 8 mirrors 000891 1787076881.645237 trunc from 0x030967 to 0x0041b4 (0x2c7b2 bytes) 000893 1787076881.855542 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x41b5): Mirror 13: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x41b5): Mirror 13: 0xa5f33388 (stale) Mirror 18: 0xa5f33388 (stale) Mirror 20: 0xa5f33388 (stale) Mirror 21: 0xa5f33388 (stale) Mirror 22: 0xa5f33388 (stale) Mirror 25: 0xa5f33388 (stale) Mirror 26: 0xa5f33388 (stale) CRC-32 checksum value for chunk [0, 0x41b5): Mirror 1: 0xa5f33388 Mirror 27: 0xa5f33388 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x41b5: skipped 000894 1787076883.397410 write 0x029a5d thru 0x02ef30 (0x54d4 bytes) 000895 1787076883.561685 mapread 0x01a25b thru 0x02275c (0x8502 bytes) 000897 1787076883.592521 write 0x02de61 thru 0x0390a9 (0xb249 bytes) 000899 1787076883.597019 trunc from 0x0390aa to 0x032205 (0x6ea4 bytes) 000902 1787076883.629092 mapread 0x017f4b thru 0x0197a6 (0x185c bytes) 000903 1787076883.630460 write 0x004f74 thru 0x010aaf (0xbb3c bytes) 000904 1787076883.649135 write 0x0303b3 thru 0x03e711 (0xe35f bytes) 000907 1787076883.654289 read 0x031f5a thru 0x03c734 (0xa7db bytes) 000908 1787076883.660297 MIRROR_RESYNC 9 mirrors 000910 1787076885.782031 trunc from 0x03e712 to 0x00eb91 (0x2fb80 bytes) 000911 1787076886.004489 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xeb92): Mirror 13: 0xe594cc4d (stale) Mirror 18: 0xe594cc4d (stale) Mirror 21: 0xe594cc4d (stale) Mirror 22: 0xe594cc4d (stale) Mirror 25: 0xe594cc4d (stale) Mirror 26: 0xe594cc4d (stale) Mirror 27: 0xe594cc4d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xeb92): Mirror 13: 0xe594cc4d (stale) Mirror 18: 0xe594cc4d (stale) Mirror 21: 0xe594cc4d (stale) Mirror 22: 0xe594cc4d (stale) Mirror 25: 0xe594cc4d (stale) Mirror 26: 0xe594cc4d (stale) Mirror 27: 0xe594cc4d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xeb92: skipped 000912 1787076886.672480 trunc from 0x00eb92 to 0x034481 (0x258f0 bytes) 000914 1787076886.733089 mapread 0x009573 thru 0x00adc2 (0x1850 bytes) 000915 1787076886.766887 trunc from 0x034482 to 0x0108e9 (0x23b98 bytes) 000916 1787076886.806257 write 0x02c1fe thru 0x02ff85 (0x3d88 bytes) 000917 1787076886.808227 read 0x01ef07 thru 0x023f34 (0x502e bytes) 000918 1787076886.827665 mapwrite 0x03caaf thru 0x03ffff (0x3551 bytes) 000920 1787076886.987575 write 0x02c927 thru 0x031ff0 (0x56ca bytes) 000921 1787076886.997060 trunc from 0x040000 to 0x01dabe (0x22541 bytes) 000922 1787076887.036132 MIRROR_RESYNC 8 mirrors 000924 1787076888.624340 mapwrite 0x026e3a thru 0x031631 (0xa7f8 bytes) 000929 1787076888.922198 mapread 0x02971e thru 0x02e572 (0x4e55 bytes) 000930 1787076888.932536 MIRROR_RESYNC 8 mirrors 000931 1787076890.515392 mapread 0x005850 thru 0x00c3af (0x6b60 bytes) 000932 1787076890.543626 read 0x010028 thru 0x01b0f5 (0xb0ce bytes) 000934 1787076890.573174 read 0x009a21 thru 0x00b454 (0x1a34 bytes) 000935 1787076890.582734 MIRROR_EXTEND to 9 mirrors 000937 1787076891.758839 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a3:0x0]: Invalid argument (22) 000938 1787076892.316834 MIRROR_EXTEND to 9 mirrors 000939 1787076893.603362 trunc from 0x031632 to 0x025de6 (0xb84b bytes) 000941 1787076893.818666 mapwrite 0x003301 thru 0x00bd26 (0x8a26 bytes) 000943 1787076893.965412 mapwrite 0x037731 thru 0x03ffff (0x88cf bytes) 000946 1787076894.094740 mapread 0x01e37f thru 0x029004 (0xac86 bytes) 000947 1787076894.141304 write 0x0127e8 thru 0x01eb74 (0xc38d bytes) 000948 1787076894.148304 mapread 0x03116f thru 0x0346df (0x3571 bytes) 000949 1787076894.149467 write 0x00cc71 thru 0x00d998 (0xd28 bytes) 000951 1787076894.152233 mapread 0x02f98e thru 0x035ed0 (0x6543 bytes) 000953 1787076894.159777 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0x2e75f17e (stale) Mirror 21: 0x2e75f17e (stale) Mirror 22: 0x2e75f17e (stale) Mirror 25: 0x2e75f17e (stale) Mirror 26: 0x2e75f17e (stale) Mirror 27: 0x2e75f17e (stale) Mirror 28: 0x2e75f17e (stale) Mirror 29: 0x2e75f17e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd35e61b7 Mirror 30: 0xd35e61b7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000963 1787076895.394257 MIRROR_RESYNC 10 mirrors 000964 1787076897.193938 mapwrite 0x012838 thru 0x0129b0 (0x179 bytes) 000971 1787076897.377582 mapwrite 0x00d3d1 thru 0x017d21 (0xa951 bytes) 000972 1787076897.499337 mapwrite 0x003f46 thru 0x004324 (0x3df bytes) 000973 1787076897.632857 write 0x0316b8 thru 0x035a98 (0x43e1 bytes) 000975 1787076897.645668 mapwrite 0x0084a7 thru 0x011a02 (0x955c bytes) 000978 1787076897.696535 mapread 0x03d364 thru 0x03ffff (0x2c9c bytes) 000979 1787076897.709578 mapwrite 0x0168f1 thru 0x01bcdf (0x53ef bytes) 000983 1787076897.789657 mapwrite 0x016164 thru 0x017d91 (0x1c2e bytes) 000985 1787076897.870341 read 0x004920 thru 0x00e708 (0x9de9 bytes) 000988 1787076897.878723 read 0x02ceda thru 0x02e29d (0x13c4 bytes) 000990 1787076897.915263 mapwrite 0x00fd30 thru 0x017ee2 (0x81b3 bytes) 000991 1787076898.033563 write 0x03659c thru 0x036a1c (0x481 bytes) 000992 1787076898.036386 trunc from 0x040000 to 0x03dcf8 (0x2307 bytes) 000994 1787076898.089418 mapread 0x02fdcd thru 0x035ac5 (0x5cf9 bytes) 000995 1787076898.092559 mapwrite 0x005ad2 thru 0x006a94 (0xfc3 bytes) 000997 1787076898.154245 write 0x00a143 thru 0x00f64e (0x550c bytes) 000998 1787076898.156421 read 0x02062d thru 0x02e610 (0xdfe4 bytes) 000999 1787076898.157980 write 0x01935e thru 0x01d8be (0x4561 bytes) All operations completed A-OK! PASS 70a (151s) == sanity-flr test 71: check mirror extend parent fid ==== 14:15:02 (1787076902) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 14:15:05 (1787076905) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.278528 s, 7.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198679 s, 5.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 14:15:14 (1787076914) Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre2 Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 6438880 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4120352 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 6003136 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7407648 .. Extending file size to 7923328 .. Extending file size to 8824288 .. Extending file size to 6572576 .. Extending file size to 7625984 .. Extending file size to 9038848 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7758016 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7915360 .. Extending file size to 8403648 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8829504 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 57415 57417 57418 57420 57422 192.168.204.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg405-client.virtnet /mnt/lustre2 (opts:) 192.168.204.105@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg405-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (78s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 14:16:33 (1787076993) Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre2 Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6045760 .. Extending file size to 3308288 .. done Extending file size to 8192832 .. Extending file size to 6352512 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8466944 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6998432 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8804704 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8848992 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8543424 .. Waiting 59024 59026 59027 59029 59031 done 192.168.204.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg405-client.virtnet /mnt/lustre2 (opts:) 192.168.204.105@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg405-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 14:17:52 (1787077072) Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.138404 s, 15.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.328636 s, 6.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x57a66e93 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg405-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 14:18:11 (1787077091) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.128979 s, 16.3 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 14:18:20 (1787077100) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f3:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 14:18:30 (1787077110) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.301027 s, 7.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 14:18:39 (1787077119) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.136792 s, 15.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x301:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 14:18:48 (1787077128) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.270523 s, 7.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 14:19:21 (1787077161) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg405-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950711 s, 11.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x305:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x305:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e8:0x0] } PASS 204d (57s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 14:20:18 (1787077218) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg405-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x306:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x307:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.33614 s, 6.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x306:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x307:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x306:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x307:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (63s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 14:21:21 (1787077281) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg405-server mds-ost sync done. Creating new pool test_204f oleg405-server: Pool lustre.test_204f created Adding targets to pool oleg405-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg405-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.242884 s, 8.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x308:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ea:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x308:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ea:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg405-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg405-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg405-server: Pool lustre.test_204f destroyed PASS 204f (82s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 14:22:43 (1787077363) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 14:22:54 (1787077374) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 14:23:02 (1787077382) oleg405-server: Pool lustre.test_206 created Adding targets to pool oleg405-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg405-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg405-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg405-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg405-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 14:23:30 (1787077410) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076923 s, 13.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681815 s, 15.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x907cc717 Mirror 3: 0x907cc717 PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 14:23:43 (1787077423) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 14:23:47 (1787077427) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 14:23:51 (1787077431) Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.126239 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x25e0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1cc0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1cc0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1cc0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1420000 Read this (4 bytes): BBBB 192.168.204.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg405-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 14:24:29 (1787077469) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0320868 s, 255 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 14:24:37 (1787077477) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595264 s, 17.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x218 lmm_fid: [0x200000404:0x218:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x218 lmm_fid: [0x200000404:0x218:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f5:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834456 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-18 14:24:39.000000000 -0400 Modify: 2026-08-18 14:24:39.000000000 -0400 Change: 2026-08-18 14:24:40.000000000 -0400 Birth: 2026-08-18 14:24:39.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 14:24:45 (1787077485) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 14:24:49 (1787077489) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.553777 s, 74.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0725285 s, 56.5 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 18 14:24 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21a:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 18 14:24 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 14:24:58 (1787077498) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 14:25:06 (1787077506) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1853 sec =========== 14:25:16 (1787077516) === sanity-flr: start cleanup 14:25:18 (1787077518) === === sanity-flr: finish cleanup 14:25:25 (1787077525) ===