-----============= acceptance-small: sanity-flr ============----- Wed Jul 8 21:58:42 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg225-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 21:58:53 (1783562333) === oleg225-client.virtnet: executing check_config_client /mnt/lustre oleg225-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg225-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff951210f82800.idle_timeout=debug osc.lustre-OST0001-osc-ffff951210f82800.idle_timeout=debug disable quota as required oleg225-server: oleg225-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 21:59:13 (1783562353) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7589] == sanity-flr test 0a: lfs mirror create with -N option == 21:59:15 (1783562355) PASS 0a (10s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 21:59:25 (1783562365) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 21:59:27 (1783562367) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 21:59:29 (1783562369) PASS 0d (15s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 21:59:44 (1783562384) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 21:59:46 (1783562386) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 21:59:48 (1783562388) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff951210f82800.stats=clear osc.lustre-OST0001-osc-ffff951210f82800.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 21:59:52 (1783562392) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (7s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 21:59:59 (1783562399) PASS 0j (5s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 22:00:04 (1783562404) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 0m: mirror extend with -N= option ===== 22:00:05 (1783562405) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (5s) == sanity-flr test 1: create components with setstripe options ========================================================== 22:00:10 (1783562410) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (6s) == sanity-flr test 2: create components from existing files ========================================================== 22:00:16 (1783562416) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 22:00:20 (1783562420) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 22:00:22 (1783562422) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 22:00:26 (1783562426) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 22:00:31 (1783562431) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 22:00:34 (1783562434) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.961682 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618668 s, 16.9 MB/s PASS 21 (6s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 22:00:40 (1783562440) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.144403 s, 79.9 MB/s PASS 22 (3s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 22:00:44 (1783562444) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (5s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 22:00:49 (1783562449) Stopping /mnt/lustre-ost1 (opts:) on oleg225-server oleg225-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff951210f82800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff951210f82800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg225-server: oleg225-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg225-server: oleg225-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg225-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg225-server oleg225-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff951210f82800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff951210f82800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg225-server: oleg225-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg225-server: oleg225-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg225-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (36s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 22:01:25 (1783562485) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.085391 s, 24.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg225-server oleg225-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff951210f82800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff951210f82800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg225-server: oleg225-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg225-server: oleg225-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg225-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (18s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 22:01:43 (1783562503) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg225-server oleg225-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff951210f82800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff951210f82800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg225-server: oleg225-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg225-server: oleg225-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg225-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg225-server oleg225-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff951210f82800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff951210f82800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg225-server: oleg225-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg225-server: oleg225-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg225-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (42s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 22:02:25 (1783562545) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.65286 s, 19.8 MB/s -rw-r--r-- 1 root root 52428800 Jul 8 22:02 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.525s user 0m0.001s sys 0m0.163s Stopping /mnt/lustre-ost1 (opts:) on oleg225-server oleg225-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff951210f82800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff951210f82800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.548s user 0m0.001s sys 0m0.161s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg225-server: oleg225-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg225-server: oleg225-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg225-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg225-server oleg225-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff951210f82800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff951210f82800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.684s user 0m0.001s sys 0m0.153s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg225-server: oleg225-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg225-server: oleg225-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg225-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg225-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff951210f82800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff951210f82800.ost_server_uuid in FULL state after 0 sec PASS 33b (49s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 22:03:14 (1783562594) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 22:03:16 (1783562596) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 22:03:18 (1783562598) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 22:03:19 (1783562599) PASS 35 (4s) == sanity-flr test 36a: write to mirrored files ========== 22:03:23 (1783562603) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.381438 s, 33.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (6s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 22:03:30 (1783562610) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195342 s, 53.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243081 s, 43.1 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 22:03:33 (1783562613) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 22:03:37 (1783562617) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000782956 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000626332 s, 1.6 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000806997 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0019391 s, 528 kB/s flv=10 olv=10 PASS 36d (5s) == sanity-flr test 37: mirror I/O API verification ======= 22:03:42 (1783562622) create test files with size 15736097 .. 2d1524ac23db1ca44d32e3aa54ed7905 - eebda400c727bf8fe2ce1b76dee55ece - 82c54388f1e3c727481f5debcf8c2f4e - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg225-client.virtnet /mnt/lustre (opts:) Starting client: oleg225-client.virtnet: -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre PASS 37 (9s) == sanity-flr test 38: resync ============================ 22:03:51 (1783562631) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (7s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 22:03:58 (1783562638) PASS 39 (4s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 22:04:02 (1783562642) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354482 s, 29.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151547 s, 69.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbb:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xda:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (5s) == sanity-flr test 41: lfs mirror resync check =========== 22:04:07 (1783562647) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0912997 s, 23.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.149506 s, 28.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.102931 s, 20.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (8s) == sanity-flr test 42: lfs mirror verify ================= 22:04:15 (1783562655) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 22:04:16 (1783562656) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 22:04:18 (1783562658) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122842 s, 85.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 22:04:22 (1783562662) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 22:04:24 (1783562664) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe1:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe1:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 22:04:29 (1783562669) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.136505 s, 76.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg225-client.virtnet /mnt/lustre (opts:) Starting client: oleg225-client.virtnet: -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 22:04:42 (1783562682) llite.lustre-ffff951209871000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.151777 s, 69.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg225-client.virtnet /mnt/lustre (opts:) Starting client: oleg225-client.virtnet: -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff95120824b800.xattr_cache=1 PASS 44e (6s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 22:04:48 (1783562688) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (17s) == sanity-flr test 46: Verify setstripe --copy option ==== 22:05:05 (1783562705) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (17s) == sanity-flr test 47: Verify mirror obj alloc =========== 22:05:22 (1783562722) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 22:05:23 (1783562723) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.126348 s, 24.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0603942 s, 52.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x583ac1d7 (nosync,stale) Mirror 2: 0x583ac1d7 (nosync,stale) ** verify mirror 2 doesn't change original checksum: a12652fd16ed68729292055df7e86509 - mirror 1 checksum: 8a3036c4371d82f67a31d063a8bc39be - mirror 2 checksum: a12652fd16ed68729292055df7e86509 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-08T22:05:24-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-08T22:05:24-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: a12652fd16ed68729292055df7e86509 - mirror 1 checksum: 8a3036c4371d82f67a31d063a8bc39be - mirror 2 checksum: 8a3036c4371d82f67a31d063a8bc39be - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-08T22:05:24-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-08T22:05:24-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } PASS 48 (5s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 22:05:28 (1783562728) Failing mds1 on oleg225-server Stopping /mnt/lustre-mds1 (opts:) on oleg225-server 22:05:32 (1783562732) shut down facet: mds1 facet_host: oleg225-server facet_failover_host: oleg225-server Failover mds1 to oleg225-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg225-server: oleg225-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-MDT0000 22:05:48 (1783562748) targets are mounted 22:05:48 (1783562748) facet_failover done oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230674 s, 45.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } PASS 50A (29s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 22:05:57 (1783562757) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 22:05:58 (1783562758) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 22:06:00 (1783562760) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0033595 s, 4.9 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 22:06:03 (1783562763) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 22:06:05 (1783562765) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 22:06:06 (1783562766) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (10s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 22:06:16 (1783562776) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (21s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 22:06:37 (1783562797) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (21s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 22:06:58 (1783562818) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 22:07:17 (1783562837) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 22:07:20 (1783562840) mirror create pid 53675 mirror split pid 53676 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53675 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53676 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 22:08:23 (1783562903) Seed set to 2904 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x14051 000004 1783562904.750320 trunc from 00000000 to 0x014050 (0x14051 bytes) 000008 1783562904.758268 write 0x02fa15 thru 0x03176b (0x1d57 bytes) 000011 1783562904.760045 mapread 0x00c53a thru 0x00e98e (0x2455 bytes) 000012 1783562904.767449 mapread 0x027e48 thru 0x03176b (0x9924 bytes) 000015 1783562904.772632 mapwrite 0x02a0c3 thru 0x032477 (0x83b5 bytes) 000024 1783562906.854664 read 0x014123 thru 0x020426 (0xc304 bytes) 000025 1783562906.861955 read 0x024729 thru 0x02cd13 (0x85eb bytes) truncating to largest ever: 0x1ac07 000026 1783562906.862815 trunc from 0x032478 to 0x01ac06 (0x17871 bytes) 000027 1783562906.876181 mapwrite 0x03cf6d thru 0x03ffff (0x3093 bytes) 000028 1783562906.956833 read 0x02e169 thru 0x038faa (0xae42 bytes) 000029 1783562906.958324 mapwrite 0x02dd77 thru 0x03686d (0x8af7 bytes) truncating to largest ever: 0x25a67 000030 1783562907.010351 trunc from 0x040000 to 0x025a66 (0x1a599 bytes) 000033 1783562907.036493 read 0x002ba6 thru 0x01042c (0xd887 bytes) 000035 1783562907.037979 mapread 0x01fda9 thru 0x025a66 (0x5cbe bytes) 000037 1783562907.041867 trunc from 0x025a67 to 0x0034ec (0x2257a bytes) 000040 1783562907.068467 mapread 0x002749 thru 0x0034ec (0xda4 bytes) 000041 1783562907.073953 read 0x002ccb thru 0x0034ec (0x822 bytes) 000042 1783562907.074653 read 0x002076 thru 0x002346 (0x2d1 bytes) 000043 1783562907.075279 trunc from 0x0034ed to 0x01023e (0xcd52 bytes) 000044 1783562907.091492 trunc from 0x01023f to 0x01c97b (0xc73d bytes) 000045 1783562907.105496 mapwrite 0x032695 thru 0x03e399 (0xbd05 bytes) 000046 1783562907.180752 trunc from 0x03e39a to 0x0025ad (0x3bdec bytes) 000047 1783562907.195741 mapwrite 0x013e94 thru 0x01faf8 (0xbc65 bytes) 000049 1783562907.296989 mapread 0x015941 thru 0x01b348 (0x5a08 bytes) 000052 1783562907.299023 write 0x01fb4c thru 0x022acf (0x2f84 bytes) 000053 1783562907.300506 mapwrite 0x01f30d thru 0x0217a1 (0x2495 bytes) truncating to largest ever: 0x36028 000054 1783562907.323608 trunc from 0x022ad0 to 0x036027 (0x13558 bytes) 000056 1783562907.332905 write 0x03b0a9 thru 0x03c339 (0x1291 bytes) 000057 1783562907.335780 MIRROR_EXTEND to 2 mirrors 000058 1783562907.707672 mapread 0x03832d thru 0x03c339 (0x400d bytes) 000061 1783562907.732149 MIRROR_RESYNC 2 mirrors 000062 1783562907.935571 MIRROR_RESYNC 2 mirrors 000063 1783562908.114955 trunc from 0x03c33a to 0x015b72 (0x267c7 bytes) 000066 1783562908.153842 read 0x014823 thru 0x015b72 (0x1350 bytes) 000067 1783562908.158404 write 0x03dc2a thru 0x03ffff (0x23d6 bytes) truncating to largest ever: 0x370cb 000068 1783562908.159361 trunc from 0x040000 to 0x0370ca (0x8f35 bytes) 000069 1783562908.168234 mapread 0x016401 thru 0x019093 (0x2c93 bytes) 000071 1783562908.181756 read 0x02d3d3 thru 0x0370ca (0x9cf8 bytes) 000072 1783562908.193678 write 0x020755 thru 0x025307 (0x4bb3 bytes) 000075 1783562908.197578 MIRROR_RESYNC 2 mirrors 000076 1783562908.453767 mapread 0x02b7e6 thru 0x034f02 (0x971d bytes) 000077 1783562908.463891 mapwrite 0x013b67 thru 0x015d7d (0x2217 bytes) 000078 1783562908.525039 mapread 0x00be20 thru 0x01b5d3 (0xf7b4 bytes) 000081 1783562908.535212 mapread 0x036511 thru 0x0370ca (0xbba bytes) 000082 1783562908.542711 mapread 0x016429 thru 0x026177 (0xfd4f bytes) 000084 1783562908.551759 mapread 0x024253 thru 0x032b5e (0xe90c bytes) 000086 1783562908.556313 write 0x025de1 thru 0x02b76f (0x598f bytes) 000087 1783562908.559656 MIRROR_RESYNC 2 mirrors 000088 1783562908.871621 mapread 0x00712b thru 0x00df51 (0x6e27 bytes) 000089 1783562908.890912 mapwrite 0x00c3e4 thru 0x013b6b (0x7788 bytes) 000090 1783562908.984824 trunc from 0x0370cb to 0x02c635 (0xaa95 bytes) 000091 1783562909.009746 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2c636): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c636): Mirror 2: 0xac4c7bb8 (stale) CRC-32 checksum value for chunk [0, 0x2c636): Mirror 1: 0x4764cd46 Mirror 3: 0x4764cd46 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c636: skipped 000092 1783562909.407796 mapwrite 0x00ddb8 thru 0x0119f3 (0x3c3c bytes) 000094 1783562909.480933 MIRROR_RESYNC 3 mirrors 000095 1783562909.921161 write 0x01893c thru 0x018bae (0x273 bytes) 000097 1783562909.962865 mapwrite 0x007c1c thru 0x00f27b (0x7660 bytes) 000098 1783562910.022636 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2c636): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c636): Mirror 2: 0x3128e856 (stale) Mirror 3: 0x3128e856 (stale) CRC-32 checksum value for chunk [0, 0x2c636): Mirror 1: 0x56a1566f Mirror 4: 0x56a1566f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c636: skipped 000099 1783562910.500912 write 0x0345ae thru 0x03ffff (0xba52 bytes) 000101 1783562910.557882 mapwrite 0x02dcab thru 0x03936f (0xb6c5 bytes) 000102 1783562910.605306 mapread 0x03f237 thru 0x03ffff (0xdc9 bytes) 000106 1783562910.607429 read 0x033f91 thru 0x03677a (0x27ea bytes) 000108 1783562910.609318 trunc from 0x040000 to 0x003ffd (0x3c002 bytes) 000110 1783562910.627535 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ffe): Mirror 3: 0xef0c953d (stale) Mirror 4: 0xef0c953d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ffe): Mirror 3: 0xef0c953d (stale) Mirror 4: 0xef0c953d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ffe: skipped 000113 1783562910.851483 write 0x01b4f3 thru 0x02151a (0x6028 bytes) 000114 1783562910.853225 mapwrite 0x025b11 thru 0x0319bc (0xbeac bytes) 000116 1783562910.908290 trunc from 0x0319bd to 0x002536 (0x2f486 bytes) 000117 1783562910.916300 trunc from 0x002537 to 0x01b2f5 (0x18dbf bytes) 000118 1783562910.923209 mapread 0x001723 thru 0x00954b (0x7e29 bytes) 000119 1783562910.929238 write 0x032aed thru 0x03ffff (0xd513 bytes) 000120 1783562910.930880 mapwrite 0x0103e0 thru 0x01d3df (0xd000 bytes) 000121 1783562910.992113 mapwrite 0x01e10f thru 0x027e5f (0x9d51 bytes) 000122 1783562911.042731 read 0x037afe thru 0x03fd7f (0x8282 bytes) 000123 1783562911.047331 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xcc72e8b6 (stale) Mirror 4: 0x1df78ff4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x65a196f8 Mirror 5: 0x65a196f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000125 1783562911.469432 mapwrite 0x03bca0 thru 0x03ffff (0x4360 bytes) 000126 1783562911.546382 mapwrite 0x035174 thru 0x0367da (0x1667 bytes) 000129 1783562911.572322 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xcc72e8b6 (stale) Mirror 4: 0x1df78ff4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xcc72e8b6 (stale) Mirror 4: 0x1df78ff4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000131 1783562911.721685 trunc from 0x040000 to 0x02703b (0x18fc4 bytes) 000132 1783562911.735769 read 0x0038a7 thru 0x00db3e (0xa298 bytes) 000133 1783562911.741826 mapread 0x004750 thru 0x00f183 (0xaa34 bytes) 000134 1783562911.749296 trunc from 0x02703c to 0x033494 (0xc459 bytes) 000135 1783562911.761630 write 0x013df7 thru 0x015517 (0x1721 bytes) 000136 1783562911.769933 read 0x014f83 thru 0x016d58 (0x1dd6 bytes) 000138 1783562911.784169 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33495): Mirror 4: 0x4661ac9d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33495): Mirror 4: 0x4661ac9d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33495: skipped 000141 1783562911.932552 write 0x02549f thru 0x02bd13 (0x6875 bytes) 000143 1783562911.942969 mapwrite 0x0053cd thru 0x00e41e (0x9052 bytes) 000145 1783562911.985752 mapread 0x0322db thru 0x033494 (0x11ba bytes) 000146 1783562911.991367 mapread 0x01bc04 thru 0x0210b2 (0x54af bytes) 000147 1783562911.998575 trunc from 0x033495 to 0x00bd28 (0x2776c bytes) 000150 1783562912.022872 mapread 0x004569 thru 0x00b4fd (0x6f95 bytes) 000151 1783562912.024375 write 0x0310d4 thru 0x03410f (0x303c bytes) 000152 1783562912.025235 mapwrite 0x018db8 thru 0x0194cf (0x718 bytes) 000153 1783562912.056944 write 0x00f8b4 thru 0x01f070 (0xf7bd bytes) 000154 1783562912.060832 mapwrite 0x005998 thru 0x00d266 (0x78cf bytes) 000156 1783562912.097154 write 0x017279 thru 0x01aa3a (0x37c2 bytes) 000157 1783562912.098272 trunc from 0x034110 to 0x002094 (0x3207b bytes) 000159 1783562912.109666 read 0x001fca thru 0x002094 (0xcb bytes) 000161 1783562912.110819 trunc from 0x002095 to 0x002658 (0x5c4 bytes) 000162 1783562912.121868 mapread 0x001e40 thru 0x002658 (0x819 bytes) 000165 1783562912.124415 read 0x000e3c thru 0x002658 (0x181d bytes) 000167 1783562912.129280 write 0x01e0c0 thru 0x02a8cd (0xc80e bytes) 000170 1783562912.131471 mapwrite 0x030a79 thru 0x03ece6 (0xe26e bytes) 000171 1783562912.204439 MIRROR_RESYNC 2 mirrors 000173 1783562912.447235 mapread 0x00a08e thru 0x010164 (0x60d7 bytes) 000174 1783562912.459556 mapread 0x0040e5 thru 0x00f0f5 (0xb011 bytes) 000176 1783562912.467535 write 0x00cfde thru 0x00d5e5 (0x608 bytes) 000179 1783562912.535958 trunc from 0x03ece7 to 0x0213bb (0x1d92b bytes) 000181 1783562912.573589 read 0x010eb6 thru 0x01c131 (0xb27c bytes) 000182 1783562912.578773 mapwrite 0x0260db thru 0x026a31 (0x957 bytes) 000183 1783562912.616543 read 0x01b54d thru 0x0231fa (0x7cae bytes) 000184 1783562912.625912 write 0x01b6aa thru 0x020e2f (0x5786 bytes) 000186 1783562912.627446 write 0x02b3e7 thru 0x02cff3 (0x1c0d bytes) 000188 1783562912.628918 mapread 0x022478 thru 0x0291dc (0x6d65 bytes) 000189 1783562912.636979 read 0x00fc40 thru 0x0174de (0x789f bytes) 000192 1783562912.638702 mapread 0x016ea1 thru 0x01b2e4 (0x4444 bytes) 000193 1783562912.640568 read 0x02c7f9 thru 0x02cff3 (0x7fb bytes) 000196 1783562912.642105 mapwrite 0x0126ae thru 0x01611c (0x3a6f bytes) 000198 1783562912.669856 trunc from 0x02cff4 to 0x02890d (0x46e6 bytes) 000199 1783562912.682700 write 0x01b29b thru 0x024fca (0x9d30 bytes) 000200 1783562912.683849 trunc from 0x02890e to 0x02dff5 (0x56e8 bytes) 000204 1783562912.695768 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2dff6): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dff6): Mirror 4: 0x14871825 (stale) CRC-32 checksum value for chunk [0, 0x2dff6): Mirror 1: 0x75dcd54f Mirror 5: 0x75dcd54f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dff6: skipped 000205 1783562913.092359 mapwrite 0x02a632 thru 0x031bbd (0x758c bytes) 000208 1783562913.217722 mapread 0x023a02 thru 0x028d06 (0x5305 bytes) 000209 1783562913.226931 MIRROR_RESYNC 3 mirrors 000211 1783562913.638617 read 0x00d351 thru 0x0133c8 (0x6078 bytes) 000212 1783562913.654130 MIRROR_RESYNC 3 mirrors 000213 1783562913.895783 mapread 0x02ef73 thru 0x031bbd (0x2c4b bytes) 000214 1783562913.917402 trunc from 0x031bbe to 0x024fae (0xcc0f bytes) 000215 1783562913.991485 trunc from 0x024faf to 0x014f29 (0x10085 bytes) 000219 1783562914.007933 mapread 0x00d1aa thru 0x00dd03 (0xb5a bytes) 000220 1783562914.016425 mapread 0x000a1f thru 0x00f70f (0xecf1 bytes) 000222 1783562914.027608 mapwrite 0x03f84b thru 0x03ffff (0x7b5 bytes) 000223 1783562914.096028 MIRROR_RESYNC 3 mirrors 000224 1783562914.431478 read 0x012caf thru 0x018998 (0x5cea bytes) 000225 1783562914.440457 trunc from 0x040000 to 0x01c2f2 (0x23d0d bytes) 000227 1783562914.521381 mapread 0x00a571 thru 0x018e43 (0xe8d3 bytes) 000228 1783562914.554598 read 0x00d7ac thru 0x015264 (0x7ab9 bytes) 000230 1783562914.555855 mapwrite 0x0030a1 thru 0x012bc3 (0xfb23 bytes) 000231 1783562914.607828 trunc from 0x01c2f3 to 0x0269ef (0xa6fd bytes) 000233 1783562914.622518 mapread 0x00a7cc thru 0x00e5a6 (0x3ddb bytes) 000234 1783562914.623237 write 0x016126 thru 0x01cbf6 (0x6ad1 bytes) 000235 1783562914.625035 trunc from 0x0269f0 to 0x008356 (0x1e699 bytes) 000236 1783562914.636668 trunc from 0x008357 to 0x03592b (0x2d5d5 bytes) 000240 1783562914.650312 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3592c): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3592c): Mirror 4: 0x8d70a93b (stale) Mirror 5: 0x8d70a93b (stale) CRC-32 checksum value for chunk [0, 0x3592c): Mirror 1: 0xeb8f7443 Mirror 6: 0xeb8f7443 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3592c: skipped 000242 1783562915.078141 mapread 0x02f256 thru 0x031ff6 (0x2da1 bytes) 000243 1783562915.097185 MIRROR_RESYNC 4 mirrors 000244 1783562915.476290 write 0x029b46 thru 0x02f9ce (0x5e89 bytes) 000245 1783562915.525125 mapread 0x00ffeb thru 0x01eb80 (0xeb96 bytes) 000246 1783562915.540663 mapread 0x021874 thru 0x031566 (0xfcf3 bytes) 000247 1783562915.552204 mapwrite 0x0268a1 thru 0x032c0e (0xc36e bytes) 000249 1783562915.592681 trunc from 0x03592c to 0x00896b (0x2cfc0 bytes) 000250 1783562915.622407 mapwrite 0x02ab4d thru 0x02c88a (0x1d3e bytes) truncating to largest ever: 0x3bddb 000251 1783562915.661712 trunc from 0x02c88b to 0x03bdda (0xf550 bytes) 000252 1783562915.671317 read 0x037690 thru 0x03ba5d (0x43ce bytes) 000253 1783562915.676248 mapread 0x005cee thru 0x00992c (0x3c3f bytes) 000257 1783562915.683660 mapread 0x0230c0 thru 0x0306b3 (0xd5f4 bytes) 000259 1783562915.694003 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bddb): Mirror 4: 0xdbdea519 (stale) Mirror 6: 0xdbdea519 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bddb): Mirror 4: 0xdbdea519 (stale) Mirror 6: 0xdbdea519 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bddb: skipped 000261 1783562915.833119 mapwrite 0x0187f9 thru 0x023a4b (0xb253 bytes) 000263 1783562915.879873 write 0x012116 thru 0x012916 (0x801 bytes) 000264 1783562915.884030 trunc from 0x03bddb to 0x0284e6 (0x138f4 bytes) 000267 1783562915.898329 mapwrite 0x00641a thru 0x00e8fb (0x84e2 bytes) 000270 1783562915.948014 write 0x010d7b thru 0x01a2e0 (0x9566 bytes) 000272 1783562915.951372 mapwrite 0x02b61b thru 0x03b589 (0xff6f bytes) 000273 1783562916.018377 read 0x0344be thru 0x03b589 (0x70cc bytes) 000275 1783562916.019354 mapread 0x0354a8 thru 0x03b589 (0x60e2 bytes) 000280 1783562916.021968 write 0x017b02 thru 0x01fc18 (0x8117 bytes) 000282 1783562916.023189 read 0x02bb42 thru 0x034533 (0x89f2 bytes) 000283 1783562916.023621 trunc from 0x03b58a to 0x00dcba (0x2d8cf bytes) 000286 1783562916.038990 trunc from 0x00dcbb to 0x00f56e (0x18b4 bytes) 000287 1783562916.048646 trunc from 0x00f56f to 0x014434 (0x4ec6 bytes) 000289 1783562916.066970 write 0x03eeb2 thru 0x03ffff (0x114e bytes) 000290 1783562916.067865 read 0x0009a4 thru 0x00a9d9 (0xa036 bytes) 000292 1783562916.078394 write 0x01cdf4 thru 0x02b551 (0xe75e bytes) 000294 1783562916.079765 write 0x00614d thru 0x01481f (0xe6d3 bytes) 000296 1783562916.083550 write 0x01a5c1 thru 0x01ce8b (0x28cb bytes) 000297 1783562916.084302 read 0x00f025 thru 0x015cf8 (0x6cd4 bytes) 000298 1783562916.084733 mapread 0x00585f thru 0x005f94 (0x736 bytes) 000299 1783562916.085501 trunc from 0x040000 to 0x0389ad (0x7652 bytes) 000300 1783562916.094117 read 0x0305a5 thru 0x032b76 (0x25d2 bytes) 000301 1783562916.094648 trunc from 0x0389ae to 0x00b9f2 (0x2cfbb bytes) 000302 1783562916.101548 read 0x008945 thru 0x00b9f2 (0x30ae bytes) 000303 1783562916.102106 mapwrite 0x03d26e thru 0x03ffff (0x2d92 bytes) 000305 1783562916.159630 trunc from 0x040000 to 0x0265b3 (0x19a4c bytes) 000306 1783562916.177228 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x265b4): Mirror 6: 0x1961b31a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x265b4): Mirror 6: 0x1961b31a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x265b4: skipped 000307 1783562916.385815 mapread 0x021f2b thru 0x0220e5 (0x1bb bytes) 000312 1783562916.402942 read 0x01a7b3 thru 0x01aedc (0x72a bytes) 000313 1783562916.408423 mapread 0x011a57 thru 0x0167e5 (0x4d8f bytes) 000314 1783562916.420685 trunc from 0x0265b4 to 0x01aa61 (0xbb52 bytes) 000317 1783562916.444401 read 0x007e07 thru 0x01315d (0xb357 bytes) 000320 1783562916.455623 trunc from 0x01aa62 to 0x00df6f (0xcaf2 bytes) 000321 1783562916.486918 mapwrite 0x01f3f0 thru 0x02b0b6 (0xbcc7 bytes) 000323 1783562916.584278 trunc from 0x02b0b7 to 0x016c11 (0x144a5 bytes) 000327 1783562916.605529 read 0x008f81 thru 0x00cc93 (0x3d13 bytes) 000328 1783562916.606216 trunc from 0x016c12 to 0x0257d6 (0xebc5 bytes) 000329 1783562916.622153 read 0x0201db thru 0x0257d6 (0x55fc bytes) 000331 1783562916.640782 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2db:0x0]: Invalid argument (22) 000332 1783562916.739519 MIRROR_EXTEND to 2 mirrors 000333 1783562917.082902 write 0x005eee thru 0x0129c4 (0xcad7 bytes) 000336 1783562917.136500 mapread 0x006fc0 thru 0x016e25 (0xfe66 bytes) 000337 1783562917.160048 write 0x034b08 thru 0x0366ad (0x1ba6 bytes) 000338 1783562917.161417 mapread 0x036103 thru 0x0366ad (0x5ab bytes) 000339 1783562917.162615 trunc from 0x0366ae to 0x01b7f9 (0x1aeb4 bytes) 000342 1783562917.188409 mapread 0x005c24 thru 0x013252 (0xd62f bytes) 000344 1783562917.190252 mapwrite 0x017ed9 thru 0x0221f9 (0xa321 bytes) 000347 1783562917.266860 read 0x01279b thru 0x017d7e (0x55e4 bytes) 000353 1783562917.281689 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2db:0x0]: Invalid argument (22) 000356 1783562917.379270 write 0x017bd0 thru 0x023563 (0xb994 bytes) 000358 1783562917.388268 write 0x026232 thru 0x0340da (0xdea9 bytes) 000359 1783562917.389888 read 0x02d1c4 thru 0x032c56 (0x5a93 bytes) 000366 1783562917.393807 write 0x037164 thru 0x03ffff (0x8e9c bytes) 000367 1783562917.395449 read 0x02be50 thru 0x033527 (0x76d8 bytes) 000369 1783562917.396755 trunc from 0x040000 to 0x0398fa (0x6705 bytes) 000371 1783562917.412970 trunc from 0x0398fb to 0x0137e0 (0x2611a bytes) 000372 1783562917.431444 trunc from 0x0137e1 to 0x0043dc (0xf404 bytes) 000375 1783562917.442428 trunc from 0x0043dd to 0x034d3e (0x30962 bytes) 000376 1783562917.457171 mapwrite 0x03bb22 thru 0x03ffff (0x44de bytes) 000378 1783562917.540815 mapread 0x020910 thru 0x02187a (0xf6b bytes) 000380 1783562917.542015 trunc from 0x040000 to 0x002c01 (0x3d3fe bytes) 000382 1783562917.552178 write 0x00acb0 thru 0x00df87 (0x32d8 bytes) 000384 1783562917.553778 trunc from 0x00df88 to 0x018f80 (0xaff9 bytes) 000385 1783562917.561559 write 0x02c98d thru 0x032876 (0x5eea bytes) 000387 1783562917.563421 write 0x037e21 thru 0x03ffff (0x81df bytes) 000390 1783562917.565718 write 0x00f9f2 thru 0x0169fe (0x700d bytes) 000391 1783562917.572107 write 0x00c842 thru 0x00cffb (0x7ba bytes) 000392 1783562917.572747 mapwrite 0x037fa2 thru 0x03e99d (0x69fc bytes) 000396 1783562917.610118 write 0x036792 thru 0x038de6 (0x2655 bytes) 000399 1783562917.617036 mapread 0x02b1c9 thru 0x02ff36 (0x4d6e bytes) 000400 1783562917.620019 trunc from 0x040000 to 0x01edb9 (0x21246 bytes) 000401 1783562917.630578 trunc from 0x01edba to 0x015a5e (0x935b bytes) 000402 1783562917.639518 mapwrite 0x026c9f thru 0x0296bc (0x2a1e bytes) 000404 1783562917.709656 trunc from 0x0296bd to 0x02ba9f (0x23e3 bytes) 000405 1783562917.733760 MIRROR_EXTEND to 2 mirrors 000407 1783562918.050278 read 0x0222dd thru 0x02ba9f (0x97c3 bytes) 000409 1783562918.063793 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2db:0x0]: Invalid argument (22) 000411 1783562918.197822 read 0x01e410 thru 0x02ba9f (0xd690 bytes) 000412 1783562918.204893 read 0x01b97d thru 0x01f5b5 (0x3c39 bytes) 000413 1783562918.213987 write 0x00efc4 thru 0x01dd17 (0xed54 bytes) 000414 1783562918.233692 write 0x021baf thru 0x025cf1 (0x4143 bytes) 000415 1783562918.243466 write 0x0020bd thru 0x00671e (0x4662 bytes) 000418 1783562918.252317 trunc from 0x02baa0 to 0x009da1 (0x21cfe bytes) 000421 1783562918.265814 read 0x003ce5 thru 0x009da1 (0x60bd bytes) 000424 1783562918.272746 write 0x01499b thru 0x020593 (0xbbf9 bytes) 000425 1783562918.274877 read 0x002469 thru 0x0111de (0xed76 bytes) 000427 1783562918.281705 trunc from 0x020594 to 0x015fd0 (0xa5c3 bytes) 000428 1783562918.295020 trunc from 0x015fd1 to 0x00e8cb (0x7705 bytes) 000430 1783562918.309962 write 0x02b144 thru 0x02e64d (0x350a bytes) 000432 1783562918.311936 trunc from 0x02e64e to 0x034966 (0x6319 bytes) 000434 1783562918.323461 write 0x036dfb thru 0x039174 (0x237a bytes) 000437 1783562918.325929 mapwrite 0x00a09d thru 0x01222e (0x8192 bytes) 000439 1783562918.395318 trunc from 0x039175 to 0x03bb91 (0x2a1d bytes) 000443 1783562918.412426 read 0x002f31 thru 0x003f13 (0xfe3 bytes) truncating to largest ever: 0x3f94a 000444 1783562918.412906 trunc from 0x03bb92 to 0x03f949 (0x3db8 bytes) 000446 1783562918.432950 read 0x027005 thru 0x02ec61 (0x7c5d bytes) 000449 1783562918.436340 mapread 0x000661 thru 0x00df47 (0xd8e7 bytes) 000451 1783562918.445194 mapwrite 0x03c911 thru 0x03d1a5 (0x895 bytes) 000452 1783562918.492833 read 0x013c5c thru 0x016c6f (0x3014 bytes) 000455 1783562918.496798 read 0x01a6ef thru 0x027814 (0xd126 bytes) 000456 1783562918.497384 mapwrite 0x038bc3 thru 0x03b89e (0x2cdc bytes) 000457 1783562918.532825 mapwrite 0x02564c thru 0x033a4f (0xe404 bytes) 000458 1783562918.598302 read 0x0095a9 thru 0x00a693 (0x10eb bytes) 000459 1783562918.600954 mapread 0x01f063 thru 0x02474a (0x56e8 bytes) 000461 1783562918.603176 trunc from 0x03f94a to 0x00b100 (0x34849 bytes) 000462 1783562918.624510 mapwrite 0x039ae8 thru 0x03ffff (0x6518 bytes) 000463 1783562918.688455 trunc from 0x040000 to 0x02ff86 (0x10079 bytes) 000464 1783562918.713487 mapwrite 0x027ef9 thru 0x02c5fd (0x4705 bytes) 000465 1783562918.766200 MIRROR_EXTEND to 2 mirrors 000467 1783562919.183800 read 0x026dc4 thru 0x02b7b8 (0x49f5 bytes) 000471 1783562919.192501 mapread 0x006266 thru 0x0118c7 (0xb662 bytes) 000475 1783562919.220420 mapwrite 0x00f67c thru 0x01d659 (0xdfde bytes) 000476 1783562919.339883 read 0x00f29d thru 0x01588f (0x65f3 bytes) 000477 1783562919.340598 write 0x03b03a thru 0x03ffff (0x4fc6 bytes) 000478 1783562919.342416 mapwrite 0x03bba4 thru 0x03e096 (0x24f3 bytes) 000479 1783562919.389671 read 0x020cdd thru 0x02d72b (0xca4f bytes) 000480 1783562919.401427 mapread 0x021c7d thru 0x026f97 (0x531b bytes) 000483 1783562919.405349 mapread 0x00bc1a thru 0x0193e9 (0xd7d0 bytes) 000485 1783562919.407663 mapwrite 0x0248b0 thru 0x02b508 (0x6c59 bytes) 000487 1783562919.454171 read 0x001338 thru 0x00d83f (0xc508 bytes) 000488 1783562919.455676 mapwrite 0x02e664 thru 0x033725 (0x50c2 bytes) 000489 1783562919.501122 mapread 0x014e42 thru 0x016dab (0x1f6a bytes) 000490 1783562919.511530 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2db:0x0]: Invalid argument (22) 000493 1783562919.606845 mapread 0x02e4a4 thru 0x034533 (0x6090 bytes) 000495 1783562919.619938 trunc from 0x040000 to 0x02103f (0x1efc0 bytes) 000496 1783562919.656933 write 0x007540 thru 0x016f72 (0xfa33 bytes) 000497 1783562919.670491 mapread 0x016ce6 thru 0x02103f (0xa35a bytes) 000498 1783562919.690164 mapwrite 0x028df9 thru 0x030da6 (0x7fae bytes) 000499 1783562919.746198 mapwrite 0x011239 thru 0x01b80b (0xa5d3 bytes) 000502 1783562919.801706 MIRROR_EXTEND to 2 mirrors 000503 1783562920.267523 mapread 0x01fe26 thru 0x025e19 (0x5ff4 bytes) 000505 1783562920.286756 trunc from 0x030da7 to 0x034326 (0x3580 bytes) 000506 1783562920.354530 read 0x01a09f thru 0x01f9f4 (0x5956 bytes) 000507 1783562920.359686 trunc from 0x034327 to 0x007ad0 (0x2c856 bytes) 000508 1783562920.375612 trunc from 0x007ad1 to 0x017365 (0xf895 bytes) 000509 1783562920.388052 mapwrite 0x03e487 thru 0x03ffff (0x1b79 bytes) 000510 1783562920.426573 mapread 0x017aa7 thru 0x01e91f (0x6e79 bytes) 000511 1783562920.433741 read 0x039c48 thru 0x03d3a9 (0x3762 bytes) 000512 1783562920.434365 read 0x02c7fb thru 0x034440 (0x7c46 bytes) 000513 1783562920.440917 mapread 0x03bc69 thru 0x03ffff (0x4397 bytes) 000514 1783562920.441859 mapwrite 0x0141a7 thru 0x017f22 (0x3d7c bytes) 000515 1783562920.463829 mapread 0x02507e thru 0x0300eb (0xb06e bytes) 000516 1783562920.465169 trunc from 0x040000 to 0x02f32d (0x10cd2 bytes) 000517 1783562920.473732 mapwrite 0x03d8e0 thru 0x03ffff (0x2720 bytes) 000518 1783562920.518925 mapwrite 0x032b95 thru 0x03ffff (0xd46b bytes) 000520 1783562920.598287 trunc from 0x040000 to 0x02cef0 (0x1310f bytes) 000522 1783562920.608994 mapwrite 0x03ef4b thru 0x03ffff (0x10b5 bytes) 000523 1783562920.651299 write 0x0342f7 thru 0x03f2dd (0xafe7 bytes) 000524 1783562920.656158 mapread 0x02c02e thru 0x038580 (0xc553 bytes) 000527 1783562920.683680 mapwrite 0x03a709 thru 0x03c2df (0x1bd7 bytes) 000528 1783562920.707731 write 0x0328d5 thru 0x03b4b7 (0x8be3 bytes) 000529 1783562920.708884 write 0x0020ce thru 0x00dd6f (0xbca2 bytes) 000530 1783562920.709951 trunc from 0x040000 to 0x011925 (0x2e6da bytes) 000531 1783562920.719994 write 0x03729d thru 0x03ffff (0x8d63 bytes) 000534 1783562920.724732 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x19da9f7e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7d21cbd0 Mirror 3: 0x7d21cbd0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000536 1783562921.162882 mapwrite 0x00499e thru 0x0077da (0x2e3d bytes) 000537 1783562921.245375 write 0x001c38 thru 0x005a13 (0x3ddc bytes) 000539 1783562921.247603 read 0x00e3ed thru 0x01dc5f (0xf873 bytes) 000541 1783562921.256831 read 0x03ec0d thru 0x03ffff (0x13f3 bytes) 000542 1783562921.265591 read 0x00a929 thru 0x012b91 (0x8269 bytes) 000543 1783562921.266053 mapwrite 0x0342ca thru 0x038308 (0x403f bytes) 000544 1783562921.296187 MIRROR_RESYNC 3 mirrors 000545 1783562921.845154 write 0x018f5e thru 0x0230f3 (0xa196 bytes) 000546 1783562921.923828 write 0x031c01 thru 0x034c50 (0x3050 bytes) 000547 1783562921.940172 write 0x03ff44 thru 0x03ffff (0xbc bytes) 000554 1783562921.948285 mapread 0x034297 thru 0x0392ac (0x5016 bytes) 000555 1783562921.966898 mapwrite 0x01b4e4 thru 0x025ab5 (0xa5d2 bytes) 000557 1783562922.015814 write 0x01bcb5 thru 0x026dc0 (0xb10c bytes) 000558 1783562922.019501 MIRROR_RESYNC 3 mirrors 000559 1783562922.548491 write 0x007627 thru 0x0170c2 (0xfa9c bytes) 000561 1783562922.585919 trunc from 0x040000 to 0x02df71 (0x1208e bytes) 000563 1783562922.600002 MIRROR_RESYNC 3 mirrors 000565 1783562923.044475 read 0x0074d5 thru 0x0158bc (0xe3e8 bytes) 000566 1783562923.053987 trunc from 0x02df72 to 0x005015 (0x28f5c bytes) 000568 1783562923.114452 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x5016): Mirror 3: 0x9f4a5d9b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5016): Mirror 3: 0x9f4a5d9b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5016: skipped 000570 1783562923.294718 read 0x003b48 thru 0x005015 (0x14ce bytes) 000571 1783562923.299665 mapread 0x00372d thru 0x005015 (0x18e9 bytes) 000572 1783562923.300873 trunc from 0x005016 to 0x00c81c (0x7807 bytes) 000573 1783562923.310676 mapread 0x0050fd thru 0x00c81c (0x7720 bytes) 000577 1783562923.331742 MIRROR_RESYNC 2 mirrors 000579 1783562923.679739 write 0x01535c thru 0x023132 (0xddd7 bytes) 000580 1783562923.726691 trunc from 0x023133 to 0x021579 (0x1bb9 bytes) 000581 1783562923.758292 mapread 0x01ff3e thru 0x021477 (0x153a bytes) 000583 1783562923.762647 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2db:0x0]: Invalid argument (22) 000584 1783562923.860216 mapwrite 0x01b170 thru 0x02ab6e (0xf9ff bytes) 000585 1783562923.924819 mapwrite 0x0242a2 thru 0x0269c4 (0x2723 bytes) 000586 1783562923.940588 write 0x029248 thru 0x02ee60 (0x5c19 bytes) 000587 1783562923.941599 write 0x033c34 thru 0x03456a (0x937 bytes) 000589 1783562923.942980 mapwrite 0x005681 thru 0x00e784 (0x9104 bytes) 000590 1783562923.977394 mapread 0x00775e thru 0x014d12 (0xd5b5 bytes) 000591 1783562923.984103 mapwrite 0x010e5d thru 0x015a47 (0x4beb bytes) 000594 1783562924.011606 trunc from 0x03456b to 0x02ae4e (0x971c bytes) 000595 1783562924.020928 write 0x004d6b thru 0x00d4ab (0x8741 bytes) 000596 1783562924.022239 write 0x03734a thru 0x03f8b8 (0x856f bytes) 000598 1783562924.024151 mapread 0x008d4a thru 0x00fedd (0x7194 bytes) 000599 1783562924.025519 read 0x00b7c1 thru 0x00cd46 (0x1586 bytes) 000600 1783562924.026158 mapread 0x01ac2d thru 0x029edd (0xf2b1 bytes) 000601 1783562924.027275 trunc from 0x03f8b9 to 0x026d7d (0x18b3b bytes) 000604 1783562924.040952 mapwrite 0x0042fb thru 0x005edf (0x1be5 bytes) 000606 1783562924.063494 read 0x00c631 thru 0x01b108 (0xead8 bytes) 000607 1783562924.064232 trunc from 0x026d7e to 0x0245b8 (0x27c5 bytes) 000608 1783562924.076007 trunc from 0x0245b9 to 0x02f48b (0xaed3 bytes) 000609 1783562924.093632 mapread 0x00a706 thru 0x01752f (0xce2a bytes) 000611 1783562924.095572 write 0x000e3d thru 0x005eb1 (0x5075 bytes) 000613 1783562924.105443 mapwrite 0x00d0ae thru 0x01c1b4 (0xf107 bytes) 000615 1783562924.142041 mapwrite 0x0374e5 thru 0x03ffff (0x8b1b bytes) 000617 1783562924.192537 read 0x03cd5f thru 0x03ffff (0x32a1 bytes) 000618 1783562924.193320 mapread 0x03a950 thru 0x03ffff (0x56b0 bytes) 000620 1783562924.195196 mapwrite 0x022297 thru 0x0233ab (0x1115 bytes) 000623 1783562924.224308 mapwrite 0x030231 thru 0x037e21 (0x7bf1 bytes) 000624 1783562924.264576 read 0x022453 thru 0x022928 (0x4d6 bytes) 000625 1783562924.265429 mapwrite 0x03dc29 thru 0x03ffff (0x23d7 bytes) 000627 1783562924.298892 mapwrite 0x036472 thru 0x03854c (0x20db bytes) 000628 1783562924.336362 mapread 0x03b5d1 thru 0x03fff8 (0x4a28 bytes) 000629 1783562924.337592 mapwrite 0x021a98 thru 0x02c062 (0xa5cb bytes) 000632 1783562924.377618 mapread 0x003d9c thru 0x00771a (0x397f bytes) 000633 1783562924.378858 mapwrite 0x036ca3 thru 0x03ffff (0x935d bytes) 000634 1783562924.411773 mapread 0x012d22 thru 0x015fab (0x328a bytes) 000635 1783562924.412549 mapwrite 0x010acd thru 0x01f7e8 (0xed1c bytes) 000636 1783562924.441359 mapread 0x01e684 thru 0x026544 (0x7ec1 bytes) 000637 1783562924.442504 mapread 0x02c21f thru 0x035a2d (0x980f bytes) 000638 1783562924.443645 mapread 0x0270fc thru 0x02bdec (0x4cf1 bytes) 000641 1783562924.448383 MIRROR_EXTEND to 2 mirrors 000643 1783562924.696323 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2db:0x0]: Invalid argument (22) 000646 1783562924.807489 trunc from 0x040000 to 0x02f14e (0x10eb1 bytes) 000650 1783562924.831595 MIRROR_EXTEND to 2 mirrors 000653 1783562925.069166 write 0x0134ab thru 0x01ac56 (0x77ac bytes) 000654 1783562925.118042 trunc from 0x02f14f to 0x00b599 (0x23bb5 bytes) 000655 1783562925.138920 read 0x006d2f thru 0x00868f (0x1961 bytes) 000656 1783562925.146233 MIRROR_RESYNC 2 mirrors 000658 1783562925.460472 mapread 0x004448 thru 0x0048a9 (0x462 bytes) 000659 1783562925.463395 read 0x001ef0 thru 0x006493 (0x45a4 bytes) 000660 1783562925.466805 read 0x003a57 thru 0x00b599 (0x7b43 bytes) 000661 1783562925.470873 trunc from 0x00b59a to 0x0319fe (0x26465 bytes) 000662 1783562925.508337 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x319ff): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x319ff): Mirror 2: 0xc83f823 (stale) CRC-32 checksum value for chunk [0, 0x319ff): Mirror 1: 0xc83f823 Mirror 3: 0xc83f823 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x319ff: skipped 000665 1783562925.917416 write 0x000fb4 thru 0x005481 (0x44ce bytes) 000666 1783562925.966939 mapwrite 0x01f90f thru 0x020e54 (0x1546 bytes) 000667 1783562926.009984 read 0x02f80c thru 0x0319fe (0x21f3 bytes) 000669 1783562926.016432 mapwrite 0x019b32 thru 0x01dc5b (0x412a bytes) 000670 1783562926.054377 mapwrite 0x030cbe thru 0x036b56 (0x5e99 bytes) 000671 1783562926.092948 trunc from 0x036b57 to 0x02ed14 (0x7e42 bytes) 000672 1783562926.103426 trunc from 0x02ed15 to 0x02b630 (0x36e4 bytes) 000676 1783562926.112167 trunc from 0x02b631 to 0x02393d (0x7cf3 bytes) 000678 1783562926.126622 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2393e): Mirror 3: 0x956ec014 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2393e): Mirror 3: 0x956ec014 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2393e: skipped 000680 1783562926.271515 mapread 0x01e249 thru 0x02393d (0x56f5 bytes) 000684 1783562926.284615 trunc from 0x02393e to 0x0094a2 (0x1a49b bytes) 000685 1783562926.292836 mapread 0x003e79 thru 0x0094a2 (0x562a bytes) 000686 1783562926.305867 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x94a3): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x94a3): Mirror 3: 0x157a12e (stale) CRC-32 checksum value for chunk [0, 0x94a3): Mirror 1: 0x570e0d38 Mirror 4: 0x570e0d38 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x94a3: skipped 000689 1783562926.669951 mapwrite 0x03778b thru 0x03ffff (0x8875 bytes) 000691 1783562926.738768 read 0x033e5a thru 0x03ffff (0xc1a6 bytes) 000692 1783562926.739502 mapread 0x03b3fd thru 0x03ffff (0x4c03 bytes) 000693 1783562926.741526 mapread 0x00b18e thru 0x00e0c9 (0x2f3c bytes) 000694 1783562926.746704 trunc from 0x040000 to 0x017020 (0x28fdf bytes) 000696 1783562926.756331 trunc from 0x017021 to 0x030211 (0x191f1 bytes) 000697 1783562926.766038 mapwrite 0x01c179 thru 0x023f14 (0x7d9c bytes) 000698 1783562926.792209 read 0x003063 thru 0x00cb0a (0x9aa8 bytes) 000699 1783562926.798586 mapread 0x00a470 thru 0x0188f8 (0xe489 bytes) 000700 1783562926.799742 mapread 0x02de41 thru 0x030211 (0x23d1 bytes) 000701 1783562926.803325 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x30212): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30212): Mirror 3: 0xfa3ef0d4 (stale) Mirror 4: 0xd6554f16 (stale) CRC-32 checksum value for chunk [0, 0x30212): Mirror 1: 0x13ba56bd Mirror 5: 0x13ba56bd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30212: skipped 000702 1783562927.121971 mapread 0x01eb90 thru 0x01f6b5 (0xb26 bytes) 000703 1783562927.126665 trunc from 0x030212 to 0x03bcf7 (0xbae6 bytes) 000704 1783562927.162991 MIRROR_RESYNC 4 mirrors 000706 1783562927.517501 read 0x00a8bc thru 0x00c860 (0x1fa5 bytes) 000707 1783562927.522847 trunc from 0x03bcf8 to 0x02025a (0x1ba9d bytes) 000708 1783562927.557642 read 0x01e4df thru 0x0201cc (0x1cee bytes) 000711 1783562927.571886 MIRROR_RESYNC 4 mirrors 000715 1783562927.905648 mapwrite 0x008899 thru 0x00f880 (0x6fe8 bytes) 000717 1783562927.992230 read 0x0126a6 thru 0x014f8b (0x28e6 bytes) 000718 1783562927.995492 mapread 0x00c0ef thru 0x00ce79 (0xd8b bytes) 000719 1783562927.996476 mapwrite 0x032d88 thru 0x03ffff (0xd278 bytes) 000722 1783562928.055427 read 0x03fbc0 thru 0x03ffff (0x440 bytes) 000723 1783562928.058996 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xad53659f (stale) Mirror 5: 0xad53659f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xad53659f (stale) Mirror 5: 0xad53659f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000724 1783562928.230534 mapread 0x0154fd thru 0x01abc2 (0x56c6 bytes) 000725 1783562928.241486 read 0x019c2e thru 0x01dc6d (0x4040 bytes) 000726 1783562928.242199 write 0x00f37f thru 0x011642 (0x22c4 bytes) 000727 1783562928.248940 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xad53659f (stale) Mirror 5: 0xad53659f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x98b9abe0 Mirror 6: 0x98b9abe0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000729 1783562928.604266 trunc from 0x040000 to 0x00c59e (0x33a61 bytes) 000732 1783562928.641534 trunc from 0x00c59f to 0x006c99 (0x5905 bytes) 000735 1783562928.653848 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6c9a): Mirror 3: 0x4d8bd1c4 (stale) Mirror 6: 0x4d8bd1c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6c9a): Mirror 3: 0x4d8bd1c4 (stale) Mirror 6: 0x4d8bd1c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6c9a: skipped 000736 1783562928.831003 write 0x001c25 thru 0x00ae6a (0x9246 bytes) 000740 1783562928.835290 mapwrite 0x032817 thru 0x038496 (0x5c80 bytes) 000741 1783562928.878891 mapread 0x02cbab thru 0x038496 (0xb8ec bytes) 000742 1783562928.888796 mapwrite 0x00e596 thru 0x01dbea (0xf655 bytes) 000743 1783562928.931007 MIRROR_RESYNC 3 mirrors 000745 1783562929.288617 MIRROR_EXTEND to 4 mirrors 000746 1783562929.547658 write 0x02f7f6 thru 0x03dc96 (0xe4a1 bytes) 000747 1783562929.570741 mapread 0x019af9 thru 0x02618b (0xc693 bytes) 000750 1783562929.583549 read 0x021941 thru 0x02f4ce (0xdb8e bytes) 000751 1783562929.586931 write 0x03fc5d thru 0x03ffff (0x3a3 bytes) 000752 1783562929.592580 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x3e887da5 (stale) Mirror 6: 0x3e887da5 (stale) Mirror 7: 0x3e887da5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2d5f8493 Mirror 8: 0x2d5f8493 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000753 1783562929.968698 mapread 0x030fc2 thru 0x0360fb (0x513a bytes) 000754 1783562929.984475 MIRROR_RESYNC 5 mirrors 000759 1783562930.355002 mapread 0x0109e2 thru 0x01346b (0x2a8a bytes) 000760 1783562930.365116 write 0x033495 thru 0x03ffc2 (0xcb2e bytes) 000762 1783562930.402649 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2d5f8493 (stale) Mirror 6: 0x2d5f8493 (stale) Mirror 7: 0x2d5f8493 (stale) Mirror 8: 0x2d5f8493 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf200a487 Mirror 9: 0xf200a487 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000763 1783562930.815419 mapwrite 0x01e78e thru 0x022d6b (0x45de bytes) 000765 1783562930.888814 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2d5f8493 (stale) Mirror 6: 0x2d5f8493 (stale) Mirror 7: 0x2d5f8493 (stale) Mirror 8: 0x2d5f8493 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2d5f8493 (stale) Mirror 6: 0x2d5f8493 (stale) Mirror 7: 0x2d5f8493 (stale) Mirror 8: 0x2d5f8493 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000766 1783562931.126323 mapread 0x00f532 thru 0x013e4c (0x491b bytes) 000769 1783562931.141489 mapread 0x01e303 thru 0x0267f2 (0x84f0 bytes) 000770 1783562931.147487 mapread 0x032153 thru 0x03cbec (0xaa9a bytes) 000771 1783562931.160244 mapwrite 0x02f290 thru 0x03281f (0x3590 bytes) 000772 1783562931.179593 mapread 0x0147be thru 0x021dfe (0xd641 bytes) 000773 1783562931.180542 mapread 0x028873 thru 0x0302bc (0x7a4a bytes) 000777 1783562931.185192 read 0x00f26d thru 0x01c327 (0xd0bb bytes) 000778 1783562931.185828 read 0x02bdfe thru 0x02d71f (0x1922 bytes) 000779 1783562931.186595 mapwrite 0x03b9cb thru 0x03ffff (0x4635 bytes) 000780 1783562931.206525 mapread 0x00a4c7 thru 0x00d90a (0x3444 bytes) 000784 1783562931.209121 mapread 0x03d758 thru 0x03ffff (0x28a8 bytes) 000785 1783562931.209709 mapread 0x03ff4f thru 0x03ffff (0xb1 bytes) 000787 1783562931.210838 mapwrite 0x038ad1 thru 0x03ffff (0x752f bytes) 000789 1783562931.231263 mapread 0x006933 thru 0x011192 (0xa860 bytes) 000790 1783562931.237187 mapwrite 0x037010 thru 0x03a61b (0x360c bytes) 000793 1783562931.263134 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2d5f8493 (stale) Mirror 6: 0x2d5f8493 (stale) Mirror 7: 0x2d5f8493 (stale) Mirror 8: 0x2d5f8493 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1273c722 Mirror 9: 0x1273c722 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000794 1783562931.589727 write 0x0180c8 thru 0x01f7f8 (0x7731 bytes) 000795 1783562931.628210 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x2d5f8493 (stale) Mirror 7: 0x2d5f8493 (stale) Mirror 8: 0x2d5f8493 (stale) Mirror 9: 0x1273c722 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x2d5f8493 (stale) Mirror 7: 0x2d5f8493 (stale) Mirror 8: 0x2d5f8493 (stale) Mirror 9: 0x1273c722 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000797 1783562931.774254 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x2d5f8493 (stale) Mirror 7: 0x2d5f8493 (stale) Mirror 8: 0x2d5f8493 (stale) Mirror 9: 0x1273c722 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfa92682c Mirror 10: 0xfa92682c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000798 1783562932.125525 mapread 0x0168ac thru 0x017398 (0xaed bytes) 000799 1783562932.133676 mapwrite 0x00d354 thru 0x01a6f8 (0xd3a5 bytes) 000800 1783562932.215665 mapwrite 0x0182bf thru 0x021a63 (0x97a5 bytes) 000802 1783562932.239592 trunc from 0x040000 to 0x0334b3 (0xcb4c bytes) 000803 1783562932.261536 write 0x00f65b thru 0x01b97e (0xc324 bytes) 000804 1783562932.264738 MIRROR_RESYNC 6 mirrors 000806 1783562932.705888 mapread 0x00ce56 thru 0x00fb49 (0x2cf4 bytes) 000808 1783562932.716643 write 0x00d59c thru 0x01476a (0x71cf bytes) 000811 1783562932.757736 MIRROR_RESYNC 6 mirrors 000814 1783562933.211942 mapread 0x02d834 thru 0x031a82 (0x424f bytes) 000818 1783562933.229533 mapwrite 0x005a35 thru 0x006d4b (0x1317 bytes) 000819 1783562933.326175 mapread 0x02342c thru 0x0244d2 (0x10a7 bytes) 000820 1783562933.335533 write 0x03c712 thru 0x03ffff (0x38ee bytes) 000821 1783562933.337534 mapwrite 0x0201d0 thru 0x02579c (0x55cd bytes) 000827 1783562933.371190 mapwrite 0x03b100 thru 0x03ffff (0x4f00 bytes) 000828 1783562933.400581 mapwrite 0x02acb4 thru 0x038b62 (0xdeaf bytes) 000829 1783562933.436546 mapwrite 0x02dcaf thru 0x0320d1 (0x4423 bytes) 000832 1783562933.458753 write 0x01be9a thru 0x01dfc0 (0x2127 bytes) 000834 1783562933.462789 trunc from 0x040000 to 0x01dc67 (0x22398 bytes) 000837 1783562933.476261 read 0x011663 thru 0x01b8fd (0xa29b bytes) 000838 1783562933.481129 mapread 0x006c17 thru 0x00fe7e (0x9268 bytes) 000839 1783562933.482401 trunc from 0x01dc68 to 0x030b7c (0x12f15 bytes) 000842 1783562933.492390 mapread 0x009647 thru 0x010544 (0x6efe bytes) 000843 1783562933.493818 write 0x02f082 thru 0x038cb9 (0x9c38 bytes) 000844 1783562933.498603 read 0x022654 thru 0x030348 (0xdcf5 bytes) 000845 1783562933.513603 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x38cba): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38cba): Mirror 6: 0xa70b5d75 (stale) Mirror 7: 0xa70b5d75 (stale) Mirror 8: 0xa70b5d75 (stale) Mirror 9: 0xa70b5d75 (stale) Mirror 10: 0xa70b5d75 (stale) CRC-32 checksum value for chunk [0, 0x38cba): Mirror 1: 0x4b4e9964 Mirror 11: 0x4b4e9964 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38cba: skipped 000847 1783562934.051155 write 0x038fbf thru 0x03ffff (0x7041 bytes) 000850 1783562934.106485 trunc from 0x040000 to 0x00567f (0x3a980 bytes) 000852 1783562934.148778 mapread 0x002a90 thru 0x00567f (0x2bf0 bytes) 000853 1783562934.163862 read 0x002a9a thru 0x00567f (0x2be6 bytes) 000856 1783562934.165746 mapread 0x003fbc thru 0x00567f (0x16c4 bytes) 000859 1783562934.168548 read 0x000537 thru 0x00567f (0x5149 bytes) 000861 1783562934.174932 write 0x0349ea thru 0x03ffff (0xb616 bytes) 000863 1783562934.177705 write 0x006c62 thru 0x00b98a (0x4d29 bytes) 000864 1783562934.192904 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x801cdeef (stale) Mirror 7: 0x801cdeef (stale) Mirror 8: 0x801cdeef (stale) Mirror 9: 0x801cdeef (stale) Mirror 10: 0x801cdeef (stale) Mirror 11: 0x80601f7d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4ddfefd2 Mirror 12: 0x4ddfefd2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000865 1783562934.732426 mapwrite 0x0239b2 thru 0x0240d0 (0x71f bytes) 000866 1783562934.830247 write 0x026295 thru 0x030862 (0xa5ce bytes) 000867 1783562934.844496 mapwrite 0x028823 thru 0x03874b (0xff29 bytes) 000870 1783562934.886542 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x801cdeef (stale) Mirror 7: 0x801cdeef (stale) Mirror 8: 0x801cdeef (stale) Mirror 9: 0x801cdeef (stale) Mirror 10: 0x801cdeef (stale) Mirror 12: 0x4ddfefd2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x801cdeef (stale) Mirror 7: 0x801cdeef (stale) Mirror 8: 0x801cdeef (stale) Mirror 9: 0x801cdeef (stale) Mirror 10: 0x801cdeef (stale) Mirror 12: 0x4ddfefd2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000873 1783562935.226478 mapwrite 0x0212fd thru 0x02dec2 (0xcbc6 bytes) 000874 1783562935.265230 mapread 0x00a751 thru 0x015df9 (0xb6a9 bytes) 000875 1783562935.284561 write 0x01c5e9 thru 0x0266e3 (0xa0fb bytes) 000876 1783562935.285997 mapwrite 0x02de37 thru 0x03af3e (0xd108 bytes) 000877 1783562935.348279 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x801cdeef (stale) Mirror 7: 0x801cdeef (stale) Mirror 8: 0x801cdeef (stale) Mirror 9: 0x801cdeef (stale) Mirror 10: 0x801cdeef (stale) Mirror 12: 0x4ddfefd2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x535b4055 Mirror 13: 0x535b4055 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000878 1783562935.780956 mapread 0x0008af thru 0x003776 (0x2ec8 bytes) 000879 1783562935.791031 mapwrite 0x00fd81 thru 0x01d073 (0xd2f3 bytes) 000880 1783562935.867745 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x801cdeef (stale) Mirror 7: 0x801cdeef (stale) Mirror 8: 0x801cdeef (stale) Mirror 10: 0x801cdeef (stale) Mirror 12: 0x4ddfefd2 (stale) Mirror 13: 0x535b4055 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x801cdeef (stale) Mirror 7: 0x801cdeef (stale) Mirror 8: 0x801cdeef (stale) Mirror 10: 0x801cdeef (stale) Mirror 12: 0x4ddfefd2 (stale) Mirror 13: 0x535b4055 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000881 1783562936.113600 read 0x02d44b thru 0x03569f (0x8255 bytes) 000882 1783562936.118726 mapread 0x0351c4 thru 0x0398e2 (0x471f bytes) 000883 1783562936.134587 read 0x035d15 thru 0x03b54a (0x5836 bytes) 000884 1783562936.139473 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x801cdeef (stale) Mirror 8: 0x801cdeef (stale) Mirror 10: 0x801cdeef (stale) Mirror 12: 0x4ddfefd2 (stale) Mirror 13: 0x535b4055 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x801cdeef (stale) Mirror 8: 0x801cdeef (stale) Mirror 10: 0x801cdeef (stale) Mirror 12: 0x4ddfefd2 (stale) Mirror 13: 0x535b4055 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000885 1783562936.346102 read 0x0338f8 thru 0x03cad1 (0x91da bytes) 000889 1783562936.351426 mapread 0x03027b thru 0x039221 (0x8fa7 bytes) 000891 1783562936.360576 trunc from 0x040000 to 0x00d98e (0x32671 bytes) 000892 1783562936.372362 mapread 0x002aba thru 0x00d98e (0xaed5 bytes) 000893 1783562936.380578 read 0x008a2f thru 0x00d98e (0x4f60 bytes) 000894 1783562936.381255 mapread 0x00a9e4 thru 0x00d98e (0x2fab bytes) 000895 1783562936.382022 write 0x029add thru 0x02e7a6 (0x4cca bytes) 000896 1783562936.387933 MIRROR_RESYNC 6 mirrors 000898 1783562936.828427 MIRROR_RESYNC 6 mirrors 000899 1783562937.000764 write 0x0114b6 thru 0x018752 (0x729d bytes) 000900 1783562937.043570 mapread 0x027701 thru 0x02e7a6 (0x70a6 bytes) 000901 1783562937.055646 trunc from 0x02e7a7 to 0x005117 (0x2968f bytes) 000903 1783562937.074331 mapread 0x0005c3 thru 0x005117 (0x4b55 bytes) 000904 1783562937.079153 mapwrite 0x031749 thru 0x03489d (0x3155 bytes) 000905 1783562937.115440 mapwrite 0x0169c1 thru 0x020a1a (0xa05a bytes) 000906 1783562937.155774 read 0x0255f6 thru 0x032611 (0xd01c bytes) 000907 1783562937.161384 read 0x02ed8c thru 0x03489d (0x5b12 bytes) 000908 1783562937.162016 write 0x000faa thru 0x007e81 (0x6ed8 bytes) 000910 1783562937.165793 mapread 0x0316f2 thru 0x03489d (0x31ac bytes) 000911 1783562937.166862 read 0x0343b6 thru 0x03489d (0x4e8 bytes) 000913 1783562937.168154 mapread 0x0101f9 thru 0x0188e3 (0x86eb bytes) 000914 1783562937.169355 read 0x01b716 thru 0x020c82 (0x556d bytes) 000915 1783562937.170216 read 0x0174f5 thru 0x02331f (0xbe2b bytes) 000918 1783562937.172185 write 0x030a15 thru 0x0342ee (0x38da bytes) 000919 1783562937.173273 mapread 0x000277 thru 0x00595a (0x56e4 bytes) 000920 1783562937.173989 write 0x0283bf thru 0x02917d (0xdbf bytes) 000923 1783562937.175753 mapread 0x00188c thru 0x010aaf (0xf224 bytes) 000924 1783562937.183024 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3489e): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3489e): Mirror 7: 0x9bc10baf (stale) Mirror 8: 0x9bc10baf (stale) Mirror 10: 0x9bc10baf (stale) Mirror 12: 0x9bc10baf (stale) Mirror 13: 0x9bc10baf (stale) CRC-32 checksum value for chunk [0, 0x3489e): Mirror 1: 0x4c46afb0 Mirror 14: 0x4c46afb0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3489e: skipped 000927 1783562937.526991 trunc from 0x03489e to 0x00a5a5 (0x2a2f8 bytes) 000928 1783562937.567939 write 0x014266 thru 0x022aa1 (0xe83c bytes) 000930 1783562937.570732 trunc from 0x022aa2 to 0x01d31d (0x5784 bytes) 000931 1783562937.579415 trunc from 0x01d31e to 0x0009c3 (0x1c95a bytes) 000932 1783562937.587004 mapwrite 0x02a013 thru 0x034715 (0xa703 bytes) 000935 1783562937.624390 mapread 0x00db59 thru 0x014c9e (0x7146 bytes) 000937 1783562937.632883 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34716): Mirror 7: 0xe97283bb (stale) Mirror 8: 0xe97283bb (stale) Mirror 10: 0xe97283bb (stale) Mirror 12: 0xe97283bb (stale) Mirror 13: 0xe97283bb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34716): Mirror 7: 0xe97283bb (stale) Mirror 8: 0xe97283bb (stale) Mirror 10: 0xe97283bb (stale) Mirror 12: 0xe97283bb (stale) Mirror 13: 0xe97283bb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34716: skipped 000938 1783562937.857568 mapread 0x009b2a thru 0x00b74e (0x1c25 bytes) 000939 1783562937.876605 read 0x0069f8 thru 0x00af61 (0x456a bytes) 000941 1783562937.877815 write 0x02a178 thru 0x02eba9 (0x4a32 bytes) 000942 1783562937.883400 mapwrite 0x01d6cb thru 0x02a21b (0xcb51 bytes) 000943 1783562937.929983 MIRROR_RESYNC 6 mirrors 000944 1783562938.615718 mapread 0x02710a thru 0x02b749 (0x4640 bytes) 000945 1783562938.626666 trunc from 0x034716 to 0x01b946 (0x18dcf bytes) 000946 1783562938.663980 mapread 0x01072c thru 0x01393c (0x3211 bytes) 000947 1783562938.668904 mapread 0x00a764 thru 0x00d575 (0x2e12 bytes) 000948 1783562938.669739 mapread 0x0135e0 thru 0x0163ad (0x2dce bytes) 000949 1783562938.672697 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b947): Mirror 8: 0x627f7c3d (stale) Mirror 10: 0x627f7c3d (stale) Mirror 12: 0x627f7c3d (stale) Mirror 13: 0x627f7c3d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b947): Mirror 8: 0x627f7c3d (stale) Mirror 10: 0x627f7c3d (stale) Mirror 12: 0x627f7c3d (stale) Mirror 13: 0x627f7c3d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b947: skipped 000950 1783562938.943052 write 0x0197ea thru 0x01f2a3 (0x5aba bytes) 000954 1783562938.948055 mapread 0x00d28f thru 0x011657 (0x43c9 bytes) 000955 1783562938.959665 read 0x01ab7d thru 0x01f2a3 (0x4727 bytes) 000958 1783562938.961024 read 0x00a879 thru 0x00ef91 (0x4719 bytes) 000959 1783562938.962749 trunc from 0x01f2a4 to 0x001890 (0x1da13 bytes) 000960 1783562938.973701 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1891): Mirror 8: 0x36a7d555 (stale) Mirror 10: 0x36a7d555 (stale) Mirror 12: 0x36a7d555 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1891): Mirror 8: 0x36a7d555 (stale) Mirror 10: 0x36a7d555 (stale) Mirror 12: 0x36a7d555 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1891: skipped 000962 1783562939.171966 write 0x004e1f thru 0x013945 (0xeb27 bytes) 000964 1783562939.175671 mapread 0x005180 thru 0x010a73 (0xb8f4 bytes) 000965 1783562939.177238 write 0x00f674 thru 0x018205 (0x8b92 bytes) 000966 1783562939.190677 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18206): Mirror 8: 0x235ea106 (stale) Mirror 12: 0x235ea106 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18206): Mirror 8: 0x235ea106 (stale) Mirror 12: 0x235ea106 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18206: skipped 000968 1783562939.404165 write 0x02ce18 thru 0x03791b (0xab04 bytes) 000972 1783562939.407490 mapread 0x006a68 thru 0x007c2f (0x11c8 bytes) 000973 1783562939.413734 write 0x015121 thru 0x01f924 (0xa804 bytes) 000974 1783562939.422666 mapread 0x033b58 thru 0x03477e (0xc27 bytes) 000976 1783562939.424521 mapread 0x03383c thru 0x036e52 (0x3617 bytes) 000977 1783562939.425667 trunc from 0x03791c to 0x0356e9 (0x2232 bytes) 000978 1783562939.435645 mapread 0x029c92 thru 0x0323f8 (0x8767 bytes) 000979 1783562939.447843 read 0x02da80 thru 0x0356e9 (0x7c6a bytes) 000980 1783562939.448749 mapread 0x02b8b7 thru 0x02fc0d (0x4357 bytes) 000982 1783562939.455478 MIRROR_RESYNC 3 mirrors 000983 1783562939.940885 write 0x014c85 thru 0x01ad43 (0x60bf bytes) 000984 1783562940.011024 write 0x019d98 thru 0x01bc0e (0x1e77 bytes) 000986 1783562940.019673 MIRROR_RESYNC 3 mirrors 000987 1783562940.508717 read 0x001ce0 thru 0x010abd (0xedde bytes) 000988 1783562940.520603 read 0x020bb0 thru 0x02fd27 (0xf178 bytes) 000989 1783562940.530761 mapread 0x01fd32 thru 0x029f71 (0xa240 bytes) 000991 1783562940.532183 write 0x00675e thru 0x01302e (0xc8d1 bytes) 000993 1783562940.576623 write 0x019a60 thru 0x01d145 (0x36e6 bytes) 000996 1783562940.585738 mapwrite 0x03392b thru 0x03aaa9 (0x717f bytes) 000997 1783562940.652312 mapwrite 0x00dcba thru 0x011af6 (0x3e3d bytes) 000998 1783562940.685950 mapwrite 0x021bb8 thru 0x022503 (0x94c bytes) 000999 1783562940.726660 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2db:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3aaaa): Mirror 12: 0xc4f15ac2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aaaa): Mirror 12: 0xc4f15ac2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aaaa: skipped 001000 1783562940.920482 mapread 0x035966 thru 0x03aaa9 (0x5144 bytes) All operations completed A-OK! PASS 70a (40s) == sanity-flr test 71: check mirror extend parent fid ==== 22:09:03 (1783562943) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 72: check append on FLR file ========== 22:09:05 (1783562945) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x487:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104906 s, 20.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4a4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x487:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102079 s, 10.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4a4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x488:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x487:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4a4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x488:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x487:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4a5:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (3s) == sanity-flr test 200a: stress test ===================== 22:09:09 (1783562949) Starting client: oleg225-client.virtnet: -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre2 Starting client: oleg225-client.virtnet: -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 1775136 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5052672 .. Extending file size to 9208128 .. Extending file size to 1837184 .. Extending file size to 8266720 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8612064 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57624 57625 57627 57629 57632 done 192.168.202.125@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg225-client.virtnet /mnt/lustre2 (opts:) 192.168.202.125@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg225-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (67s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 22:10:16 (1783563016) Starting client: oleg225-client.virtnet: -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre2 Starting client: oleg225-client.virtnet: -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4492896 .. Extending file size to 575840 .. Extending file size to 1453600 .. Extending file size to 8221024 .. done Extending file size to 7852544 .. Extending file size to 8221312 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8462240 .. Extending file size to 8303392 .. Extending file size to 8468736 .. Extending file size to 8917312 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8459232 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8942720 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8728928 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9165344 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8945376 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8987424 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 59953 59954 59956 59958 59960 failed 192.168.202.125@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg225-client.virtnet /mnt/lustre2 (opts:) 192.168.202.125@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg225-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (67s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 22:11:23 (1783563083) Starting client: oleg225-client.virtnet: -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2944 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2944 2205440 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0839434 s, 25.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.149394 s, 14.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x4750abbb (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.125@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg225-client.virtnet /mnt/lustre2 (opts:) PASS 200c (8s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 22:11:31 (1783563091) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0438086 s, 47.9 MB/s PASS 202 (4s) == sanity-flr test 203: mirror file preserve mirror ID === 22:11:35 (1783563095) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x31f:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x49e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x49e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x49f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x4ba:0x0] } PASS 203 (4s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 22:11:39 (1783563099) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0786947 s, 26.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4a0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4bb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4bc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (4s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 22:11:43 (1783563103) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0826276 s, 25.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4a1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4a2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4bd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4a1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4a2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4bd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4be:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 22:11:46 (1783563106) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.074756 s, 28.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4bf:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4bf:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c0:0x0] } PASS 204c (26s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 22:12:12 (1783563132) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg225-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524573 s, 20.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4a4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4a4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4a5:0x0] } PASS 204d (48s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 22:13:00 (1783563180) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg225-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0810282 s, 25.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4a6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4a6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (45s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 22:13:45 (1783563225) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg225-server mds-ost sync done. Creating new pool test_204f oleg225-server: Pool lustre.test_204f created Adding targets to pool oleg225-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg225-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0568676 s, 36.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4a7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4a7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg225-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg225-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg225-server: Pool lustre.test_204f destroyed PASS 204f (57s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 22:14:42 (1783563282) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 22:14:44 (1783563284) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 22:14:47 (1783563287) oleg225-server: Pool lustre.test_206 created Adding targets to pool oleg225-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg225-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg225-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg225-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg225-server: Pool lustre.test_206 destroyed PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 22:15:01 (1783563301) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026593 s, 39.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129126 s, 81.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x3dea73d1 Mirror 3: 0x3dea73d1 PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 22:15:04 (1783563304) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 22:15:06 (1783563306) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 22:15:07 (1783563307) Starting client: oleg225-client.virtnet: -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0299169 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1700000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x19a0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x19a0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x19a0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1240000 Read this (4 bytes): BBBB 192.168.202.125@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg225-client.virtnet /mnt/lustre2 (opts:) PASS 209a (18s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 22:15:25 (1783563325) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00397294 s, 2.1 MB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 22:15:28 (1783563328) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200406 s, 52.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x344 lmm_fid: [0x200000404:0x344:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x344 lmm_fid: [0x200000404:0x344:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4b1:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834756 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-08 22:15:29.000000000 -0400 Modify: 2026-07-08 22:15:29.000000000 -0400 Change: 2026-07-08 22:15:30.000000000 -0400 Birth: 2026-07-08 22:15:28.000000000 -0400 PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 22:15:32 (1783563332) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 22:15:33 (1783563333) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.183083 s, 224 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0162609 s, 252 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 8 22:15 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x346:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 8 22:15 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 22:15:36 (1783563336) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (3s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 22:15:39 (1783563339) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (3s) == sanity-flr test complete, duration 1019 sec =========== 22:15:42 (1783563342) === sanity-flr: start cleanup 22:15:42 (1783563342) === === sanity-flr: finish cleanup 22:15:45 (1783563345) ===