-----============= acceptance-small: sanity-flr ============----- Fri Sep 11 06:01:39 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg601-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 06:01:54 (1789120914) === oleg601-client.virtnet: executing check_config_client /mnt/lustre oleg601-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg601-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b8a61098800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b8a61098800.idle_timeout=debug disable quota as required oleg601-server: oleg601-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Setting lod.lustre-MDT0000-mdtlov.enable_compr_rotational from 0 to 1 Waiting 90s for '1' Updated after 2s: want '1' got '1' === sanity-flr: finish setup 06:02:34 (1789120954) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7657] == sanity-flr test 0a: lfs mirror create with -N option == 06:02:37 (1789120957) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 06:02:52 (1789120972) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 06:02:55 (1789120975) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 06:02:59 (1789120979) PASS 0d (20s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 06:03:19 (1789120999) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 06:03:22 (1789121002) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 06:03:24 (1789121004) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9b8a61098800.stats=clear osc.lustre-OST0001-osc-ffff9b8a61098800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 06:03:31 (1789121011) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 06:03:40 (1789121020) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 06:03:48 (1789121028) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 06:03:51 (1789121031) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (6s) == sanity-flr test 1: create components with setstripe options ========================================================== 06:03:57 (1789121037) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 06:04:05 (1789121045) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 06:04:13 (1789121053) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 06:04:17 (1789121057) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 06:04:24 (1789121064) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 06:04:34 (1789121074) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 06:04:40 (1789121080) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.38917 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129809 s, 8.1 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 06:04:49 (1789121089) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.182616 s, 57.4 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 06:04:57 (1789121097) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 06:05:09 (1789121109) Stopping /mnt/lustre-ost1 (opts:) on oleg601-server oleg601-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b8a61098800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b8a61098800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg601-server: oleg601-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg601-client: oleg601-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg601-server: oleg601-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg601-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg601-server oleg601-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b8a61098800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b8a61098800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg601-server: oleg601-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg601-client: oleg601-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg601-server: oleg601-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg601-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (64s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 06:06:13 (1789121173) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.520766 s, 16.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg601-server oleg601-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b8a61098800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b8a61098800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg601-server: oleg601-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg601-client: oleg601-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg601-server: oleg601-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg601-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (34s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 06:06:48 (1789121208) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg601-server oleg601-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b8a61098800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b8a61098800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg601-server: oleg601-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg601-client: oleg601-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg601-server: oleg601-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg601-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg601-server oleg601-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b8a61098800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b8a61098800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg601-server: oleg601-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg601-client: oleg601-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg601-server: oleg601-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg601-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (70s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 06:07:57 (1789121277) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.79359 s, 10.9 MB/s -rw-r--r-- 1 root root 52428800 Sep 11 06:08 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.873s user 0m0.000s sys 0m0.190s Stopping /mnt/lustre-ost1 (opts:) on oleg601-server oleg601-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b8a61098800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b8a61098800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.110s user 0m0.000s sys 0m0.195s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg601-server: oleg601-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg601-client: oleg601-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg601-server: oleg601-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg601-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg601-server oleg601-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b8a61098800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b8a61098800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.031s user 0m0.001s sys 0m0.188s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg601-server: oleg601-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg601-client: oleg601-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg601-server: oleg601-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg601-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg601-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b8a61098800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b8a61098800.ost_server_uuid in FULL state after 0 sec PASS 33b (84s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 06:09:21 (1789121361) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 06:09:24 (1789121364) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 06:09:27 (1789121367) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 06:09:30 (1789121370) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 06:09:39 (1789121379) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.869389 s, 14.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 06:09:51 (1789121391) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380005 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225902 s, 46.4 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 06:09:58 (1789121398) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 06:10:06 (1789121406) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000687965 s, 1.5 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00142509 s, 719 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00110108 s, 930 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000758086 s, 1.4 MB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 06:10:16 (1789121416) create test files with size 15740096 .. bb6c1633a3e7af8fa4c73f88771a75c5 - c3af1701086c0a1b0b93d8cbe0607eca - febf50f3e5259b034358fcd05b9ad3ef - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg601-client.virtnet /mnt/lustre (opts:) Starting client: oleg601-client.virtnet: -o user_xattr,flock 192.168.206.101@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 06:10:39 (1789121439) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 06:10:54 (1789121454) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 06:11:01 (1789121461) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100221 s, 10.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251037 s, 41.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (7s) == sanity-flr test 41: lfs mirror resync check =========== 06:11:08 (1789121468) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.12601 s, 16.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.28335 s, 14.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.227999 s, 9.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 06:11:22 (1789121482) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 06:11:25 (1789121485) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 06:11:28 (1789121488) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358502 s, 29.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 43c: read prefer uncompressed mirror == 06:11:37 (1789121497) llite.lustre-ffff9b8a575e3000.enable_compression=1 fail_loc=0x1425 fail_val=-1 PASS 43c (9s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 06:11:46 (1789121506) llite.lustre-ffff9b8a575e3000.enable_compression=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792488 s, 13.2 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272312 s, 38.5 MB/s **verify components PASS 43d (8s) == sanity-flr test 44a: lfs mirror split check =========== 06:11:54 (1789121514) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 06:11:57 (1789121517) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcc:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcc:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (6s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 06:12:04 (1789121524) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.219579 s, 47.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x1f:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg601-client.virtnet /mnt/lustre (opts:) Starting client: oleg601-client.virtnet: -o user_xattr,flock 192.168.206.101@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 06:12:14 (1789121534) llite.lustre-ffff9b8a60181800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.131114 s, 80.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg601-client.virtnet /mnt/lustre (opts:) Starting client: oleg601-client.virtnet: -o user_xattr,flock 192.168.206.101@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9b8a451a5000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 06:12:26 (1789121546) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (36s) == sanity-flr test 46: Verify setstripe --copy option ==== 06:13:02 (1789121582) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (31s) == sanity-flr test 47: Verify mirror obj alloc =========== 06:13:33 (1789121613) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 06:13:37 (1789121617) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.477553 s, 6.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.31876 s, 9.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x7d4e403f (nosync,stale) Mirror 2: 0x7d4e403f (nosync,stale) ** verify mirror 2 doesn't change original checksum: faff6d6259ef8e054c8fa03faf12b0b9 - mirror 1 checksum: 9c3ee52c271f1ef3fe2475d792a558b3 - mirror 2 checksum: faff6d6259ef8e054c8fa03faf12b0b9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-11T06:13:43-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-11T06:13:43-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe2:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: faff6d6259ef8e054c8fa03faf12b0b9 - mirror 1 checksum: 9c3ee52c271f1ef3fe2475d792a558b3 - mirror 2 checksum: 9c3ee52c271f1ef3fe2475d792a558b3 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-11T06:13:43-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-11T06:13:43-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe2:0x0] } PASS 48 (16s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 06:13:55 (1789121635) Failing mds1 on oleg601-server Stopping /mnt/lustre-mds1 (opts:) on oleg601-server 06:14:02 (1789121642) shut down facet: mds1 facet_host: oleg601-server facet_failover_host: oleg601-server Failover mds1 to oleg601-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg601-server: oleg601-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg601-client: oleg601-server: ssh exited with exit code 1 Started lustre-MDT0000 06:14:34 (1789121674) targets are mounted 06:14:34 (1789121674) facet_failover done oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312311 s, 33.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe3:0x0] } PASS 50A (60s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 06:14:55 (1789121695) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 06:14:59 (1789121699) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 06:15:04 (1789121704) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0146671 s, 1.1 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 06:15:11 (1789121711) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 06:15:14 (1789121714) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00744963 s, 550 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0636561 s, 64.3 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 06:15:22 (1789121722) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 06:15:46 (1789121746) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 06:16:15 (1789121775) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 06:16:40 (1789121800) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 06:17:07 (1789121827) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 06:17:15 (1789121835) mirror create pid 55792 mirror split pid 55794 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 55792 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 55794 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 06:18:23 (1789121903) Seed set to 1905 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1789121905.550304 write 0x01c2a5 thru 0x0236f9 (0x7455 bytes) 000002 1789121905.567375 write 0x032aa7 thru 0x039c2f (0x7189 bytes) 000004 1789121905.570440 write 0x02cb40 thru 0x031c50 (0x5111 bytes) 000008 1789121905.590929 mapwrite 0x00e4e2 thru 0x019c18 (0xb737 bytes) 000010 1789121905.779295 mapwrite 0x032a73 thru 0x032cfa (0x288 bytes) truncating to largest ever: 0x23c1 000015 1789121905.871145 trunc from 0x039c30 to 0x0023c0 (0x3786f bytes) 000017 1789121905.895828 read 0x000573 thru 0x0023c0 (0x1e4e bytes) 000019 1789121905.906991 MIRROR_EXTEND to 2 mirrors 000022 1789121906.684998 mapread 0x002350 thru 0x0023c0 (0x71 bytes) 000024 1789121906.690990 mapread 0x002312 thru 0x0023c0 (0xaf bytes) 000028 1789121906.692510 write 0x01ae0c thru 0x022717 (0x790c bytes) 000029 1789121906.762485 write 0x012e56 thru 0x01b88f (0x8a3a bytes) 000030 1789121906.770518 write 0x01c34e thru 0x029567 (0xd21a bytes) 000031 1789121906.772092 read 0x0034f7 thru 0x00868b (0x5195 bytes) 000032 1789121906.778649 read 0x007d1d thru 0x00c5cd (0x48b1 bytes) 000035 1789121906.788897 read 0x024b3f thru 0x028e11 (0x42d3 bytes) 000036 1789121906.789695 mapread 0x0012db thru 0x010aeb (0xf811 bytes) truncating to largest ever: 0x82b7 000037 1789121906.791283 trunc from 0x029568 to 0x0082b6 (0x212b1 bytes) 000038 1789121906.835406 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) 000040 1789121907.066535 mapread 0x0074f4 thru 0x0082b6 (0xdc3 bytes) 000041 1789121907.087093 write 0x03be97 thru 0x03ffff (0x4169 bytes) 000043 1789121907.089151 read 0x027160 thru 0x02c980 (0x5821 bytes) 000044 1789121907.113895 read 0x02e2d2 thru 0x03173f (0x346e bytes) 000047 1789121907.132370 read 0x03a5f1 thru 0x03edc3 (0x47d3 bytes) 000048 1789121907.134209 read 0x03aa85 thru 0x03ffff (0x557b bytes) 000049 1789121907.135018 read 0x0025a2 thru 0x00a570 (0x7fcf bytes) 000050 1789121907.135735 read 0x019c12 thru 0x01fbef (0x5fde bytes) 000052 1789121907.147550 MIRROR_EXTEND to 2 mirrors 000054 1789121908.622859 MIRROR_EXTEND to 3 mirrors 000055 1789121909.781804 mapwrite 0x00665b thru 0x0096b9 (0x305f bytes) truncating to largest ever: 0x3948f 000057 1789121909.950529 trunc from 0x040000 to 0x03948e (0x6b71 bytes) 000059 1789121909.984018 trunc from 0x03948f to 0x033770 (0x5d1e bytes) 000063 1789121910.014523 write 0x024a10 thru 0x03086b (0xbe5c bytes) 000064 1789121910.026231 trunc from 0x033771 to 0x020490 (0x132e0 bytes) 000066 1789121910.044220 write 0x002dd9 thru 0x008c7b (0x5ea3 bytes) 000067 1789121910.045467 read 0x00c1d8 thru 0x01b13b (0xef64 bytes) 000070 1789121910.058559 trunc from 0x020491 to 0x00af12 (0x1557e bytes) 000071 1789121910.082977 mapread 0x001510 thru 0x0067c7 (0x52b8 bytes) 000072 1789121910.087023 mapwrite 0x018190 thru 0x01d72b (0x559c bytes) 000073 1789121910.185023 mapwrite 0x013507 thru 0x02059b (0xd095 bytes) 000075 1789121910.305670 write 0x032cac thru 0x03f091 (0xc3e6 bytes) 000077 1789121910.319100 trunc from 0x03f092 to 0x0074c7 (0x37bca bytes) 000078 1789121910.355037 trunc from 0x0074c8 to 0x018cf0 (0x11829 bytes) 000080 1789121910.389846 mapread 0x00b4eb thru 0x00c165 (0xc7b bytes) 000084 1789121910.421319 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18cf1): Mirror 3: 0x7620c618 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18cf1): Mirror 3: 0x7620c618 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18cf1: skipped 000085 1789121910.871576 mapwrite 0x0371ec thru 0x03ffff (0x8e14 bytes) 000086 1789121910.997440 mapwrite 0x02b0e3 thru 0x035a36 (0xa954 bytes) 000087 1789121911.119551 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) 000088 1789121911.451838 write 0x02bd9d thru 0x030eaf (0x5113 bytes) 000091 1789121911.479376 write 0x002689 thru 0x00dee1 (0xb859 bytes) 000092 1789121911.509007 read 0x01f0ed thru 0x02b420 (0xc334 bytes) 000093 1789121911.531874 read 0x00fc15 thru 0x01dad4 (0xdec0 bytes) 000094 1789121911.546705 write 0x032c11 thru 0x03da88 (0xae78 bytes) 000098 1789121911.552836 read 0x001ba2 thru 0x0065bc (0x4a1b bytes) 000100 1789121911.555204 write 0x008583 thru 0x015ba6 (0xd624 bytes) truncating to largest ever: 0x3d0de 000102 1789121911.559521 trunc from 0x040000 to 0x03d0dd (0x2f22 bytes) 000103 1789121911.582017 mapwrite 0x008a8a thru 0x01776d (0xece4 bytes) 000108 1789121911.662883 mapwrite 0x005c82 thru 0x006be2 (0xf61 bytes) 000111 1789121911.787337 read 0x0214ed thru 0x026980 (0x5494 bytes) 000112 1789121911.793314 read 0x03cbec thru 0x03d0dd (0x4f2 bytes) 000113 1789121911.794826 mapwrite 0x014f4b thru 0x021db9 (0xce6f bytes) 000114 1789121911.882167 mapread 0x007af7 thru 0x00df2f (0x6439 bytes) 000115 1789121911.883207 read 0x007332 thru 0x011667 (0xa336 bytes) 000116 1789121911.883813 read 0x02a4c3 thru 0x032bb2 (0x86f0 bytes) 000118 1789121911.886673 read 0x039b75 thru 0x03d0dd (0x3569 bytes) 000119 1789121911.887852 write 0x019cd8 thru 0x02950e (0xf837 bytes) 000121 1789121911.893725 mapread 0x019904 thru 0x01f465 (0x5b62 bytes) 000122 1789121911.896157 write 0x03ae8f thru 0x03ffff (0x5171 bytes) 000123 1789121911.897603 mapread 0x009005 thru 0x00c3eb (0x33e7 bytes) 000124 1789121911.904840 mapread 0x003434 thru 0x003b7e (0x74b bytes) 000126 1789121911.907276 write 0x01f3da thru 0x02cd3f (0xd966 bytes) 000127 1789121911.917401 MIRROR_EXTEND to 2 mirrors 000128 1789121913.071608 trunc from 0x040000 to 0x0060f4 (0x39f0b bytes) 000133 1789121913.216938 trunc from 0x0060f5 to 0x0118b1 (0xb7bd bytes) 000134 1789121913.262762 mapwrite 0x016f6d thru 0x025368 (0xe3fc bytes) 000136 1789121913.366733 write 0x005922 thru 0x0086d1 (0x2db0 bytes) 000137 1789121913.384173 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x25369): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25369): Mirror 2: 0xc8dbba01 (stale) CRC-32 checksum value for chunk [0, 0x25369): Mirror 1: 0x726f38 Mirror 3: 0x726f38 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25369: skipped 000138 1789121914.953517 mapwrite 0x036b98 thru 0x03ffff (0x9468 bytes) 000143 1789121915.149887 read 0x00b154 thru 0x012d50 (0x7bfd bytes) truncating to largest ever: 0x3d30a 000145 1789121915.182327 trunc from 0x040000 to 0x03d309 (0x2cf6 bytes) 000146 1789121915.237435 mapwrite 0x036cde thru 0x037022 (0x345 bytes) 000147 1789121915.376072 trunc from 0x03d30a to 0x000da9 (0x3c560 bytes) 000148 1789121915.432477 write 0x01b575 thru 0x023b1c (0x85a8 bytes) 000149 1789121915.433940 mapread 0x0001f2 thru 0x008ae8 (0x88f7 bytes) 000151 1789121915.469062 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23b1d): Mirror 3: 0xe8abb7e8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23b1d): Mirror 3: 0xe8abb7e8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23b1d: skipped 000152 1789121916.002665 trunc from 0x023b1d to 0x007aa8 (0x1c074 bytes) 000153 1789121916.042328 trunc from 0x007aa9 to 0x0326ee (0x2ac46 bytes) 000154 1789121916.065847 write 0x010966 thru 0x020382 (0xfa1d bytes) 000155 1789121916.084017 read 0x01559a thru 0x02499e (0xf405 bytes) 000158 1789121916.100571 read 0x022bd0 thru 0x031850 (0xec81 bytes) 000160 1789121916.129604 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) 000161 1789121916.352171 read 0x020a67 thru 0x023074 (0x260e bytes) 000163 1789121916.364597 read 0x0318f0 thru 0x0326ee (0xdff bytes) 000165 1789121916.411991 MIRROR_EXTEND to 2 mirrors 000166 1789121917.973809 MIRROR_EXTEND to 3 mirrors 000168 1789121918.816174 mapread 0x023b89 thru 0x0251ef (0x1667 bytes) 000170 1789121918.837200 mapwrite 0x01fb15 thru 0x022361 (0x284d bytes) 000173 1789121919.029243 read 0x01a458 thru 0x0255a2 (0xb14b bytes) 000174 1789121919.034474 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x326ef): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x326ef): Mirror 2: 0x67df7aa3 (stale) Mirror 3: 0x67df7aa3 (stale) CRC-32 checksum value for chunk [0, 0x326ef): Mirror 1: 0x6ee27d4b Mirror 4: 0x6ee27d4b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x326ef: skipped 000175 1789121920.081936 write 0x018e4e thru 0x023ebe (0xb071 bytes) 000177 1789121920.181989 read 0x02b319 thru 0x0326ee (0x73d6 bytes) 000178 1789121920.190676 write 0x038195 thru 0x03ffff (0x7e6b bytes) 000180 1789121920.199158 write 0x034d50 thru 0x03ffff (0xb2b0 bytes) 000181 1789121920.208801 read 0x0017c7 thru 0x00233b (0xb75 bytes) 000182 1789121920.217843 mapwrite 0x0274f3 thru 0x029f03 (0x2a11 bytes) 000183 1789121920.308416 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf72becd8 (stale) Mirror 4: 0xb153566d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf72becd8 (stale) Mirror 4: 0xb153566d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000186 1789121920.777064 mapread 0x02c1da thru 0x035a84 (0x98ab bytes) 000187 1789121920.811830 write 0x0164e1 thru 0x0234a8 (0xcfc8 bytes) 000188 1789121920.823789 write 0x01e8b8 thru 0x02916a (0xa8b3 bytes) 000189 1789121920.825401 write 0x003ea9 thru 0x009fb1 (0x6109 bytes) 000190 1789121920.838576 mapwrite 0x01e98a thru 0x022ac2 (0x4139 bytes) 000192 1789121920.966643 write 0x02f84c thru 0x03f5d5 (0xfd8a bytes) 000194 1789121920.997699 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf72becd8 (stale) Mirror 4: 0xb153566d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbb007e48 Mirror 5: 0xbb007e48 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000197 1789121922.405070 mapwrite 0x017404 thru 0x020557 (0x9154 bytes) 000199 1789121922.600338 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf72becd8 (stale) Mirror 4: 0xb153566d (stale) Mirror 5: 0xbb007e48 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4a528830 Mirror 6: 0x4a528830 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000200 1789121924.149773 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf72becd8 (stale) Mirror 4: 0xb153566d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf72becd8 (stale) Mirror 4: 0xb153566d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4a528830 Mirror 6: 0x4a528830 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000210 1789121924.973782 write 0x00f95b thru 0x018aa9 (0x914f bytes) 000212 1789121925.209183 trunc from 0x040000 to 0x032ca5 (0xd35a bytes) 000213 1789121925.255307 read 0x016b4f thru 0x01ac30 (0x40e2 bytes) 000214 1789121925.263497 mapwrite 0x015058 thru 0x01e98a (0x9933 bytes) 000215 1789121925.382235 read 0x016eb9 thru 0x025735 (0xe87d bytes) 000216 1789121925.383761 write 0x009b76 thru 0x012719 (0x8ba4 bytes) 000217 1789121925.400962 write 0x030e74 thru 0x0332f9 (0x2486 bytes) 000218 1789121925.401935 mapwrite 0x034ada thru 0x03dcb7 (0x91de bytes) 000220 1789121925.603144 write 0x029775 thru 0x02f63c (0x5ec8 bytes) 000221 1789121925.604265 mapread 0x026a51 thru 0x02d9e2 (0x6f92 bytes) 000223 1789121925.607762 trunc from 0x03dcb8 to 0x0152ae (0x28a09 bytes) 000225 1789121925.652684 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x152af): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x152af): Mirror 3: 0x6c727c46 (stale) Mirror 4: 0x6c727c46 (stale) Mirror 6: 0x198b4a22 (stale) CRC-32 checksum value for chunk [0, 0x152af): Mirror 1: 0x9c2ecf22 Mirror 7: 0x9c2ecf22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x152af: skipped 000226 1789121926.997222 write 0x03ad0f thru 0x03ffff (0x52f1 bytes) 000227 1789121927.096036 mapread 0x018ebd thru 0x01eaae (0x5bf2 bytes) 000229 1789121927.121260 mapwrite 0x00bf2c thru 0x00eebf (0x2f94 bytes) 000231 1789121927.201252 write 0x03f71b thru 0x03ffff (0x8e5 bytes) 000232 1789121927.202559 read 0x00554b thru 0x00a40a (0x4ec0 bytes) 000234 1789121927.203872 write 0x00c56e thru 0x018236 (0xbcc9 bytes) 000236 1789121927.206042 mapwrite 0x035b5d thru 0x03f1a4 (0x9648 bytes) 000237 1789121927.263674 trunc from 0x040000 to 0x039c25 (0x63da bytes) 000238 1789121927.304535 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39c26): Mirror 3: 0x1da7ac51 (stale) Mirror 6: 0xca966eff (stale) Mirror 7: 0xafe84f88 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39c26): Mirror 3: 0x1da7ac51 (stale) Mirror 6: 0xca966eff (stale) Mirror 7: 0xafe84f88 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39c26: skipped 000239 1789121927.888973 read 0x014185 thru 0x020ea7 (0xcd23 bytes) 000240 1789121927.926994 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39c26): Mirror 6: 0xca966eff (stale) Mirror 7: 0xafe84f88 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39c26): Mirror 6: 0xca966eff (stale) Mirror 7: 0xafe84f88 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39c26: skipped 000241 1789121928.261919 read 0x02043e thru 0x023057 (0x2c1a bytes) 000243 1789121928.270280 write 0x010809 thru 0x01b887 (0xb07f bytes) 000244 1789121928.295034 mapread 0x01a8d1 thru 0x027cbd (0xd3ed bytes) 000247 1789121928.348800 read 0x018ab7 thru 0x023232 (0xa77c bytes) 000248 1789121928.351008 trunc from 0x039c26 to 0x02629c (0x13989 bytes) 000249 1789121928.372063 trunc from 0x02629d to 0x0216d5 (0x4bc7 bytes) 000250 1789121928.399624 MIRROR_RESYNC 3 mirrors 000252 1789121929.278431 read 0x00848d thru 0x016804 (0xe378 bytes) 000254 1789121929.291160 mapwrite 0x017b93 thru 0x01d962 (0x5dd0 bytes) 000255 1789121929.545484 write 0x02be4f thru 0x03a9ab (0xeb5d bytes) 000256 1789121929.548998 write 0x00257c thru 0x008eb2 (0x6937 bytes) 000260 1789121929.573514 trunc from 0x03a9ac to 0x023c85 (0x16d26 bytes) 000262 1789121929.637083 mapread 0x002f2f thru 0x00a3ae (0x7480 bytes) 000265 1789121929.652694 read 0x004034 thru 0x00ba56 (0x7a23 bytes) 000266 1789121929.653422 mapread 0x01020d thru 0x0184a1 (0x8295 bytes) 000267 1789121929.655368 trunc from 0x023c86 to 0x026701 (0x2a7c bytes) 000270 1789121929.680714 read 0x015888 thru 0x01c343 (0x6abc bytes) 000271 1789121929.682131 trunc from 0x026702 to 0x02c34c (0x5c4b bytes) 000272 1789121929.702838 mapwrite 0x013f07 thru 0x0197c7 (0x58c1 bytes) 000273 1789121929.807049 mapwrite 0x013cfe thru 0x0169bd (0x2cc0 bytes) 000275 1789121929.904546 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c34d): Mirror 7: 0x1d3336bc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c34d): Mirror 7: 0x1d3336bc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c34d: skipped 000276 1789121930.300763 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2c34d): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c34d): Mirror 7: 0x1d3336bc (stale) CRC-32 checksum value for chunk [0, 0x2c34d): Mirror 1: 0xeaf99fd2 Mirror 8: 0xeaf99fd2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c34d: skipped 000277 1789121931.714724 mapwrite 0x0395c0 thru 0x03ffff (0x6a40 bytes) 000278 1789121932.075028 mapread 0x03cc70 thru 0x03ffff (0x3390 bytes) 000280 1789121932.081024 mapread 0x02b96f thru 0x02f220 (0x38b2 bytes) 000281 1789121932.116571 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x1079f81 (stale) Mirror 8: 0x8c1e60b1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x34169d2c Mirror 9: 0x34169d2c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000286 1789121933.339727 read 0x016b9f thru 0x02642e (0xf890 bytes) 000287 1789121933.359517 read 0x015fd2 thru 0x016d02 (0xd31 bytes) 000291 1789121933.389476 read 0x0318ca thru 0x038ab5 (0x71ec bytes) 000293 1789121933.395274 write 0x0107ef thru 0x018853 (0x8065 bytes) 000294 1789121933.630774 mapwrite 0x028b28 thru 0x02b6c5 (0x2b9e bytes) 000295 1789121933.709469 mapread 0x033a52 thru 0x03755a (0x3b09 bytes) 000296 1789121933.712196 trunc from 0x040000 to 0x020261 (0x1fd9e bytes) 000297 1789121933.759302 MIRROR_RESYNC 4 mirrors 000301 1789121935.178899 read 0x001ce0 thru 0x00d5ef (0xb910 bytes) 000304 1789121935.210176 trunc from 0x020262 to 0x037cc7 (0x17a66 bytes) 000307 1789121935.307948 read 0x037684 thru 0x037cc7 (0x644 bytes) 000308 1789121935.327912 mapread 0x0173dd thru 0x01d942 (0x6566 bytes) 000309 1789121935.334290 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x37cc8): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37cc8): Mirror 7: 0x73950c89 (stale) Mirror 8: 0x73950c89 (stale) Mirror 9: 0x73950c89 (stale) CRC-32 checksum value for chunk [0, 0x37cc8): Mirror 1: 0x73950c89 Mirror 10: 0x73950c89 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37cc8: skipped 000311 1789121936.811759 mapread 0x027da7 thru 0x02dc6b (0x5ec5 bytes) 000312 1789121936.842812 MIRROR_RESYNC 5 mirrors 000314 1789121938.001245 mapwrite 0x0385dd thru 0x03d696 (0x50ba bytes) 000316 1789121938.245755 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3d697): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d697): Mirror 7: 0x9a129ef9 (stale) Mirror 8: 0x9a129ef9 (stale) Mirror 9: 0x9a129ef9 (stale) Mirror 10: 0x9a129ef9 (stale) CRC-32 checksum value for chunk [0, 0x3d697): Mirror 1: 0x275c6e13 Mirror 11: 0x275c6e13 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d697: skipped 000318 1789121939.295720 mapread 0x010b90 thru 0x01446d (0x38de bytes) 000321 1789121939.369059 trunc from 0x03d697 to 0x001ee5 (0x3b7b1 bytes) 000322 1789121939.520491 mapread 0x001a04 thru 0x001ee5 (0x4e2 bytes) 000323 1789121939.528816 trunc from 0x001ee6 to 0x0185b4 (0x166cf bytes) 000324 1789121939.564208 read 0x0030ff thru 0x0072ec (0x41ee bytes) 000329 1789121939.581157 read 0x013408 thru 0x0185b4 (0x51ad bytes) 000330 1789121939.593017 mapread 0x00a677 thru 0x0185b4 (0xdf3e bytes) 000331 1789121939.609707 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x185b5): Mirror 7: 0x6798c75c (stale) Mirror 8: 0x6798c75c (stale) Mirror 10: 0x6798c75c (stale) Mirror 11: 0x6798c75c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x185b5): Mirror 7: 0x6798c75c (stale) Mirror 8: 0x6798c75c (stale) Mirror 10: 0x6798c75c (stale) Mirror 11: 0x6798c75c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x185b5: skipped 000332 1789121940.124441 write 0x026e7f thru 0x027a53 (0xbd5 bytes) 000334 1789121940.125882 trunc from 0x027a54 to 0x02894c (0xef9 bytes) 000335 1789121940.161042 trunc from 0x02894d to 0x01286f (0x160dd bytes) 000336 1789121940.188724 trunc from 0x012870 to 0x00e291 (0x45de bytes) 000337 1789121940.232536 read 0x007a15 thru 0x00e291 (0x687d bytes) 000338 1789121940.245015 write 0x003d78 thru 0x013319 (0xf5a2 bytes) 000339 1789121940.251678 trunc from 0x01331a to 0x02b667 (0x1834e bytes) 000341 1789121940.279646 write 0x00ed84 thru 0x01e37b (0xf5f8 bytes) 000342 1789121940.286804 mapwrite 0x012f7e thru 0x01e4a6 (0xb529 bytes) 000344 1789121940.363888 write 0x0315fb thru 0x03c4ed (0xaef3 bytes) 000346 1789121940.377944 MIRROR_RESYNC 5 mirrors 000347 1789121942.196087 mapwrite 0x023674 thru 0x02e0f4 (0xaa81 bytes) 000348 1789121942.492813 MIRROR_RESYNC 5 mirrors 000350 1789121943.768018 write 0x03f879 thru 0x03ffff (0x787 bytes) 000351 1789121943.851811 mapwrite 0x02f759 thru 0x034f07 (0x57af bytes) 000354 1789121943.914537 mapread 0x028f44 thru 0x02e716 (0x57d3 bytes) 000359 1789121943.930705 write 0x034a64 thru 0x03ffff (0xb59c bytes) 000361 1789121943.933070 trunc from 0x040000 to 0x008835 (0x377ca bytes) 000363 1789121943.958355 read 0x0051a2 thru 0x008835 (0x3694 bytes) 000364 1789121943.966104 trunc from 0x008836 to 0x03be1d (0x335e8 bytes) 000366 1789121943.986367 mapwrite 0x03500c thru 0x03a928 (0x591d bytes) 000367 1789121944.069333 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3be1e): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3be1e): Mirror 7: 0x342b2fd9 (stale) Mirror 8: 0x342b2fd9 (stale) Mirror 10: 0x342b2fd9 (stale) Mirror 11: 0x342b2fd9 (stale) CRC-32 checksum value for chunk [0, 0x3be1e): Mirror 1: 0xfa22f900 Mirror 12: 0xfa22f900 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3be1e: skipped 000368 1789121945.432606 mapwrite 0x02f3f3 thru 0x036f1c (0x7b2a bytes) 000369 1789121945.643099 mapread 0x012a4d thru 0x01dcf4 (0xb2a8 bytes) 000370 1789121945.668375 write 0x01b066 thru 0x01ce52 (0x1ded bytes) 000374 1789121945.680371 mapread 0x033ce6 thru 0x03be1d (0x8138 bytes) 000375 1789121945.687357 trunc from 0x03be1e to 0x0067c8 (0x35655 bytes) 000376 1789121945.736883 mapread 0x00417e thru 0x0052f0 (0x1173 bytes) 000378 1789121945.762442 MIRROR_RESYNC 6 mirrors 000379 1789121946.930778 trunc from 0x0067c9 to 0x0168d1 (0x10109 bytes) 000380 1789121947.021058 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x168d2): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x168d2): Mirror 7: 0x9ad6964 (stale) Mirror 8: 0x9ad6964 (stale) Mirror 10: 0x9ad6964 (stale) Mirror 11: 0x9ad6964 (stale) Mirror 12: 0x9ad6964 (stale) CRC-32 checksum value for chunk [0, 0x168d2): Mirror 1: 0x9ad6964 Mirror 13: 0x9ad6964 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x168d2: skipped 000382 1789121948.362053 write 0x03def0 thru 0x03ffff (0x2110 bytes) 000383 1789121948.456977 read 0x02d0bb thru 0x034faa (0x7ef0 bytes) 000387 1789121948.470592 trunc from 0x040000 to 0x02b0b2 (0x14f4d bytes) 000391 1789121948.492490 trunc from 0x02b0b3 to 0x00aa7f (0x20633 bytes) 000393 1789121948.518147 write 0x0136f9 thru 0x014255 (0xb5d bytes) 000394 1789121948.519632 mapwrite 0x02c0d7 thru 0x03be9d (0xfdc7 bytes) 000395 1789121948.621615 read 0x0309c2 thru 0x03860f (0x7c4e bytes) 000400 1789121948.625853 mapwrite 0x022af9 thru 0x0283a2 (0x58aa bytes) 000401 1789121948.674400 MIRROR_RESYNC 7 mirrors 000403 1789121950.674905 mapwrite 0x0118c2 thru 0x01c1bf (0xa8fe bytes) 000405 1789121950.897860 write 0x01c4ec thru 0x024788 (0x829d bytes) 000406 1789121950.921735 mapread 0x012c85 thru 0x01c20d (0x9589 bytes) 000408 1789121950.931924 MIRROR_RESYNC 7 mirrors 000410 1789121952.226992 trunc from 0x03be9e to 0x0296f5 (0x127a8 bytes) 000411 1789121952.332943 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x296f6): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x296f6): Mirror 7: 0x9ab2f14f (stale) Mirror 8: 0x9ab2f14f (stale) Mirror 10: 0x9ab2f14f (stale) Mirror 11: 0x9ab2f14f (stale) Mirror 12: 0x9ab2f14f (stale) Mirror 13: 0x9ab2f14f (stale) CRC-32 checksum value for chunk [0, 0x296f6): Mirror 1: 0x9ab2f14f Mirror 14: 0x9ab2f14f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x296f6: skipped 000413 1789121953.883609 trunc from 0x0296f6 to 0x025baf (0x3b46 bytes) 000414 1789121954.103323 mapwrite 0x013687 thru 0x01fbd5 (0xc54f bytes) 000415 1789121954.245876 write 0x0083b1 thru 0x008da9 (0x9f9 bytes) 000416 1789121954.268929 MIRROR_RESYNC 8 mirrors 000417 1789121955.765162 mapread 0x01e59c thru 0x022342 (0x3da7 bytes) 000419 1789121955.791270 mapwrite 0x034924 thru 0x03ffff (0xb6dc bytes) 000420 1789121955.969560 mapwrite 0x0253c7 thru 0x027cba (0x28f4 bytes) 000421 1789121956.025883 mapwrite 0x024eb7 thru 0x032ced (0xde37 bytes) 000422 1789121956.074049 write 0x02fe42 thru 0x03a24b (0xa40a bytes) 000424 1789121956.080425 read 0x0078cf thru 0x017165 (0xf897 bytes) 000425 1789121956.091799 mapread 0x03a85c thru 0x03ffff (0x57a4 bytes) 000426 1789121956.093310 trunc from 0x040000 to 0x014d1f (0x2b2e0 bytes) 000428 1789121956.124031 mapread 0x004568 thru 0x005cca (0x1763 bytes) 000429 1789121956.134826 trunc from 0x014d20 to 0x007abf (0xd260 bytes) 000430 1789121956.152384 trunc from 0x007ac0 to 0x02646b (0x1e9ac bytes) 000434 1789121956.176617 read 0x00058e thru 0x001d56 (0x17c9 bytes) 000437 1789121956.223229 mapwrite 0x00a74d thru 0x00d72f (0x2fe3 bytes) 000438 1789121956.318001 write 0x03f79d thru 0x03ffff (0x863 bytes) 000439 1789121956.319695 mapread 0x025b49 thru 0x027b4d (0x2005 bytes) 000441 1789121956.333988 mapwrite 0x001f26 thru 0x010b32 (0xec0d bytes) 000443 1789121956.415186 read 0x03f722 thru 0x03ffff (0x8de bytes) 000445 1789121956.417184 read 0x005e40 thru 0x011c34 (0xbdf5 bytes) 000448 1789121956.419940 mapread 0x015c0c thru 0x01775c (0x1b51 bytes) 000449 1789121956.421565 mapread 0x01ee8b thru 0x01f487 (0x5fd bytes) 000451 1789121956.424211 mapwrite 0x0079e6 thru 0x010e7d (0x9498 bytes) 000453 1789121956.460157 mapwrite 0x00146a thru 0x0090a6 (0x7c3d bytes) 000454 1789121956.525530 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xffd0f375 (stale) Mirror 10: 0xffd0f375 (stale) Mirror 11: 0xffd0f375 (stale) Mirror 12: 0xffd0f375 (stale) Mirror 13: 0xffd0f375 (stale) Mirror 14: 0xffd0f375 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xffd0f375 (stale) Mirror 10: 0xffd0f375 (stale) Mirror 11: 0xffd0f375 (stale) Mirror 12: 0xffd0f375 (stale) Mirror 13: 0xffd0f375 (stale) Mirror 14: 0xffd0f375 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000455 1789121957.200012 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xffd0f375 (stale) Mirror 10: 0xffd0f375 (stale) Mirror 11: 0xffd0f375 (stale) Mirror 12: 0xffd0f375 (stale) Mirror 13: 0xffd0f375 (stale) Mirror 14: 0xffd0f375 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2a215f5d Mirror 15: 0x2a215f5d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000456 1789121958.511973 trunc from 0x040000 to 0x0326eb (0xd914 bytes) 000457 1789121958.610075 write 0x0209d2 thru 0x0227bc (0x1deb bytes) 000458 1789121958.620620 write 0x007d2c thru 0x00d3b1 (0x5686 bytes) 000459 1789121958.630792 mapread 0x00043d thru 0x00916f (0x8d33 bytes) 000461 1789121958.672062 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x326ec): Mirror 8: 0xdb565ab (stale) Mirror 10: 0xdb565ab (stale) Mirror 12: 0xdb565ab (stale) Mirror 13: 0xdb565ab (stale) Mirror 14: 0xdb565ab (stale) Mirror 15: 0x237de8dc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x326ec): Mirror 8: 0xdb565ab (stale) Mirror 10: 0xdb565ab (stale) Mirror 12: 0xdb565ab (stale) Mirror 13: 0xdb565ab (stale) Mirror 14: 0xdb565ab (stale) Mirror 15: 0x237de8dc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x326ec: skipped 000462 1789121959.300431 read 0x02f6f2 thru 0x0326eb (0x2ffa bytes) 000463 1789121959.308851 mapread 0x016c10 thru 0x01792d (0xd1e bytes) 000464 1789121959.323058 trunc from 0x0326ec to 0x035f05 (0x381a bytes) 000465 1789121959.340433 trunc from 0x035f06 to 0x01d420 (0x18ae5 bytes) 000467 1789121959.359713 read 0x007b3d thru 0x00fbf7 (0x80bb bytes) 000468 1789121959.370361 mapwrite 0x020918 thru 0x0296e4 (0x8dcd bytes) 000469 1789121959.457431 mapwrite 0x033da0 thru 0x03d56a (0x97cb bytes) 000470 1789121959.597999 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3d56b): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d56b): Mirror 8: 0x5b239935 (stale) Mirror 10: 0x5b239935 (stale) Mirror 12: 0x5b239935 (stale) Mirror 13: 0x5b239935 (stale) Mirror 14: 0x5b239935 (stale) Mirror 15: 0xd538c12a (stale) CRC-32 checksum value for chunk [0, 0x3d56b): Mirror 1: 0x32a6285a Mirror 16: 0x32a6285a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d56b: skipped 000471 1789121960.434828 mapwrite 0x01b0e1 thru 0x01d510 (0x2430 bytes) 000476 1789121960.611591 write 0x02a740 thru 0x02e815 (0x40d6 bytes) 000477 1789121960.619766 trunc from 0x03d56b to 0x01d857 (0x1fd13 bytes) 000478 1789121960.647175 mapread 0x0113c8 thru 0x014a2a (0x3663 bytes) 000479 1789121960.658176 mapwrite 0x035c52 thru 0x0368bc (0xc6b bytes) 000481 1789121960.750550 trunc from 0x0368bd to 0x008823 (0x2e099 bytes) 000483 1789121960.781800 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x8824): Mirror 10: 0x70f81ecb (stale) Mirror 12: 0x70f81ecb (stale) Mirror 13: 0x70f81ecb (stale) Mirror 14: 0x70f81ecb (stale) Mirror 15: 0x615f8aee (stale) Mirror 16: 0x81fe69c8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8824): Mirror 10: 0x70f81ecb (stale) Mirror 12: 0x70f81ecb (stale) Mirror 13: 0x70f81ecb (stale) Mirror 14: 0x70f81ecb (stale) Mirror 15: 0x615f8aee (stale) Mirror 16: 0x81fe69c8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8824: skipped 000484 1789121961.481419 trunc from 0x008824 to 0x016d46 (0xe523 bytes) 000486 1789121961.535760 trunc from 0x016d47 to 0x038134 (0x213ee bytes) 000487 1789121961.585991 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38135): Mirror 10: 0x5ac7fe5c (stale) Mirror 12: 0x5ac7fe5c (stale) Mirror 13: 0x5ac7fe5c (stale) Mirror 15: 0x66bf20c2 (stale) Mirror 16: 0x7d4ab89a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38135): Mirror 10: 0x5ac7fe5c (stale) Mirror 12: 0x5ac7fe5c (stale) Mirror 13: 0x5ac7fe5c (stale) Mirror 15: 0x66bf20c2 (stale) Mirror 16: 0x7d4ab89a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38135: skipped 000488 1789121962.227908 mapwrite 0x024390 thru 0x02df3b (0x9bac bytes) 000490 1789121962.342323 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x38135): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38135): Mirror 10: 0x5ac7fe5c (stale) Mirror 12: 0x5ac7fe5c (stale) Mirror 13: 0x5ac7fe5c (stale) Mirror 15: 0x66bf20c2 (stale) Mirror 16: 0x7d4ab89a (stale) CRC-32 checksum value for chunk [0, 0x38135): Mirror 1: 0xa48d41fd Mirror 17: 0xa48d41fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38135: skipped 000492 1789121963.741739 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38135): Mirror 12: 0x5ac7fe5c (stale) Mirror 13: 0x5ac7fe5c (stale) Mirror 15: 0x66bf20c2 (stale) Mirror 16: 0x7d4ab89a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38135): Mirror 12: 0x5ac7fe5c (stale) Mirror 13: 0x5ac7fe5c (stale) Mirror 15: 0x66bf20c2 (stale) Mirror 16: 0x7d4ab89a (stale) CRC-32 checksum value for chunk [0, 0x38135): Mirror 1: 0xa48d41fd Mirror 17: 0xa48d41fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38135: skipped 000494 1789121964.517209 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x38135): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38135): Mirror 12: 0x5ac7fe5c (stale) Mirror 13: 0x5ac7fe5c (stale) Mirror 15: 0x66bf20c2 (stale) Mirror 16: 0x7d4ab89a (stale) CRC-32 checksum value for chunk [0, 0x38135): Mirror 1: 0xa48d41fd Mirror 17: 0xa48d41fd Mirror 18: 0xa48d41fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38135: skipped 000496 1789121966.346018 write 0x016212 thru 0x01b535 (0x5324 bytes) 000499 1789121966.502729 read 0x028450 thru 0x02b84c (0x33fd bytes) 000501 1789121966.532236 mapwrite 0x016624 thru 0x018651 (0x202e bytes) 000502 1789121966.629462 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38135): Mirror 12: 0x5ac7fe5c (stale) Mirror 15: 0x66bf20c2 (stale) Mirror 16: 0x7d4ab89a (stale) Mirror 17: 0xa48d41fd (stale) Mirror 18: 0xa48d41fd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38135): Mirror 12: 0x5ac7fe5c (stale) Mirror 15: 0x66bf20c2 (stale) Mirror 16: 0x7d4ab89a (stale) Mirror 17: 0xa48d41fd (stale) Mirror 18: 0xa48d41fd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38135: skipped 000503 1789121967.153225 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x38135): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38135): Mirror 12: 0x5ac7fe5c (stale) Mirror 15: 0x66bf20c2 (stale) Mirror 16: 0x7d4ab89a (stale) Mirror 17: 0xa48d41fd (stale) Mirror 18: 0xa48d41fd (stale) CRC-32 checksum value for chunk [0, 0x38135): Mirror 1: 0xed79cf0f Mirror 19: 0xed79cf0f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38135: skipped 000505 1789121968.455075 read 0x005d55 thru 0x00f539 (0x97e5 bytes) 000506 1789121968.468433 trunc from 0x038135 to 0x0295ff (0xeb35 bytes) 000508 1789121968.616218 write 0x02a7f0 thru 0x031d65 (0x7576 bytes) 000509 1789121968.628218 trunc from 0x031d66 to 0x02056f (0x117f6 bytes) 000511 1789121968.679967 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20570): Mirror 12: 0xac6fc58e (stale) Mirror 15: 0xa683b787 (stale) Mirror 17: 0xdba187e6 (stale) Mirror 18: 0xdba187e6 (stale) Mirror 19: 0x9d218c71 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20570): Mirror 12: 0xac6fc58e (stale) Mirror 15: 0xa683b787 (stale) Mirror 17: 0xdba187e6 (stale) Mirror 18: 0xdba187e6 (stale) Mirror 19: 0x9d218c71 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20570: skipped 000512 1789121969.256438 mapread 0x01950d thru 0x02042a (0x6f1e bytes) 000513 1789121969.283396 MIRROR_RESYNC 6 mirrors 000515 1789121970.677749 mapread 0x00641d thru 0x013534 (0xd118 bytes) 000517 1789121970.724691 write 0x01908c thru 0x024cab (0xbc20 bytes) 000522 1789121970.820339 mapwrite 0x036e6f thru 0x03f15b (0x82ed bytes) 000523 1789121970.996726 read 0x0118b2 thru 0x0151a5 (0x38f4 bytes) 000524 1789121971.017738 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3f15c): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f15c): Mirror 12: 0x974ef10f (stale) Mirror 15: 0x974ef10f (stale) Mirror 17: 0x974ef10f (stale) Mirror 18: 0x974ef10f (stale) Mirror 19: 0x974ef10f (stale) CRC-32 checksum value for chunk [0, 0x3f15c): Mirror 1: 0x7b6dfaf1 Mirror 20: 0x7b6dfaf1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f15c: skipped 000525 1789121972.157203 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3f15c): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f15c): Mirror 12: 0x974ef10f (stale) Mirror 15: 0x974ef10f (stale) Mirror 17: 0x974ef10f (stale) Mirror 18: 0x974ef10f (stale) Mirror 19: 0x974ef10f (stale) CRC-32 checksum value for chunk [0, 0x3f15c): Mirror 1: 0x7b6dfaf1 Mirror 20: 0x7b6dfaf1 Mirror 21: 0x7b6dfaf1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f15c: skipped 000526 1789121973.269141 trunc from 0x03f15c to 0x009337 (0x35e24 bytes) 000527 1789121973.384846 mapread 0x00816c thru 0x009337 (0x11cc bytes) 000528 1789121973.394360 trunc from 0x009338 to 0x015fad (0xcc76 bytes) 000529 1789121973.412464 mapwrite 0x01a6b5 thru 0x021969 (0x72b5 bytes) 000531 1789121973.492435 mapwrite 0x000ed1 thru 0x00a755 (0x9885 bytes) 000534 1789121973.580227 MIRROR_RESYNC 8 mirrors 000535 1789121975.739214 MIRROR_RESYNC 8 mirrors 000536 1789121976.732425 mapwrite 0x00ee9d thru 0x019564 (0xa6c8 bytes) 000537 1789121977.088262 mapread 0x00246e thru 0x00dbf5 (0xb788 bytes) 000540 1789121977.109740 trunc from 0x02196a to 0x01db61 (0x3e08 bytes) 000541 1789121977.155908 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1db62): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1db62): Mirror 12: 0x57bb0234 (stale) Mirror 15: 0x57bb0234 (stale) Mirror 17: 0x57bb0234 (stale) Mirror 18: 0x57bb0234 (stale) Mirror 19: 0x57bb0234 (stale) Mirror 20: 0x57bb0234 (stale) Mirror 21: 0x57bb0234 (stale) CRC-32 checksum value for chunk [0, 0x1db62): Mirror 1: 0x69cbe474 Mirror 22: 0x69cbe474 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1db62: skipped 000542 1789121978.226597 trunc from 0x01db62 to 0x01d7d1 (0x390 bytes) 000543 1789121978.382755 mapread 0x01ae75 thru 0x01d7d1 (0x295d bytes) 000544 1789121978.403068 mapread 0x01552b thru 0x01d7d1 (0x82a7 bytes) 000546 1789121978.405800 write 0x022b73 thru 0x02e80a (0xbc98 bytes) 000547 1789121978.408940 read 0x001752 thru 0x00a9ea (0x9299 bytes) 000548 1789121978.424534 write 0x02dc2f thru 0x03373e (0x5b10 bytes) 000549 1789121978.425936 write 0x01e44d thru 0x02d13b (0xecef bytes) 000551 1789121978.431886 trunc from 0x03373f to 0x000c25 (0x32b19 bytes) 000553 1789121978.461567 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0xc26): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc26): Mirror 12: 0x3a6180ba (stale) Mirror 15: 0x3a6180ba (stale) Mirror 17: 0x3a6180ba (stale) Mirror 18: 0x3a6180ba (stale) Mirror 19: 0x3a6180ba (stale) Mirror 20: 0x3a6180ba (stale) Mirror 21: 0x3a6180ba (stale) Mirror 22: 0x3a6180ba (stale) CRC-32 checksum value for chunk [0, 0xc26): Mirror 1: 0x3a6180ba Mirror 23: 0x3a6180ba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc26: skipped 000555 1789121979.572519 write 0x02095e thru 0x02abf6 (0xa299 bytes) 000556 1789121979.654726 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2abf7): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2abf7): Mirror 12: 0xc29c53ef (stale) Mirror 15: 0xc29c53ef (stale) Mirror 17: 0xc29c53ef (stale) Mirror 18: 0xc29c53ef (stale) Mirror 19: 0xc29c53ef (stale) Mirror 20: 0xc29c53ef (stale) Mirror 21: 0xc29c53ef (stale) Mirror 22: 0x8dcb8a79 (stale) Mirror 23: 0x9f0c8ee7 (stale) CRC-32 checksum value for chunk [0, 0x2abf7): Mirror 1: 0x8f41ad5a Mirror 24: 0x8f41ad5a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2abf7: skipped 000559 1789121980.897170 write 0x0006e3 thru 0x00f532 (0xee50 bytes) 000560 1789121981.009434 write 0x0255c6 thru 0x027d3e (0x2779 bytes) 000561 1789121981.019835 read 0x009da2 thru 0x016b70 (0xcdcf bytes) 000564 1789121981.036734 mapwrite 0x0121c8 thru 0x015121 (0x2f5a bytes) 000565 1789121981.102641 mapwrite 0x021c3e thru 0x02d68b (0xba4e bytes) 000567 1789121981.259417 trunc from 0x02d68c to 0x01e1a4 (0xf4e7 bytes) 000568 1789121981.281984 mapread 0x003952 thru 0x004da5 (0x1454 bytes) 000569 1789121981.282887 write 0x025998 thru 0x030cc1 (0xb32a bytes) 000571 1789121981.290641 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30cc2): Mirror 15: 0xe2a6015d (stale) Mirror 17: 0xe2a6015d (stale) Mirror 18: 0xe2a6015d (stale) Mirror 19: 0xe2a6015d (stale) Mirror 20: 0xe2a6015d (stale) Mirror 21: 0xe2a6015d (stale) Mirror 22: 0xfc93f77 (stale) Mirror 23: 0x7f2ec260 (stale) Mirror 24: 0xdd6f9628 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30cc2): Mirror 15: 0xe2a6015d (stale) Mirror 17: 0xe2a6015d (stale) Mirror 18: 0xe2a6015d (stale) Mirror 19: 0xe2a6015d (stale) Mirror 20: 0xe2a6015d (stale) Mirror 21: 0xe2a6015d (stale) Mirror 22: 0xfc93f77 (stale) Mirror 23: 0x7f2ec260 (stale) Mirror 24: 0xdd6f9628 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30cc2: skipped 000573 1789121981.942098 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x30cc2): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30cc2): Mirror 15: 0xe2a6015d (stale) Mirror 17: 0xe2a6015d (stale) Mirror 18: 0xe2a6015d (stale) Mirror 19: 0xe2a6015d (stale) Mirror 20: 0xe2a6015d (stale) Mirror 21: 0xe2a6015d (stale) Mirror 22: 0xfc93f77 (stale) Mirror 23: 0x7f2ec260 (stale) Mirror 24: 0xdd6f9628 (stale) CRC-32 checksum value for chunk [0, 0x30cc2): Mirror 1: 0xfcde9a6b Mirror 25: 0xfcde9a6b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30cc2: skipped 000576 1789121983.202922 read 0x00ce17 thru 0x01a5a3 (0xd78d bytes) 000578 1789121983.213059 trunc from 0x030cc2 to 0x00b015 (0x25cac bytes) 000581 1789121983.453452 mapread 0x00a799 thru 0x00b015 (0x87d bytes) 000582 1789121983.488757 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb016): Mirror 17: 0xc25f57bf (stale) Mirror 18: 0xc25f57bf (stale) Mirror 19: 0xc25f57bf (stale) Mirror 20: 0xc25f57bf (stale) Mirror 21: 0xc25f57bf (stale) Mirror 22: 0xc25f57bf (stale) Mirror 23: 0xf98bc81 (stale) Mirror 24: 0xf98bc81 (stale) Mirror 25: 0x2bf1925a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb016): Mirror 17: 0xc25f57bf (stale) Mirror 18: 0xc25f57bf (stale) Mirror 19: 0xc25f57bf (stale) Mirror 20: 0xc25f57bf (stale) Mirror 21: 0xc25f57bf (stale) Mirror 22: 0xc25f57bf (stale) Mirror 23: 0xf98bc81 (stale) Mirror 24: 0xf98bc81 (stale) Mirror 25: 0x2bf1925a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb016: skipped 000584 1789121984.065778 write 0x02eeec thru 0x03c901 (0xda16 bytes) 000585 1789121984.068163 read 0x009816 thru 0x010612 (0x6dfd bytes) 000586 1789121984.081821 read 0x01f6f6 thru 0x022c1c (0x3527 bytes) 000589 1789121984.093987 mapwrite 0x00899a thru 0x0144ad (0xbb14 bytes) 000590 1789121984.170883 read 0x0284ae thru 0x02bb62 (0x36b5 bytes) 000591 1789121984.171653 trunc from 0x03c902 to 0x01bfb7 (0x2094a bytes) 000593 1789121984.204169 mapwrite 0x01948f thru 0x01fc4c (0x67be bytes) 000594 1789121984.278913 read 0x015ca3 thru 0x01fc4c (0x9faa bytes) 000596 1789121984.281045 write 0x03a185 thru 0x03c97a (0x27f6 bytes) 000599 1789121984.284416 mapwrite 0x023bf2 thru 0x0249d6 (0xde5 bytes) 000600 1789121984.339307 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3c97b): Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c97b): Mirror 17: 0x34b3ff62 (stale) Mirror 18: 0x34b3ff62 (stale) Mirror 19: 0x34b3ff62 (stale) Mirror 20: 0x34b3ff62 (stale) Mirror 21: 0x34b3ff62 (stale) Mirror 22: 0x95a116f (stale) Mirror 23: 0x5ad0199 (stale) Mirror 24: 0x94b17614 (stale) Mirror 25: 0x9bd7d16a (stale) CRC-32 checksum value for chunk [0, 0x3c97b): Mirror 1: 0x1472325a Mirror 26: 0x1472325a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c97b: skipped 000601 1789121985.225576 mapwrite 0x00b076 thru 0x00e630 (0x35bb bytes) 000604 1789121985.393863 mapread 0x03aed8 thru 0x03c97a (0x1aa3 bytes) 000606 1789121985.416475 MIRROR_RESYNC 11 mirrors 000607 1789121987.213597 trunc from 0x03c97b to 0x019239 (0x23741 bytes) 000610 1789121987.299643 mapread 0x00a31c thru 0x0132b5 (0x8f9a bytes) 000611 1789121987.336676 trunc from 0x01923a to 0x00da7d (0xb7bc bytes) 000614 1789121987.369873 mapread 0x009314 thru 0x00da7d (0x476a bytes) 000615 1789121987.371726 trunc from 0x00da7e to 0x01f07c (0x115ff bytes) 000616 1789121987.395690 MIRROR_RESYNC 11 mirrors 000617 1789121989.555249 mapread 0x0183da thru 0x019c6b (0x1892 bytes) 000619 1789121989.567345 trunc from 0x01f07d to 0x034ac8 (0x15a4c bytes) 000620 1789121989.657840 trunc from 0x034ac9 to 0x00fddf (0x24ce9 bytes) 000622 1789121989.679537 write 0x029d63 thru 0x038968 (0xec06 bytes) 000623 1789121989.682217 mapwrite 0x03cd65 thru 0x03ffff (0x329b bytes) 000624 1789121989.768177 read 0x017ad6 thru 0x01d3c0 (0x58eb bytes) 000625 1789121989.783748 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0xeebcd4b1 (stale) Mirror 19: 0xeebcd4b1 (stale) Mirror 20: 0xeebcd4b1 (stale) Mirror 21: 0xeebcd4b1 (stale) Mirror 22: 0xeebcd4b1 (stale) Mirror 23: 0xeebcd4b1 (stale) Mirror 24: 0xeebcd4b1 (stale) Mirror 25: 0xeebcd4b1 (stale) Mirror 26: 0xeebcd4b1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0xeebcd4b1 (stale) Mirror 19: 0xeebcd4b1 (stale) Mirror 20: 0xeebcd4b1 (stale) Mirror 21: 0xeebcd4b1 (stale) Mirror 22: 0xeebcd4b1 (stale) Mirror 23: 0xeebcd4b1 (stale) Mirror 24: 0xeebcd4b1 (stale) Mirror 25: 0xeebcd4b1 (stale) Mirror 26: 0xeebcd4b1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000627 1789121990.357294 mapread 0x018b95 thru 0x0219c3 (0x8e2f bytes) 000628 1789121990.380084 trunc from 0x040000 to 0x001973 (0x3e68c bytes) 000631 1789121990.401421 read 0x00100a thru 0x001973 (0x96a bytes) 000634 1789121990.411625 mapwrite 0x00b07f thru 0x00e944 (0x38c6 bytes) 000635 1789121990.488909 write 0x00b2a5 thru 0x010f04 (0x5c60 bytes) 000636 1789121990.490434 read 0x00b5cf thru 0x01064f (0x5081 bytes) 000637 1789121990.492208 mapread 0x010c16 thru 0x010f04 (0x2ef bytes) 000638 1789121990.493207 write 0x02d7b3 thru 0x035b2d (0x837b bytes) 000639 1789121990.495406 mapwrite 0x00ab64 thru 0x00e66c (0x3b09 bytes) 000640 1789121990.555751 read 0x00b21d thru 0x00d864 (0x2648 bytes) 000641 1789121990.556456 read 0x0063d1 thru 0x01304a (0xcc7a bytes) 000642 1789121990.570707 write 0x0094fb thru 0x01142e (0x7f34 bytes) 000643 1789121990.573578 mapread 0x01b5c2 thru 0x01bc8b (0x6ca bytes) 000644 1789121990.574975 write 0x029a37 thru 0x029ce2 (0x2ac bytes) 000646 1789121990.578395 mapwrite 0x006293 thru 0x009ee7 (0x3c55 bytes) 000648 1789121990.637791 mapwrite 0x01025f thru 0x0126a0 (0x2442 bytes) 000649 1789121990.681148 MIRROR_RESYNC 10 mirrors 000650 1789121992.670001 mapwrite 0x007b3d thru 0x00ca11 (0x4ed5 bytes) 000652 1789121992.899444 trunc from 0x035b2e to 0x0211bc (0x14971 bytes) 000653 1789121992.962314 mapwrite 0x00c9f0 thru 0x01505a (0x866b bytes) 000656 1789121993.075356 mapwrite 0x024feb thru 0x02fad6 (0xaaec bytes) 000657 1789121993.179541 write 0x020e07 thru 0x02833b (0x7535 bytes) 000658 1789121993.181262 write 0x00eb85 thru 0x012f51 (0x43cd bytes) 000659 1789121993.185377 mapread 0x01ab38 thru 0x01b828 (0xcf1 bytes) 000662 1789121993.195629 write 0x00b981 thru 0x00dc53 (0x22d3 bytes) 000663 1789121993.196577 mapwrite 0x0047a5 thru 0x014197 (0xf9f3 bytes) 000664 1789121993.305624 mapread 0x005686 thru 0x00fb77 (0xa4f2 bytes) 000667 1789121993.307805 write 0x02b6fd thru 0x02ffba (0x48be bytes) 000668 1789121993.309890 read 0x00d4d2 thru 0x011f92 (0x4ac1 bytes) 000669 1789121993.310848 write 0x025e0c thru 0x0288f6 (0x2aeb bytes) 000670 1789121993.312099 mapwrite 0x00dbf0 thru 0x01dbd4 (0xffe5 bytes) 000671 1789121993.355217 write 0x000706 thru 0x001f25 (0x1820 bytes) 000674 1789121993.360384 mapwrite 0x006230 thru 0x01205e (0xbe2f bytes) 000675 1789121993.412690 mapread 0x006f69 thru 0x013cd6 (0xcd6e bytes) 000677 1789121993.415548 read 0x008796 thru 0x00f639 (0x6ea4 bytes) 000678 1789121993.418172 mapwrite 0x027a1a thru 0x02bdd9 (0x43c0 bytes) 000681 1789121993.485260 mapwrite 0x0306fa thru 0x03b492 (0xad99 bytes) 000682 1789121993.580401 MIRROR_RESYNC 10 mirrors 000683 1789121995.292798 write 0x03f8e7 thru 0x03ffff (0x719 bytes) 000685 1789121995.376395 MIRROR_RESYNC 10 mirrors 000688 1789121997.061351 MIRROR_RESYNC 10 mirrors 000689 1789121998.026032 trunc from 0x040000 to 0x01624f (0x29db0 bytes) 000690 1789121998.147542 trunc from 0x016250 to 0x00f7ba (0x6a95 bytes) 000691 1789121998.178234 mapread 0x0077e6 thru 0x00be1b (0x4636 bytes) 000692 1789121998.208323 MIRROR_RESYNC 10 mirrors 000693 1789121999.867764 mapread 0x0057e2 thru 0x00810a (0x2929 bytes) 000695 1789121999.891858 mapread 0x001643 thru 0x00f7ba (0xe178 bytes) 000696 1789121999.897836 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) 000697 1789122000.485071 mapwrite 0x02617c thru 0x028a8d (0x2912 bytes) 000699 1789122000.663066 trunc from 0x028a8e to 0x032896 (0x9e09 bytes) 000701 1789122000.689085 write 0x02bcd0 thru 0x034054 (0x8385 bytes) 000702 1789122000.706948 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34055): Mirror 19: 0x575cca1e (stale) Mirror 20: 0x575cca1e (stale) Mirror 21: 0x575cca1e (stale) Mirror 23: 0x575cca1e (stale) Mirror 24: 0x575cca1e (stale) Mirror 25: 0x575cca1e (stale) Mirror 26: 0x575cca1e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34055): Mirror 19: 0x575cca1e (stale) Mirror 20: 0x575cca1e (stale) Mirror 21: 0x575cca1e (stale) Mirror 23: 0x575cca1e (stale) Mirror 24: 0x575cca1e (stale) Mirror 25: 0x575cca1e (stale) Mirror 26: 0x575cca1e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34055: skipped 000704 1789122001.315349 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34055): Mirror 19: 0x575cca1e (stale) Mirror 20: 0x575cca1e (stale) Mirror 21: 0x575cca1e (stale) Mirror 23: 0x575cca1e (stale) Mirror 24: 0x575cca1e (stale) Mirror 25: 0x575cca1e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34055): Mirror 19: 0x575cca1e (stale) Mirror 20: 0x575cca1e (stale) Mirror 21: 0x575cca1e (stale) Mirror 23: 0x575cca1e (stale) Mirror 24: 0x575cca1e (stale) Mirror 25: 0x575cca1e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34055: skipped 000705 1789122001.789215 mapread 0x028553 thru 0x0291eb (0xc99 bytes) 000706 1789122001.800773 read 0x0312cc thru 0x034054 (0x2d89 bytes) 000708 1789122001.807345 write 0x017c93 thru 0x0189b4 (0xd22 bytes) 000709 1789122001.817584 mapwrite 0x01a35f thru 0x029d20 (0xf9c2 bytes) 000713 1789122001.967206 read 0x0305e3 thru 0x034054 (0x3a72 bytes) 000714 1789122002.015714 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34055): Mirror 19: 0x575cca1e (stale) Mirror 20: 0x575cca1e (stale) Mirror 23: 0x575cca1e (stale) Mirror 24: 0x575cca1e (stale) Mirror 25: 0x575cca1e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34055): Mirror 19: 0x575cca1e (stale) Mirror 20: 0x575cca1e (stale) Mirror 23: 0x575cca1e (stale) Mirror 24: 0x575cca1e (stale) Mirror 25: 0x575cca1e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34055: skipped 000715 1789122002.559833 write 0x03a798 thru 0x03ffff (0x5868 bytes) 000716 1789122002.562062 write 0x03b791 thru 0x03ffff (0x486f bytes) 000717 1789122002.563482 mapread 0x03855e thru 0x03ffff (0x7aa2 bytes) 000720 1789122002.581902 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x2e759969 (stale) Mirror 20: 0x2e759969 (stale) Mirror 23: 0x2e759969 (stale) Mirror 24: 0x2e759969 (stale) Mirror 25: 0x2e759969 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xefc6d88c Mirror 26: 0xefc6d88c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000721 1789122003.840626 read 0x011fa4 thru 0x02095f (0xe9bc bytes) 000723 1789122003.853678 trunc from 0x040000 to 0x025393 (0x1ac6c bytes) 000724 1789122003.963830 mapwrite 0x010c78 thru 0x01a82b (0x9bb4 bytes) 000725 1789122004.061064 mapread 0x00a9bc thru 0x018d34 (0xe379 bytes) 000726 1789122004.081186 trunc from 0x025394 to 0x025772 (0x3df bytes) 000728 1789122004.105510 read 0x019edc thru 0x01db1b (0x3c40 bytes) 000729 1789122004.119752 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25773): Mirror 19: 0x28d90284 (stale) Mirror 20: 0x28d90284 (stale) Mirror 23: 0x28d90284 (stale) Mirror 24: 0x28d90284 (stale) Mirror 26: 0x68d3e421 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25773): Mirror 19: 0x28d90284 (stale) Mirror 20: 0x28d90284 (stale) Mirror 23: 0x28d90284 (stale) Mirror 24: 0x28d90284 (stale) Mirror 26: 0x68d3e421 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25773: skipped 000730 1789122004.620683 mapread 0x005945 thru 0x00f530 (0x9bec bytes) 000731 1789122004.650907 mapwrite 0x003600 thru 0x01071c (0xd11d bytes) 000732 1789122004.766915 read 0x01c666 thru 0x022197 (0x5b32 bytes) 000733 1789122004.799976 read 0x01c0bc thru 0x02288d (0x67d2 bytes) 000734 1789122004.800943 mapwrite 0x03df88 thru 0x03ffff (0x2078 bytes) 000735 1789122004.953005 read 0x01cb6f thru 0x024791 (0x7c23 bytes) 000736 1789122004.956716 mapwrite 0x009ff5 thru 0x019cb8 (0xfcc4 bytes) 000738 1789122005.026822 mapwrite 0x036f06 thru 0x039255 (0x2350 bytes) 000741 1789122005.076518 mapwrite 0x00f204 thru 0x01e925 (0xf722 bytes) 000746 1789122005.143050 mapread 0x005fca thru 0x015277 (0xf2ae bytes) 000747 1789122005.145720 trunc from 0x040000 to 0x03024b (0xfdb4 bytes) 000748 1789122005.167257 mapread 0x025e34 thru 0x028f54 (0x3121 bytes) 000750 1789122005.168736 trunc from 0x03024c to 0x00f5f3 (0x20c58 bytes) 000752 1789122005.189694 write 0x007ee8 thru 0x00ad7e (0x2e97 bytes) 000756 1789122005.199978 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf5f4): Mirror 20: 0xfa1b0c83 (stale) Mirror 23: 0xfa1b0c83 (stale) Mirror 24: 0xfa1b0c83 (stale) Mirror 26: 0xfa1b0c83 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf5f4): Mirror 20: 0xfa1b0c83 (stale) Mirror 23: 0xfa1b0c83 (stale) Mirror 24: 0xfa1b0c83 (stale) Mirror 26: 0xfa1b0c83 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf5f4: skipped 000757 1789122005.642421 read 0x008378 thru 0x00acba (0x2943 bytes) 000758 1789122005.654789 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf5f4): Mirror 20: 0xfa1b0c83 (stale) Mirror 24: 0xfa1b0c83 (stale) Mirror 26: 0xfa1b0c83 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf5f4): Mirror 20: 0xfa1b0c83 (stale) Mirror 24: 0xfa1b0c83 (stale) Mirror 26: 0xfa1b0c83 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf5f4: skipped 000759 1789122006.069802 mapread 0x00b349 thru 0x00f5f3 (0x42ab bytes) 000762 1789122006.095059 trunc from 0x00f5f4 to 0x017405 (0x7e12 bytes) 000764 1789122006.129249 MIRROR_RESYNC 4 mirrors 000765 1789122007.315031 mapread 0x008fe9 thru 0x014122 (0xb13a bytes) 000766 1789122007.360547 MIRROR_RESYNC 4 mirrors 000768 1789122007.839716 mapread 0x0096c0 thru 0x00f5bc (0x5efd bytes) 000769 1789122007.841021 mapwrite 0x03b206 thru 0x03ffff (0x4dfa bytes) 000770 1789122008.060404 mapread 0x02b84d thru 0x02d16f (0x1923 bytes) 000772 1789122008.080326 read 0x03ebc0 thru 0x03ffff (0x1440 bytes) 000773 1789122008.081411 read 0x0162fb thru 0x0247c0 (0xe4c6 bytes) 000774 1789122008.104479 trunc from 0x040000 to 0x034354 (0xbcab bytes) 000775 1789122008.141815 mapread 0x017f6e thru 0x01a212 (0x22a5 bytes) 000777 1789122008.144723 mapread 0x00ce7e thru 0x01892b (0xbaae bytes) 000778 1789122008.146727 read 0x0273f7 thru 0x0298e5 (0x24ef bytes) 000779 1789122008.147843 trunc from 0x034355 to 0x0053cf (0x2ef85 bytes) 000780 1789122008.170907 mapwrite 0x0012fc thru 0x00a3d7 (0x90dc bytes) 000783 1789122008.251097 mapread 0x0017d8 thru 0x003cbb (0x24e4 bytes) 000784 1789122008.252474 mapwrite 0x030b5a thru 0x036f77 (0x641e bytes) 000786 1789122008.391242 read 0x019af6 thru 0x0222a2 (0x87ad bytes) 000788 1789122008.392511 mapread 0x008f8c thru 0x00f8f6 (0x696b bytes) 000789 1789122008.395339 trunc from 0x036f78 to 0x035b8f (0x13e8 bytes) 000790 1789122008.416087 mapread 0x01e5a1 thru 0x025864 (0x72c4 bytes) 000793 1789122008.423224 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35b90): Mirror 24: 0xa748f361 (stale) Mirror 26: 0xa748f361 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35b90): Mirror 24: 0xa748f361 (stale) Mirror 26: 0xa748f361 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35b90: skipped 000796 1789122008.739903 mapwrite 0x007fbd thru 0x012de3 (0xae27 bytes) 000797 1789122008.844317 trunc from 0x035b90 to 0x02ddbb (0x7dd4 bytes) 000798 1789122008.873295 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ddbc): Mirror 26: 0xf9bf5dc0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ddbc): Mirror 26: 0xf9bf5dc0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ddbc: skipped 000799 1789122009.170601 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2ddbc): Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ddbc): Mirror 26: 0xf9bf5dc0 (stale) CRC-32 checksum value for chunk [0, 0x2ddbc): Mirror 1: 0x72fe3c86 Mirror 27: 0x72fe3c86 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ddbc: skipped 000802 1789122010.243569 trunc from 0x02ddbc to 0x01faf0 (0xe2cb bytes) 000803 1789122010.416168 mapwrite 0x02d60f thru 0x02f298 (0x1c8a bytes) 000804 1789122010.543944 read 0x014f2b thru 0x0220a0 (0xd176 bytes) 000805 1789122010.566147 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f299): Mirror 27: 0xa6e66d9d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f299): Mirror 27: 0xa6e66d9d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f299: skipped 000811 1789122010.949522 MIRROR_RESYNC 2 mirrors 000813 1789122011.920654 mapread 0x022bc3 thru 0x029e14 (0x7252 bytes) 000817 1789122011.983626 MIRROR_RESYNC 2 mirrors 000818 1789122012.407869 write 0x032dfc thru 0x03ac29 (0x7e2e bytes) 000819 1789122012.497202 mapwrite 0x0267d6 thru 0x027e3e (0x1669 bytes) 000822 1789122012.572222 trunc from 0x03ac2a to 0x020c52 (0x19fd7 bytes) 000824 1789122012.615617 write 0x00ca5e thru 0x01a9be (0xdf61 bytes) 000825 1789122012.631453 MIRROR_RESYNC 2 mirrors 000828 1789122013.487339 MIRROR_RESYNC 2 mirrors 000829 1789122013.936613 write 0x0183a5 thru 0x01af32 (0x2b8e bytes) 000830 1789122014.057800 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x20c53): Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20c53): Mirror 27: 0xd2a8431c (stale) CRC-32 checksum value for chunk [0, 0x20c53): Mirror 1: 0xae37470 Mirror 28: 0xae37470 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20c53: skipped 000831 1789122014.965197 read 0x00f3b7 thru 0x014fd3 (0x5c1d bytes) 000834 1789122014.997492 write 0x0001c8 thru 0x00698f (0x67c8 bytes) 000835 1789122015.239702 read 0x01ab71 thru 0x020c52 (0x60e2 bytes) 000836 1789122015.256416 trunc from 0x020c53 to 0x0010d3 (0x1fb7f bytes) 000838 1789122015.310664 write 0x03b322 thru 0x03c2ab (0xf8a bytes) 000839 1789122015.311746 mapwrite 0x0231c0 thru 0x026314 (0x3155 bytes) 000842 1789122015.423749 trunc from 0x03c2ac to 0x010310 (0x2bf9b bytes) 000843 1789122015.463465 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x10311): Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10311): Mirror 27: 0x63ca3a9c (stale) Mirror 28: 0x63ca3a9c (stale) CRC-32 checksum value for chunk [0, 0x10311): Mirror 1: 0xd7173cb4 Mirror 29: 0xd7173cb4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10311: skipped 000844 1789122016.512289 read 0x00199f thru 0x0076c5 (0x5d27 bytes) 000845 1789122016.523331 trunc from 0x010311 to 0x024b12 (0x14802 bytes) 000846 1789122016.654842 mapwrite 0x008a03 thru 0x012f78 (0xa576 bytes) 000847 1789122016.795971 mapwrite 0x03d778 thru 0x03ffff (0x2888 bytes) 000848 1789122016.936696 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 27: 0xee14ad69 (stale) Mirror 28: 0x3b77b199 (stale) Mirror 29: 0xc7ed4b8e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2851be78 Mirror 30: 0x2851be78 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000849 1789122018.567388 write 0x022117 thru 0x02ed08 (0xcbf2 bytes) 000850 1789122018.665301 mapwrite 0x03aee2 thru 0x03e181 (0x32a0 bytes) 000851 1789122018.830311 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 27: 0xee14ad69 (stale) Mirror 29: 0xc7ed4b8e (stale) Mirror 30: 0x2851be78 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 27: 0xee14ad69 (stale) Mirror 29: 0xc7ed4b8e (stale) Mirror 30: 0x2851be78 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000852 1789122019.439009 write 0x01407d thru 0x01a652 (0x65d6 bytes) 000856 1789122019.492928 read 0x002af7 thru 0x00a6ac (0x7bb6 bytes) 000857 1789122019.516722 read 0x01b5b6 thru 0x021739 (0x6184 bytes) 000858 1789122019.531962 mapread 0x01442a thru 0x01a603 (0x61da bytes) 000861 1789122019.534690 trunc from 0x040000 to 0x037c93 (0x836c bytes) 000862 1789122019.552174 mapwrite 0x018160 thru 0x02417e (0xc01f bytes) truncating to largest ever: 0x3ef83 000863 1789122019.628963 trunc from 0x037c94 to 0x03ef82 (0x72ef bytes) 000864 1789122019.643973 write 0x014746 thru 0x01f7a5 (0xb060 bytes) 000865 1789122019.649396 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3ef83): Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ef83): Mirror 27: 0x6305e0fd (stale) Mirror 29: 0x121bb56b (stale) Mirror 30: 0xe4496295 (stale) CRC-32 checksum value for chunk [0, 0x3ef83): Mirror 1: 0xcaa40739 Mirror 31: 0xcaa40739 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ef83: skipped 000866 1789122021.027278 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3ef83): Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ef83): Mirror 27: 0x6305e0fd (stale) Mirror 29: 0x121bb56b (stale) Mirror 30: 0xe4496295 (stale) CRC-32 checksum value for chunk [0, 0x3ef83): Mirror 1: 0xcaa40739 Mirror 31: 0xcaa40739 Mirror 32: 0xcaa40739 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ef83: skipped 000869 1789122022.019138 write 0x008c1f thru 0x01238d (0x976f bytes) 000871 1789122022.143420 write 0x007b6e thru 0x0174f5 (0xf988 bytes) 000874 1789122022.160033 MIRROR_RESYNC 6 mirrors 000875 1789122023.806828 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) 000877 1789122024.208496 mapread 0x0052e6 thru 0x00d21b (0x7f36 bytes) 000878 1789122024.238421 trunc from 0x03ef83 to 0x03af93 (0x3fef bytes) 000879 1789122024.367385 mapread 0x01a32c thru 0x01b00b (0xce0 bytes) 000880 1789122024.402869 trunc from 0x03af94 to 0x01aa3b (0x20558 bytes) 000881 1789122024.432909 read 0x0075e0 thru 0x00a805 (0x3226 bytes) 000882 1789122024.447140 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1aa3c): Mirror 27: 0xe7e81c74 (stale) Mirror 30: 0xe7e81c74 (stale) Mirror 32: 0xe7e81c74 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1aa3c): Mirror 27: 0xe7e81c74 (stale) Mirror 30: 0xe7e81c74 (stale) Mirror 32: 0xe7e81c74 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1aa3c: skipped 000883 1789122024.860978 mapread 0x018123 thru 0x01aa3b (0x2919 bytes) 000884 1789122024.892672 mapread 0x007efb thru 0x00d567 (0x566d bytes) 000885 1789122024.906073 trunc from 0x01aa3c to 0x039446 (0x1ea0b bytes) 000887 1789122024.938202 mapread 0x0299df thru 0x02ec05 (0x5227 bytes) 000890 1789122024.957338 read 0x033498 thru 0x039446 (0x5faf bytes) 000892 1789122024.969269 mapread 0x02bb12 thru 0x0391d9 (0xd6c8 bytes) 000893 1789122024.975759 trunc from 0x039447 to 0x03e7a8 (0x5362 bytes) 000894 1789122024.995244 mapwrite 0x0201f5 thru 0x026720 (0x652c bytes) 000895 1789122025.133607 write 0x0074bd thru 0x016e65 (0xf9a9 bytes) 000896 1789122025.137413 write 0x00abb9 thru 0x00c016 (0x145e bytes) 000897 1789122025.140672 mapwrite 0x0078ef thru 0x0150f5 (0xd807 bytes) 000898 1789122025.222169 write 0x035889 thru 0x03e8bc (0x9034 bytes) 000899 1789122025.242877 MIRROR_RESYNC 4 mirrors 000902 1789122026.435555 mapread 0x00c5f2 thru 0x019680 (0xd08f bytes) 000903 1789122026.464547 read 0x00c937 thru 0x015cbe (0x9388 bytes) 000904 1789122026.469597 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) 000905 1789122026.960677 MIRROR_RESYNC 3 mirrors 000907 1789122027.615758 write 0x0176b5 thru 0x01dd96 (0x66e2 bytes) 000908 1789122027.751680 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e8bd): Mirror 32: 0x6810a9f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e8bd): Mirror 32: 0x6810a9f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e8bd: skipped 000912 1789122028.164645 mapwrite 0x00913d thru 0x00b7b2 (0x2676 bytes) 000913 1789122028.268553 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) 000916 1789122028.473581 write 0x02c81a thru 0x036ccb (0xa4b2 bytes) 000918 1789122028.491361 read 0x013488 thru 0x01428b (0xe04 bytes) truncating to largest ever: 0x3f477 000922 1789122028.507742 trunc from 0x03e8bd to 0x03f476 (0xbba bytes) 000923 1789122028.561142 write 0x03f46b thru 0x03ffff (0xb95 bytes) 000926 1789122028.569454 write 0x020149 thru 0x02a781 (0xa639 bytes) 000928 1789122028.582616 trunc from 0x040000 to 0x01cdfe (0x23201 bytes) 000930 1789122028.601455 write 0x00a46a thru 0x00db9d (0x3734 bytes) 000931 1789122028.612174 mapread 0x008be9 thru 0x012b3b (0x9f53 bytes) 000933 1789122028.631868 trunc from 0x01cdff to 0x02930d (0xc50f bytes) 000934 1789122028.650367 mapread 0x023884 thru 0x02930d (0x5a8a bytes) 000935 1789122028.660554 read 0x00117c thru 0x006697 (0x551c bytes) 000937 1789122028.669638 write 0x0306d8 thru 0x03d489 (0xcdb2 bytes) 000939 1789122028.673832 read 0x00d74c thru 0x013c8c (0x6541 bytes) 000940 1789122028.675191 write 0x0173ed thru 0x0267e2 (0xf3f6 bytes) 000941 1789122028.677650 read 0x00ce13 thru 0x016000 (0x91ee bytes) 000943 1789122028.679350 mapread 0x009e1b thru 0x014294 (0xa47a bytes) 000946 1789122028.683791 write 0x02b5d8 thru 0x031380 (0x5da9 bytes) 000948 1789122028.700168 write 0x00b524 thru 0x016db1 (0xb88e bytes) 000950 1789122028.708782 mapread 0x003706 thru 0x00a0e6 (0x69e1 bytes) 000951 1789122028.713036 trunc from 0x03d48a to 0x00f227 (0x2e262 bytes) 000952 1789122028.749734 mapwrite 0x003269 thru 0x00f022 (0xbdba bytes) 000955 1789122028.835385 trunc from 0x00f228 to 0x01ab1c (0xb8f5 bytes) 000958 1789122028.851271 read 0x0169be thru 0x01ab1c (0x415f bytes) 000959 1789122028.861375 trunc from 0x01ab1d to 0x0255fa (0xaade bytes) 000960 1789122028.883935 mapwrite 0x028e04 thru 0x02de3e (0x503b bytes) 000963 1789122029.001639 mapread 0x00a9ce thru 0x00d32c (0x295f bytes) 000965 1789122029.018294 MIRROR_EXTEND to 2 mirrors 000967 1789122029.905311 MIRROR_EXTEND to 3 mirrors 000968 1789122031.008906 write 0x036349 thru 0x03e502 (0x81ba bytes) 000969 1789122031.125666 write 0x021fbd thru 0x02f1ef (0xd233 bytes) 000970 1789122031.138500 trunc from 0x03e503 to 0x033791 (0xad71 bytes) 000971 1789122031.168922 read 0x022f0d thru 0x032ecf (0xffc3 bytes) 000972 1789122031.174607 read 0x031185 thru 0x033791 (0x260d bytes) 000973 1789122031.187381 mapwrite 0x000142 thru 0x00d543 (0xd402 bytes) 000974 1789122031.258660 MIRROR_RESYNC 3 mirrors 000975 1789122032.430507 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) 000976 1789122032.819421 MIRROR_EXTEND to 3 mirrors 000978 1789122033.835149 trunc from 0x033792 to 0x0090ad (0x2a6e4 bytes) 000979 1789122033.962216 mapwrite 0x0235e0 thru 0x031cc8 (0xe6e9 bytes) 000982 1789122034.119425 MIRROR_RESYNC 3 mirrors 000985 1789122035.262826 write 0x022983 thru 0x026835 (0x3eb3 bytes) 000986 1789122035.343371 trunc from 0x031cc9 to 0x012637 (0x1f691 bytes) 000987 1789122035.367209 trunc from 0x012638 to 0x01b2c8 (0x8c91 bytes) 000988 1789122035.387712 MIRROR_RESYNC 3 mirrors 000989 1789122036.213001 write 0x00db97 thru 0x01d153 (0xf5bd bytes) 000990 1789122036.336946 mapwrite 0x022827 thru 0x025b0a (0x32e4 bytes) 000992 1789122036.465526 read 0x0257e7 thru 0x025b0a (0x324 bytes) 000993 1789122036.466828 trunc from 0x025b0b to 0x0198d9 (0xc231 bytes) 000994 1789122036.488435 trunc from 0x0198da to 0x028c3a (0xf361 bytes) 000995 1789122036.510495 mapwrite 0x01c085 thru 0x01ed80 (0x2cfc bytes) 000997 1789122036.574134 write 0x00119d thru 0x006cff (0x5b63 bytes) 000998 1789122036.583894 mapwrite 0x03cbed thru 0x03ffff (0x3413 bytes) 000999 1789122036.658296 read 0x02def8 thru 0x0380ff (0xa208 bytes) All operations completed A-OK! PASS 70a (139s) == sanity-flr test 71: check mirror extend parent fid ==== 06:20:42 (1789122042) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 06:20:45 (1789122045) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.282751 s, 7.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175851 s, 6.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 06:20:55 (1789122055) Starting client: oleg601-client.virtnet: -o user_xattr,flock 192.168.206.101@tcp:/lustre /mnt/lustre2 Starting client: oleg601-client.virtnet: -o user_xattr,flock 192.168.206.101@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 3690336 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3947776 .. failed Extending file size to 6136288 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7492448 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6693696 .. Extending file size to 7525888 .. Extending file size to 7258720 .. done Extending file size to 8585024 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8128608 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8218880 .. Extending file size to 8707904 .. Extending file size to 8727904 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8749664 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8839712 .. Waiting 59019 59020 59021 59023 59025 done 192.168.206.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg601-client.virtnet /mnt/lustre2 (opts:) 192.168.206.101@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg601-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 06:22:17 (1789122137) Starting client: oleg601-client.virtnet: -o user_xattr,flock 192.168.206.101@tcp:/lustre /mnt/lustre2 Starting client: oleg601-client.virtnet: -o user_xattr,flock 192.168.206.101@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5144512 .. Extending file size to 1748128 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8400384 .. Extending file size to 5165632 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6596352 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7586880 .. Extending file size to 8547776 .. Extending file size to 8305120 .. done Extending file size to 8448256 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9265536 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8602592 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8625568 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9044736 .. Waiting 60723 60724 60727 60728 60731 failed 192.168.206.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg601-client.virtnet /mnt/lustre2 (opts:) 192.168.206.101@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg601-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 06:23:38 (1789122218) Starting client: oleg601-client.virtnet: -o user_xattr,flock 192.168.206.101@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.128935 s, 16.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.353396 s, 5.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0xb084fdc7 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.206.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg601-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 06:23:59 (1789122239) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.157708 s, 13.3 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 06:24:08 (1789122248) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x200:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f8:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } PASS 203 (10s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 06:24:18 (1789122258) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.287546 s, 7.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (11s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 06:24:28 (1789122268) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.245249 s, 8.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 06:24:38 (1789122278) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.264745 s, 7.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x318:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x318:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x319:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 06:25:12 (1789122312) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg601-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144723 s, 7.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fd:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fd:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fe:0x0] } PASS 204d (64s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 06:26:16 (1789122376) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg601-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.371176 s, 5.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ff:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ff:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (64s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 06:27:20 (1789122440) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg601-server mds-ost sync done. Creating new pool test_204f oleg601-server: Pool lustre.test_204f created Adding targets to pool oleg601-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg601-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.320241 s, 6.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x300:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x300:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg601-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg601-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg601-server: Pool lustre.test_204f destroyed PASS 204f (79s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 06:28:39 (1789122519) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 06:28:46 (1789122526) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 06:28:54 (1789122534) oleg601-server: Pool lustre.test_206 created Adding targets to pool oleg601-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg601-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg601-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg601-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg601-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 06:29:22 (1789122562) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813117 s, 12.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074979 s, 14.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x274cf71e Mirror 3: 0x274cf71e PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 06:29:33 (1789122573) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 06:29:37 (1789122577) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 06:29:41 (1789122581) Starting client: oleg601-client.virtnet: -o user_xattr,flock 192.168.206.101@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.172896 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xbe0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x700000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x700000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x700000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x7e0000 Read this (4 bytes): BBBB 192.168.206.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg601-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 06:30:18 (1789122618) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0124979 s, 655 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 06:30:26 (1789122626) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100886 s, 10.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x225 lmm_fid: [0x200000404:0x225:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x225 lmm_fid: [0x200000404:0x225:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30b:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834469 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 06:30:28.000000000 -0400 Modify: 2026-09-11 06:30:28.000000000 -0400 Change: 2026-09-11 06:30:30.000000000 -0400 Birth: 2026-09-11 06:30:29.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 06:30:36 (1789122636) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 06:30:39 (1789122639) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.777647 s, 52.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0577929 s, 70.9 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 11 06:30 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x227:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 11 06:30 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 06:30:48 (1789122648) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 06:30:56 (1789122656) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (12s) == sanity-flr test complete, duration 1768 sec =========== 06:31:08 (1789122668) osc.lustre-OST0000-osc-ffff9b8a451a5000.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 osc.lustre-OST0001-osc-ffff9b8a451a5000.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 obdfilter.lustre-OST0000.stats_compr= snapshot_time 1789122672.678606047 secs.nsecs start_time 1789121308.716155494 secs.nsecs elapsed_time 1363.962450553 secs.nsecs uncompressed_objects 2786 samples [reqs] obdfilter.lustre-OST0001.stats_compr= snapshot_time 1789122672.678667257 secs.nsecs start_time 1789121338.808802503 secs.nsecs elapsed_time 1333.869864754 secs.nsecs uncompressed_objects 2797 samples [reqs] === sanity-flr: start cleanup 06:31:12 (1789122672) === === sanity-flr: finish cleanup 06:31:20 (1789122680) ===