-----============= acceptance-small: sanity-flr ============----- Fri Sep 11 05:03:50 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg440-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 05:04:10 (1789117450) === oleg440-client.virtnet: executing check_config_client /mnt/lustre oleg440-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg440-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff900583e9a800.idle_timeout=debug osc.lustre-OST0001-osc-ffff900583e9a800.idle_timeout=debug disable quota as required oleg440-server: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Setting lod.lustre-MDT0000-mdtlov.enable_compr_rotational from 0 to 1 Waiting 90s for '1' Updated after 2s: want '1' got '1' === sanity-flr: finish setup 05:04:59 (1789117499) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7657] == sanity-flr test 0a: lfs mirror create with -N option == 05:05:04 (1789117504) PASS 0a (20s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 05:05:24 (1789117524) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 05:05:27 (1789117527) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 05:05:31 (1789117531) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 05:05:58 (1789117558) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (5s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 05:06:02 (1789117562) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 05:06:06 (1789117566) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff900583e9a800.stats=clear osc.lustre-OST0001-osc-ffff900583e9a800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 05:06:15 (1789117575) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (14s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 05:06:30 (1789117590) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 05:06:39 (1789117599) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 05:06:42 (1789117602) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 05:06:50 (1789117610) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 05:07:01 (1789117621) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 05:07:10 (1789117630) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 05:07:13 (1789117633) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 05:07:21 (1789117641) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 05:07:33 (1789117653) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 05:07:41 (1789117661) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 1.84445 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163065 s, 6.4 MB/s PASS 21 (13s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 05:07:54 (1789117674) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.293726 s, 35.7 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 05:08:03 (1789117683) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 05:08:13 (1789117693) Stopping /mnt/lustre-ost1 (opts:) on oleg440-server oleg440-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff900583e9a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff900583e9a800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg440-server: oleg440-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg440-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg440-server oleg440-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff900583e9a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff900583e9a800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg440-server: oleg440-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg440-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (75s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 05:09:28 (1789117768) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.507694 s, 16.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg440-server oleg440-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff900583e9a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff900583e9a800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg440-server: oleg440-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg440-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 05:10:12 (1789117812) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg440-server oleg440-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff900583e9a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff900583e9a800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg440-server: oleg440-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg440-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg440-server oleg440-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff900583e9a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff900583e9a800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg440-server: oleg440-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg440-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (71s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 05:11:24 (1789117884) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.94447 s, 13.3 MB/s -rw-r--r-- 1 root root 52428800 Sep 11 05:11 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.838s user 0m0.002s sys 0m0.182s Stopping /mnt/lustre-ost1 (opts:) on oleg440-server oleg440-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff900583e9a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff900583e9a800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.501s user 0m0.001s sys 0m0.214s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg440-server: oleg440-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg440-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg440-server oleg440-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff900583e9a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff900583e9a800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.232s user 0m0.000s sys 0m0.165s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg440-server: oleg440-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg440-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg440-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff900583e9a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff900583e9a800.ost_server_uuid in FULL state after 0 sec PASS 33b (93s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 05:12:57 (1789117977) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 05:13:01 (1789117981) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 05:13:04 (1789117984) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 05:13:08 (1789117988) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 05:13:16 (1789117996) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.674696 s, 18.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 05:13:28 (1789118008) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358999 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293219 s, 35.8 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 05:13:35 (1789118015) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 05:13:43 (1789118023) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00220588 s, 464 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00077773 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00207551 s, 493 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00165388 s, 619 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 05:13:53 (1789118033) create test files with size 15749481 .. 35603897c17868b0069e6a8d07228f14 - 5996d5166487f01e68a64257bb946641 - 200db9a4f89017a87382ff54035a4eb6 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg440-client.virtnet /mnt/lustre (opts:) Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre PASS 37 (21s) == sanity-flr test 38: resync ============================ 05:14:14 (1789118054) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 05:14:31 (1789118071) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 05:14:39 (1789118079) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124915 s, 8.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518453 s, 20.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 05:14:49 (1789118089) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.103703 s, 20.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.387489 s, 10.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.211516 s, 9.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (20s) == sanity-flr test 42: lfs mirror verify ================= 05:15:08 (1789118108) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 05:15:12 (1789118112) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 05:15:15 (1789118115) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232835 s, 45.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 43c: read prefer uncompressed mirror == 05:15:23 (1789118123) llite.lustre-ffff90058ac22800.enable_compression=1 fail_loc=0x1425 fail_val=-1 PASS 43c (7s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 05:15:31 (1789118131) llite.lustre-ffff90058ac22800.enable_compression=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547064 s, 19.2 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298188 s, 35.2 MB/s **verify components PASS 43d (8s) == sanity-flr test 44a: lfs mirror split check =========== 05:15:39 (1789118139) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 05:15:42 (1789118142) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe2:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc8:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc8:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 05:15:52 (1789118152) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.317825 s, 33.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x1f:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg440-client.virtnet /mnt/lustre (opts:) Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 05:16:05 (1789118165) llite.lustre-ffff900582c12000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.281474 s, 37.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg440-client.virtnet /mnt/lustre (opts:) Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff900598414000.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 05:16:19 (1789118179) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (42s) == sanity-flr test 46: Verify setstripe --copy option ==== 05:17:01 (1789118221) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (37s) == sanity-flr test 47: Verify mirror obj alloc =========== 05:17:38 (1789118258) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 05:17:41 (1789118261) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.327175 s, 9.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.165804 s, 19.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x13e73f44 (nosync,stale) Mirror 2: 0x13e73f44 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 83a355735472d50083b7dec6ef249e94 - mirror 1 checksum: 45f008d127e65ab48f7aebfb44ee3611 - mirror 2 checksum: 83a355735472d50083b7dec6ef249e94 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-11T05:17:44-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-11T05:17:44-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 83a355735472d50083b7dec6ef249e94 - mirror 1 checksum: 45f008d127e65ab48f7aebfb44ee3611 - mirror 2 checksum: 45f008d127e65ab48f7aebfb44ee3611 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-11T05:17:44-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-11T05:17:44-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 05:17:52 (1789118272) Failing mds1 on oleg440-server Stopping /mnt/lustre-mds1 (opts:) on oleg440-server 05:17:59 (1789118279) shut down facet: mds1 facet_host: oleg440-server facet_failover_host: oleg440-server Failover mds1 to oleg440-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0000 05:18:31 (1789118311) targets are mounted 05:18:31 (1789118311) facet_failover done oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442152 s, 23.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } PASS 50A (57s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 05:18:49 (1789118329) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 05:18:52 (1789118332) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 05:18:56 (1789118336) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00532757 s, 3.1 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 05:19:04 (1789118344) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 05:19:08 (1789118348) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0137982 s, 297 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0939725 s, 43.6 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 05:19:16 (1789118356) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 05:19:40 (1789118380) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 05:20:08 (1789118408) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 05:20:32 (1789118432) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 05:20:56 (1789118456) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 05:21:05 (1789118465) mirror create pid 55820 mirror split pid 55821 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 55820 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 55821 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 05:22:13 (1789118533) Seed set to 8535 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0xc178 000002 1789118535.589707 trunc from 00000000 to 0x00c177 (0xc178 bytes) 000004 1789118535.608995 mapread 0x0068dc thru 0x00c177 (0x589c bytes) 000005 1789118535.631266 write 0x0031a8 thru 0x00b755 (0x85ae bytes) 000006 1789118535.632671 mapread 0x0025b8 thru 0x009198 (0x6be1 bytes) 000007 1789118535.634272 read 0x008336 thru 0x00941f (0x10ea bytes) 000012 1789118535.638421 write 0x01701a thru 0x01daa5 (0x6a8c bytes) 000014 1789118535.650162 read 0x0097ad thru 0x00c0db (0x292f bytes) 000016 1789118535.651209 write 0x012f04 thru 0x01f46d (0xc56a bytes) 000029 1789118535.668755 write 0x00cac1 thru 0x0124a6 (0x59e6 bytes) 000032 1789118535.673454 mapwrite 0x03d0ff thru 0x03ffff (0x2f01 bytes) 000033 1789118535.775287 mapwrite 0x000f01 thru 0x00fd04 (0xee04 bytes) truncating to largest ever: 0x35d7d 000034 1789118535.823947 trunc from 0x040000 to 0x035d7c (0xa283 bytes) 000035 1789118535.840251 write 0x033bee thru 0x03ffff (0xc412 bytes) 000036 1789118535.841821 mapread 0x02aa96 thru 0x033ade (0x9049 bytes) 000039 1789118535.845264 mapread 0x0047d7 thru 0x007d99 (0x35c3 bytes) 000042 1789118535.847751 trunc from 0x040000 to 0x014b45 (0x2b4ba bytes) 000045 1789118535.883605 trunc from 0x014b46 to 0x00e497 (0x66ae bytes) 000046 1789118535.904590 mapwrite 0x00bf32 thru 0x0194c5 (0xd594 bytes) 000048 1789118536.204348 write 0x028f1c thru 0x038f14 (0xfff9 bytes) 000049 1789118536.225695 write 0x01bbd4 thru 0x01da07 (0x1e34 bytes) 000051 1789118536.258283 read 0x024718 thru 0x032f40 (0xe829 bytes) 000053 1789118536.284935 write 0x02cadf thru 0x02ddd9 (0x12fb bytes) 000055 1789118536.287304 mapwrite 0x00adec thru 0x015a3b (0xac50 bytes) 000057 1789118536.350805 mapwrite 0x02a16c thru 0x02f792 (0x5627 bytes) 000061 1789118536.420891 mapwrite 0x03a11e thru 0x03ffff (0x5ee2 bytes) 000062 1789118536.497459 mapwrite 0x00a502 thru 0x00c4e2 (0x1fe1 bytes) 000063 1789118536.565662 trunc from 0x040000 to 0x029fb5 (0x1604a bytes) 000065 1789118536.615516 read 0x0110ac thru 0x01915e (0x80b3 bytes) 000067 1789118536.617335 read 0x001cd5 thru 0x006c62 (0x4f8e bytes) 000070 1789118536.629383 read 0x00b959 thru 0x01463e (0x8ce6 bytes) 000073 1789118536.635734 mapwrite 0x007f58 thru 0x014b38 (0xcbe1 bytes) 000074 1789118536.733364 trunc from 0x029fb6 to 0x011727 (0x1888e bytes) 000075 1789118536.754827 trunc from 0x011728 to 0x001b8f (0xfb98 bytes) 000076 1789118536.789176 write 0x0170f0 thru 0x025b95 (0xeaa6 bytes) 000077 1789118536.791368 read 0x022c6e thru 0x025b95 (0x2f28 bytes) 000078 1789118536.794390 mapwrite 0x035dbd thru 0x03ffff (0xa243 bytes) 000079 1789118536.960942 write 0x004c71 thru 0x008e8d (0x421d bytes) 000080 1789118536.961893 read 0x0291ec thru 0x02b594 (0x23a9 bytes) 000081 1789118536.962449 read 0x010ce8 thru 0x01fda8 (0xf0c1 bytes) 000082 1789118536.969695 read 0x0251f4 thru 0x02a06b (0x4e78 bytes) 000083 1789118536.971711 trunc from 0x040000 to 0x00c632 (0x339cd bytes) 000084 1789118537.013778 read 0x00b04f thru 0x00c632 (0x15e4 bytes) 000085 1789118537.017778 mapwrite 0x03c80e thru 0x03ffff (0x37f2 bytes) 000086 1789118537.122433 write 0x02a668 thru 0x02ff59 (0x58f2 bytes) 000088 1789118537.129264 MIRROR_EXTEND to 2 mirrors 000089 1789118537.763927 mapwrite 0x022e77 thru 0x032740 (0xf8ca bytes) 000091 1789118537.930859 mapwrite 0x037ce1 thru 0x03ffff (0x831f bytes) 000092 1789118537.984689 write 0x01880d thru 0x0264ad (0xdca1 bytes) 000096 1789118537.996374 mapwrite 0x02e52f thru 0x03d846 (0xf318 bytes) 000097 1789118538.069336 write 0x0259b8 thru 0x030428 (0xaa71 bytes) 000098 1789118538.071640 write 0x00b261 thru 0x017f17 (0xccb7 bytes) 000101 1789118538.083397 mapwrite 0x03c255 thru 0x03ffff (0x3dab bytes) 000102 1789118538.129107 MIRROR_RESYNC 2 mirrors 000105 1789118538.667861 mapread 0x0306ed thru 0x03ce26 (0xc73a bytes) 000107 1789118538.709457 trunc from 0x040000 to 0x0029d3 (0x3d62c bytes) 000108 1789118538.866149 write 0x02042d thru 0x020f0d (0xae1 bytes) 000109 1789118538.873658 read 0x00673f thru 0x00ea15 (0x82d7 bytes) 000110 1789118538.885356 mapread 0x014eac thru 0x020e87 (0xbfdc bytes) 000111 1789118538.922211 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) 000112 1789118539.183988 mapread 0x00715e thru 0x00ae42 (0x3ce5 bytes) 000113 1789118539.217788 read 0x001dfb thru 0x007060 (0x5266 bytes) 000114 1789118539.222823 mapread 0x014649 thru 0x01d053 (0x8a0b bytes) 000116 1789118539.244937 read 0x00b247 thru 0x0165e6 (0xb3a0 bytes) 000117 1789118539.250157 trunc from 0x020f0e to 0x003b4f (0x1d3be bytes) 000120 1789118539.297554 trunc from 0x003b50 to 0x01ff6e (0x1c41f bytes) 000122 1789118539.321407 mapread 0x00fb9d thru 0x011f11 (0x2375 bytes) 000126 1789118539.345795 mapread 0x00ea81 thru 0x0108ad (0x1e2d bytes) 000127 1789118539.346721 write 0x002681 thru 0x006a72 (0x43f2 bytes) 000131 1789118539.351655 mapwrite 0x01f0c5 thru 0x02094b (0x1887 bytes) 000133 1789118539.443953 mapread 0x01660f thru 0x02094b (0xa33d bytes) 000134 1789118539.445463 mapread 0x00b7bd thru 0x01162e (0x5e72 bytes) 000139 1789118539.450316 read 0x01b5b8 thru 0x01c7c5 (0x120e bytes) 000140 1789118539.451174 mapwrite 0x00a4d7 thru 0x00c90f (0x2439 bytes) 000142 1789118539.505402 trunc from 0x02094c to 0x032321 (0x119d6 bytes) 000143 1789118539.531086 mapread 0x00f97b thru 0x01a544 (0xabca bytes) 000144 1789118539.547502 write 0x03e716 thru 0x03ffff (0x18ea bytes) 000145 1789118539.552739 mapread 0x01ea7b thru 0x0201d2 (0x1758 bytes) 000148 1789118539.560883 trunc from 0x040000 to 0x02a9cc (0x15633 bytes) 000149 1789118539.621545 write 0x02a671 thru 0x0324ff (0x7e8f bytes) 000150 1789118539.633986 trunc from 0x032500 to 0x01e162 (0x1439d bytes) 000151 1789118539.656456 mapread 0x00d29b thru 0x0137ee (0x6554 bytes) 000153 1789118539.658954 read 0x016787 thru 0x01d3e7 (0x6c61 bytes) 000155 1789118539.661469 read 0x01987f thru 0x01e162 (0x48e4 bytes) 000162 1789118539.668407 read 0x007911 thru 0x016fc2 (0xf6b2 bytes) 000163 1789118539.669985 read 0x00389e thru 0x00e7c8 (0xaf2b bytes) 000167 1789118539.672902 trunc from 0x01e163 to 0x00386f (0x1a8f3 bytes) 000173 1789118539.714365 write 0x00c963 thru 0x00e3d7 (0x1a75 bytes) 000175 1789118539.726706 mapread 0x0099f4 thru 0x00e3d7 (0x49e4 bytes) 000177 1789118539.745448 read 0x0088e5 thru 0x00e3d7 (0x5af3 bytes) 000178 1789118539.746298 mapread 0x00008c thru 0x0044ad (0x4422 bytes) 000179 1789118539.747433 write 0x036f66 thru 0x03ffff (0x909a bytes) 000180 1789118539.749570 mapread 0x03636b thru 0x03bd7d (0x5a13 bytes) 000182 1789118539.751629 read 0x038ed9 thru 0x03ffff (0x7127 bytes) 000183 1789118539.752651 mapwrite 0x00e451 thru 0x01be73 (0xda23 bytes) 000185 1789118539.803831 read 0x02f44a thru 0x037fa5 (0x8b5c bytes) 000186 1789118539.804439 trunc from 0x040000 to 0x034f07 (0xb0f8 bytes) 000187 1789118539.817520 read 0x0121a2 thru 0x02109a (0xeef9 bytes) 000188 1789118539.822569 MIRROR_EXTEND to 2 mirrors 000190 1789118540.659380 MIRROR_RESYNC 2 mirrors 000191 1789118540.967136 mapwrite 0x01626c thru 0x020d0b (0xaaa0 bytes) 000192 1789118541.133718 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x34f08): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34f08): Mirror 2: 0xa0e172da (stale) CRC-32 checksum value for chunk [0, 0x34f08): Mirror 1: 0xd3df758 Mirror 3: 0xd3df758 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34f08: skipped 000194 1789118541.867637 read 0x025226 thru 0x026530 (0x130b bytes) 000195 1789118541.877270 read 0x001b93 thru 0x0118f3 (0xfd61 bytes) 000196 1789118541.887675 write 0x00695e thru 0x00be6c (0x550f bytes) 000197 1789118541.983046 trunc from 0x034f08 to 0x024c01 (0x10306 bytes) 000198 1789118542.011681 write 0x023c34 thru 0x02b3c4 (0x7791 bytes) 000199 1789118542.026914 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2b3c5): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b3c5): Mirror 2: 0xbfb7a35 (stale) Mirror 3: 0xeb4db2cb (stale) CRC-32 checksum value for chunk [0, 0x2b3c5): Mirror 1: 0x4dd30238 Mirror 4: 0x4dd30238 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b3c5: skipped 000200 1789118542.887666 trunc from 0x02b3c5 to 0x02f287 (0x3ec3 bytes) 000201 1789118542.971805 write 0x00d1db thru 0x011c10 (0x4a36 bytes) truncating to largest ever: 0x37e44 000203 1789118542.981723 trunc from 0x02f288 to 0x037e43 (0x8bbc bytes) 000204 1789118543.023626 trunc from 0x037e44 to 0x02451d (0x13926 bytes) 000206 1789118543.040854 mapwrite 0x01a77c thru 0x020a58 (0x62dd bytes) 000209 1789118543.101365 write 0x020acb thru 0x02566c (0x4ba2 bytes) 000210 1789118543.102422 read 0x019391 thru 0x01cf33 (0x3ba3 bytes) 000212 1789118543.104656 mapread 0x0233d1 thru 0x02566c (0x229c bytes) 000214 1789118543.107358 mapwrite 0x00aaa7 thru 0x00e25b (0x37b5 bytes) 000215 1789118543.156324 trunc from 0x02566d to 0x01bac7 (0x9ba5 bytes) 000216 1789118543.179401 read 0x0062ee thru 0x010a6f (0xa782 bytes) 000217 1789118543.181680 mapwrite 0x01076c thru 0x01f9fe (0xf293 bytes) 000219 1789118543.274466 MIRROR_RESYNC 4 mirrors 000220 1789118544.281866 read 0x00ed50 thru 0x01cc7b (0xdf2c bytes) 000222 1789118544.290339 write 0x010661 thru 0x01da9c (0xd43c bytes) 000223 1789118544.410082 mapwrite 0x02e7fd thru 0x034337 (0x5b3b bytes) 000224 1789118544.589145 mapwrite 0x00704c thru 0x0094f4 (0x24a9 bytes) 000225 1789118544.638926 mapread 0x0062c2 thru 0x0123eb (0xc12a bytes) 000227 1789118544.640449 mapwrite 0x015c95 thru 0x01f24b (0x95b7 bytes) 000228 1789118544.689105 mapwrite 0x01bbf7 thru 0x0234c7 (0x78d1 bytes) 000229 1789118544.733584 write 0x0076a7 thru 0x00f812 (0x816c bytes) 000230 1789118544.735398 write 0x00707d thru 0x014cd1 (0xdc55 bytes) 000231 1789118544.737897 read 0x01f625 thru 0x0225d2 (0x2fae bytes) 000232 1789118544.750911 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x34338): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34338): Mirror 2: 0x921107f9 (stale) Mirror 3: 0x921107f9 (stale) Mirror 4: 0x921107f9 (stale) CRC-32 checksum value for chunk [0, 0x34338): Mirror 1: 0xa28f2938 Mirror 5: 0xa28f2938 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34338: skipped 000233 1789118545.697567 mapwrite 0x027b02 thru 0x036be8 (0xf0e7 bytes) 000234 1789118545.962341 mapread 0x00e3e0 thru 0x014f26 (0x6b47 bytes) 000236 1789118546.009302 MIRROR_RESYNC 5 mirrors 000239 1789118547.126863 trunc from 0x036be9 to 0x02fc8e (0x6f5a bytes) 000241 1789118547.300947 mapwrite 0x0195fd thru 0x01d2b1 (0x3cb5 bytes) 000242 1789118547.426467 read 0x01e787 thru 0x020c31 (0x24ab bytes) 000243 1789118547.427277 trunc from 0x02fc8f to 0x02fd61 (0xd3 bytes) 000244 1789118547.485228 mapread 0x00ebc4 thru 0x0155b3 (0x69f0 bytes) 000247 1789118547.558920 trunc from 0x02fd62 to 0x01b3a7 (0x149ba bytes) 000248 1789118547.580953 mapwrite 0x00dc2a thru 0x00dd41 (0x118 bytes) 000249 1789118547.671902 mapread 0x004f91 thru 0x006a73 (0x1ae3 bytes) 000251 1789118547.687297 read 0x009364 thru 0x015d9d (0xca3a bytes) 000252 1789118547.696246 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1b3a8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b3a8): Mirror 2: 0x63db405d (stale) Mirror 3: 0x63db405d (stale) Mirror 4: 0x63db405d (stale) Mirror 5: 0x63db405d (stale) CRC-32 checksum value for chunk [0, 0x1b3a8): Mirror 1: 0x79a10a40 Mirror 6: 0x79a10a40 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b3a8: skipped 000253 1789118548.752452 mapwrite 0x027201 thru 0x030e1d (0x9c1d bytes) 000254 1789118548.957368 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x30e1e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30e1e): Mirror 2: 0xd57c020b (stale) Mirror 3: 0xd57c020b (stale) Mirror 4: 0xd57c020b (stale) Mirror 5: 0xd57c020b (stale) Mirror 6: 0x706d8739 (stale) CRC-32 checksum value for chunk [0, 0x30e1e): Mirror 1: 0x173facb0 Mirror 7: 0x173facb0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30e1e: skipped 000258 1789118549.885603 trunc from 0x030e1e to 0x03219e (0x1381 bytes) 000259 1789118549.993171 trunc from 0x03219f to 0x023268 (0xef36 bytes) 000260 1789118550.012612 trunc from 0x023269 to 0x02130c (0x1f5c bytes) 000261 1789118550.031048 mapwrite 0x016f6f thru 0x01c9e7 (0x5a79 bytes) 000262 1789118550.114116 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2130d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2130d): Mirror 2: 0xed52f106 (stale) Mirror 3: 0xed52f106 (stale) Mirror 4: 0xed52f106 (stale) Mirror 5: 0xed52f106 (stale) Mirror 6: 0x92322c63 (stale) Mirror 7: 0x92322c63 (stale) CRC-32 checksum value for chunk [0, 0x2130d): Mirror 1: 0x54e4d26d Mirror 8: 0x54e4d26d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2130d: skipped 000263 1789118551.043279 read 0x0022bc thru 0x00c352 (0xa097 bytes) 000264 1789118551.064465 mapread 0x007cd3 thru 0x012c9c (0xafca bytes) 000265 1789118551.089706 trunc from 0x02130d to 0x01cb0b (0x4801 bytes) 000266 1789118551.269789 mapwrite 0x037665 thru 0x03ffff (0x899b bytes) 000267 1789118551.461677 read 0x0017e2 thru 0x00c9e4 (0xb203 bytes) 000271 1789118551.511257 mapwrite 0x03bede thru 0x03f2be (0x33e1 bytes) 000272 1789118551.604973 MIRROR_RESYNC 8 mirrors 000273 1789118553.430680 MIRROR_EXTEND to 9 mirrors 000274 1789118554.383181 trunc from 0x040000 to 0x033c0d (0xc3f2 bytes) 000275 1789118554.476417 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33c0e): Mirror 3: 0x8da60a0b (stale) Mirror 4: 0x8da60a0b (stale) Mirror 5: 0x8da60a0b (stale) Mirror 6: 0x8da60a0b (stale) Mirror 7: 0x8da60a0b (stale) Mirror 8: 0x8da60a0b (stale) Mirror 9: 0x8da60a0b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33c0e): Mirror 3: 0x8da60a0b (stale) Mirror 4: 0x8da60a0b (stale) Mirror 5: 0x8da60a0b (stale) Mirror 6: 0x8da60a0b (stale) Mirror 7: 0x8da60a0b (stale) Mirror 8: 0x8da60a0b (stale) Mirror 9: 0x8da60a0b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33c0e: skipped 000278 1789118554.952649 mapread 0x01cc20 thru 0x025413 (0x87f4 bytes) 000279 1789118554.972242 write 0x014af4 thru 0x01f52a (0xaa37 bytes) 000280 1789118554.986268 mapwrite 0x03f744 thru 0x03ffff (0x8bc bytes) 000281 1789118555.063954 write 0x027b1a thru 0x02d6f6 (0x5bdd bytes) 000282 1789118555.079032 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf4b55b5 (stale) Mirror 4: 0xf4b55b5 (stale) Mirror 5: 0xf4b55b5 (stale) Mirror 6: 0xf4b55b5 (stale) Mirror 7: 0xf4b55b5 (stale) Mirror 8: 0xf4b55b5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf4b55b5 (stale) Mirror 4: 0xf4b55b5 (stale) Mirror 5: 0xf4b55b5 (stale) Mirror 6: 0xf4b55b5 (stale) Mirror 7: 0xf4b55b5 (stale) Mirror 8: 0xf4b55b5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000283 1789118555.605077 write 0x0054ef thru 0x00e28f (0x8da1 bytes) 000286 1789118555.621704 mapread 0x03bba8 thru 0x03ffff (0x4458 bytes) 000287 1789118555.641694 read 0x01e5be thru 0x02b28b (0xccce bytes) 000288 1789118555.657390 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf4b55b5 (stale) Mirror 4: 0xf4b55b5 (stale) Mirror 5: 0xf4b55b5 (stale) Mirror 6: 0xf4b55b5 (stale) Mirror 7: 0xf4b55b5 (stale) Mirror 8: 0xf4b55b5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x18888517 Mirror 9: 0x18888517 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000289 1789118556.793678 write 0x02e221 thru 0x036649 (0x8429 bytes) 000290 1789118556.889793 mapwrite 0x0205f8 thru 0x02617e (0x5b87 bytes) 000292 1789118556.972595 mapread 0x03d3aa thru 0x03ffff (0x2c56 bytes) 000293 1789118556.984207 trunc from 0x040000 to 0x033217 (0xcde8 bytes) 000294 1789118557.008983 write 0x012f76 thru 0x014db1 (0x1e3c bytes) 000295 1789118557.025967 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33218): Mirror 3: 0x43152932 (stale) Mirror 4: 0x43152932 (stale) Mirror 5: 0x43152932 (stale) Mirror 6: 0x43152932 (stale) Mirror 8: 0x43152932 (stale) Mirror 9: 0x2ad9526f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33218): Mirror 3: 0x43152932 (stale) Mirror 4: 0x43152932 (stale) Mirror 5: 0x43152932 (stale) Mirror 6: 0x43152932 (stale) Mirror 8: 0x43152932 (stale) Mirror 9: 0x2ad9526f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33218: skipped 000298 1789118557.598380 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33218): Mirror 3: 0x43152932 (stale) Mirror 4: 0x43152932 (stale) Mirror 6: 0x43152932 (stale) Mirror 8: 0x43152932 (stale) Mirror 9: 0x2ad9526f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33218): Mirror 3: 0x43152932 (stale) Mirror 4: 0x43152932 (stale) Mirror 6: 0x43152932 (stale) Mirror 8: 0x43152932 (stale) Mirror 9: 0x2ad9526f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33218: skipped 000300 1789118557.970219 mapwrite 0x004e22 thru 0x00c032 (0x7211 bytes) 000301 1789118558.070835 mapread 0x020c8c thru 0x02ea66 (0xdddb bytes) 000305 1789118558.113837 mapread 0x00e130 thru 0x01ca0e (0xe8df bytes) 000306 1789118558.126844 mapwrite 0x0343dd thru 0x03ffff (0xbc23 bytes) 000307 1789118558.241625 mapwrite 0x017074 thru 0x01c537 (0x54c4 bytes) 000310 1789118558.297464 MIRROR_RESYNC 6 mirrors 000312 1789118559.569178 mapread 0x000328 thru 0x00ab7e (0xa857 bytes) 000313 1789118559.602852 write 0x018c36 thru 0x0241c4 (0xb58f bytes) 000314 1789118559.745625 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7a859dbb (stale) Mirror 4: 0x7a859dbb (stale) Mirror 6: 0x7a859dbb (stale) Mirror 8: 0x7a859dbb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7a859dbb (stale) Mirror 4: 0x7a859dbb (stale) Mirror 6: 0x7a859dbb (stale) Mirror 8: 0x7a859dbb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000315 1789118560.210129 mapwrite 0x03c614 thru 0x03ffff (0x39ec bytes) 000316 1789118560.331253 write 0x0227c3 thru 0x02efb6 (0xc7f4 bytes) 000317 1789118560.354031 trunc from 0x040000 to 0x001cf9 (0x3e306 bytes) 000318 1789118560.380663 write 0x03cbcc thru 0x03ffff (0x3434 bytes) 000319 1789118560.382885 write 0x005c38 thru 0x0150f8 (0xf4c1 bytes) 000322 1789118560.420208 MIRROR_RESYNC 5 mirrors 000323 1789118561.514293 trunc from 0x040000 to 0x01ffc5 (0x2003a bytes) 000324 1789118561.604917 mapread 0x0195ee thru 0x01b17b (0x1b8e bytes) 000329 1789118561.678850 trunc from 0x01ffc6 to 0x03319a (0x131d5 bytes) 000330 1789118561.717767 read 0x021526 thru 0x02f7e6 (0xe2c1 bytes) 000331 1789118561.730284 read 0x01d14e thru 0x026541 (0x93f4 bytes) 000332 1789118561.758977 trunc from 0x03319b to 0x0079e3 (0x2b7b7 bytes) 000335 1789118561.792164 read 0x0052f1 thru 0x0079e3 (0x26f3 bytes) 000336 1789118561.794060 write 0x01d591 thru 0x021b14 (0x4584 bytes) 000339 1789118561.798843 trunc from 0x021b15 to 0x02c7b4 (0xaca0 bytes) 000340 1789118561.819937 trunc from 0x02c7b5 to 0x01097b (0x1be39 bytes) 000342 1789118561.840966 mapread 0x0033e0 thru 0x006379 (0x2f9a bytes) 000344 1789118561.857988 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1097c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1097c): Mirror 3: 0x8e9de086 (stale) Mirror 4: 0x8e9de086 (stale) Mirror 6: 0x8e9de086 (stale) Mirror 8: 0x8e9de086 (stale) CRC-32 checksum value for chunk [0, 0x1097c): Mirror 1: 0x22ef8e85 Mirror 9: 0x22ef8e85 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1097c: skipped 000345 1789118562.928205 MIRROR_RESYNC 6 mirrors 000347 1789118564.253914 trunc from 0x01097c to 0x000924 (0x10057 bytes) 000354 1789118564.372848 MIRROR_RESYNC 6 mirrors 000355 1789118565.531003 write 0x0255be thru 0x02b9aa (0x63ed bytes) 000356 1789118565.603650 mapwrite 0x0054df thru 0x0079f2 (0x2514 bytes) 000359 1789118565.758315 mapwrite 0x014acd thru 0x01b93c (0x6e70 bytes) truncating to largest ever: 0x3b587 000361 1789118565.850853 trunc from 0x02b9ab to 0x03b586 (0xfbdc bytes) 000363 1789118565.900037 MIRROR_RESYNC 6 mirrors 000364 1789118567.341908 write 0x032f52 thru 0x03ce48 (0x9ef7 bytes) 000365 1789118567.462288 read 0x024314 thru 0x02b9bb (0x76a8 bytes) 000367 1789118567.485873 read 0x03ad48 thru 0x03b41c (0x6d5 bytes) 000368 1789118567.486889 read 0x01dddd thru 0x02b416 (0xd63a bytes) 000370 1789118567.498297 mapwrite 0x024d53 thru 0x025ba6 (0xe54 bytes) 000372 1789118567.568808 mapread 0x01431b thru 0x01f54f (0xb235 bytes) 000374 1789118567.571363 read 0x01cc10 thru 0x024c4c (0x803d bytes) 000375 1789118567.577185 MIRROR_RESYNC 6 mirrors 000377 1789118568.638759 write 0x03f598 thru 0x03ffff (0xa68 bytes) 000378 1789118568.804909 mapread 0x010cbc thru 0x018334 (0x7679 bytes) 000380 1789118568.833806 mapwrite 0x02463d thru 0x03354e (0xef12 bytes) 000382 1789118568.904194 mapwrite 0x010e30 thru 0x01b54b (0xa71c bytes) 000383 1789118568.950986 mapwrite 0x00cb86 thru 0x0114cd (0x4948 bytes) 000384 1789118568.992002 trunc from 0x040000 to 0x0287d8 (0x17827 bytes) 000385 1789118569.012798 write 0x038b77 thru 0x03ffff (0x7489 bytes) 000386 1789118569.014124 read 0x03880b thru 0x039f09 (0x16ff bytes) 000387 1789118569.014672 write 0x00b444 thru 0x012fbe (0x7b7b bytes) 000389 1789118569.018657 write 0x00402d thru 0x00a98d (0x6961 bytes) 000391 1789118569.027325 mapwrite 0x0389b6 thru 0x03afce (0x2619 bytes) 000392 1789118569.068389 trunc from 0x040000 to 0x01c25b (0x23da4 bytes) 000393 1789118569.084694 read 0x01ae9e thru 0x01c25b (0x13be bytes) 000394 1789118569.090024 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1c25c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c25c): Mirror 3: 0x7590b84 (stale) Mirror 4: 0x7590b84 (stale) Mirror 6: 0x7590b84 (stale) Mirror 8: 0x7590b84 (stale) Mirror 9: 0x7590b84 (stale) CRC-32 checksum value for chunk [0, 0x1c25c): Mirror 1: 0x4ac75d56 Mirror 10: 0x4ac75d56 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c25c: skipped 000395 1789118569.742122 write 0x00f8a2 thru 0x011ca4 (0x2403 bytes) 000396 1789118569.843268 trunc from 0x01c25c to 0x025e08 (0x9bad bytes) 000397 1789118569.886783 write 0x03f214 thru 0x03ffff (0xdec bytes) 000400 1789118569.888688 read 0x003e7f thru 0x00c460 (0x85e2 bytes) 000401 1789118569.898714 write 0x03fd54 thru 0x03ffff (0x2ac bytes) 000402 1789118569.899921 trunc from 0x040000 to 0x0251aa (0x1ae55 bytes) 000403 1789118569.917960 write 0x00402b thru 0x0116ac (0xd682 bytes) 000404 1789118569.920015 trunc from 0x0251ab to 0x02b073 (0x5ec9 bytes) 000405 1789118569.973834 write 0x00b032 thru 0x01753b (0xc50a bytes) 000406 1789118569.987041 trunc from 0x02b074 to 0x01dbb6 (0xd4bd bytes) 000407 1789118570.021118 mapwrite 0x00a5a6 thru 0x0124e8 (0x7f43 bytes) 000410 1789118570.088745 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1dbb7): Mirror 3: 0xc2097760 (stale) Mirror 4: 0xc2097760 (stale) Mirror 6: 0xc2097760 (stale) Mirror 8: 0xc2097760 (stale) Mirror 9: 0xc2097760 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dbb7): Mirror 3: 0xc2097760 (stale) Mirror 4: 0xc2097760 (stale) Mirror 6: 0xc2097760 (stale) Mirror 8: 0xc2097760 (stale) Mirror 9: 0xc2097760 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dbb7: skipped 000414 1789118570.634771 trunc from 0x01dbb7 to 0x02e63a (0x10a84 bytes) 000416 1789118570.650789 mapread 0x00b79c thru 0x01752a (0xbd8f bytes) 000419 1789118570.682002 read 0x01ecd4 thru 0x01f637 (0x964 bytes) 000422 1789118570.687923 mapread 0x00e006 thru 0x015804 (0x77ff bytes) 000424 1789118570.690042 read 0x02ab31 thru 0x02e63a (0x3b0a bytes) 000425 1789118570.703123 MIRROR_RESYNC 6 mirrors 000426 1789118572.168415 mapread 0x00db9f thru 0x0108ff (0x2d61 bytes) 000428 1789118572.202135 write 0x035f95 thru 0x03ee72 (0x8ede bytes) 000429 1789118572.329782 mapread 0x02f49d thru 0x03a2d6 (0xae3a bytes) 000432 1789118572.345691 trunc from 0x03ee73 to 0x036b74 (0x82fe bytes) 000435 1789118572.366059 mapread 0x00b4c3 thru 0x01516c (0x9caa bytes) 000436 1789118572.390093 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x36b75): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36b75): Mirror 3: 0x1c52518a (stale) Mirror 4: 0x1c52518a (stale) Mirror 6: 0x1c52518a (stale) Mirror 8: 0x1c52518a (stale) Mirror 9: 0x1c52518a (stale) CRC-32 checksum value for chunk [0, 0x36b75): Mirror 1: 0x6fa11cba Mirror 10: 0x6fa11cba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36b75: skipped 000437 1789118573.272852 mapwrite 0x002bab thru 0x00ae15 (0x826b bytes) 000439 1789118573.604030 mapwrite 0x002702 thru 0x003203 (0xb02 bytes) 000440 1789118573.692981 mapread 0x0089c0 thru 0x00d870 (0x4eb1 bytes) 000444 1789118573.699028 trunc from 0x036b75 to 0x03a0dc (0x3568 bytes) 000445 1789118573.759010 trunc from 0x03a0dd to 0x02faa3 (0xa639 bytes) 000446 1789118573.782703 mapwrite 0x02cca5 thru 0x0313f2 (0x474e bytes) 000447 1789118573.869748 mapwrite 0x029e66 thru 0x035c8a (0xbe25 bytes) 000448 1789118573.969150 write 0x021536 thru 0x02928c (0x7d57 bytes) 000452 1789118573.977925 read 0x02f5b8 thru 0x032719 (0x3162 bytes) 000454 1789118573.984718 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x35c8b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35c8b): Mirror 3: 0x7d5d2cd0 (stale) Mirror 4: 0x7d5d2cd0 (stale) Mirror 6: 0x7d5d2cd0 (stale) Mirror 8: 0x7d5d2cd0 (stale) Mirror 9: 0x7d5d2cd0 (stale) Mirror 10: 0x7d5d2cd0 (stale) CRC-32 checksum value for chunk [0, 0x35c8b): Mirror 1: 0x2e5c84a1 Mirror 11: 0x2e5c84a1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35c8b: skipped 000456 1789118574.955257 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x35c8b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35c8b): Mirror 3: 0x7d5d2cd0 (stale) Mirror 4: 0x7d5d2cd0 (stale) Mirror 6: 0x7d5d2cd0 (stale) Mirror 8: 0x7d5d2cd0 (stale) Mirror 9: 0x7d5d2cd0 (stale) Mirror 10: 0x7d5d2cd0 (stale) CRC-32 checksum value for chunk [0, 0x35c8b): Mirror 1: 0x2e5c84a1 Mirror 11: 0x2e5c84a1 Mirror 12: 0x2e5c84a1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35c8b: skipped 000458 1789118575.961893 mapread 0x00a68c thru 0x010213 (0x5b88 bytes) 000460 1789118575.982745 read 0x012f30 thru 0x018ba8 (0x5c79 bytes) 000461 1789118575.992281 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x35c8b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35c8b): Mirror 3: 0x7d5d2cd0 (stale) Mirror 4: 0x7d5d2cd0 (stale) Mirror 6: 0x7d5d2cd0 (stale) Mirror 8: 0x7d5d2cd0 (stale) Mirror 9: 0x7d5d2cd0 (stale) Mirror 10: 0x7d5d2cd0 (stale) CRC-32 checksum value for chunk [0, 0x35c8b): Mirror 1: 0x2e5c84a1 Mirror 11: 0x2e5c84a1 Mirror 12: 0x2e5c84a1 Mirror 13: 0x2e5c84a1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35c8b: skipped 000462 1789118577.251721 write 0x014557 thru 0x017ed2 (0x397c bytes) 000463 1789118577.413975 mapwrite 0x006aee thru 0x00d602 (0x6b15 bytes) 000464 1789118577.488445 read 0x018cee thru 0x020511 (0x7824 bytes) 000468 1789118577.504686 write 0x007123 thru 0x00b9b3 (0x4891 bytes) 000469 1789118577.506687 write 0x02eef3 thru 0x034e87 (0x5f95 bytes) 000473 1789118577.521122 mapwrite 0x02ed3c thru 0x03d866 (0xeb2b bytes) 000474 1789118577.597114 mapread 0x028b6b thru 0x029b45 (0xfdb bytes) 000475 1789118577.617008 write 0x034fbf thru 0x03e96a (0x99ac bytes) 000476 1789118577.619137 mapread 0x0336f8 thru 0x039d2f (0x6638 bytes) 000477 1789118577.620434 write 0x013afe thru 0x0200a5 (0xc5a8 bytes) 000478 1789118577.628656 trunc from 0x03e96b to 0x02b3e3 (0x13587 bytes) 000479 1789118577.647977 mapread 0x00a9c6 thru 0x016df7 (0xc432 bytes) 000480 1789118577.657654 mapwrite 0x010d01 thru 0x01c6d9 (0xb9d9 bytes) 000482 1789118577.717758 write 0x03c6bc thru 0x03ffff (0x3944 bytes) 000483 1789118577.719608 mapread 0x01335d thru 0x0196dc (0x6380 bytes) 000487 1789118577.730620 read 0x00ea93 thru 0x01d545 (0xeab3 bytes) 000488 1789118577.731517 read 0x03e902 thru 0x03ff4e (0x164d bytes) 000489 1789118577.732745 mapread 0x0003f1 thru 0x000c37 (0x847 bytes) 000490 1789118577.745192 read 0x00b449 thru 0x00c6a2 (0x125a bytes) 000492 1789118577.746348 read 0x032a84 thru 0x03f643 (0xcbc0 bytes) 000494 1789118577.749191 read 0x0376b9 thru 0x0379e1 (0x329 bytes) 000496 1789118577.750207 read 0x005f70 thru 0x00f91d (0x99ae bytes) 000497 1789118577.750726 mapwrite 0x00a75c thru 0x00f5bc (0x4e61 bytes) 000498 1789118577.825941 write 0x008402 thru 0x0125c4 (0xa1c3 bytes) 000499 1789118577.829557 mapwrite 0x0075e8 thru 0x0169ed (0xf406 bytes) 000500 1789118577.911205 trunc from 0x040000 to 0x01dcab (0x22354 bytes) truncating to largest ever: 0x3e17f 000501 1789118577.937791 trunc from 0x01dcac to 0x03e17e (0x204d3 bytes) 000504 1789118577.963835 mapwrite 0x03a442 thru 0x03a4fa (0xb9 bytes) 000506 1789118578.046101 mapwrite 0x0347af thru 0x039327 (0x4b79 bytes) 000508 1789118578.192358 mapread 0x02b88f thru 0x03af96 (0xf708 bytes) 000509 1789118578.197738 write 0x00336b thru 0x004c5e (0x18f4 bytes) 000510 1789118578.212428 MIRROR_RESYNC 10 mirrors 000511 1789118580.139641 mapwrite 0x00dfab thru 0x0171c6 (0x921c bytes) 000512 1789118580.583437 trunc from 0x03e17f to 0x022d50 (0x1b42e bytes) 000514 1789118580.665506 mapread 0x004754 thru 0x005a39 (0x12e6 bytes) 000516 1789118580.684042 trunc from 0x022d51 to 0x0388bd (0x15b6d bytes) 000518 1789118580.700749 write 0x02417a thru 0x02e191 (0xa018 bytes) 000519 1789118580.711196 read 0x027c40 thru 0x02d239 (0x55fa bytes) 000521 1789118580.713226 trunc from 0x0388be to 0x01977e (0x1f13f bytes) 000522 1789118580.734119 trunc from 0x01977f to 0x0277fa (0xe07c bytes) 000523 1789118580.761368 mapread 0x015429 thru 0x024bde (0xf7b6 bytes) 000524 1789118580.782586 mapread 0x00cc25 thru 0x011813 (0x4bef bytes) 000526 1789118580.785360 mapwrite 0x018366 thru 0x020bed (0x8888 bytes) 000528 1789118580.839868 mapread 0x02772a thru 0x0277fa (0xd1 bytes) 000529 1789118580.846337 MIRROR_RESYNC 10 mirrors 000531 1789118582.330333 MIRROR_RESYNC 10 mirrors 000534 1789118582.975884 mapwrite 0x0025da thru 0x0053a1 (0x2dc8 bytes) 000535 1789118583.142753 MIRROR_RESYNC 10 mirrors 000537 1789118584.637855 read 0x011f2d thru 0x019e6c (0x7f40 bytes) 000538 1789118584.645720 read 0x01d4f7 thru 0x0277fa (0xa304 bytes) 000539 1789118584.658813 trunc from 0x0277fb to 0x03688c (0xf092 bytes) 000540 1789118584.743332 MIRROR_RESYNC 10 mirrors 000542 1789118586.286189 read 0x020107 thru 0x02caed (0xc9e7 bytes) 000544 1789118586.295012 write 0x00690e thru 0x006d1d (0x410 bytes) 000547 1789118586.363979 mapwrite 0x026af0 thru 0x02a156 (0x3667 bytes) 000549 1789118586.438356 read 0x01fceb thru 0x02d841 (0xdb57 bytes) 000550 1789118586.461064 mapwrite 0x001073 thru 0x008696 (0x7624 bytes) 000551 1789118586.502721 trunc from 0x03688d to 0x0300c5 (0x67c7 bytes) 000552 1789118586.521617 write 0x03561a thru 0x039163 (0x3b4a bytes) 000553 1789118586.524451 write 0x031ccc thru 0x0375be (0x58f3 bytes) 000554 1789118586.531834 write 0x01e202 thru 0x02d550 (0xf34f bytes) 000555 1789118586.535420 read 0x008584 thru 0x017d1b (0xf798 bytes) 000556 1789118586.536231 read 0x013fa5 thru 0x01520b (0x1267 bytes) 000558 1789118586.541704 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x39164): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39164): Mirror 3: 0x3ea11776 (stale) Mirror 4: 0x3ea11776 (stale) Mirror 6: 0x3ea11776 (stale) Mirror 8: 0x3ea11776 (stale) Mirror 9: 0x3ea11776 (stale) Mirror 10: 0x3ea11776 (stale) Mirror 11: 0x3ea11776 (stale) Mirror 12: 0x3ea11776 (stale) Mirror 13: 0x3ea11776 (stale) CRC-32 checksum value for chunk [0, 0x39164): Mirror 1: 0xe8cdc996 Mirror 14: 0xe8cdc996 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39164: skipped 000559 1789118587.574967 mapread 0x02d020 thru 0x030546 (0x3527 bytes) 000560 1789118587.594219 mapread 0x01b714 thru 0x02a3a6 (0xec93 bytes) 000561 1789118587.625118 read 0x011822 thru 0x0141cd (0x29ac bytes) 000562 1789118587.632185 mapwrite 0x020a67 thru 0x02f2e5 (0xe87f bytes) 000565 1789118587.842261 trunc from 0x039164 to 0x032249 (0x6f1a bytes) 000566 1789118587.886675 read 0x008f24 thru 0x011e39 (0x8f16 bytes) 000567 1789118587.904573 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3224a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3224a): Mirror 3: 0xdb426757 (stale) Mirror 4: 0xdb426757 (stale) Mirror 6: 0xdb426757 (stale) Mirror 8: 0xdb426757 (stale) Mirror 9: 0xdb426757 (stale) Mirror 10: 0xdb426757 (stale) Mirror 11: 0xdb426757 (stale) Mirror 12: 0xdb426757 (stale) Mirror 13: 0xdb426757 (stale) Mirror 14: 0xc1a7ea7c (stale) CRC-32 checksum value for chunk [0, 0x3224a): Mirror 1: 0xbbbb9522 Mirror 15: 0xbbbb9522 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3224a: skipped 000568 1789118589.253739 write 0x03b401 thru 0x03ff2a (0x4b2a bytes) 000569 1789118589.346748 write 0x032daa thru 0x035efa (0x3151 bytes) 000570 1789118589.361952 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ff2b): Mirror 3: 0x13dbf735 (stale) Mirror 4: 0x13dbf735 (stale) Mirror 6: 0x13dbf735 (stale) Mirror 8: 0x13dbf735 (stale) Mirror 9: 0x13dbf735 (stale) Mirror 10: 0x13dbf735 (stale) Mirror 12: 0x13dbf735 (stale) Mirror 13: 0x13dbf735 (stale) Mirror 14: 0xdf1e800f (stale) Mirror 15: 0xdb5a4bda (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ff2b): Mirror 3: 0x13dbf735 (stale) Mirror 4: 0x13dbf735 (stale) Mirror 6: 0x13dbf735 (stale) Mirror 8: 0x13dbf735 (stale) Mirror 9: 0x13dbf735 (stale) Mirror 10: 0x13dbf735 (stale) Mirror 12: 0x13dbf735 (stale) Mirror 13: 0x13dbf735 (stale) Mirror 14: 0xdf1e800f (stale) Mirror 15: 0xdb5a4bda (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ff2b: skipped 000572 1789118590.132106 read 0x0330a5 thru 0x03b380 (0x82dc bytes) 000575 1789118590.154090 mapwrite 0x0087d1 thru 0x00d2bf (0x4aef bytes) 000577 1789118590.285347 mapwrite 0x03ba25 thru 0x03ffff (0x45db bytes) 000578 1789118590.390136 write 0x010e1b thru 0x01d5de (0xc7c4 bytes) 000579 1789118590.398277 read 0x035acf thru 0x03a69c (0x4bce bytes) 000580 1789118590.398975 trunc from 0x040000 to 0x00dbfb (0x32404 bytes) 000582 1789118590.427724 mapread 0x003b4c thru 0x00dbfb (0xa0b0 bytes) 000583 1789118590.428778 read 0x00a1bd thru 0x00dbfb (0x3a3f bytes) 000585 1789118590.430813 mapread 0x0055fa thru 0x00b1bc (0x5bc3 bytes) 000589 1789118590.435590 write 0x0131d8 thru 0x018a9b (0x58c4 bytes) 000590 1789118590.437073 mapwrite 0x0220ac thru 0x024f3c (0x2e91 bytes) 000592 1789118590.557842 read 0x0099b0 thru 0x00da75 (0x40c6 bytes) 000594 1789118590.558896 mapread 0x00ec2b thru 0x0185c7 (0x999d bytes) 000595 1789118590.561013 mapread 0x0015f4 thru 0x00dfa5 (0xc9b2 bytes) 000596 1789118590.569703 mapwrite 0x0148e2 thru 0x023b7d (0xf29c bytes) 000598 1789118590.659690 mapread 0x003c30 thru 0x010076 (0xc447 bytes) 000601 1789118590.664635 read 0x019794 thru 0x01e0a6 (0x4913 bytes) 000602 1789118590.665575 write 0x00fd0b thru 0x010d39 (0x102f bytes) 000603 1789118590.668977 mapread 0x0096b9 thru 0x01899c (0xf2e4 bytes) 000604 1789118590.670710 mapwrite 0x025e40 thru 0x02b77c (0x593d bytes) 000607 1789118590.783864 write 0x0098c7 thru 0x00d12c (0x3866 bytes) 000608 1789118590.786620 write 0x01a008 thru 0x02354f (0x9548 bytes) 000609 1789118590.792076 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b77d): Mirror 3: 0x3232dcc1 (stale) Mirror 6: 0x3232dcc1 (stale) Mirror 8: 0x3232dcc1 (stale) Mirror 9: 0x3232dcc1 (stale) Mirror 10: 0x3232dcc1 (stale) Mirror 12: 0x3232dcc1 (stale) Mirror 13: 0x3232dcc1 (stale) Mirror 14: 0xf8f583ef (stale) Mirror 15: 0xcb8d9fbb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b77d): Mirror 3: 0x3232dcc1 (stale) Mirror 6: 0x3232dcc1 (stale) Mirror 8: 0x3232dcc1 (stale) Mirror 9: 0x3232dcc1 (stale) Mirror 10: 0x3232dcc1 (stale) Mirror 12: 0x3232dcc1 (stale) Mirror 13: 0x3232dcc1 (stale) Mirror 14: 0xf8f583ef (stale) Mirror 15: 0xcb8d9fbb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b77d: skipped 000610 1789118591.895441 write 0x00586c thru 0x00a556 (0x4ceb bytes) 000611 1789118591.919526 mapwrite 0x00dea9 thru 0x014ca9 (0x6e01 bytes) 000612 1789118591.989966 write 0x010e97 thru 0x01b97d (0xaae7 bytes) 000615 1789118591.997319 mapread 0x013abb thru 0x021d3e (0xe284 bytes) 000616 1789118592.012223 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b77d): Mirror 3: 0x3232dcc1 (stale) Mirror 6: 0x3232dcc1 (stale) Mirror 8: 0x3232dcc1 (stale) Mirror 10: 0x3232dcc1 (stale) Mirror 12: 0x3232dcc1 (stale) Mirror 13: 0x3232dcc1 (stale) Mirror 14: 0xf8f583ef (stale) Mirror 15: 0xcb8d9fbb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b77d): Mirror 3: 0x3232dcc1 (stale) Mirror 6: 0x3232dcc1 (stale) Mirror 8: 0x3232dcc1 (stale) Mirror 10: 0x3232dcc1 (stale) Mirror 12: 0x3232dcc1 (stale) Mirror 13: 0x3232dcc1 (stale) Mirror 14: 0xf8f583ef (stale) Mirror 15: 0xcb8d9fbb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b77d: skipped 000618 1789118592.507938 read 0x012c69 thru 0x01f9d0 (0xcd68 bytes) 000621 1789118592.521687 trunc from 0x02b77d to 0x02eff0 (0x3874 bytes) 000624 1789118592.541977 write 0x035f4b thru 0x03ffff (0xa0b5 bytes) 000625 1789118592.544384 write 0x001c50 thru 0x00ec18 (0xcfc9 bytes) 000626 1789118592.557319 read 0x013931 thru 0x01f3d7 (0xbaa7 bytes) 000627 1789118592.564842 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x768cbf49 (stale) Mirror 6: 0x768cbf49 (stale) Mirror 8: 0x768cbf49 (stale) Mirror 10: 0x768cbf49 (stale) Mirror 12: 0x768cbf49 (stale) Mirror 13: 0x768cbf49 (stale) Mirror 14: 0xa8b04171 (stale) Mirror 15: 0x5f7a5819 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd09c5bbf Mirror 16: 0xd09c5bbf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000629 1789118593.932469 mapwrite 0x021108 thru 0x025177 (0x4070 bytes) 000631 1789118594.121470 trunc from 0x040000 to 0x017e9a (0x28165 bytes) 000632 1789118594.147134 write 0x02772f thru 0x030b67 (0x9439 bytes) 000633 1789118594.152550 mapread 0x0265b9 thru 0x030b67 (0xa5af bytes) 000635 1789118594.161325 read 0x01dc8d thru 0x021f0c (0x4280 bytes) 000637 1789118594.167169 trunc from 0x030b68 to 0x011983 (0x1f1e4 bytes) 000640 1789118594.193934 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x11984): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11984): Mirror 3: 0x913ea3a2 (stale) Mirror 6: 0x913ea3a2 (stale) Mirror 8: 0x913ea3a2 (stale) Mirror 10: 0x913ea3a2 (stale) Mirror 12: 0x913ea3a2 (stale) Mirror 13: 0x913ea3a2 (stale) Mirror 14: 0x556492cc (stale) Mirror 15: 0x556492cc (stale) Mirror 16: 0x10e227ed (stale) CRC-32 checksum value for chunk [0, 0x11984): Mirror 1: 0x10e227ed Mirror 17: 0x10e227ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11984: skipped 000642 1789118595.053923 trunc from 0x011984 to 0x01933e (0x79bb bytes) 000643 1789118595.187315 read 0x006fae thru 0x013d7d (0xcdd0 bytes) 000644 1789118595.197050 mapwrite 0x00bbca thru 0x011f54 (0x638b bytes) 000645 1789118595.245818 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x1933f): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1933f): Mirror 3: 0xe452b4b3 (stale) Mirror 6: 0xe452b4b3 (stale) Mirror 8: 0xe452b4b3 (stale) Mirror 10: 0xe452b4b3 (stale) Mirror 12: 0xe452b4b3 (stale) Mirror 13: 0xe452b4b3 (stale) Mirror 14: 0x8c24535 (stale) Mirror 15: 0x8c24535 (stale) Mirror 16: 0x5f9b4c94 (stale) Mirror 17: 0xd1a002d6 (stale) CRC-32 checksum value for chunk [0, 0x1933f): Mirror 1: 0x2c0b5c19 Mirror 18: 0x2c0b5c19 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1933f: skipped 000646 1789118596.301442 mapread 0x004ac5 thru 0x012cb1 (0xe1ed bytes) 000647 1789118596.354894 mapread 0x000fc4 thru 0x001c77 (0xcb4 bytes) 000649 1789118596.363979 mapread 0x010a2c thru 0x01933e (0x8913 bytes) 000654 1789118596.365945 write 0x033bfa thru 0x036b7a (0x2f81 bytes) 000656 1789118596.558799 MIRROR_RESYNC 12 mirrors 000659 1789118598.977823 MIRROR_EXTEND to 13 mirrors 000660 1789118600.123406 mapwrite 0x021c81 thru 0x02d506 (0xb886 bytes) 000668 1789118600.282260 trunc from 0x036b7b to 0x01cba1 (0x19fd9 bytes) 000670 1789118600.306316 write 0x02bfc1 thru 0x02e2b0 (0x22f0 bytes) 000671 1789118600.308471 write 0x0164f6 thru 0x016742 (0x24d bytes) 000673 1789118600.317055 mapread 0x017d2e thru 0x02171d (0x99f0 bytes) 000674 1789118600.333418 trunc from 0x02e2b1 to 0x01af96 (0x1331a bytes) 000676 1789118600.353183 trunc from 0x01af97 to 0x023077 (0x80e1 bytes) 000679 1789118600.374153 trunc from 0x023078 to 0x015c9c (0xd3db bytes) 000680 1789118600.388403 mapread 0x00c540 thru 0x0104a8 (0x3f69 bytes) 000681 1789118600.403844 MIRROR_SPLIT mirror 4 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15c9d): Mirror 3: 0xcd155daf (stale) Mirror 6: 0xcd155daf (stale) Mirror 8: 0xcd155daf (stale) Mirror 12: 0xcd155daf (stale) Mirror 13: 0xcd155daf (stale) Mirror 14: 0xcd155daf (stale) Mirror 15: 0xcd155daf (stale) Mirror 16: 0xcd155daf (stale) Mirror 17: 0xcd155daf (stale) Mirror 18: 0xcd155daf (stale) Mirror 19: 0xcd155daf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15c9d): Mirror 3: 0xcd155daf (stale) Mirror 6: 0xcd155daf (stale) Mirror 8: 0xcd155daf (stale) Mirror 12: 0xcd155daf (stale) Mirror 13: 0xcd155daf (stale) Mirror 14: 0xcd155daf (stale) Mirror 15: 0xcd155daf (stale) Mirror 16: 0xcd155daf (stale) Mirror 17: 0xcd155daf (stale) Mirror 18: 0xcd155daf (stale) Mirror 19: 0xcd155daf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15c9d: skipped 000685 1789118600.959047 write 0x014f56 thru 0x0174c8 (0x2573 bytes) 000686 1789118600.966149 write 0x00e5c4 thru 0x01c933 (0xe370 bytes) 000687 1789118600.979942 MIRROR_RESYNC 12 mirrors 000688 1789118603.359203 mapwrite 0x013c4e thru 0x01c986 (0x8d39 bytes) 000689 1789118603.654798 trunc from 0x01c987 to 0x02c8d5 (0xff4f bytes) 000690 1789118603.684600 mapwrite 0x01917f thru 0x019e3e (0xcc0 bytes) 000691 1789118603.757031 read 0x01a6c1 thru 0x029087 (0xe9c7 bytes) 000692 1789118603.779536 write 0x028f8c thru 0x0313c8 (0x843d bytes) 000693 1789118603.781023 trunc from 0x0313c9 to 0x024488 (0xcf40 bytes) 000694 1789118603.842230 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x24489): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24489): Mirror 3: 0xfaa76fc4 (stale) Mirror 6: 0xfaa76fc4 (stale) Mirror 8: 0xfaa76fc4 (stale) Mirror 12: 0xfaa76fc4 (stale) Mirror 13: 0xfaa76fc4 (stale) Mirror 14: 0xfaa76fc4 (stale) Mirror 15: 0xfaa76fc4 (stale) Mirror 16: 0xfaa76fc4 (stale) Mirror 17: 0xfaa76fc4 (stale) Mirror 18: 0xfaa76fc4 (stale) Mirror 19: 0xfaa76fc4 (stale) CRC-32 checksum value for chunk [0, 0x24489): Mirror 1: 0xe5810630 Mirror 20: 0xe5810630 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24489: skipped 000696 1789118605.161043 trunc from 0x024489 to 0x02f0d2 (0xac4a bytes) 000697 1789118605.287593 MIRROR_RESYNC 13 mirrors 000698 1789118608.368828 write 0x01fa41 thru 0x024ffb (0x55bb bytes) 000702 1789118608.451158 write 0x01307d thru 0x017403 (0x4387 bytes) 000706 1789118608.470082 mapwrite 0x03eb66 thru 0x03ffff (0x149a bytes) 000707 1789118608.600953 MIRROR_RESYNC 13 mirrors 000711 1789118611.118325 write 0x00f56a thru 0x016d09 (0x77a0 bytes) 000714 1789118611.239468 trunc from 0x040000 to 0x023468 (0x1cb97 bytes) 000719 1789118611.268908 MIRROR_SPLIT mirror 4 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23469): Mirror 3: 0x1b544b9f (stale) Mirror 6: 0x1b544b9f (stale) Mirror 8: 0x1b544b9f (stale) Mirror 13: 0x1b544b9f (stale) Mirror 14: 0x1b544b9f (stale) Mirror 15: 0x1b544b9f (stale) Mirror 16: 0x1b544b9f (stale) Mirror 17: 0x1b544b9f (stale) Mirror 18: 0x1b544b9f (stale) Mirror 19: 0x1b544b9f (stale) Mirror 20: 0x1b544b9f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23469): Mirror 3: 0x1b544b9f (stale) Mirror 6: 0x1b544b9f (stale) Mirror 8: 0x1b544b9f (stale) Mirror 13: 0x1b544b9f (stale) Mirror 14: 0x1b544b9f (stale) Mirror 15: 0x1b544b9f (stale) Mirror 16: 0x1b544b9f (stale) Mirror 17: 0x1b544b9f (stale) Mirror 18: 0x1b544b9f (stale) Mirror 19: 0x1b544b9f (stale) Mirror 20: 0x1b544b9f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23469: skipped 000723 1789118611.875436 write 0x01ce04 thru 0x01fd64 (0x2f61 bytes) 000725 1789118611.886998 read 0x002152 thru 0x00e160 (0xc00f bytes) 000729 1789118611.903107 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x23469): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23469): Mirror 3: 0x1b544b9f (stale) Mirror 6: 0x1b544b9f (stale) Mirror 8: 0x1b544b9f (stale) Mirror 13: 0x1b544b9f (stale) Mirror 14: 0x1b544b9f (stale) Mirror 15: 0x1b544b9f (stale) Mirror 16: 0x1b544b9f (stale) Mirror 17: 0x1b544b9f (stale) Mirror 18: 0x1b544b9f (stale) Mirror 19: 0x1b544b9f (stale) Mirror 20: 0x1b544b9f (stale) CRC-32 checksum value for chunk [0, 0x23469): Mirror 1: 0xeb0bb64b Mirror 21: 0xeb0bb64b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23469: skipped 000730 1789118612.968982 MIRROR_RESYNC 13 mirrors 000731 1789118615.458146 read 0x017854 thru 0x022395 (0xab42 bytes) 000733 1789118615.485786 mapwrite 0x010b95 thru 0x01403b (0x34a7 bytes) 000734 1789118615.688647 write 0x01e414 thru 0x023a05 (0x55f2 bytes) 000737 1789118615.705960 mapwrite 0x0121dc thru 0x016376 (0x419b bytes) 000738 1789118615.756702 read 0x023240 thru 0x023a05 (0x7c6 bytes) 000739 1789118615.757287 read 0x002d18 thru 0x007373 (0x465c bytes) 000740 1789118615.767531 write 0x00fd37 thru 0x01aa6e (0xad38 bytes) 000741 1789118615.768752 mapwrite 0x00df12 thru 0x010830 (0x291f bytes) 000744 1789118615.885931 mapwrite 0x017d5b thru 0x0184a8 (0x74e bytes) 000745 1789118615.947927 write 0x02dbef thru 0x034d00 (0x7112 bytes) 000747 1789118615.963242 MIRROR_SPLIT mirror 8 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34d01): Mirror 3: 0xe7589204 (stale) Mirror 6: 0xe7589204 (stale) Mirror 8: 0xe7589204 (stale) Mirror 13: 0xe7589204 (stale) Mirror 14: 0xe7589204 (stale) Mirror 15: 0xe7589204 (stale) Mirror 16: 0xe7589204 (stale) Mirror 18: 0xe7589204 (stale) Mirror 19: 0xe7589204 (stale) Mirror 20: 0xe7589204 (stale) Mirror 21: 0xe7589204 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34d01): Mirror 3: 0xe7589204 (stale) Mirror 6: 0xe7589204 (stale) Mirror 8: 0xe7589204 (stale) Mirror 13: 0xe7589204 (stale) Mirror 14: 0xe7589204 (stale) Mirror 15: 0xe7589204 (stale) Mirror 16: 0xe7589204 (stale) Mirror 18: 0xe7589204 (stale) Mirror 19: 0xe7589204 (stale) Mirror 20: 0xe7589204 (stale) Mirror 21: 0xe7589204 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34d01: skipped 000748 1789118616.856379 read 0x01f3e7 thru 0x02b672 (0xc28c bytes) 000749 1789118616.869371 trunc from 0x034d01 to 0x005ef4 (0x2ee0c bytes) 000752 1789118616.897917 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x5ef5): Mirror 6: 0xd754b008 (stale) Mirror 8: 0xd754b008 (stale) Mirror 13: 0xd754b008 (stale) Mirror 14: 0xd754b008 (stale) Mirror 15: 0xd754b008 (stale) Mirror 16: 0xd754b008 (stale) Mirror 18: 0xd754b008 (stale) Mirror 19: 0xd754b008 (stale) Mirror 20: 0xd754b008 (stale) Mirror 21: 0xd754b008 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5ef5): Mirror 6: 0xd754b008 (stale) Mirror 8: 0xd754b008 (stale) Mirror 13: 0xd754b008 (stale) Mirror 14: 0xd754b008 (stale) Mirror 15: 0xd754b008 (stale) Mirror 16: 0xd754b008 (stale) Mirror 18: 0xd754b008 (stale) Mirror 19: 0xd754b008 (stale) Mirror 20: 0xd754b008 (stale) Mirror 21: 0xd754b008 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5ef5: skipped 000753 1789118617.545523 mapread 0x0048b3 thru 0x005ef4 (0x1642 bytes) 000755 1789118617.562801 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x5ef5): Mirror 6: 0xd754b008 (stale) Mirror 8: 0xd754b008 (stale) Mirror 13: 0xd754b008 (stale) Mirror 14: 0xd754b008 (stale) Mirror 15: 0xd754b008 (stale) Mirror 16: 0xd754b008 (stale) Mirror 18: 0xd754b008 (stale) Mirror 19: 0xd754b008 (stale) Mirror 21: 0xd754b008 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5ef5): Mirror 6: 0xd754b008 (stale) Mirror 8: 0xd754b008 (stale) Mirror 13: 0xd754b008 (stale) Mirror 14: 0xd754b008 (stale) Mirror 15: 0xd754b008 (stale) Mirror 16: 0xd754b008 (stale) Mirror 18: 0xd754b008 (stale) Mirror 19: 0xd754b008 (stale) Mirror 21: 0xd754b008 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5ef5: skipped 000762 1789118618.146030 MIRROR_RESYNC 10 mirrors 000763 1789118620.095583 write 0x007edc thru 0x00c594 (0x46b9 bytes) 000766 1789118620.189178 write 0x02f0f0 thru 0x03be60 (0xcd71 bytes) 000768 1789118620.191887 mapwrite 0x010cf5 thru 0x01972c (0x8a38 bytes) 000771 1789118620.268912 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3be61): Mirror 6: 0x8c70c908 (stale) Mirror 13: 0x8c70c908 (stale) Mirror 14: 0x8c70c908 (stale) Mirror 15: 0x8c70c908 (stale) Mirror 16: 0x8c70c908 (stale) Mirror 18: 0x8c70c908 (stale) Mirror 19: 0x8c70c908 (stale) Mirror 21: 0x8c70c908 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3be61): Mirror 6: 0x8c70c908 (stale) Mirror 13: 0x8c70c908 (stale) Mirror 14: 0x8c70c908 (stale) Mirror 15: 0x8c70c908 (stale) Mirror 16: 0x8c70c908 (stale) Mirror 18: 0x8c70c908 (stale) Mirror 19: 0x8c70c908 (stale) Mirror 21: 0x8c70c908 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3be61: skipped 000772 1789118620.745958 write 0x00b71b thru 0x00f45f (0x3d45 bytes) 000774 1789118620.754800 mapread 0x003100 thru 0x004958 (0x1859 bytes) 000775 1789118620.762749 mapwrite 0x02750c thru 0x02c9a7 (0x549c bytes) 000776 1789118620.817814 write 0x03c865 thru 0x03ffff (0x379b bytes) 000778 1789118620.819868 read 0x029f17 thru 0x030725 (0x680f bytes) 000779 1789118620.835439 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x41e5644a (stale) Mirror 13: 0x41e5644a (stale) Mirror 14: 0x41e5644a (stale) Mirror 15: 0x41e5644a (stale) Mirror 16: 0x41e5644a (stale) Mirror 18: 0x41e5644a (stale) Mirror 19: 0x41e5644a (stale) Mirror 21: 0x41e5644a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe8f9bd11 Mirror 22: 0xe8f9bd11 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000781 1789118621.906194 mapread 0x018a10 thru 0x01dba4 (0x5195 bytes) 000782 1789118621.932693 mapread 0x03067a thru 0x03c6ec (0xc073 bytes) 000783 1789118621.956418 mapread 0x014a11 thru 0x01c385 (0x7975 bytes) 000784 1789118621.957863 trunc from 0x040000 to 0x039a25 (0x65da bytes) 000785 1789118622.093896 write 0x018fdc thru 0x0196ec (0x711 bytes) 000786 1789118622.122976 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x39a26): Mirror 6: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a26): Mirror 6: 0xfe54229b (stale) Mirror 13: 0xfe54229b (stale) Mirror 14: 0xfe54229b (stale) Mirror 15: 0xfe54229b (stale) Mirror 16: 0xfe54229b (stale) Mirror 18: 0xfe54229b (stale) Mirror 19: 0xfe54229b (stale) Mirror 21: 0xfe54229b (stale) Mirror 22: 0x79bee9c (stale) CRC-32 checksum value for chunk [0, 0x39a26): Mirror 1: 0xb274c112 Mirror 23: 0xb274c112 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a26: skipped 000787 1789118623.393364 mapwrite 0x01c22a thru 0x027f0a (0xbce1 bytes) 000791 1789118623.597604 mapread 0x01f293 thru 0x02945f (0xa1cd bytes) 000793 1789118623.600256 read 0x001ceb thru 0x001f05 (0x21b bytes) 000795 1789118623.610026 MIRROR_RESYNC 11 mirrors 000797 1789118625.250228 read 0x01d8e7 thru 0x023abd (0x61d7 bytes) 000798 1789118625.260925 mapwrite 0x02115f thru 0x028f24 (0x7dc6 bytes) 000799 1789118625.471921 trunc from 0x039a26 to 0x01a1ff (0x1f826 bytes) 000800 1789118625.542079 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a200): Mirror 6: 0x331c0169 (stale) Mirror 13: 0x331c0169 (stale) Mirror 14: 0x331c0169 (stale) Mirror 16: 0x331c0169 (stale) Mirror 18: 0x331c0169 (stale) Mirror 19: 0x331c0169 (stale) Mirror 21: 0x331c0169 (stale) Mirror 22: 0x331c0169 (stale) Mirror 23: 0x331c0169 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a200): Mirror 6: 0x331c0169 (stale) Mirror 13: 0x331c0169 (stale) Mirror 14: 0x331c0169 (stale) Mirror 16: 0x331c0169 (stale) Mirror 18: 0x331c0169 (stale) Mirror 19: 0x331c0169 (stale) Mirror 21: 0x331c0169 (stale) Mirror 22: 0x331c0169 (stale) Mirror 23: 0x331c0169 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a200: skipped 000801 1789118626.179339 mapwrite 0x02a0de thru 0x02e393 (0x42b6 bytes) 000802 1789118626.322086 mapread 0x017aa5 thru 0x025117 (0xd673 bytes) 000803 1789118626.371004 mapread 0x00db44 thru 0x00ead9 (0xf96 bytes) 000804 1789118626.390406 mapread 0x007525 thru 0x0153c0 (0xde9c bytes) 000805 1789118626.398057 mapwrite 0x019ca4 thru 0x023e3d (0xa19a bytes) 000806 1789118626.439950 mapread 0x0027c3 thru 0x00b964 (0x91a2 bytes) 000807 1789118626.447179 write 0x033246 thru 0x03b23d (0x7ff8 bytes) 000808 1789118626.448575 read 0x0118d0 thru 0x012a7f (0x11b0 bytes) 000811 1789118626.452072 trunc from 0x03b23e to 0x037f64 (0x32d9 bytes) 000812 1789118626.465854 mapwrite 0x012f51 thru 0x01a433 (0x74e3 bytes) 000813 1789118626.536977 mapread 0x021e56 thru 0x02a6ce (0x8879 bytes) 000814 1789118626.543924 trunc from 0x037f65 to 0x028eb0 (0xf0b4 bytes) 000815 1789118626.586290 trunc from 0x028eb1 to 0x008245 (0x20c6b bytes) 000819 1789118626.629276 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x8246): Mirror 6: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8246): Mirror 6: 0x1eb61901 (stale) Mirror 13: 0x1eb61901 (stale) Mirror 14: 0x1eb61901 (stale) Mirror 16: 0x1eb61901 (stale) Mirror 18: 0x1eb61901 (stale) Mirror 19: 0x1eb61901 (stale) Mirror 21: 0x1eb61901 (stale) Mirror 22: 0x1eb61901 (stale) Mirror 23: 0x1eb61901 (stale) CRC-32 checksum value for chunk [0, 0x8246): Mirror 1: 0x1eb61901 Mirror 24: 0x1eb61901 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8246: skipped 000820 1789118627.603852 trunc from 0x008246 to 0x00648b (0x1dba bytes) 000821 1789118627.748115 trunc from 0x00648c to 0x038b06 (0x3267b bytes) 000822 1789118627.782704 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38b07): Mirror 6: 0xfde8cb5c (stale) Mirror 13: 0xfde8cb5c (stale) Mirror 14: 0xfde8cb5c (stale) Mirror 18: 0xfde8cb5c (stale) Mirror 19: 0xfde8cb5c (stale) Mirror 21: 0xfde8cb5c (stale) Mirror 22: 0xfde8cb5c (stale) Mirror 23: 0xfde8cb5c (stale) Mirror 24: 0xc1227b32 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38b07): Mirror 6: 0xfde8cb5c (stale) Mirror 13: 0xfde8cb5c (stale) Mirror 14: 0xfde8cb5c (stale) Mirror 18: 0xfde8cb5c (stale) Mirror 19: 0xfde8cb5c (stale) Mirror 21: 0xfde8cb5c (stale) Mirror 22: 0xfde8cb5c (stale) Mirror 23: 0xfde8cb5c (stale) Mirror 24: 0xc1227b32 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38b07: skipped 000823 1789118628.562621 write 0x02c1c2 thru 0x02dfc4 (0x1e03 bytes) 000824 1789118628.575660 read 0x011062 thru 0x02086d (0xf80c bytes) 000825 1789118628.585656 mapread 0x025c76 thru 0x029b00 (0x3e8b bytes) 000827 1789118628.608402 MIRROR_RESYNC 10 mirrors 000828 1789118630.765135 write 0x025af6 thru 0x02ab17 (0x5022 bytes) 000829 1789118630.880094 write 0x035d22 thru 0x03da80 (0x7d5f bytes) 000830 1789118630.885375 trunc from 0x03da81 to 0x037b82 (0x5efe bytes) 000832 1789118630.910219 mapread 0x01d698 thru 0x0231b7 (0x5b20 bytes) 000833 1789118630.933093 mapread 0x000009 thru 0x002b75 (0x2b6d bytes) 000834 1789118630.946114 read 0x00019f thru 0x006729 (0x658b bytes) 000835 1789118630.947262 mapread 0x0029aa thru 0x00e1ff (0xb856 bytes) 000836 1789118630.949728 trunc from 0x037b83 to 0x014147 (0x23a3b bytes) 000838 1789118630.967050 trunc from 0x014148 to 0x00f109 (0x503e bytes) 000841 1789118630.991473 trunc from 0x00f10a to 0x01887a (0x9771 bytes) 000845 1789118631.022806 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x1887b): Mirror 6: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1887b): Mirror 6: 0xf9eeebed (stale) Mirror 13: 0xf9eeebed (stale) Mirror 14: 0xf9eeebed (stale) Mirror 18: 0xf9eeebed (stale) Mirror 19: 0xf9eeebed (stale) Mirror 21: 0xf9eeebed (stale) Mirror 22: 0xf9eeebed (stale) Mirror 23: 0xf9eeebed (stale) Mirror 24: 0xf9eeebed (stale) CRC-32 checksum value for chunk [0, 0x1887b): Mirror 1: 0xf9eeebed Mirror 25: 0xf9eeebed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1887b: skipped 000846 1789118632.163299 read 0x00c043 thru 0x0132bb (0x7279 bytes) 000847 1789118632.176859 write 0x02d0b8 thru 0x03c4e1 (0xf42a bytes) 000849 1789118632.315325 write 0x00adf1 thru 0x0160ff (0xb30f bytes) 000850 1789118632.332783 mapread 0x005990 thru 0x01356c (0xdbdd bytes) 000854 1789118632.385673 read 0x01c78b thru 0x02862d (0xbea3 bytes) 000856 1789118632.413151 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c4e2): Mirror 6: 0x171e9942 (stale) Mirror 13: 0x171e9942 (stale) Mirror 14: 0x171e9942 (stale) Mirror 18: 0x171e9942 (stale) Mirror 19: 0x171e9942 (stale) Mirror 22: 0x171e9942 (stale) Mirror 23: 0x171e9942 (stale) Mirror 24: 0x171e9942 (stale) Mirror 25: 0xd7c9b749 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c4e2): Mirror 6: 0x171e9942 (stale) Mirror 13: 0x171e9942 (stale) Mirror 14: 0x171e9942 (stale) Mirror 18: 0x171e9942 (stale) Mirror 19: 0x171e9942 (stale) Mirror 22: 0x171e9942 (stale) Mirror 23: 0x171e9942 (stale) Mirror 24: 0x171e9942 (stale) Mirror 25: 0xd7c9b749 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c4e2: skipped 000857 1789118633.203895 MIRROR_RESYNC 10 mirrors 000860 1789118634.989760 trunc from 0x03c4e2 to 0x0275bf (0x14f22 bytes) 000862 1789118635.152614 write 0x00668a thru 0x00b83b (0x51b2 bytes) 000863 1789118635.168797 trunc from 0x0275c0 to 0x025ff6 (0x15c9 bytes) 000864 1789118635.251743 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25ff7): Mirror 6: 0x130b948c (stale) Mirror 13: 0x130b948c (stale) Mirror 14: 0x130b948c (stale) Mirror 18: 0x130b948c (stale) Mirror 19: 0x130b948c (stale) Mirror 22: 0x130b948c (stale) Mirror 23: 0x130b948c (stale) Mirror 25: 0x130b948c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25ff7): Mirror 6: 0x130b948c (stale) Mirror 13: 0x130b948c (stale) Mirror 14: 0x130b948c (stale) Mirror 18: 0x130b948c (stale) Mirror 19: 0x130b948c (stale) Mirror 22: 0x130b948c (stale) Mirror 23: 0x130b948c (stale) Mirror 25: 0x130b948c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25ff7: skipped 000866 1789118636.094003 read 0x00a913 thru 0x0160a9 (0xb797 bytes) 000867 1789118636.113551 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x25ff7): Mirror 6: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25ff7): Mirror 6: 0x130b948c (stale) Mirror 13: 0x130b948c (stale) Mirror 14: 0x130b948c (stale) Mirror 18: 0x130b948c (stale) Mirror 19: 0x130b948c (stale) Mirror 22: 0x130b948c (stale) Mirror 23: 0x130b948c (stale) Mirror 25: 0x130b948c (stale) CRC-32 checksum value for chunk [0, 0x25ff7): Mirror 1: 0x311e5ce9 Mirror 26: 0x311e5ce9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25ff7: skipped 000868 1789118637.148343 write 0x036986 thru 0x03dcb0 (0x732b bytes) 000869 1789118637.283133 write 0x02130a thru 0x03020c (0xef03 bytes) 000870 1789118637.302941 mapwrite 0x02923a thru 0x036526 (0xd2ed bytes) 000871 1789118637.384106 read 0x00093d thru 0x001c49 (0x130d bytes) 000875 1789118637.397475 trunc from 0x03dcb1 to 0x03a933 (0x337d bytes) 000876 1789118637.439007 mapread 0x004610 thru 0x01117b (0xcb6c bytes) 000877 1789118637.474732 write 0x004a79 thru 0x0126e0 (0xdc68 bytes) 000879 1789118637.477603 read 0x03a53e thru 0x03a933 (0x3f6 bytes) 000880 1789118637.478298 trunc from 0x03a934 to 0x026be3 (0x13d50 bytes) 000882 1789118637.510223 mapwrite 0x002b7f thru 0x01114c (0xe5ce bytes) 000884 1789118637.596375 trunc from 0x026be4 to 0x02e471 (0x788e bytes) 000887 1789118637.633264 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e472): Mirror 6: 0x1dc54cc5 (stale) Mirror 13: 0x1dc54cc5 (stale) Mirror 14: 0x1dc54cc5 (stale) Mirror 18: 0x1dc54cc5 (stale) Mirror 19: 0x1dc54cc5 (stale) Mirror 22: 0x1dc54cc5 (stale) Mirror 23: 0x1dc54cc5 (stale) Mirror 26: 0x713a2c94 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e472): Mirror 6: 0x1dc54cc5 (stale) Mirror 13: 0x1dc54cc5 (stale) Mirror 14: 0x1dc54cc5 (stale) Mirror 18: 0x1dc54cc5 (stale) Mirror 19: 0x1dc54cc5 (stale) Mirror 22: 0x1dc54cc5 (stale) Mirror 23: 0x1dc54cc5 (stale) Mirror 26: 0x713a2c94 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e472: skipped 000888 1789118638.235223 trunc from 0x02e472 to 0x016dcf (0x176a2 bytes) 000889 1789118638.275297 read 0x00cd24 thru 0x016dcf (0xa0ac bytes) 000891 1789118638.305863 trunc from 0x016dd0 to 0x029325 (0x12556 bytes) 000894 1789118638.342130 read 0x019111 thru 0x01ea25 (0x5915 bytes) 000895 1789118638.363567 mapread 0x0225e8 thru 0x023d1c (0x1735 bytes) 000896 1789118638.365312 trunc from 0x029326 to 0x00a3a7 (0x1ef7e bytes) 000897 1789118638.394646 mapread 0x005718 thru 0x008a04 (0x32ed bytes) 000898 1789118638.402125 MIRROR_RESYNC 9 mirrors 000901 1789118640.037966 mapread 0x008951 thru 0x00a3a7 (0x1a57 bytes) 000902 1789118640.064085 mapread 0x00205a thru 0x00a3a7 (0x834e bytes) 000904 1789118640.076811 read 0x006a01 thru 0x0092d3 (0x28d3 bytes) 000905 1789118640.077513 mapread 0x005e83 thru 0x00a3a7 (0x4525 bytes) 000907 1789118640.080371 read 0x008d9d thru 0x00a3a7 (0x160b bytes) 000909 1789118640.083454 mapread 0x002c1e thru 0x0053de (0x27c1 bytes) 000911 1789118640.091422 mapread 0x00289b thru 0x00a3a7 (0x7b0d bytes) 000913 1789118640.092904 write 0x016ffc thru 0x022aab (0xbab0 bytes) 000914 1789118640.253696 read 0x01333d thru 0x017ff9 (0x4cbd bytes) 000916 1789118640.268789 write 0x004347 thru 0x0061e5 (0x1e9f bytes) 000919 1789118640.272013 trunc from 0x022aac to 0x0344e5 (0x11a3a bytes) 000920 1789118640.296253 mapread 0x00320c thru 0x007a7c (0x4871 bytes) 000921 1789118640.297687 mapwrite 0x00d7e7 thru 0x01251e (0x4d38 bytes) 000923 1789118640.373900 trunc from 0x0344e6 to 0x03d7d4 (0x92ef bytes) 000924 1789118640.391366 read 0x02c0eb thru 0x031774 (0x568a bytes) 000925 1789118640.406662 trunc from 0x03d7d5 to 0x001671 (0x3c163 bytes) 000926 1789118640.423044 mapread 0x0009e1 thru 0x001671 (0xc91 bytes) 000931 1789118640.433815 write 0x02428a thru 0x028e9a (0x4c11 bytes) 000935 1789118640.440775 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x28e9b): Mirror 6: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28e9b): Mirror 6: 0xc2bebf49 (stale) Mirror 13: 0xc2bebf49 (stale) Mirror 14: 0xc2bebf49 (stale) Mirror 18: 0xc2bebf49 (stale) Mirror 19: 0xc2bebf49 (stale) Mirror 22: 0xc2bebf49 (stale) Mirror 23: 0xc2bebf49 (stale) Mirror 26: 0xc2bebf49 (stale) CRC-32 checksum value for chunk [0, 0x28e9b): Mirror 1: 0x2e4de752 Mirror 27: 0x2e4de752 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28e9b: skipped 000936 1789118641.248131 mapwrite 0x006c6a thru 0x008d10 (0x20a7 bytes) 000937 1789118641.511777 MIRROR_RESYNC 10 mirrors 000938 1789118643.361261 mapread 0x014b81 thru 0x017c7c (0x30fc bytes) 000940 1789118643.385018 read 0x02366f thru 0x028e9a (0x582c bytes) 000941 1789118643.396314 mapread 0x013093 thru 0x01e2fb (0xb269 bytes) 000944 1789118643.398608 read 0x004301 thru 0x00598a (0x168a bytes) 000946 1789118643.415395 MIRROR_RESYNC 10 mirrors 000947 1789118644.206019 MIRROR_EXTEND to 11 mirrors 000948 1789118645.263287 mapread 0x00ce2f thru 0x01bbf2 (0xedc4 bytes) 000949 1789118645.317223 write 0x028dc6 thru 0x03649f (0xd6da bytes) 000956 1789118645.403838 write 0x016bd1 thru 0x025660 (0xea90 bytes) 000957 1789118645.416176 read 0x02e330 thru 0x033ad2 (0x57a3 bytes) 000960 1789118645.417762 read 0x007276 thru 0x014a70 (0xd7fb bytes) 000961 1789118645.449252 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x364a0): Mirror 6: 0x8ccf110b (stale) Mirror 13: 0x8ccf110b (stale) Mirror 14: 0x8ccf110b (stale) Mirror 18: 0x8ccf110b (stale) Mirror 19: 0x8ccf110b (stale) Mirror 22: 0x8ccf110b (stale) Mirror 26: 0x8ccf110b (stale) Mirror 27: 0x8ccf110b (stale) Mirror 28: 0x8ccf110b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x364a0): Mirror 6: 0x8ccf110b (stale) Mirror 13: 0x8ccf110b (stale) Mirror 14: 0x8ccf110b (stale) Mirror 18: 0x8ccf110b (stale) Mirror 19: 0x8ccf110b (stale) Mirror 22: 0x8ccf110b (stale) Mirror 26: 0x8ccf110b (stale) Mirror 27: 0x8ccf110b (stale) Mirror 28: 0x8ccf110b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x364a0: skipped 000962 1789118646.097332 mapwrite 0x032c80 thru 0x03ffff (0xd380 bytes) 000963 1789118646.205382 mapwrite 0x02d9d5 thru 0x03031a (0x2946 bytes) 000964 1789118646.283182 MIRROR_RESYNC 10 mirrors 000966 1789118648.206843 mapread 0x00fefa thru 0x01d9f5 (0xdafc bytes) 000967 1789118648.240241 trunc from 0x040000 to 0x01a6b2 (0x2594d bytes) 000969 1789118648.333122 mapread 0x00f947 thru 0x0169ac (0x7066 bytes) 000970 1789118648.341399 read 0x018c2f thru 0x01a6b2 (0x1a84 bytes) 000971 1789118648.354841 read 0x00ebae thru 0x014383 (0x57d6 bytes) 000972 1789118648.355626 read 0x0004eb thru 0x01001f (0xfb35 bytes) 000973 1789118648.368885 write 0x0279cd thru 0x0377d8 (0xfe0c bytes) 000975 1789118648.372391 write 0x021343 thru 0x02e302 (0xcfc0 bytes) 000976 1789118648.380660 write 0x027a94 thru 0x033548 (0xbab5 bytes) 000977 1789118648.383407 mapwrite 0x023b6c thru 0x027f37 (0x43cc bytes) 000978 1789118648.441404 mapwrite 0x034bfd thru 0x03c1fa (0x75fe bytes) 000984 1789118648.549859 read 0x006931 thru 0x00d918 (0x6fe8 bytes) 000985 1789118648.555206 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3c1fb): Mirror 6: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c1fb): Mirror 6: 0x3593859f (stale) Mirror 13: 0x3593859f (stale) Mirror 14: 0x3593859f (stale) Mirror 18: 0x3593859f (stale) Mirror 19: 0x3593859f (stale) Mirror 22: 0x3593859f (stale) Mirror 26: 0x3593859f (stale) Mirror 27: 0x3593859f (stale) Mirror 28: 0x3593859f (stale) CRC-32 checksum value for chunk [0, 0x3c1fb): Mirror 1: 0x543399e3 Mirror 29: 0x543399e3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c1fb: skipped 000986 1789118649.521598 write 0x02d9bf thru 0x035f38 (0x857a bytes) 000987 1789118649.632590 mapread 0x02fc3b thru 0x03b6c2 (0xba88 bytes) 000988 1789118649.656597 read 0x02e0a1 thru 0x031ab1 (0x3a11 bytes) 000989 1789118649.662074 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c1fb): Mirror 6: 0x3593859f (stale) Mirror 13: 0x3593859f (stale) Mirror 14: 0x3593859f (stale) Mirror 18: 0x3593859f (stale) Mirror 19: 0x3593859f (stale) Mirror 22: 0x3593859f (stale) Mirror 27: 0x3593859f (stale) Mirror 28: 0x3593859f (stale) Mirror 29: 0x543399e3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c1fb): Mirror 6: 0x3593859f (stale) Mirror 13: 0x3593859f (stale) Mirror 14: 0x3593859f (stale) Mirror 18: 0x3593859f (stale) Mirror 19: 0x3593859f (stale) Mirror 22: 0x3593859f (stale) Mirror 27: 0x3593859f (stale) Mirror 28: 0x3593859f (stale) Mirror 29: 0x543399e3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c1fb: skipped 000990 1789118650.303149 mapwrite 0x00eb7e thru 0x01c44b (0xd8ce bytes) 000991 1789118650.433852 mapwrite 0x006965 thru 0x00a4b2 (0x3b4e bytes) 000994 1789118650.484520 mapread 0x00bd6d thru 0x00be77 (0x10b bytes) 000995 1789118650.489460 MIRROR_RESYNC 10 mirrors 000996 1789118652.186614 trunc from 0x03c1fb to 0x00f6b4 (0x2cb46 bytes) 000997 1789118652.301595 trunc from 0x00f6b5 to 0x02c18d (0x1cad9 bytes) 001000 1789118652.325354 read 0x015ae5 thru 0x020311 (0xa82d bytes) All operations completed A-OK! PASS 70a (124s) == sanity-flr test 71: check mirror extend parent fid ==== 05:24:17 (1789118657) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 05:24:20 (1789118660) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.239043 s, 8.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149514 s, 7.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 05:24:30 (1789118670) Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre2 Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5901632 .. Extending file size to 3828448 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 5847264 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7994784 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8542496 .. Extending file size to 9145184 .. Extending file size to 6632640 .. Extending file size to 7761792 .. Extending file size to 9170848 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9211296 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58872 58874 58875 58876 58878 failed 192.168.204.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg440-client.virtnet /mnt/lustre2 (opts:) 192.168.204.140@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg440-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (77s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 05:25:47 (1789118747) Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre2 Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3505056 .. Extending file size to 4738240 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6800512 .. Extending file size to 8121216 .. Extending file size to 8415552 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7810784 .. Extending file size to 9397376 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8966144 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 60812 60813 60815 60818 60819 192.168.204.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg440-client.virtnet /mnt/lustre2 (opts:) 192.168.204.140@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg440-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (75s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 05:27:03 (1789118823) Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0891662 s, 23.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.359123 s, 5.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0xb026478e (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.204.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg440-client.virtnet /mnt/lustre2 (opts:) PASS 200c (15s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 05:27:18 (1789118838) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0772325 s, 27.2 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 05:27:24 (1789118844) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ce:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c4:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 05:27:31 (1789118851) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.26021 s, 8.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 05:27:38 (1789118858) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.172738 s, 12.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 05:27:45 (1789118865) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.208754 s, 10.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } PASS 204c (30s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 05:28:15 (1789118895) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg440-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099008 s, 10.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } PASS 204d (58s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 05:29:13 (1789118953) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg440-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.38292 s, 5.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (55s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 05:30:08 (1789119008) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg440-server mds-ost sync done. Creating new pool test_204f oleg440-server: Pool lustre.test_204f created Adding targets to pool oleg440-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg440-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.387925 s, 5.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg440-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg440-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg440-server: Pool lustre.test_204f destroyed PASS 204f (76s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 05:31:24 (1789119084) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 05:31:30 (1789119090) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 05:31:36 (1789119096) oleg440-server: Pool lustre.test_206 created Adding targets to pool oleg440-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg440-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg440-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg440-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg440-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 05:32:02 (1789119122) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110588 s, 9.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751061 s, 14.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xed43ff6f Mirror 3: 0xed43ff6f PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 05:32:13 (1789119133) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 05:32:17 (1789119137) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 05:32:21 (1789119141) Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0844529 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x19b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1f60000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1f60000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1f60000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xe50000 Read this (4 bytes): BBBB 192.168.204.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg440-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 05:32:57 (1789119177) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0125543 s, 653 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 05:33:05 (1789119185) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482413 s, 21.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f3 lmm_fid: [0x200000404:0x1f3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f3 lmm_fid: [0x200000404:0x1f3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834419 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 05:33:06.000000000 -0400 Modify: 2026-09-11 05:33:06.000000000 -0400 Change: 2026-09-11 05:33:08.000000000 -0400 Birth: 2026-09-11 05:33:06.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 05:33:13 (1789119193) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 05:33:16 (1789119196) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.57337 s, 71.4 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0616554 s, 66.4 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 11 05:33 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f5:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 11 05:33 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 05:33:24 (1789119204) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 05:33:32 (1789119212) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1790 sec =========== 05:33:42 (1789119222) osc.lustre-OST0000-osc-ffff900598414000.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 osc.lustre-OST0001-osc-ffff900598414000.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 obdfilter.lustre-OST0000.stats_compr= snapshot_time 1789119224.774757064 secs.nsecs start_time 1789117915.829305140 secs.nsecs elapsed_time 1308.945451924 secs.nsecs uncompressed_objects 3150 samples [reqs] obdfilter.lustre-OST0001.stats_compr= snapshot_time 1789119224.774829349 secs.nsecs start_time 1789117950.323419761 secs.nsecs elapsed_time 1274.451409588 secs.nsecs uncompressed_objects 3143 samples [reqs] === sanity-flr: start cleanup 05:33:45 (1789119225) === === sanity-flr: finish cleanup 05:33:52 (1789119232) ===