-----============= acceptance-small: sanity-flr ============----- Fri Sep 25 22:53:56 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg132-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 22:54:16 (1790391256) === oleg132-client.virtnet: executing check_config_client /mnt/lustre oleg132-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg132-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff952144c6e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff952144c6e800.idle_timeout=debug disable quota as required oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 22:54:52 (1790391292) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7637] == sanity-flr test 0a: lfs mirror create with -N option == 22:54:54 (1790391294) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 22:55:11 (1790391311) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 22:55:14 (1790391314) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 22:55:18 (1790391318) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 22:55:40 (1790391340) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 22:55:43 (1790391343) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 22:55:46 (1790391346) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff952144c6e800.stats=clear osc.lustre-OST0001-osc-ffff952144c6e800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 22:55:54 (1790391354) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 22:56:04 (1790391364) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 22:56:13 (1790391373) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 22:56:16 (1790391376) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 22:56:23 (1790391383) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 22:56:32 (1790391392) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 22:56:40 (1790391400) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 22:56:43 (1790391403) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 22:56:50 (1790391410) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (9s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 22:57:04 (1790391424) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 22:57:12 (1790391432) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 1.53233 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12377 s, 8.5 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 22:57:22 (1790391442) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.404972 s, 51.8 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 22:57:31 (1790391451) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 22:57:41 (1790391461) Stopping /mnt/lustre-ost1 (opts:) on oleg132-server oleg132-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff952144c6e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff952144c6e800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg132-server: oleg132-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg132-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg132-server oleg132-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff952144c6e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff952144c6e800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg132-server: oleg132-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg132-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (76s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 22:58:57 (1790391537) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.358213 s, 26.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg132-server oleg132-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff952144c6e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff952144c6e800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg132-server: oleg132-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg132-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 22:59:41 (1790391581) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg132-server oleg132-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff952144c6e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff952144c6e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg132-server: oleg132-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg132-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg132-server oleg132-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff952144c6e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff952144c6e800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg132-server: oleg132-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg132-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (86s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 23:01:07 (1790391667) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.57861 s, 11.5 MB/s -rw-r--r-- 1 root root 52428800 Sep 25 23:01 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.065s user 0m0.001s sys 0m0.156s Stopping /mnt/lustre-ost1 (opts:) on oleg132-server oleg132-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff952144c6e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff952144c6e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.911s user 0m0.001s sys 0m0.153s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg132-server: oleg132-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg132-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg132-server oleg132-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff952144c6e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff952144c6e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.052s user 0m0.001s sys 0m0.170s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg132-server: oleg132-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg132-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg132-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff952144c6e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff952144c6e800.ost_server_uuid in FULL state after 0 sec PASS 33b (92s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 23:02:39 (1790391759) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 23:02:43 (1790391763) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 23:02:47 (1790391767) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 23:02:50 (1790391770) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 23:02:58 (1790391778) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.446815 s, 28.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 23:03:10 (1790391790) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271336 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245234 s, 42.8 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 23:03:17 (1790391797) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 23:03:27 (1790391807) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00099981 s, 1.0 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000709956 s, 1.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00074766 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000762928 s, 1.3 MB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 23:03:38 (1790391818) create test files with size 15751196 .. 128eefa2ef0a1c26df21110b729e2c04 - 82e0147a6127c93fae23b2fb17344e5a - 0bca26b5faccb1fc898c1f8af6d498c4 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg132-client.virtnet /mnt/lustre (opts:) Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 23:04:05 (1790391845) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 23:04:20 (1790391860) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 23:04:28 (1790391868) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12108 s, 8.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xda:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xdb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xb8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525236 s, 20.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xb9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xba:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xdc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 23:04:37 (1790391877) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.126979 s, 16.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.300095 s, 14.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.186998 s, 11.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 23:04:55 (1790391895) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 23:04:58 (1790391898) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 23:05:01 (1790391901) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281958 s, 37.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 23:05:09 (1790391909) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 23:05:12 (1790391912) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc1:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe3:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 23:05:22 (1790391922) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.390728 s, 26.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg132-client.virtnet /mnt/lustre (opts:) Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 23:05:34 (1790391934) llite.lustre-ffff9521603d1800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.379488 s, 27.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg132-client.virtnet /mnt/lustre (opts:) Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff952151206800.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 23:05:47 (1790391947) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (44s) == sanity-flr test 46: Verify setstripe --copy option ==== 23:06:32 (1790391992) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (39s) == sanity-flr test 47: Verify mirror obj alloc =========== 23:07:11 (1790392031) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 23:07:15 (1790392035) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.275864 s, 11.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.145981 s, 21.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x2e422e08 (nosync,stale) Mirror 2: 0x2e422e08 (nosync,stale) ** verify mirror 2 doesn't change original checksum: d08a829f91b92450f10b01103647b084 - mirror 1 checksum: dddd7a9ee8afc04344b996104107e89c - mirror 2 checksum: d08a829f91b92450f10b01103647b084 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-25T23:07:18-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-25T23:07:18-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf9:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: d08a829f91b92450f10b01103647b084 - mirror 1 checksum: dddd7a9ee8afc04344b996104107e89c - mirror 2 checksum: dddd7a9ee8afc04344b996104107e89c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-25T23:07:18-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-25T23:07:18-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf9:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 23:07:28 (1790392048) Failing mds1 on oleg132-server Stopping /mnt/lustre-mds1 (opts:) on oleg132-server 23:07:35 (1790392055) shut down facet: mds1 facet_host: oleg132-server facet_failover_host: oleg132-server Failover mds1 to oleg132-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0000 23:07:59 (1790392079) targets are mounted 23:07:59 (1790392079) facet_failover done oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250193 s, 41.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xfa:0x0] } PASS 50A (47s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 23:08:16 (1790392096) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 23:08:19 (1790392099) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 23:08:22 (1790392102) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00838901 s, 2.0 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 23:08:29 (1790392109) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 23:08:33 (1790392113) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00682683 s, 600 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0672441 s, 60.9 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 23:08:41 (1790392121) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (19s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 23:09:00 (1790392140) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 23:09:28 (1790392168) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 23:09:51 (1790392191) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 23:10:15 (1790392215) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 23:10:21 (1790392221) mirror create pid 54341 mirror split pid 54343 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54341 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54343 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (65s) == sanity-flr test 70a: flr mode fsx test ================ 23:11:26 (1790392286) Seed set to 2288 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1790392288.654385 mapwrite 0x00d56f thru 0x010d16 (0x37a8 bytes) 000004 1790392288.735513 punch from 0x00cd5d to 0x00cf24 (0x1c8 bytes) 000007 1790392288.741525 read 0x0008d9 thru 0x01029a (0xf9c2 bytes) 000008 1790392288.754409 mapread 0x00f3f8 thru 0x010d16 (0x191f bytes) 000009 1790392288.755377 mapread 0x006395 thru 0x0087a9 (0x2415 bytes) 000010 1790392288.762649 write 0x0043fe thru 0x0090c2 (0x4cc5 bytes) 000012 1790392288.764093 read 0x0097f7 thru 0x00ee06 (0x5610 bytes) 000013 1790392288.764649 mapread 0x0063a2 thru 0x00a121 (0x3d80 bytes) 000014 1790392288.766390 mapread 0x00e67c thru 0x010d16 (0x269b bytes) 000017 1790392288.775054 MIRROR_EXTEND to 2 mirrors 000018 1790392289.453933 write 0x02dc15 thru 0x034d45 (0x7131 bytes) 000019 1790392289.525728 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x34d46): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34d46): Mirror 2: 0xaa5f9122 (stale) CRC-32 checksum value for chunk [0, 0x34d46): Mirror 1: 0xb9d3bc72 Mirror 3: 0xb9d3bc72 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34d46: skipped truncating to largest ever: 0x286fc 000020 1790392290.389017 trunc from 0x034d46 to 0x0286fb (0xc64a bytes) 000022 1790392290.456095 mapwrite 0x0113e6 thru 0x01ea55 (0xd670 bytes) 000024 1790392290.529268 mapread 0x00a2f7 thru 0x01518c (0xae96 bytes) 000025 1790392290.545106 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x286fc): Mirror 2: 0xbdb8cdd8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x286fc): Mirror 2: 0xbdb8cdd8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x286fc: skipped 000026 1790392290.809109 write 0x02d84b thru 0x0383e8 (0xab9e bytes) 000028 1790392290.814723 trunc from 0x0383e9 to 0x01bb01 (0x1c8e7 bytes) 000029 1790392290.831247 read 0x004771 thru 0x00a258 (0x5ae8 bytes) 000030 1790392290.836119 punch from 0x0134c8 to 0x01bb01 (0x863a bytes) 000031 1790392290.839434 mapread 0x007f38 thru 0x01100f (0x90d8 bytes) 000034 1790392290.862256 mapwrite 0x01e951 thru 0x02703f (0x86ef bytes) 000035 1790392290.933181 trunc from 0x027040 to 0x0190f5 (0xdf4a bytes) 000036 1790392290.942335 trunc from 0x0190f6 to 0x011d58 (0x739d bytes) 000037 1790392290.955146 trunc from 0x011d59 to 0x001c4b (0x1010d bytes) 000038 1790392290.965644 read 0x0018fb thru 0x001c4b (0x351 bytes) 000040 1790392290.972021 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1c4c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c4c): Mirror 2: 0x5c343873 (stale) CRC-32 checksum value for chunk [0, 0x1c4c): Mirror 1: 0x5c343873 Mirror 3: 0x5c343873 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c4c: skipped 000044 1790392291.648092 read 0x000a44 thru 0x000bde (0x19b bytes) 000047 1790392291.666533 write 0x01d625 thru 0x027499 (0x9e75 bytes) 000050 1790392291.725676 mapwrite 0x026a70 thru 0x0275b2 (0xb43 bytes) 000051 1790392291.781702 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x275b3): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x275b3): Mirror 2: 0x3f021589 (stale) Mirror 3: 0xcc6c9ac3 (stale) CRC-32 checksum value for chunk [0, 0x275b3): Mirror 1: 0xa0ef1c3 Mirror 4: 0xa0ef1c3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x275b3: skipped truncating to largest ever: 0x3c8b3 000052 1790392292.425464 trunc from 0x0275b3 to 0x03c8b2 (0x15300 bytes) 000053 1790392292.490715 MIRROR_RESYNC 4 mirrors 000054 1790392293.437740 MIRROR_RESYNC 4 mirrors 000056 1790392293.806295 mapread 0x011c49 thru 0x021698 (0xfa50 bytes) 000057 1790392293.833060 trunc from 0x03c8b3 to 0x01faf0 (0x1cdc2 bytes) 000058 1790392293.922806 read 0x00733b thru 0x0155b5 (0xe27b bytes) 000059 1790392293.933315 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1faf1): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1faf1): Mirror 2: 0x77d2cf5b (stale) Mirror 3: 0x77d2cf5b (stale) Mirror 4: 0x77d2cf5b (stale) CRC-32 checksum value for chunk [0, 0x1faf1): Mirror 1: 0x77d2cf5b Mirror 5: 0x77d2cf5b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1faf1: skipped 000060 1790392294.542198 write 0x001aa8 thru 0x00b3b9 (0x9912 bytes) 000062 1790392294.634831 write 0x010cdd thru 0x01d487 (0xc7ab bytes) 000063 1790392294.644356 punch from 0x00a13e to 0x00d477 (0x333a bytes) 000064 1790392294.657334 mapread 0x00ef66 thru 0x01dd89 (0xee24 bytes) 000065 1790392294.667587 mapread 0x0165ce thru 0x017d9f (0x17d2 bytes) 000066 1790392294.668541 mapread 0x00ea52 thru 0x011171 (0x2720 bytes) 000067 1790392294.670129 read 0x01d040 thru 0x01faf0 (0x2ab1 bytes) 000068 1790392294.677385 trunc from 0x01faf1 to 0x0142a3 (0xb84d bytes) 000069 1790392294.690547 punch from 0x00e3d0 to 0x0142a3 (0x5ed4 bytes) 000070 1790392294.701903 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x142a4): Mirror 3: 0x17056b57 (stale) Mirror 4: 0x17056b57 (stale) Mirror 5: 0x17056b57 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x142a4): Mirror 3: 0x17056b57 (stale) Mirror 4: 0x17056b57 (stale) Mirror 5: 0x17056b57 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x142a4: skipped 000071 1790392294.992741 write 0x00f14e thru 0x017af6 (0x89a9 bytes) 000074 1790392295.001045 trunc from 0x017af7 to 0x006c57 (0x10e9f bytes) 000075 1790392295.029262 punch from 0x0042e0 to 0x006c57 (0x2978 bytes) 000076 1790392295.034201 MIRROR_RESYNC 4 mirrors 000077 1790392295.920068 mapwrite 0x026ee0 thru 0x02980b (0x292c bytes) 000079 1790392296.042118 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2980c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2980c): Mirror 3: 0x2b207291 (stale) Mirror 4: 0x2b207291 (stale) Mirror 5: 0x2b207291 (stale) CRC-32 checksum value for chunk [0, 0x2980c): Mirror 1: 0x8702f036 Mirror 6: 0x8702f036 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2980c: skipped 000080 1790392296.826749 trunc from 0x02980c to 0x014c02 (0x14c09 bytes) 000083 1790392296.914338 write 0x00d289 thru 0x019f49 (0xccc1 bytes) 000084 1790392296.927259 write 0x022e26 thru 0x026e35 (0x4010 bytes) 000088 1790392296.931113 read 0x021056 thru 0x026e35 (0x5de0 bytes) 000089 1790392296.938290 mapwrite 0x0196fe thru 0x01e12b (0x4a2e bytes) 000092 1790392297.013899 mapread 0x00d0c9 thru 0x00eaab (0x19e3 bytes) 000093 1790392297.020906 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x26e36): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26e36): Mirror 3: 0x147947a0 (stale) Mirror 4: 0x147947a0 (stale) Mirror 5: 0x147947a0 (stale) Mirror 6: 0x147947a0 (stale) CRC-32 checksum value for chunk [0, 0x26e36): Mirror 1: 0xdc73aa23 Mirror 7: 0xdc73aa23 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26e36: skipped 000094 1790392297.849575 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26e36): Mirror 4: 0x147947a0 (stale) Mirror 5: 0x147947a0 (stale) Mirror 6: 0x147947a0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26e36): Mirror 4: 0x147947a0 (stale) Mirror 5: 0x147947a0 (stale) Mirror 6: 0x147947a0 (stale) CRC-32 checksum value for chunk [0, 0x26e36): Mirror 1: 0xdc73aa23 Mirror 7: 0xdc73aa23 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26e36: skipped 000096 1790392298.387093 write 0x02d049 thru 0x02d080 (0x38 bytes) 000098 1790392298.455741 read 0x009c32 thru 0x00ec97 (0x5066 bytes) 000099 1790392298.464426 write 0x02e56e thru 0x03a8ff (0xc392 bytes) 000101 1790392298.466327 read 0x0377ff thru 0x03a8ff (0x3101 bytes) 000102 1790392298.467937 punch from 0x0160a8 to 0x01cd84 (0x6cdd bytes) 000104 1790392298.479908 mapread 0x01453e thru 0x01d3ee (0x8eb1 bytes) 000105 1790392298.498906 mapwrite 0x012d54 thru 0x01f516 (0xc7c3 bytes) 000107 1790392298.592446 mapread 0x00c8a2 thru 0x01be9e (0xf5fd bytes) 000108 1790392298.597147 punch from 0x0055fb to 0x009b6a (0x4570 bytes) 000109 1790392298.602300 trunc from 0x03a900 to 0x02c4f9 (0xe406 bytes) 000110 1790392298.634431 mapwrite 0x022d36 thru 0x024b50 (0x1e1b bytes) 000112 1790392298.756467 read 0x01a4bf thru 0x01cd9c (0x28de bytes) 000114 1790392298.766322 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2c4fa): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c4fa): Mirror 4: 0xe4e2bb1c (stale) Mirror 5: 0xe4e2bb1c (stale) Mirror 6: 0x94a6b333 (stale) Mirror 7: 0x5e7478e5 (stale) CRC-32 checksum value for chunk [0, 0x2c4fa): Mirror 1: 0xbd383c5d Mirror 8: 0xbd383c5d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c4fa: skipped 000115 1790392299.877548 write 0x017d47 thru 0x01b6f1 (0x39ab bytes) 000120 1790392299.956830 punch from 0x023ca4 to 0x02c4f9 (0x8856 bytes) 000121 1790392299.973654 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c4fa): Mirror 4: 0xe4e2bb1c (stale) Mirror 5: 0xe4e2bb1c (stale) Mirror 7: 0x5e7478e5 (stale) Mirror 8: 0xbd383c5d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c4fa): Mirror 4: 0xe4e2bb1c (stale) Mirror 5: 0xe4e2bb1c (stale) Mirror 7: 0x5e7478e5 (stale) Mirror 8: 0xbd383c5d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c4fa: skipped 000122 1790392300.328977 mapwrite 0x038e85 thru 0x03ffff (0x717b bytes) 000123 1790392300.446214 punch from 0x03f033 to 0x03fefe (0xecc bytes) 000124 1790392300.452024 punch from 0x0188b9 to 0x01cc40 (0x4388 bytes) 000125 1790392300.455511 write 0x02af8f thru 0x036e07 (0xbe79 bytes) 000127 1790392300.467629 MIRROR_RESYNC 5 mirrors 000128 1790392302.937457 read 0x02fb13 thru 0x0351c1 (0x56af bytes) 000129 1790392302.945617 mapwrite 0x032ea5 thru 0x03ffff (0xd15b bytes) 000131 1790392303.119445 trunc from 0x040000 to 0x03004b (0xffb4 bytes) 000132 1790392303.136731 trunc from 0x03004c to 0x009320 (0x26d2b bytes) 000133 1790392303.150207 trunc from 0x009321 to 0x01ca96 (0x13776 bytes) 000134 1790392303.162103 trunc from 0x01ca97 to 0x01f19a (0x2704 bytes) 000135 1790392303.176232 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1f19b): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f19b): Mirror 4: 0xeffa38c8 (stale) Mirror 5: 0xeffa38c8 (stale) Mirror 7: 0xeffa38c8 (stale) Mirror 8: 0xeffa38c8 (stale) CRC-32 checksum value for chunk [0, 0x1f19b): Mirror 1: 0x9ce187c5 Mirror 9: 0x9ce187c5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f19b: skipped 000136 1790392304.022863 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f19b): Mirror 4: 0xeffa38c8 (stale) Mirror 7: 0xeffa38c8 (stale) Mirror 8: 0xeffa38c8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f19b): Mirror 4: 0xeffa38c8 (stale) Mirror 7: 0xeffa38c8 (stale) Mirror 8: 0xeffa38c8 (stale) CRC-32 checksum value for chunk [0, 0x1f19b): Mirror 1: 0x9ce187c5 Mirror 9: 0x9ce187c5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f19b: skipped 000138 1790392304.534137 mapread 0x011803 thru 0x014159 (0x2957 bytes) 000139 1790392304.581562 write 0x028a0b thru 0x02c9da (0x3fd0 bytes) 000140 1790392304.658691 mapread 0x018c33 thru 0x01c3a1 (0x376f bytes) 000141 1790392304.673418 read 0x027d56 thru 0x02c9da (0x4c85 bytes) 000142 1790392304.678269 punch from 0x007e14 to 0x010dc7 (0x8fb4 bytes) 000143 1790392304.688012 write 0x034020 thru 0x03f0ba (0xb09b bytes) 000144 1790392304.689419 trunc from 0x03f0bb to 0x02190b (0x1d7af bytes) 000145 1790392304.703769 read 0x00b9da thru 0x01ad13 (0xf33a bytes) 000146 1790392304.711419 trunc from 0x02190c to 0x0039c4 (0x1df47 bytes) 000147 1790392304.724415 punch from 0x0033c2 to 0x0039c4 (0x603 bytes) 000148 1790392304.733662 write 0x02cdc5 thru 0x036026 (0x9262 bytes) 000151 1790392304.736720 mapread 0x01ddc6 thru 0x02c480 (0xe6bb bytes) 000153 1790392304.766329 trunc from 0x036027 to 0x00d69b (0x2898b bytes) 000154 1790392304.784120 mapwrite 0x03e02c thru 0x03ffff (0x1fd4 bytes) 000155 1790392304.842289 write 0x0071f9 thru 0x00813e (0xf46 bytes) 000156 1790392304.849871 read 0x03a361 thru 0x03ffff (0x5c9f bytes) 000158 1790392304.851310 punch from 0x03dbb6 to 0x03ebc7 (0x1012 bytes) 000161 1790392304.862155 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x659aabf9 (stale) Mirror 7: 0x659aabf9 (stale) Mirror 8: 0x659aabf9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x659aabf9 (stale) Mirror 7: 0x659aabf9 (stale) Mirror 8: 0x659aabf9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000162 1790392305.166951 read 0x0257bd thru 0x02aedd (0x5721 bytes) 000163 1790392305.179090 read 0x03afac thru 0x03bb35 (0xb8a bytes) 000164 1790392305.192047 read 0x00c90d thru 0x016dc9 (0xa4bd bytes) 000166 1790392305.193173 read 0x030fcb thru 0x034020 (0x3056 bytes) 000167 1790392305.193736 mapwrite 0x03bf17 thru 0x03ffff (0x40e9 bytes) 000168 1790392305.247545 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x659aabf9 (stale) Mirror 8: 0x659aabf9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x659aabf9 (stale) Mirror 8: 0x659aabf9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000169 1790392305.492984 read 0x01b459 thru 0x01d1c2 (0x1d6a bytes) 000171 1790392305.500194 write 0x00e16d thru 0x01b059 (0xceed bytes) 000172 1790392305.506819 read 0x02cc9c thru 0x0374d3 (0xa838 bytes) 000173 1790392305.520302 read 0x036e88 thru 0x039bb9 (0x2d32 bytes) 000174 1790392305.521095 mapread 0x033b31 thru 0x034a13 (0xee3 bytes) 000175 1790392305.523924 mapwrite 0x00fa26 thru 0x0164a9 (0x6a84 bytes) 000176 1790392305.584709 read 0x03fefb thru 0x03ffff (0x105 bytes) 000177 1790392305.585279 write 0x014313 thru 0x014e17 (0xb05 bytes) 000178 1790392305.586071 mapread 0x037af0 thru 0x03ffff (0x8510 bytes) 000179 1790392305.586904 punch from 0x0100f3 to 0x015d5e (0x5c6c bytes) 000180 1790392305.598865 punch from 0x024883 to 0x032fd5 (0xe753 bytes) 000181 1790392305.602391 mapwrite 0x00fed7 thru 0x01fa2c (0xfb56 bytes) 000184 1790392305.662106 mapread 0x03ef31 thru 0x03ffff (0x10cf bytes) 000185 1790392305.662754 mapwrite 0x03440a thru 0x03c82f (0x8426 bytes) 000186 1790392305.707147 write 0x00b35b thru 0x010341 (0x4fe7 bytes) 000188 1790392305.709155 punch from 0x00283a to 0x00baea (0x92b1 bytes) 000190 1790392305.721931 punch from 0x01c931 to 0x02c662 (0xfd32 bytes) 000191 1790392305.726093 read 0x035189 thru 0x03ffff (0xae77 bytes) 000192 1790392305.726752 mapwrite 0x0007d5 thru 0x000f35 (0x761 bytes) 000194 1790392305.765132 mapread 0x00569b thru 0x00f74f (0xa0b5 bytes) 000195 1790392305.775244 mapwrite 0x0018d8 thru 0x005923 (0x404c bytes) 000196 1790392305.820182 punch from 0x002b5d to 0x007159 (0x45fd bytes) 000197 1790392305.825019 write 0x010117 thru 0x01e4ee (0xe3d8 bytes) 000199 1790392305.827341 mapread 0x01a35b thru 0x01ba18 (0x16be bytes) 000202 1790392305.829953 mapwrite 0x00d828 thru 0x0104fb (0x2cd4 bytes) 000203 1790392305.901768 MIRROR_RESYNC 3 mirrors 000204 1790392306.792554 write 0x004509 thru 0x0099c4 (0x54bc bytes) 000205 1790392306.837025 trunc from 0x040000 to 0x032b00 (0xd4ff bytes) 000206 1790392306.854035 mapwrite 0x039e30 thru 0x03c5c2 (0x2793 bytes) 000208 1790392306.930793 trunc from 0x03c5c3 to 0x001a64 (0x3ab5e bytes) 000209 1790392306.948467 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1a65): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a65): Mirror 7: 0x895eebd0 (stale) Mirror 8: 0x895eebd0 (stale) CRC-32 checksum value for chunk [0, 0x1a65): Mirror 1: 0x895eebd0 Mirror 9: 0x895eebd0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a65: skipped 000210 1790392307.847793 mapread 0x000c72 thru 0x001a64 (0xdf3 bytes) 000211 1790392307.863300 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1a65): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a65): Mirror 7: 0x895eebd0 (stale) Mirror 8: 0x895eebd0 (stale) CRC-32 checksum value for chunk [0, 0x1a65): Mirror 1: 0x895eebd0 Mirror 9: 0x895eebd0 Mirror 10: 0x895eebd0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a65: skipped 000213 1790392308.460825 mapwrite 0x00eca6 thru 0x011fd1 (0x332c bytes) 000214 1790392308.589023 read 0x002a31 thru 0x00f7a4 (0xcd74 bytes) 000215 1790392308.599980 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x11fd2): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11fd2): Mirror 7: 0xcbc0042e (stale) Mirror 8: 0xcbc0042e (stale) Mirror 9: 0xb0147470 (stale) Mirror 10: 0xb0147470 (stale) CRC-32 checksum value for chunk [0, 0x11fd2): Mirror 1: 0x64d384bc Mirror 11: 0x64d384bc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11fd2: skipped 000220 1790392309.312787 read 0x005785 thru 0x00bb0b (0x6387 bytes) 000221 1790392309.328076 MIRROR_RESYNC 6 mirrors 000222 1790392310.483720 write 0x01d29e thru 0x024508 (0x726b bytes) 000224 1790392310.600556 trunc from 0x024509 to 0x0014f3 (0x23015 bytes) 000225 1790392310.641778 write 0x00e009 thru 0x01010b (0x2103 bytes) 000227 1790392310.646773 MIRROR_RESYNC 6 mirrors 000229 1790392311.828636 write 0x037e3a thru 0x03fb4e (0x7d15 bytes) 000230 1790392311.909182 mapread 0x025e84 thru 0x027a34 (0x1bb1 bytes) 000232 1790392311.934927 punch from 0x03dfc1 to 0x03fb4e (0x1b8e bytes) 000233 1790392311.951036 write 0x036052 thru 0x03ffff (0x9fae bytes) 000235 1790392311.963911 read 0x0079ac thru 0x015c33 (0xe288 bytes) 000236 1790392311.971373 mapread 0x02d35c thru 0x0332c4 (0x5f69 bytes) 000237 1790392311.979709 write 0x03f5ed thru 0x03ffff (0xa13 bytes) 000240 1790392311.981426 write 0x020058 thru 0x020ce0 (0xc89 bytes) 000241 1790392311.983930 trunc from 0x040000 to 0x01cf49 (0x230b6 bytes) 000242 1790392312.002213 mapwrite 0x00ba4d thru 0x017a1c (0xbfd0 bytes) truncating to largest ever: 0x3ed41 000244 1790392312.061261 trunc from 0x01cf4a to 0x03ed40 (0x21df7 bytes) 000245 1790392312.079129 mapread 0x010012 thru 0x013b21 (0x3b10 bytes) 000247 1790392312.083397 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ed41): Mirror 8: 0x7e2fe4c4 (stale) Mirror 9: 0x7e2fe4c4 (stale) Mirror 10: 0x7e2fe4c4 (stale) Mirror 11: 0x7e2fe4c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed41): Mirror 8: 0x7e2fe4c4 (stale) Mirror 9: 0x7e2fe4c4 (stale) Mirror 10: 0x7e2fe4c4 (stale) Mirror 11: 0x7e2fe4c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed41: skipped 000249 1790392312.388601 read 0x03c544 thru 0x03ed40 (0x27fd bytes) 000250 1790392312.393583 mapwrite 0x01aecb thru 0x020ae8 (0x5c1e bytes) 000251 1790392312.461866 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3ed41): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed41): Mirror 8: 0x7e2fe4c4 (stale) Mirror 9: 0x7e2fe4c4 (stale) Mirror 10: 0x7e2fe4c4 (stale) Mirror 11: 0x7e2fe4c4 (stale) CRC-32 checksum value for chunk [0, 0x3ed41): Mirror 1: 0x7601906 Mirror 12: 0x7601906 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed41: skipped 000252 1790392313.191093 MIRROR_RESYNC 6 mirrors 000253 1790392314.938365 MIRROR_EXTEND to 7 mirrors 000254 1790392316.056021 punch from 0x00eed6 to 0x01674d (0x7878 bytes) 000255 1790392316.153294 write 0x0120ef thru 0x01e6e0 (0xc5f2 bytes) 000256 1790392316.165516 read 0x00af2d thru 0x00f76e (0x4842 bytes) 000257 1790392316.173815 read 0x030ef4 thru 0x0357f5 (0x4902 bytes) 000258 1790392316.192039 mapwrite 0x02c4e0 thru 0x0357c0 (0x92e1 bytes) 000259 1790392316.275715 mapwrite 0x020a6e thru 0x026ba7 (0x613a bytes) 000260 1790392316.337404 read 0x02c3df thru 0x0326e6 (0x6308 bytes) 000261 1790392316.338001 write 0x030dac thru 0x03ffff (0xf254 bytes) 000262 1790392316.339093 mapwrite 0x00c41a thru 0x0136f2 (0x72d9 bytes) 000263 1790392316.410016 mapread 0x00b1e3 thru 0x00fb5a (0x4978 bytes) 000264 1790392316.411105 trunc from 0x040000 to 0x02f666 (0x10999 bytes) 000265 1790392316.468820 trunc from 0x02f667 to 0x02a5ca (0x509c bytes) 000266 1790392316.490594 read 0x00e081 thru 0x00eebf (0xe3f bytes) 000267 1790392316.491388 read 0x00bb51 thru 0x01af72 (0xf422 bytes) 000268 1790392316.497895 MIRROR_RESYNC 7 mirrors 000269 1790392318.215185 punch from 0x007c4c to 0x00a8b7 (0x2c6c bytes) 000270 1790392318.284256 punch from 0x029f73 to 0x02a5ca (0x658 bytes) 000271 1790392318.288640 mapwrite 0x02593c thru 0x02e834 (0x8ef9 bytes) 000272 1790392318.368636 read 0x01a7f4 thru 0x01cca6 (0x24b3 bytes) 000275 1790392318.375370 write 0x005a45 thru 0x007266 (0x1822 bytes) 000276 1790392318.385872 mapwrite 0x007c51 thru 0x0134b3 (0xb863 bytes) 000277 1790392318.457897 mapread 0x001a66 thru 0x00faa6 (0xe041 bytes) 000278 1790392318.459116 mapwrite 0x0170db thru 0x020631 (0x9557 bytes) 000279 1790392318.509730 MIRROR_RESYNC 7 mirrors 000280 1790392320.134998 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fa:0x0]: Invalid argument (22) 000281 1790392320.539044 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fa:0x0]: Invalid argument (22) 000283 1790392320.805757 MIRROR_EXTEND to 6 mirrors 000284 1790392321.359666 mapwrite 0x00889e thru 0x00fe12 (0x7575 bytes) 000285 1790392321.508712 trunc from 0x02e835 to 0x019f3d (0x148f7 bytes) 000286 1790392321.531342 mapwrite 0x00701e thru 0x008ea2 (0x1e85 bytes) 000287 1790392321.570553 trunc from 0x019f3e to 0x015483 (0x4aba bytes) 000288 1790392321.599220 read 0x012efe thru 0x015483 (0x2586 bytes) 000289 1790392321.623153 mapread 0x00b259 thru 0x015483 (0xa22b bytes) 000290 1790392321.628291 write 0x01f2bd thru 0x02a9de (0xb722 bytes) 000291 1790392321.634546 punch from 0x002d0d to 0x00f481 (0xc775 bytes) 000292 1790392321.651204 mapread 0x01bdc7 thru 0x02a7c9 (0xea03 bytes) 000293 1790392321.673757 mapread 0x01e04c thru 0x023929 (0x58de bytes) 000294 1790392321.674627 trunc from 0x02a9df to 0x01e369 (0xc675 bytes) 000295 1790392321.699515 mapread 0x0079cc thru 0x013812 (0xbe47 bytes) 000296 1790392321.712453 write 0x022f1a thru 0x0306a0 (0xd787 bytes) 000298 1790392321.715073 punch from 0x01511e to 0x017b49 (0x2a2c bytes) 000300 1790392321.734836 read 0x00dcd8 thru 0x012535 (0x485e bytes) 000304 1790392321.736996 read 0x016778 thru 0x01c7b5 (0x603e bytes) 000306 1790392321.745069 read 0x0030e1 thru 0x010a32 (0xd952 bytes) 000307 1790392321.745700 mapread 0x019858 thru 0x01bade (0x2287 bytes) 000308 1790392321.746523 trunc from 0x0306a1 to 0x030f6c (0x8cc bytes) 000310 1790392321.764699 trunc from 0x030f6d to 0x00a1a6 (0x26dc6 bytes) 000312 1790392321.785071 trunc from 0x00a1a7 to 0x026063 (0x1bebd bytes) 000316 1790392321.816345 mapread 0x0184ab thru 0x01d246 (0x4d9c bytes) 000317 1790392321.845167 write 0x0256c1 thru 0x034cc2 (0xf602 bytes) 000318 1790392321.853797 read 0x02ac5e thru 0x034cc2 (0xa065 bytes) 000319 1790392321.863398 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34cc3): Mirror 8: 0xa276c753 (stale) Mirror 9: 0xa276c753 (stale) Mirror 10: 0xa276c753 (stale) Mirror 13: 0xa276c753 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34cc3): Mirror 8: 0xa276c753 (stale) Mirror 9: 0xa276c753 (stale) Mirror 10: 0xa276c753 (stale) Mirror 13: 0xa276c753 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34cc3: skipped 000320 1790392322.309822 trunc from 0x034cc3 to 0x03c954 (0x7c92 bytes) 000321 1790392322.338208 mapwrite 0x0213d4 thru 0x0309f2 (0xf61f bytes) 000322 1790392322.412067 trunc from 0x03c955 to 0x0148d2 (0x28082 bytes) 000324 1790392322.441625 write 0x0199eb thru 0x019f52 (0x568 bytes) 000325 1790392322.442786 punch from 0x010d92 to 0x019f52 (0x91c1 bytes) 000326 1790392322.450292 write 0x01c48a thru 0x028ccd (0xc844 bytes) 000327 1790392322.452204 read 0x0241b4 thru 0x024441 (0x28e bytes) 000329 1790392322.453600 trunc from 0x028cce to 0x01bd07 (0xcfc6 bytes) 000330 1790392322.470742 read 0x00f348 thru 0x01bd07 (0xc9c0 bytes) 000331 1790392322.480630 mapread 0x008a46 thru 0x00c2ca (0x3885 bytes) 000334 1790392322.483183 write 0x025b7f thru 0x02e2f8 (0x877a bytes) 000335 1790392322.489178 MIRROR_RESYNC 5 mirrors 000336 1790392324.048007 mapwrite 0x00f190 thru 0x0109e3 (0x1854 bytes) 000337 1790392324.183710 MIRROR_RESYNC 5 mirrors 000338 1790392325.828250 write 0x03c3ec thru 0x03d91b (0x1530 bytes) 000339 1790392325.870669 mapread 0x00ff8c thru 0x015dfd (0x5e72 bytes) 000340 1790392325.880945 mapwrite 0x004ec2 thru 0x008b81 (0x3cc0 bytes) 000341 1790392325.918390 mapwrite 0x01afaa thru 0x026376 (0xb3cd bytes) 000345 1790392325.972104 read 0x005cbd thru 0x009b4d (0x3e91 bytes) 000346 1790392325.972926 mapwrite 0x00611b thru 0x00c888 (0x676e bytes) 000347 1790392326.021911 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3d91c): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d91c): Mirror 8: 0x738d8145 (stale) Mirror 9: 0x738d8145 (stale) Mirror 10: 0x738d8145 (stale) Mirror 13: 0x738d8145 (stale) CRC-32 checksum value for chunk [0, 0x3d91c): Mirror 1: 0xf7e1d35a Mirror 14: 0xf7e1d35a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d91c: skipped 000348 1790392327.351576 read 0x00150b thru 0x0102f5 (0xedeb bytes) 000352 1790392327.387742 mapread 0x036823 thru 0x038a96 (0x2274 bytes) 000353 1790392327.414915 read 0x006ce4 thru 0x0092a6 (0x25c3 bytes) 000354 1790392327.420850 punch from 0x00570d to 0x014f96 (0xf88a bytes) 000356 1790392327.499256 mapread 0x010526 thru 0x01cd9c (0xc877 bytes) 000357 1790392327.519005 mapwrite 0x00930a thru 0x00f1c4 (0x5ebb bytes) 000359 1790392327.595848 trunc from 0x03d91c to 0x030976 (0xcfa5 bytes) 000361 1790392327.617224 mapread 0x02357e thru 0x030976 (0xd3f9 bytes) 000363 1790392327.620510 mapwrite 0x0131af thru 0x01921c (0x606e bytes) 000364 1790392327.665995 read 0x008409 thru 0x00f363 (0x6f5b bytes) truncating to largest ever: 0x3ffe9 000365 1790392327.666655 trunc from 0x030977 to 0x03ffe8 (0xf672 bytes) 000366 1790392327.686855 read 0x03b032 thru 0x03f622 (0x45f1 bytes) 000367 1790392327.692804 mapwrite 0x03004d thru 0x03486f (0x4823 bytes) 000368 1790392327.730458 mapwrite 0x016646 thru 0x01e3fc (0x7db7 bytes) 000369 1790392327.810570 mapread 0x021771 thru 0x02b524 (0x9db4 bytes) 000371 1790392327.817661 MIRROR_RESYNC 6 mirrors 000373 1790392330.456897 read 0x026339 thru 0x032e51 (0xcb19 bytes) 000374 1790392330.465649 write 0x001ac6 thru 0x006ef1 (0x542c bytes) 000376 1790392330.587954 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3ffe9): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ffe9): Mirror 8: 0x65dfb04e (stale) Mirror 9: 0x65dfb04e (stale) Mirror 10: 0x65dfb04e (stale) Mirror 13: 0x65dfb04e (stale) Mirror 14: 0x65dfb04e (stale) CRC-32 checksum value for chunk [0, 0x3ffe9): Mirror 1: 0xb6766eb5 Mirror 15: 0xb6766eb5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ffe9: skipped 000377 1790392331.906324 write 0x03e512 thru 0x03ee30 (0x91f bytes) 000378 1790392331.977405 punch from 0x02d105 to 0x030191 (0x308d bytes) 000379 1790392332.004104 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3ffe9): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ffe9): Mirror 8: 0x65dfb04e (stale) Mirror 9: 0x65dfb04e (stale) Mirror 10: 0x65dfb04e (stale) Mirror 13: 0x65dfb04e (stale) Mirror 14: 0x65dfb04e (stale) Mirror 15: 0xb6766eb5 (stale) CRC-32 checksum value for chunk [0, 0x3ffe9): Mirror 1: 0xa6ee1f7d Mirror 16: 0xa6ee1f7d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ffe9: skipped 000380 1790392333.321800 punch from 0x034f94 to 0x0380c6 (0x3133 bytes) 000382 1790392333.390439 mapread 0x0161c5 thru 0x01d59e (0x73da bytes) 000383 1790392333.402877 write 0x039789 thru 0x03ffff (0x6877 bytes) 000384 1790392333.407140 trunc from 0x040000 to 0x0112b6 (0x2ed49 bytes) 000385 1790392333.423198 mapread 0x00d4ab thru 0x00f4f1 (0x2047 bytes) 000386 1790392333.432992 write 0x01dbc1 thru 0x02514b (0x758b bytes) 000390 1790392333.440928 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2514c): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2514c): Mirror 8: 0xffe58aca (stale) Mirror 9: 0xffe58aca (stale) Mirror 10: 0xffe58aca (stale) Mirror 13: 0xffe58aca (stale) Mirror 14: 0xffe58aca (stale) Mirror 15: 0xa3e22a0d (stale) Mirror 16: 0xa3e22a0d (stale) CRC-32 checksum value for chunk [0, 0x2514c): Mirror 1: 0xd5c8a5d Mirror 17: 0xd5c8a5d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2514c: skipped 000391 1790392334.645011 read 0x010439 thru 0x01c9e6 (0xc5ae bytes) 000394 1790392334.677101 trunc from 0x02514c to 0x00b584 (0x19bc7 bytes) 000395 1790392334.777989 read 0x0036c2 thru 0x00a080 (0x69bf bytes) 000396 1790392334.792580 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb585): Mirror 8: 0x1da0763f (stale) Mirror 10: 0x1da0763f (stale) Mirror 13: 0x1da0763f (stale) Mirror 14: 0x1da0763f (stale) Mirror 15: 0x8978c9c4 (stale) Mirror 16: 0x8978c9c4 (stale) Mirror 17: 0x8978c9c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb585): Mirror 8: 0x1da0763f (stale) Mirror 10: 0x1da0763f (stale) Mirror 13: 0x1da0763f (stale) Mirror 14: 0x1da0763f (stale) Mirror 15: 0x8978c9c4 (stale) Mirror 16: 0x8978c9c4 (stale) Mirror 17: 0x8978c9c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb585: skipped 000397 1790392335.215001 trunc from 0x00b585 to 0x038f75 (0x2d9f1 bytes) 000399 1790392335.228747 read 0x033b65 thru 0x036c0a (0x30a6 bytes) 000402 1790392335.233979 punch from 0x01beea to 0x020f83 (0x509a bytes) 000403 1790392335.236919 write 0x01b405 thru 0x01d212 (0x1e0e bytes) 000406 1790392335.243328 trunc from 0x038f76 to 0x03e20d (0x5298 bytes) 000411 1790392335.257455 punch from 0x021139 to 0x0237c0 (0x2688 bytes) 000412 1790392335.269098 mapread 0x010ba6 thru 0x02096e (0xfdc9 bytes) 000413 1790392335.297974 write 0x02d238 thru 0x03adec (0xdbb5 bytes) 000414 1790392335.312521 mapwrite 0x008fbd thru 0x0133db (0xa41f bytes) 000415 1790392335.400828 MIRROR_RESYNC 8 mirrors 000416 1790392339.614226 punch from 0x030762 to 0x0347f5 (0x4094 bytes) 000417 1790392339.761641 mapwrite 0x01600a thru 0x01889b (0x2892 bytes) 000420 1790392339.925291 write 0x016093 thru 0x01df4c (0x7eba bytes) 000422 1790392339.927037 trunc from 0x03e20e to 0x006cd9 (0x37534 bytes) 000423 1790392339.967444 trunc from 0x006cda to 0x03a5b6 (0x338dd bytes) 000424 1790392339.987561 punch from 0x0360c4 to 0x03a5b6 (0x44f3 bytes) 000425 1790392339.992882 trunc from 0x03a5b7 to 0x03938c (0x122a bytes) 000426 1790392340.014064 trunc from 0x03938d to 0x00502d (0x3435f bytes) 000428 1790392340.041284 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x502e): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x502e): Mirror 8: 0xc8c89e31 (stale) Mirror 10: 0xc8c89e31 (stale) Mirror 13: 0xc8c89e31 (stale) Mirror 14: 0xc8c89e31 (stale) Mirror 15: 0xc8c89e31 (stale) Mirror 16: 0xc8c89e31 (stale) Mirror 17: 0xc8c89e31 (stale) CRC-32 checksum value for chunk [0, 0x502e): Mirror 1: 0xc8c89e31 Mirror 18: 0xc8c89e31 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x502e: skipped 000431 1790392340.850791 mapwrite 0x01e2d7 thru 0x02d315 (0xf03f bytes) 000432 1790392340.988807 punch from 0x01fe22 to 0x02d315 (0xd4f4 bytes) 000433 1790392340.993506 read 0x02cbef thru 0x02d315 (0x727 bytes) 000434 1790392340.999168 mapread 0x02bc3d thru 0x02d315 (0x16d9 bytes) 000435 1790392341.008007 write 0x0127a5 thru 0x01c223 (0x9a7f bytes) 000436 1790392341.014136 write 0x030672 thru 0x036285 (0x5c14 bytes) 000437 1790392341.021995 MIRROR_RESYNC 9 mirrors 000438 1790392344.261452 write 0x00e661 thru 0x01ccd7 (0xe677 bytes) 000439 1790392344.349443 trunc from 0x036286 to 0x029d93 (0xc4f2 bytes) 000441 1790392344.373877 mapread 0x012c2b thru 0x019aed (0x6ec3 bytes) 000442 1790392344.376893 write 0x00c08b thru 0x015ebc (0x9e32 bytes) 000443 1790392344.386917 mapwrite 0x037a8e thru 0x039207 (0x177a bytes) 000444 1790392344.515605 mapread 0x015e92 thru 0x01fd0a (0x9e79 bytes) 000445 1790392344.527881 MIRROR_RESYNC 9 mirrors 000447 1790392347.524900 MIRROR_EXTEND to 10 mirrors 000448 1790392348.496897 punch from 0x014ffc to 0x020e9c (0xbea1 bytes) 000449 1790392348.590623 trunc from 0x039208 to 0x030179 (0x908e bytes) 000450 1790392348.614192 trunc from 0x03017a to 0x010a1d (0x1f75c bytes) 000451 1790392348.624693 punch from 0x004eb9 to 0x00cc66 (0x7dae bytes) 000452 1790392348.627855 read 0x00b382 thru 0x010a1d (0x569c bytes) 000455 1790392348.632665 read 0x000a6e thru 0x008170 (0x7703 bytes) 000457 1790392348.637536 punch from 0x0105c5 to 0x010a1d (0x459 bytes) 000458 1790392348.641005 punch from 0x005cd8 to 0x010a1d (0xad46 bytes) 000459 1790392348.651865 punch from 0x010a06 to 0x010a1d (0x18 bytes) 000460 1790392348.656623 read 0x00463d thru 0x00bd52 (0x7716 bytes) 000461 1790392348.662780 mapread 0x005294 thru 0x008ff1 (0x3d5e bytes) 000463 1790392348.664091 read 0x003563 thru 0x004852 (0x12f0 bytes) 000465 1790392348.664968 mapread 0x009b9c thru 0x00c848 (0x2cad bytes) 000468 1790392348.667209 write 0x012187 thru 0x01222f (0xa9 bytes) 000470 1790392348.668908 trunc from 0x012230 to 0x00c59f (0x5c90 bytes) 000471 1790392348.682758 write 0x038a07 thru 0x03ffff (0x75f9 bytes) 000472 1790392348.684677 mapread 0x038c3c thru 0x03ffff (0x73c4 bytes) 000475 1790392348.687633 write 0x032e84 thru 0x03ffff (0xd17c bytes) 000476 1790392348.704188 read 0x00dbbd thru 0x01d93a (0xfd7e bytes) 000477 1790392348.705055 mapwrite 0x0312e0 thru 0x03d161 (0xbe82 bytes) 000478 1790392348.764181 mapwrite 0x012c2e thru 0x020913 (0xdce6 bytes) 000479 1790392348.819070 trunc from 0x040000 to 0x0276e4 (0x1891b bytes) 000480 1790392348.834765 write 0x003b1c thru 0x00d610 (0x9af5 bytes) 000481 1790392348.836334 mapwrite 0x003a05 thru 0x00b297 (0x7893 bytes) 000482 1790392348.892845 mapwrite 0x006699 thru 0x00c853 (0x61bb bytes) 000485 1790392348.950226 read 0x0103c4 thru 0x016e4a (0x6a87 bytes) 000486 1790392348.951444 read 0x000645 thru 0x00a599 (0x9f55 bytes) 000487 1790392348.952735 write 0x01b2f9 thru 0x024800 (0x9508 bytes) 000489 1790392348.955120 punch from 0x020805 to 0x026610 (0x5e0c bytes) 000490 1790392348.972626 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x276e5): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x276e5): Mirror 8: 0x14dca94b (stale) Mirror 10: 0x14dca94b (stale) Mirror 13: 0x14dca94b (stale) Mirror 14: 0x14dca94b (stale) Mirror 15: 0x14dca94b (stale) Mirror 16: 0x14dca94b (stale) Mirror 17: 0x14dca94b (stale) Mirror 18: 0x14dca94b (stale) Mirror 19: 0x14dca94b (stale) CRC-32 checksum value for chunk [0, 0x276e5): Mirror 1: 0xd396f26c Mirror 20: 0xd396f26c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x276e5: skipped 000491 1790392350.164620 mapread 0x01e7d4 thru 0x020624 (0x1e51 bytes) 000493 1790392350.220770 MIRROR_RESYNC 11 mirrors 000495 1790392352.443193 read 0x01010d thru 0x01721b (0x710f bytes) 000497 1790392352.452067 trunc from 0x0276e5 to 0x000d47 (0x2699d bytes) 000499 1790392352.528884 read 0x000727 thru 0x000d47 (0x621 bytes) 000500 1790392352.534161 mapread 0x000a26 thru 0x000d47 (0x322 bytes) 000503 1790392352.536060 mapwrite 0x00662b thru 0x00bed3 (0x58a9 bytes) 000505 1790392352.611667 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0xbed4): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbed4): Mirror 8: 0xc467bd1a (stale) Mirror 10: 0xc467bd1a (stale) Mirror 13: 0xc467bd1a (stale) Mirror 14: 0xc467bd1a (stale) Mirror 15: 0xc467bd1a (stale) Mirror 16: 0xc467bd1a (stale) Mirror 17: 0xc467bd1a (stale) Mirror 18: 0xc467bd1a (stale) Mirror 19: 0xc467bd1a (stale) Mirror 20: 0xc467bd1a (stale) CRC-32 checksum value for chunk [0, 0xbed4): Mirror 1: 0x9bc6f3a7 Mirror 21: 0x9bc6f3a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbed4: skipped 000507 1790392353.467987 mapread 0x004e56 thru 0x006bc7 (0x1d72 bytes) 000508 1790392353.498124 read 0x006e7d thru 0x00bed3 (0x5057 bytes) 000509 1790392353.505749 write 0x03f2d7 thru 0x03fc51 (0x97b bytes) 000511 1790392353.576324 mapwrite 0x0034de thru 0x0051f8 (0x1d1b bytes) 000512 1790392353.649652 mapread 0x000f34 thru 0x00c3a8 (0xb475 bytes) 000513 1790392353.654980 mapread 0x0325f2 thru 0x03eedb (0xc8ea bytes) 000514 1790392353.674901 MIRROR_RESYNC 12 mirrors 000515 1790392356.453764 mapwrite 0x039520 thru 0x03ffff (0x6ae0 bytes) 000516 1790392356.536640 write 0x0192a4 thru 0x02750d (0xe26a bytes) 000517 1790392356.544064 punch from 0x00c951 to 0x01b27b (0xe92b bytes) 000518 1790392356.551864 trunc from 0x040000 to 0x013ff7 (0x2c008 bytes) 000519 1790392356.564038 mapread 0x00e2ad thru 0x013ff7 (0x5d4b bytes) 000520 1790392356.569343 read 0x009792 thru 0x013ff7 (0xa866 bytes) 000521 1790392356.569804 punch from 0x009d8e to 0x013ff7 (0xa26a bytes) 000523 1790392356.573595 trunc from 0x013ff8 to 0x012e7c (0x117b bytes) 000525 1790392356.584432 write 0x007a2f thru 0x010764 (0x8d36 bytes) 000526 1790392356.589218 mapwrite 0x02233e thru 0x02f31c (0xcfdf bytes) 000527 1790392356.642943 mapwrite 0x0356c3 thru 0x03ffff (0xa93d bytes) 000528 1790392356.688095 trunc from 0x040000 to 0x00bdf8 (0x34207 bytes) 000529 1790392356.698380 trunc from 0x00bdf9 to 0x017cf7 (0xbeff bytes) 000530 1790392356.708555 mapwrite 0x030a8d thru 0x03ffff (0xf573 bytes) 000531 1790392356.760758 trunc from 0x040000 to 0x0269ff (0x19600 bytes) 000533 1790392356.770884 mapwrite 0x02156c thru 0x024c73 (0x3708 bytes) 000534 1790392356.806257 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x26a00): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26a00): Mirror 8: 0x1ac1e9c8 (stale) Mirror 10: 0x1ac1e9c8 (stale) Mirror 13: 0x1ac1e9c8 (stale) Mirror 14: 0x1ac1e9c8 (stale) Mirror 15: 0x1ac1e9c8 (stale) Mirror 16: 0x1ac1e9c8 (stale) Mirror 17: 0x1ac1e9c8 (stale) Mirror 18: 0x1ac1e9c8 (stale) Mirror 19: 0x1ac1e9c8 (stale) Mirror 20: 0x1ac1e9c8 (stale) Mirror 21: 0x1ac1e9c8 (stale) CRC-32 checksum value for chunk [0, 0x26a00): Mirror 1: 0x6a98ca0b Mirror 22: 0x6a98ca0b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26a00: skipped 000535 1790392357.616308 mapread 0x0086ec thru 0x01043f (0x7d54 bytes) 000536 1790392357.659842 mapwrite 0x02d27e thru 0x0372a7 (0xa02a bytes) 000537 1790392357.809380 write 0x02a4fe thru 0x03a2d7 (0xfdda bytes) 000538 1790392357.810645 trunc from 0x03a2d8 to 0x01e3a7 (0x1bf30 bytes) 000539 1790392357.823173 mapread 0x01950e thru 0x01e3a7 (0x4e9a bytes) 000540 1790392357.829696 mapwrite 0x00d18d thru 0x012ff5 (0x5e69 bytes) 000542 1790392357.893640 mapwrite 0x00365a thru 0x003fc1 (0x968 bytes) 000543 1790392357.924140 mapwrite 0x03a721 thru 0x03ffff (0x58df bytes) 000544 1790392357.988730 read 0x0133f5 thru 0x01de70 (0xaa7c bytes) 000545 1790392357.989577 read 0x03efa1 thru 0x03ffff (0x105f bytes) 000548 1790392357.991344 trunc from 0x040000 to 0x02df30 (0x120cf bytes) 000549 1790392358.009701 read 0x024be2 thru 0x02df30 (0x934f bytes) 000550 1790392358.018223 punch from 0x01775c to 0x024c1f (0xd4c4 bytes) 000552 1790392358.031847 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x2df31): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2df31): Mirror 8: 0x7dc331fe (stale) Mirror 10: 0x7dc331fe (stale) Mirror 13: 0x7dc331fe (stale) Mirror 14: 0x7dc331fe (stale) Mirror 15: 0x7dc331fe (stale) Mirror 16: 0x7dc331fe (stale) Mirror 17: 0x7dc331fe (stale) Mirror 18: 0x7dc331fe (stale) Mirror 19: 0x7dc331fe (stale) Mirror 20: 0x7dc331fe (stale) Mirror 21: 0x7dc331fe (stale) Mirror 22: 0x41434326 (stale) CRC-32 checksum value for chunk [0, 0x2df31): Mirror 1: 0x88915157 Mirror 23: 0x88915157 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2df31: skipped 000553 1790392359.215409 write 0x016339 thru 0x023783 (0xd44b bytes) 000555 1790392359.312192 write 0x031755 thru 0x033e5a (0x2706 bytes) 000557 1790392359.315221 read 0x00bd8b thru 0x010c81 (0x4ef7 bytes) 000558 1790392359.326886 mapwrite 0x0128be thru 0x022495 (0xfbd8 bytes) 000559 1790392359.394727 read 0x03237c thru 0x033e5a (0x1adf bytes) 000560 1790392359.395349 mapwrite 0x010717 thru 0x018561 (0x7e4b bytes) 000561 1790392359.436971 mapwrite 0x03567c thru 0x036d23 (0x16a8 bytes) 000564 1790392359.505566 punch from 0x029b5c to 0x036d23 (0xd1c8 bytes) 000565 1790392359.510663 mapread 0x012231 thru 0x0165c2 (0x4392 bytes) 000566 1790392359.511747 mapread 0x021bf3 thru 0x02b499 (0x98a7 bytes) 000567 1790392359.525534 trunc from 0x036d24 to 0x00b70b (0x2b618 bytes) 000568 1790392359.542824 punch from 0x00a8e9 to 0x00b70b (0xe23 bytes) 000569 1790392359.546790 mapwrite 0x004fcd thru 0x00778a (0x27be bytes) 000571 1790392359.586375 MIRROR_SPLIT mirror 7 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb70c): Mirror 8: 0x404a51bb (stale) Mirror 10: 0x404a51bb (stale) Mirror 13: 0x404a51bb (stale) Mirror 14: 0x404a51bb (stale) Mirror 15: 0x404a51bb (stale) Mirror 16: 0x404a51bb (stale) Mirror 18: 0x404a51bb (stale) Mirror 19: 0x404a51bb (stale) Mirror 20: 0x404a51bb (stale) Mirror 21: 0x404a51bb (stale) Mirror 22: 0xba12d943 (stale) Mirror 23: 0x44975a05 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb70c): Mirror 8: 0x404a51bb (stale) Mirror 10: 0x404a51bb (stale) Mirror 13: 0x404a51bb (stale) Mirror 14: 0x404a51bb (stale) Mirror 15: 0x404a51bb (stale) Mirror 16: 0x404a51bb (stale) Mirror 18: 0x404a51bb (stale) Mirror 19: 0x404a51bb (stale) Mirror 20: 0x404a51bb (stale) Mirror 21: 0x404a51bb (stale) Mirror 22: 0xba12d943 (stale) Mirror 23: 0x44975a05 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb70c: skipped 000573 1790392359.979685 mapwrite 0x013d37 thru 0x014802 (0xacc bytes) 000575 1790392360.050796 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x14803): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14803): Mirror 8: 0x1703f78 (stale) Mirror 10: 0x1703f78 (stale) Mirror 13: 0x1703f78 (stale) Mirror 14: 0x1703f78 (stale) Mirror 15: 0x1703f78 (stale) Mirror 16: 0x1703f78 (stale) Mirror 18: 0x1703f78 (stale) Mirror 19: 0x1703f78 (stale) Mirror 20: 0x1703f78 (stale) Mirror 21: 0x1703f78 (stale) Mirror 22: 0x175e8608 (stale) Mirror 23: 0x6d1d421e (stale) CRC-32 checksum value for chunk [0, 0x14803): Mirror 1: 0xc160433c Mirror 24: 0xc160433c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14803: skipped 000576 1790392361.037178 mapread 0x013dec thru 0x014802 (0xa17 bytes) 000577 1790392361.052872 MIRROR_SPLIT mirror 6 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14803): Mirror 8: 0x1703f78 (stale) Mirror 10: 0x1703f78 (stale) Mirror 13: 0x1703f78 (stale) Mirror 14: 0x1703f78 (stale) Mirror 15: 0x1703f78 (stale) Mirror 18: 0x1703f78 (stale) Mirror 19: 0x1703f78 (stale) Mirror 20: 0x1703f78 (stale) Mirror 21: 0x1703f78 (stale) Mirror 22: 0x175e8608 (stale) Mirror 23: 0x6d1d421e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14803): Mirror 8: 0x1703f78 (stale) Mirror 10: 0x1703f78 (stale) Mirror 13: 0x1703f78 (stale) Mirror 14: 0x1703f78 (stale) Mirror 15: 0x1703f78 (stale) Mirror 18: 0x1703f78 (stale) Mirror 19: 0x1703f78 (stale) Mirror 20: 0x1703f78 (stale) Mirror 21: 0x1703f78 (stale) Mirror 22: 0x175e8608 (stale) Mirror 23: 0x6d1d421e (stale) CRC-32 checksum value for chunk [0, 0x14803): Mirror 1: 0xc160433c Mirror 24: 0xc160433c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14803: skipped 000578 1790392361.875930 mapread 0x0127d2 thru 0x014802 (0x2031 bytes) 000580 1790392361.940935 mapread 0x007f90 thru 0x0096f2 (0x1763 bytes) 000581 1790392361.961265 mapwrite 0x033d20 thru 0x03ec6e (0xaf4f bytes) 000582 1790392362.085342 write 0x0126c3 thru 0x017f0c (0x584a bytes) 000583 1790392362.099188 punch from 0x020ffc to 0x02afc5 (0x9fca bytes) 000585 1790392362.112588 trunc from 0x03ec6f to 0x02e5c0 (0x106ae bytes) 000586 1790392362.149065 mapwrite 0x0124a9 thru 0x01e1a0 (0xbcf8 bytes) 000588 1790392362.218958 punch from 0x02b083 to 0x02e5c0 (0x353e bytes) 000590 1790392362.227406 read 0x02b81b thru 0x02e5c0 (0x2da6 bytes) 000592 1790392362.233433 punch from 0x00e45a to 0x0160ab (0x7c52 bytes) 000595 1790392362.245089 trunc from 0x02e5c1 to 0x03cd3e (0xe77e bytes) 000596 1790392362.260767 trunc from 0x03cd3f to 0x0394cd (0x3871 bytes) 000597 1790392362.296437 punch from 0x014b28 to 0x01c49c (0x7975 bytes) 000598 1790392362.304025 punch from 0x0225f2 to 0x02938e (0x6d9d bytes) 000599 1790392362.316793 mapread 0x02bd72 thru 0x03929f (0xd52e bytes) 000601 1790392362.338384 mapwrite 0x0359eb thru 0x03ffff (0xa615 bytes) 000606 1790392362.439706 trunc from 0x040000 to 0x014020 (0x2bfdf bytes) 000607 1790392362.465464 trunc from 0x014021 to 0x03e874 (0x2a854 bytes) 000608 1790392362.492105 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x3e875): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e875): Mirror 8: 0x97a4e4c1 (stale) Mirror 10: 0x97a4e4c1 (stale) Mirror 13: 0x97a4e4c1 (stale) Mirror 14: 0x97a4e4c1 (stale) Mirror 15: 0x97a4e4c1 (stale) Mirror 18: 0x97a4e4c1 (stale) Mirror 19: 0x97a4e4c1 (stale) Mirror 20: 0x97a4e4c1 (stale) Mirror 21: 0x97a4e4c1 (stale) Mirror 22: 0x16667b6a (stale) Mirror 23: 0xb49e3f53 (stale) Mirror 24: 0x378fac55 (stale) CRC-32 checksum value for chunk [0, 0x3e875): Mirror 1: 0x8b26ee1e Mirror 25: 0x8b26ee1e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e875: skipped 000609 1790392363.597746 punch from 0x03a647 to 0x03e874 (0x422e bytes) 000610 1790392363.646069 mapread 0x00f274 thru 0x01002a (0xdb7 bytes) 000611 1790392363.653739 trunc from 0x03e875 to 0x0166b6 (0x281be bytes) 000612 1790392363.677899 mapread 0x01536d thru 0x0166b6 (0x134a bytes) 000615 1790392363.686725 trunc from 0x0166b7 to 0x02ed14 (0x1865e bytes) 000616 1790392363.697887 mapread 0x024e28 thru 0x02ab1d (0x5cf6 bytes) 000619 1790392363.708736 MIRROR_RESYNC 14 mirrors 000620 1790392367.461849 read 0x01e141 thru 0x0262d9 (0x8199 bytes) 000622 1790392367.468428 punch from 0x01b5bd to 0x02a429 (0xee6d bytes) 000625 1790392367.559150 read 0x009759 thru 0x01765b (0xdf03 bytes) 000626 1790392367.576512 MIRROR_SPLIT mirror 5 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ed15): Mirror 8: 0x1cf5edab (stale) Mirror 10: 0x1cf5edab (stale) Mirror 13: 0x1cf5edab (stale) Mirror 14: 0x1cf5edab (stale) Mirror 18: 0x1cf5edab (stale) Mirror 19: 0x1cf5edab (stale) Mirror 20: 0x1cf5edab (stale) Mirror 21: 0x1cf5edab (stale) Mirror 22: 0x1cf5edab (stale) Mirror 23: 0x1cf5edab (stale) Mirror 24: 0x1cf5edab (stale) Mirror 25: 0x1cf5edab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ed15): Mirror 8: 0x1cf5edab (stale) Mirror 10: 0x1cf5edab (stale) Mirror 13: 0x1cf5edab (stale) Mirror 14: 0x1cf5edab (stale) Mirror 18: 0x1cf5edab (stale) Mirror 19: 0x1cf5edab (stale) Mirror 20: 0x1cf5edab (stale) Mirror 21: 0x1cf5edab (stale) Mirror 22: 0x1cf5edab (stale) Mirror 23: 0x1cf5edab (stale) Mirror 24: 0x1cf5edab (stale) Mirror 25: 0x1cf5edab (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ed15: skipped 000627 1790392368.156026 trunc from 0x02ed15 to 0x0332f7 (0x45e3 bytes) 000628 1790392368.176289 read 0x00073f thru 0x0025d3 (0x1e95 bytes) 000629 1790392368.183834 write 0x022605 thru 0x026d36 (0x4732 bytes) 000630 1790392368.195087 write 0x00854a thru 0x015ee8 (0xd99f bytes) 000631 1790392368.208317 mapread 0x00a31e thru 0x00b9a0 (0x1683 bytes) 000633 1790392368.214059 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x332f8): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x332f8): Mirror 8: 0x87be59b9 (stale) Mirror 10: 0x87be59b9 (stale) Mirror 13: 0x87be59b9 (stale) Mirror 14: 0x87be59b9 (stale) Mirror 18: 0x87be59b9 (stale) Mirror 19: 0x87be59b9 (stale) Mirror 20: 0x87be59b9 (stale) Mirror 21: 0x87be59b9 (stale) Mirror 22: 0x87be59b9 (stale) Mirror 23: 0x87be59b9 (stale) Mirror 24: 0x87be59b9 (stale) Mirror 25: 0x87be59b9 (stale) CRC-32 checksum value for chunk [0, 0x332f8): Mirror 1: 0x57f164fb Mirror 26: 0x57f164fb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x332f8: skipped 000634 1790392369.427263 mapwrite 0x01554d thru 0x01a960 (0x5414 bytes) 000636 1790392369.576094 punch from 0x01e297 to 0x028231 (0x9f9b bytes) 000637 1790392369.580420 read 0x01ca49 thru 0x01cce8 (0x2a0 bytes) 000638 1790392369.584070 MIRROR_RESYNC 14 mirrors 000639 1790392372.588218 mapwrite 0x01d60d thru 0x01fac7 (0x24bb bytes) 000640 1790392372.807062 punch from 0x00a4e1 to 0x013951 (0x9471 bytes) 000641 1790392372.818303 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x332f8): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x332f8): Mirror 8: 0x88263ae9 (stale) Mirror 10: 0x88263ae9 (stale) Mirror 13: 0x88263ae9 (stale) Mirror 14: 0x88263ae9 (stale) Mirror 18: 0x88263ae9 (stale) Mirror 19: 0x88263ae9 (stale) Mirror 20: 0x88263ae9 (stale) Mirror 21: 0x88263ae9 (stale) Mirror 22: 0x88263ae9 (stale) Mirror 23: 0x88263ae9 (stale) Mirror 24: 0x88263ae9 (stale) Mirror 25: 0x88263ae9 (stale) Mirror 26: 0x88263ae9 (stale) CRC-32 checksum value for chunk [0, 0x332f8): Mirror 1: 0xfcef0d3b Mirror 27: 0xfcef0d3b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x332f8: skipped 000643 1790392374.282398 read 0x004d4f thru 0x0051f8 (0x4aa bytes) 000644 1790392374.310174 MIRROR_RESYNC 15 mirrors 000645 1790392379.108045 trunc from 0x0332f8 to 0x00a123 (0x291d4 bytes) 000646 1790392379.204215 mapwrite 0x02e89d thru 0x03b7fa (0xcf5e bytes) 000647 1790392379.297064 mapread 0x01b7af thru 0x022367 (0x6bb9 bytes) 000648 1790392379.306080 mapread 0x012ab2 thru 0x017a79 (0x4fc8 bytes) 000649 1790392379.324678 trunc from 0x03b7fb to 0x02e10f (0xd6eb bytes) 000650 1790392379.341664 trunc from 0x02e110 to 0x031276 (0x3167 bytes) 000652 1790392379.354344 mapread 0x00769a thru 0x0142cb (0xcc32 bytes) 000654 1790392379.360932 mapwrite 0x03bbe6 thru 0x03ffff (0x441a bytes) 000656 1790392379.446407 read 0x006a0b thru 0x00f00d (0x8603 bytes) 000657 1790392379.448943 mapread 0x0287dd thru 0x02f578 (0x6d9c bytes) 000658 1790392379.461932 trunc from 0x040000 to 0x00a7c6 (0x35839 bytes) 000660 1790392379.481066 mapwrite 0x03c775 thru 0x03ffff (0x388b bytes) 000661 1790392379.555405 mapwrite 0x002e7f thru 0x005a05 (0x2b87 bytes) 000662 1790392379.597357 MIRROR_RESYNC 15 mirrors 000663 1790392382.822416 read 0x02d8e0 thru 0x0380bd (0xa7de bytes) 000665 1790392382.830933 mapread 0x03ae4e thru 0x03fcd1 (0x4e84 bytes) 000670 1790392382.848644 punch from 0x01b772 to 0x028ddb (0xd66a bytes) 000671 1790392382.909108 trunc from 0x040000 to 0x00cc0e (0x333f1 bytes) 000673 1790392382.927426 read 0x006ab2 thru 0x00cc0e (0x615d bytes) 000674 1790392382.932616 write 0x009505 thru 0x0162d7 (0xcdd3 bytes) 000675 1790392382.934541 mapwrite 0x02f08b thru 0x02faca (0xa40 bytes) 000677 1790392383.041268 trunc from 0x02facb to 0x024812 (0xb2b8 bytes) 000678 1790392383.053123 read 0x008365 thru 0x017d3e (0xf9da bytes) 000679 1790392383.053737 punch from 0x00769f to 0x00dba9 (0x650b bytes) 000682 1790392383.057568 mapread 0x00fd90 thru 0x014d67 (0x4fd8 bytes) 000684 1790392383.058803 mapwrite 0x01fb93 thru 0x02da6d (0xdedb bytes) 000685 1790392383.154997 read 0x019202 thru 0x01ed2c (0x5b2b bytes) 000686 1790392383.156839 trunc from 0x02da6e to 0x0358a6 (0x7e39 bytes) 000687 1790392383.170681 read 0x01343b thru 0x0151a0 (0x1d66 bytes) 000688 1790392383.171267 mapwrite 0x0326b7 thru 0x032ffd (0x947 bytes) 000690 1790392383.219429 mapwrite 0x022b2a thru 0x026ad6 (0x3fad bytes) 000691 1790392383.259155 mapread 0x021849 thru 0x025a32 (0x41ea bytes) 000692 1790392383.260213 write 0x034643 thru 0x0355e8 (0xfa6 bytes) 000693 1790392383.265360 MIRROR_RESYNC 15 mirrors 000697 1790392389.448311 MIRROR_SPLIT mirror 2 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fa:0x0]: Invalid argument (22) 000699 1790392389.871420 mapread 0x0325ca thru 0x0358a6 (0x32dd bytes) 000701 1790392389.890954 punch from 0x01e896 to 0x0230bd (0x4828 bytes) 000703 1790392389.998050 mapread 0x033d7c thru 0x034ca0 (0xf25 bytes) 000704 1790392390.017620 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x358a7): Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x358a7): Mirror 8: 0x996851fd (stale) Mirror 13: 0x996851fd (stale) Mirror 14: 0x996851fd (stale) Mirror 18: 0x996851fd (stale) Mirror 19: 0x996851fd (stale) Mirror 20: 0x996851fd (stale) Mirror 21: 0x996851fd (stale) Mirror 22: 0x996851fd (stale) Mirror 23: 0x996851fd (stale) Mirror 24: 0x996851fd (stale) Mirror 25: 0x996851fd (stale) Mirror 26: 0x996851fd (stale) Mirror 27: 0x996851fd (stale) CRC-32 checksum value for chunk [0, 0x358a7): Mirror 1: 0x67fbb418 Mirror 28: 0x67fbb418 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x358a7: skipped 000706 1790392391.343296 read 0x0064f1 thru 0x00d51d (0x702d bytes) 000707 1790392391.358117 MIRROR_SPLIT mirror 13 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x358a7): Mirror 8: 0x996851fd (stale) Mirror 13: 0x996851fd (stale) Mirror 14: 0x996851fd (stale) Mirror 18: 0x996851fd (stale) Mirror 19: 0x996851fd (stale) Mirror 20: 0x996851fd (stale) Mirror 21: 0x996851fd (stale) Mirror 22: 0x996851fd (stale) Mirror 23: 0x996851fd (stale) Mirror 24: 0x996851fd (stale) Mirror 25: 0x996851fd (stale) Mirror 26: 0x996851fd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x358a7): Mirror 8: 0x996851fd (stale) Mirror 13: 0x996851fd (stale) Mirror 14: 0x996851fd (stale) Mirror 18: 0x996851fd (stale) Mirror 19: 0x996851fd (stale) Mirror 20: 0x996851fd (stale) Mirror 21: 0x996851fd (stale) Mirror 22: 0x996851fd (stale) Mirror 23: 0x996851fd (stale) Mirror 24: 0x996851fd (stale) Mirror 25: 0x996851fd (stale) Mirror 26: 0x996851fd (stale) CRC-32 checksum value for chunk [0, 0x358a7): Mirror 1: 0x67fbb418 Mirror 28: 0x67fbb418 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x358a7: skipped 000708 1790392392.060615 mapwrite 0x008db5 thru 0x00b788 (0x29d4 bytes) 000709 1790392392.216727 read 0x01e110 thru 0x021183 (0x3074 bytes) 000710 1790392392.232245 mapwrite 0x00667d thru 0x010647 (0x9fcb bytes) 000711 1790392392.299828 write 0x01d478 thru 0x020e4b (0x39d4 bytes) 000712 1790392392.312545 MIRROR_RESYNC 14 mirrors 000713 1790392398.065046 punch from 0x028a1a to 0x034ccc (0xc2b3 bytes) 000714 1790392398.132529 write 0x03ba80 thru 0x03fd09 (0x428a bytes) 000715 1790392398.134443 write 0x020e15 thru 0x0235c9 (0x27b5 bytes) 000716 1790392398.141384 read 0x035d31 thru 0x0362e4 (0x5b4 bytes) 000717 1790392398.146688 mapread 0x03711c thru 0x03fd09 (0x8bee bytes) 000718 1790392398.157315 write 0x00d301 thru 0x0166a0 (0x93a0 bytes) 000720 1790392398.167704 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x3fd0a): Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd0a): Mirror 8: 0x5b06e633 (stale) Mirror 13: 0x5b06e633 (stale) Mirror 14: 0x5b06e633 (stale) Mirror 18: 0x5b06e633 (stale) Mirror 19: 0x5b06e633 (stale) Mirror 20: 0x5b06e633 (stale) Mirror 21: 0x5b06e633 (stale) Mirror 22: 0x5b06e633 (stale) Mirror 23: 0x5b06e633 (stale) Mirror 24: 0x5b06e633 (stale) Mirror 25: 0x5b06e633 (stale) Mirror 26: 0x5b06e633 (stale) Mirror 28: 0x5b06e633 (stale) CRC-32 checksum value for chunk [0, 0x3fd0a): Mirror 1: 0xf289a73f Mirror 29: 0xf289a73f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd0a: skipped 000721 1790392399.126954 mapread 0x031365 thru 0x03c3a7 (0xb043 bytes) 000722 1790392399.174074 mapread 0x01408a thru 0x01e791 (0xa708 bytes) 000724 1790392399.207693 punch from 0x006e9a to 0x0099d3 (0x2b3a bytes) 000725 1790392399.271392 trunc from 0x03fd0a to 0x024bf8 (0x1b111 bytes) 000728 1790392399.302874 mapread 0x000c56 thru 0x00a5d6 (0x9981 bytes) 000729 1790392399.311012 trunc from 0x024bf9 to 0x0226a6 (0x2552 bytes) 000730 1790392399.330566 mapread 0x01508f thru 0x0226a6 (0xd618 bytes) 000732 1790392399.344842 punch from 0x01bfcc to 0x0226a6 (0x66db bytes) 000733 1790392399.348509 mapread 0x000463 thru 0x0033e6 (0x2f84 bytes) 000734 1790392399.349355 mapread 0x00d17b thru 0x01b8fb (0xe781 bytes) 000735 1790392399.358749 trunc from 0x0226a7 to 0x02c2f2 (0x9c4c bytes) 000737 1790392399.378590 mapread 0x01f196 thru 0x027819 (0x8684 bytes) 000738 1790392399.385634 mapwrite 0x01962c thru 0x01bc97 (0x266c bytes) 000739 1790392399.428510 read 0x0100d4 thru 0x01601d (0x5f4a bytes) 000742 1790392399.430089 read 0x00528b thru 0x0136b0 (0xe426 bytes) 000743 1790392399.431459 write 0x022566 thru 0x0299cc (0x7467 bytes) 000745 1790392399.433627 mapread 0x022ff9 thru 0x024a87 (0x1a8f bytes) 000746 1790392399.434558 punch from 0x006f2d to 0x0089ad (0x1a81 bytes) 000748 1790392399.445269 trunc from 0x02c2f3 to 0x01f039 (0xd2b9 bytes) 000750 1790392399.459560 trunc from 0x01f03a to 0x02583b (0x6802 bytes) 000751 1790392399.473438 write 0x03b94b thru 0x03ffff (0x46b5 bytes) 000752 1790392399.475653 write 0x024dcf thru 0x025b8d (0xdbf bytes) 000753 1790392399.483230 mapwrite 0x007ac7 thru 0x007cb8 (0x1f2 bytes) 000754 1790392399.518401 mapwrite 0x03e56f thru 0x03ffff (0x1a91 bytes) 000755 1790392399.566938 read 0x00f28b thru 0x019d69 (0xaadf bytes) 000756 1790392399.567496 write 0x02c01b thru 0x0306e4 (0x46ca bytes) 000758 1790392399.570954 MIRROR_RESYNC 15 mirrors 000760 1790392406.576563 read 0x01dc63 thru 0x0265d7 (0x8975 bytes) 000763 1790392406.583685 mapwrite 0x036def thru 0x03ffff (0x9211 bytes) 000764 1790392406.800573 punch from 0x01ee24 to 0x0231ac (0x4389 bytes) 000765 1790392406.804241 write 0x016d1c thru 0x019a9d (0x2d82 bytes) 000767 1790392406.814800 mapwrite 0x03a832 thru 0x03ee22 (0x45f1 bytes) 000768 1790392406.876303 read 0x03157f thru 0x03ee23 (0xd8a5 bytes) 000770 1790392406.880710 mapread 0x023a9b thru 0x02d936 (0x9e9c bytes) 000772 1790392406.900691 write 0x0175c2 thru 0x023e42 (0xc881 bytes) 000774 1790392406.902368 mapwrite 0x003973 thru 0x0076e6 (0x3d74 bytes) 000775 1790392406.998293 mapread 0x004484 thru 0x00ae0b (0x6988 bytes) 000776 1790392406.999514 mapwrite 0x016e09 thru 0x0253fb (0xe5f3 bytes) 000777 1790392407.093635 mapwrite 0x01265d thru 0x01268a (0x2e bytes) 000778 1790392407.156622 punch from 0x02f70f to 0x037fd4 (0x88c6 bytes) 000779 1790392407.163408 trunc from 0x040000 to 0x01a063 (0x25f9c bytes) 000783 1790392407.201300 MIRROR_RESYNC 15 mirrors 000785 1790392410.195193 mapwrite 0x032e9a thru 0x03999f (0x6b06 bytes) 000787 1790392410.344628 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x399a0): Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x399a0): Mirror 8: 0x6b00aabc (stale) Mirror 13: 0x6b00aabc (stale) Mirror 14: 0x6b00aabc (stale) Mirror 18: 0x6b00aabc (stale) Mirror 19: 0x6b00aabc (stale) Mirror 20: 0x6b00aabc (stale) Mirror 21: 0x6b00aabc (stale) Mirror 22: 0x6b00aabc (stale) Mirror 23: 0x6b00aabc (stale) Mirror 24: 0x6b00aabc (stale) Mirror 25: 0x6b00aabc (stale) Mirror 26: 0x6b00aabc (stale) Mirror 28: 0x6b00aabc (stale) Mirror 29: 0x6b00aabc (stale) CRC-32 checksum value for chunk [0, 0x399a0): Mirror 1: 0x3ed22af2 Mirror 30: 0x3ed22af2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x399a0: skipped 000788 1790392411.401232 mapwrite 0x02aa4e thru 0x034c15 (0xa1c8 bytes) 000790 1790392411.563677 punch from 0x0148f9 to 0x0200b7 (0xb7bf bytes) 000794 1790392411.570359 mapread 0x029817 thru 0x0377b0 (0xdf9a bytes) 000796 1790392411.571787 mapwrite 0x036259 thru 0x03a0c8 (0x3e70 bytes) 000797 1790392411.652207 write 0x03315a thru 0x03ffff (0xcea6 bytes) 000798 1790392411.653673 read 0x022bf7 thru 0x02e0f5 (0xb4ff bytes) 000799 1790392411.661322 trunc from 0x040000 to 0x0183ec (0x27c13 bytes) 000801 1790392411.673335 mapwrite 0x00515c thru 0x01095f (0xb804 bytes) 000803 1790392411.732387 mapwrite 0x023204 thru 0x02585e (0x265b bytes) 000805 1790392411.800268 mapwrite 0x009c7a thru 0x01331f (0x96a6 bytes) 000806 1790392411.869937 mapwrite 0x02a085 thru 0x039196 (0xf112 bytes) 000808 1790392411.956615 write 0x02324d thru 0x02881f (0x55d3 bytes) 000809 1790392411.959744 mapread 0x00f8ef thru 0x018d06 (0x9418 bytes) 000810 1790392411.961320 read 0x033056 thru 0x0334dc (0x487 bytes) 000811 1790392411.966206 punch from 0x019f44 to 0x029411 (0xf4ce bytes) 000812 1790392411.980580 read 0x019f50 thru 0x0221a1 (0x8252 bytes) 000813 1790392411.990457 mapread 0x037e18 thru 0x039196 (0x137f bytes) 000814 1790392411.991457 punch from 0x02a1e3 to 0x02a2dc (0xfa bytes) 000815 1790392411.996410 punch from 0x003a52 to 0x004161 (0x710 bytes) 000816 1790392412.000669 mapread 0x00d335 thru 0x00fb1a (0x27e6 bytes) 000817 1790392412.002157 write 0x00fd70 thru 0x01abae (0xae3f bytes) 000819 1790392412.004944 write 0x01d65d thru 0x01f81c (0x21c0 bytes) 000820 1790392412.006017 trunc from 0x039197 to 0x00bf46 (0x2d250 bytes) 000821 1790392412.026071 trunc from 0x00bf47 to 0x010206 (0x42c0 bytes) 000823 1790392412.045009 mapwrite 0x021a0c thru 0x021b1b (0x110 bytes) 000824 1790392412.098842 mapread 0x01ad6d thru 0x021b1b (0x6daf bytes) 000825 1790392412.134411 punch from 0x010c37 to 0x01d828 (0xcbf2 bytes) 000826 1790392412.141816 MIRROR_SPLIT mirror 5 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21b1c): Mirror 8: 0xc500c896 (stale) Mirror 13: 0xc500c896 (stale) Mirror 14: 0xc500c896 (stale) Mirror 18: 0xc500c896 (stale) Mirror 20: 0xc500c896 (stale) Mirror 21: 0xc500c896 (stale) Mirror 22: 0xc500c896 (stale) Mirror 23: 0xc500c896 (stale) Mirror 24: 0xc500c896 (stale) Mirror 25: 0xc500c896 (stale) Mirror 26: 0xc500c896 (stale) Mirror 28: 0xc500c896 (stale) Mirror 29: 0xc500c896 (stale) Mirror 30: 0xc500c896 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21b1c): Mirror 8: 0xc500c896 (stale) Mirror 13: 0xc500c896 (stale) Mirror 14: 0xc500c896 (stale) Mirror 18: 0xc500c896 (stale) Mirror 20: 0xc500c896 (stale) Mirror 21: 0xc500c896 (stale) Mirror 22: 0xc500c896 (stale) Mirror 23: 0xc500c896 (stale) Mirror 24: 0xc500c896 (stale) Mirror 25: 0xc500c896 (stale) Mirror 26: 0xc500c896 (stale) Mirror 28: 0xc500c896 (stale) Mirror 29: 0xc500c896 (stale) Mirror 30: 0xc500c896 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21b1c: skipped 000828 1790392412.885384 mapread 0x00f7d0 thru 0x013800 (0x4031 bytes) 000829 1790392412.902459 trunc from 0x021b1c to 0x00104a (0x20ad1 bytes) 000830 1790392412.921699 trunc from 0x00104b to 0x03d208 (0x3c1be bytes) 000831 1790392412.942399 mapwrite 0x0111d4 thru 0x01c6c1 (0xb4ee bytes) 000832 1790392413.055376 MIRROR_SPLIT mirror 1 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d209): Mirror 13: 0x6d72d85e (stale) Mirror 14: 0x6d72d85e (stale) Mirror 18: 0x6d72d85e (stale) Mirror 20: 0x6d72d85e (stale) Mirror 21: 0x6d72d85e (stale) Mirror 22: 0x6d72d85e (stale) Mirror 23: 0x6d72d85e (stale) Mirror 24: 0x6d72d85e (stale) Mirror 25: 0x6d72d85e (stale) Mirror 26: 0x6d72d85e (stale) Mirror 28: 0x6d72d85e (stale) Mirror 29: 0x6d72d85e (stale) Mirror 30: 0x1a1daa64 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d209): Mirror 13: 0x6d72d85e (stale) Mirror 14: 0x6d72d85e (stale) Mirror 18: 0x6d72d85e (stale) Mirror 20: 0x6d72d85e (stale) Mirror 21: 0x6d72d85e (stale) Mirror 22: 0x6d72d85e (stale) Mirror 23: 0x6d72d85e (stale) Mirror 24: 0x6d72d85e (stale) Mirror 25: 0x6d72d85e (stale) Mirror 26: 0x6d72d85e (stale) Mirror 28: 0x6d72d85e (stale) Mirror 29: 0x6d72d85e (stale) Mirror 30: 0x1a1daa64 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d209: skipped 000833 1790392413.641135 write 0x018f2f thru 0x02416c (0xb23e bytes) 000835 1790392413.648830 write 0x016ef8 thru 0x026eb7 (0xffc0 bytes) 000836 1790392413.657321 mapread 0x016417 thru 0x02244d (0xc037 bytes) 000837 1790392413.658374 mapwrite 0x03a80e thru 0x03a9a4 (0x197 bytes) 000838 1790392413.707127 trunc from 0x03d209 to 0x00366e (0x39b9a bytes) 000839 1790392413.719510 mapwrite 0x035889 thru 0x03c364 (0x6adc bytes) 000840 1790392413.788221 trunc from 0x03c365 to 0x007c7b (0x346e9 bytes) 000842 1790392413.801297 mapread 0x006e6e thru 0x007c7b (0xe0e bytes) 000843 1790392413.814657 punch from 0x006a5e to 0x007c7b (0x121e bytes) 000847 1790392413.823195 trunc from 0x007c7c to 0x020489 (0x1880e bytes) 000848 1790392413.851612 punch from 0x00df1e to 0x00f4c4 (0x15a7 bytes) 000851 1790392413.855148 punch from 0x01703f to 0x01a4a1 (0x3463 bytes) 000853 1790392413.859726 punch from 0x01794a to 0x01de1e (0x64d5 bytes) 000854 1790392413.864660 trunc from 0x02048a to 0x012fe9 (0xd4a0 bytes) 000857 1790392413.877891 write 0x014b09 thru 0x01f25f (0xa757 bytes) 000858 1790392413.879694 mapwrite 0x037f36 thru 0x03e8db (0x69a6 bytes) 000859 1790392413.987492 punch from 0x0284b5 to 0x02d1e9 (0x4d35 bytes) 000860 1790392413.992896 read 0x033ebf thru 0x03e8db (0xaa1d bytes) 000863 1790392413.996411 trunc from 0x03e8dc to 0x010fbe (0x2d91d bytes) 000864 1790392414.019136 MIRROR_RESYNC 14 mirrors 000865 1790392416.179298 write 0x024718 thru 0x0312c8 (0xcbb1 bytes) 000866 1790392416.293511 trunc from 0x0312c9 to 0x03ad05 (0x9a3d bytes) 000867 1790392416.322454 read 0x017ab8 thru 0x018a9e (0xfe7 bytes) 000868 1790392416.329706 read 0x039d7a thru 0x03ad05 (0xf8c bytes) 000869 1790392416.342227 mapwrite 0x0246fa thru 0x02a726 (0x602d bytes) 000870 1790392416.371458 mapread 0x02932d thru 0x02ceea (0x3bbe bytes) 000871 1790392416.372186 trunc from 0x03ad06 to 0x000646 (0x3a6bf bytes) 000872 1790392416.386596 write 0x010e13 thru 0x01a388 (0x9576 bytes) 000873 1790392416.387993 write 0x000e74 thru 0x00fd43 (0xeed0 bytes) 000874 1790392416.391782 punch from 0x00d816 to 0x01a388 (0xcb73 bytes) 000875 1790392416.404012 read 0x01515a thru 0x01a388 (0x522f bytes) 000876 1790392416.412678 read 0x00bd99 thru 0x00c315 (0x57d bytes) 000878 1790392416.413748 mapwrite 0x02691e thru 0x03676e (0xfe51 bytes) 000880 1790392416.507457 MIRROR_SPLIT mirror 13 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3676f): Mirror 13: 0x3abb7cb8 (stale) Mirror 14: 0x3abb7cb8 (stale) Mirror 18: 0x3abb7cb8 (stale) Mirror 20: 0x3abb7cb8 (stale) Mirror 21: 0x3abb7cb8 (stale) Mirror 22: 0x3abb7cb8 (stale) Mirror 23: 0x3abb7cb8 (stale) Mirror 24: 0x3abb7cb8 (stale) Mirror 25: 0x3abb7cb8 (stale) Mirror 26: 0x3abb7cb8 (stale) Mirror 28: 0x3abb7cb8 (stale) Mirror 29: 0x3abb7cb8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3676f): Mirror 13: 0x3abb7cb8 (stale) Mirror 14: 0x3abb7cb8 (stale) Mirror 18: 0x3abb7cb8 (stale) Mirror 20: 0x3abb7cb8 (stale) Mirror 21: 0x3abb7cb8 (stale) Mirror 22: 0x3abb7cb8 (stale) Mirror 23: 0x3abb7cb8 (stale) Mirror 24: 0x3abb7cb8 (stale) Mirror 25: 0x3abb7cb8 (stale) Mirror 26: 0x3abb7cb8 (stale) Mirror 28: 0x3abb7cb8 (stale) Mirror 29: 0x3abb7cb8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3676f: skipped 000881 1790392417.090564 mapread 0x018386 thru 0x0202c4 (0x7f3f bytes) 000882 1790392417.105631 trunc from 0x03676f to 0x025e9a (0x108d4 bytes) 000883 1790392417.122765 read 0x01b970 thru 0x01f08f (0x3720 bytes) 000884 1790392417.127430 MIRROR_SPLIT mirror 8 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25e9b): Mirror 13: 0x80e1e4d2 (stale) Mirror 14: 0x80e1e4d2 (stale) Mirror 18: 0x80e1e4d2 (stale) Mirror 20: 0x80e1e4d2 (stale) Mirror 21: 0x80e1e4d2 (stale) Mirror 22: 0x80e1e4d2 (stale) Mirror 23: 0x80e1e4d2 (stale) Mirror 25: 0x80e1e4d2 (stale) Mirror 26: 0x80e1e4d2 (stale) Mirror 28: 0x80e1e4d2 (stale) Mirror 29: 0x80e1e4d2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25e9b): Mirror 13: 0x80e1e4d2 (stale) Mirror 14: 0x80e1e4d2 (stale) Mirror 18: 0x80e1e4d2 (stale) Mirror 20: 0x80e1e4d2 (stale) Mirror 21: 0x80e1e4d2 (stale) Mirror 22: 0x80e1e4d2 (stale) Mirror 23: 0x80e1e4d2 (stale) Mirror 25: 0x80e1e4d2 (stale) Mirror 26: 0x80e1e4d2 (stale) Mirror 28: 0x80e1e4d2 (stale) Mirror 29: 0x80e1e4d2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25e9b: skipped 000886 1790392417.626948 punch from 0x0220c9 to 0x025e9a (0x3dd2 bytes) 000887 1790392417.633048 read 0x018fcf thru 0x0224b4 (0x94e6 bytes) 000888 1790392417.639909 mapwrite 0x00e195 thru 0x01c5fb (0xe467 bytes) 000890 1790392417.711043 write 0x020464 thru 0x022ab9 (0x2656 bytes) 000891 1790392417.712350 read 0x024638 thru 0x025e9a (0x1863 bytes) 000892 1790392417.718391 mapread 0x01ec30 thru 0x025e9a (0x726b bytes) 000894 1790392417.719763 read 0x018468 thru 0x01d0f2 (0x4c8b bytes) 000896 1790392417.720960 trunc from 0x025e9b to 0x021065 (0x4e35 bytes) 000901 1790392417.737720 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21066): Mirror 13: 0xa745de72 (stale) Mirror 14: 0xa745de72 (stale) Mirror 20: 0xa745de72 (stale) Mirror 21: 0xa745de72 (stale) Mirror 22: 0xa745de72 (stale) Mirror 23: 0xa745de72 (stale) Mirror 25: 0xa745de72 (stale) Mirror 26: 0xa745de72 (stale) Mirror 28: 0xa745de72 (stale) Mirror 29: 0xa745de72 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21066): Mirror 13: 0xa745de72 (stale) Mirror 14: 0xa745de72 (stale) Mirror 20: 0xa745de72 (stale) Mirror 21: 0xa745de72 (stale) Mirror 22: 0xa745de72 (stale) Mirror 23: 0xa745de72 (stale) Mirror 25: 0xa745de72 (stale) Mirror 26: 0xa745de72 (stale) Mirror 28: 0xa745de72 (stale) Mirror 29: 0xa745de72 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21066: skipped 000902 1790392418.324648 write 0x01cf65 thru 0x0240f8 (0x7194 bytes) 000904 1790392418.337119 write 0x03df3a thru 0x03ffff (0x20c6 bytes) 000905 1790392418.338289 write 0x03a161 thru 0x03e5e6 (0x4486 bytes) 000907 1790392418.351815 MIRROR_RESYNC 11 mirrors 000908 1790392420.346360 mapwrite 0x008798 thru 0x00c6f6 (0x3f5f bytes) 000910 1790392420.523424 write 0x008c5a thru 0x00fb8d (0x6f34 bytes) 000911 1790392420.530042 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x3e7d5fbb (stale) Mirror 14: 0x3e7d5fbb (stale) Mirror 20: 0x3e7d5fbb (stale) Mirror 21: 0x3e7d5fbb (stale) Mirror 22: 0x3e7d5fbb (stale) Mirror 23: 0x3e7d5fbb (stale) Mirror 25: 0x3e7d5fbb (stale) Mirror 28: 0x3e7d5fbb (stale) Mirror 29: 0x3e7d5fbb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x3e7d5fbb (stale) Mirror 14: 0x3e7d5fbb (stale) Mirror 20: 0x3e7d5fbb (stale) Mirror 21: 0x3e7d5fbb (stale) Mirror 22: 0x3e7d5fbb (stale) Mirror 23: 0x3e7d5fbb (stale) Mirror 25: 0x3e7d5fbb (stale) Mirror 28: 0x3e7d5fbb (stale) Mirror 29: 0x3e7d5fbb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000912 1790392421.024258 write 0x020d85 thru 0x030664 (0xf8e0 bytes) 000913 1790392421.036697 read 0x03687e thru 0x03c05f (0x57e2 bytes) 000915 1790392421.048529 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x3e7d5fbb (stale) Mirror 14: 0x3e7d5fbb (stale) Mirror 20: 0x3e7d5fbb (stale) Mirror 21: 0x3e7d5fbb (stale) Mirror 22: 0x3e7d5fbb (stale) Mirror 23: 0x3e7d5fbb (stale) Mirror 25: 0x3e7d5fbb (stale) Mirror 28: 0x3e7d5fbb (stale) Mirror 29: 0x3e7d5fbb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc43d3ee Mirror 30: 0xc43d3ee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000917 1790392422.158902 mapwrite 0x02540b thru 0x029ac5 (0x46bb bytes) 000918 1790392422.411313 mapwrite 0x039dc4 thru 0x03ef34 (0x5171 bytes) 000919 1790392422.479433 MIRROR_RESYNC 11 mirrors 000920 1790392424.758176 punch from 0x03d216 to 0x03ffff (0x2dea bytes) 000921 1790392424.820036 write 0x02157c thru 0x02bed5 (0xa95a bytes) 000922 1790392424.833503 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xf1da3f23 (stale) Mirror 14: 0xf1da3f23 (stale) Mirror 20: 0xf1da3f23 (stale) Mirror 21: 0xf1da3f23 (stale) Mirror 22: 0xf1da3f23 (stale) Mirror 23: 0xf1da3f23 (stale) Mirror 25: 0xf1da3f23 (stale) Mirror 28: 0xf1da3f23 (stale) Mirror 29: 0xf1da3f23 (stale) Mirror 30: 0xf1da3f23 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6498a28f Mirror 31: 0x6498a28f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000923 1790392425.877019 write 0x019c7e thru 0x02913f (0xf4c2 bytes) 000924 1790392425.954384 write 0x003292 thru 0x00e485 (0xb1f4 bytes) 000925 1790392425.963963 trunc from 0x040000 to 0x037323 (0x8cdc bytes) 000926 1790392425.981831 mapread 0x025f84 thru 0x02f384 (0x9401 bytes) 000927 1790392425.997023 punch from 0x00e290 to 0x0170d8 (0x8e49 bytes) 000928 1790392426.012155 trunc from 0x037324 to 0x03ac3a (0x3917 bytes) 000931 1790392426.027599 punch from 0x023cbe to 0x03011a (0xc45d bytes) 000932 1790392426.033160 mapread 0x0206b8 thru 0x021a5c (0x13a5 bytes) 000934 1790392426.035241 mapread 0x00ca7d thru 0x012396 (0x591a bytes) 000935 1790392426.041648 punch from 0x00a793 to 0x00c3e3 (0x1c51 bytes) 000936 1790392426.054055 MIRROR_RESYNC 12 mirrors 000937 1790392430.229247 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fa:0x0]: Invalid argument (22) 000938 1790392430.553155 trunc from 0x03ac3b to 0x00bb6c (0x2f0ce bytes) 000939 1790392430.626767 mapwrite 0x0221f5 thru 0x0275b0 (0x53bc bytes) 000940 1790392430.712097 trunc from 0x0275b1 to 0x00429c (0x23314 bytes) 000941 1790392430.725078 read 0x000319 thru 0x00429c (0x3f84 bytes) 000943 1790392430.735946 read 0x0018f3 thru 0x00429c (0x29aa bytes) 000944 1790392430.740130 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x429d): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x429d): Mirror 13: 0x6cf5580 (stale) Mirror 14: 0x6cf5580 (stale) Mirror 20: 0x6cf5580 (stale) Mirror 21: 0x6cf5580 (stale) Mirror 22: 0x6cf5580 (stale) Mirror 23: 0x6cf5580 (stale) Mirror 25: 0x6cf5580 (stale) Mirror 28: 0x6cf5580 (stale) Mirror 29: 0x6cf5580 (stale) Mirror 31: 0x6cf5580 (stale) CRC-32 checksum value for chunk [0, 0x429d): Mirror 1: 0x6cf5580 Mirror 32: 0x6cf5580 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x429d: skipped 000945 1790392431.577221 mapread 0x002e74 thru 0x00429c (0x1429 bytes) 000946 1790392431.618814 punch from 0x003adf to 0x00429c (0x7be bytes) 000947 1790392431.698879 mapwrite 0x037a1c thru 0x03ffff (0x85e4 bytes) 000951 1790392431.749508 trunc from 0x040000 to 0x0061e7 (0x39e18 bytes) 000953 1790392431.768735 punch from 0x003f05 to 0x0061e7 (0x22e3 bytes) 000954 1790392431.777374 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x61e8): Mirror 13: 0xb35b7e32 (stale) Mirror 14: 0xb35b7e32 (stale) Mirror 20: 0xb35b7e32 (stale) Mirror 21: 0xb35b7e32 (stale) Mirror 22: 0xb35b7e32 (stale) Mirror 23: 0xb35b7e32 (stale) Mirror 28: 0xb35b7e32 (stale) Mirror 29: 0xb35b7e32 (stale) Mirror 31: 0xb35b7e32 (stale) Mirror 32: 0xa97add35 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x61e8): Mirror 13: 0xb35b7e32 (stale) Mirror 14: 0xb35b7e32 (stale) Mirror 20: 0xb35b7e32 (stale) Mirror 21: 0xb35b7e32 (stale) Mirror 22: 0xb35b7e32 (stale) Mirror 23: 0xb35b7e32 (stale) Mirror 28: 0xb35b7e32 (stale) Mirror 29: 0xb35b7e32 (stale) Mirror 31: 0xb35b7e32 (stale) Mirror 32: 0xa97add35 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x61e8: skipped 000955 1790392432.323225 mapread 0x003f05 thru 0x0061e7 (0x22e3 bytes) 000956 1790392432.340368 trunc from 0x0061e8 to 0x02ab48 (0x24961 bytes) 000958 1790392432.360286 MIRROR_RESYNC 11 mirrors 000960 1790392434.220563 write 0x0314cc thru 0x0378e1 (0x6416 bytes) 000961 1790392434.279101 mapread 0x0254b6 thru 0x02dc47 (0x8792 bytes) 000964 1790392434.297789 mapread 0x02c954 thru 0x02f948 (0x2ff5 bytes) 000965 1790392434.298682 trunc from 0x0378e2 to 0x02cba0 (0xad41 bytes) 000966 1790392434.314967 mapread 0x00f1bd thru 0x014c89 (0x5acd bytes) 000967 1790392434.323226 read 0x027331 thru 0x02c26e (0x4f3e bytes) 000968 1790392434.323895 punch from 0x026314 to 0x02907d (0x2d6a bytes) 000972 1790392434.330820 punch from 0x009431 to 0x00d955 (0x4525 bytes) 000974 1790392434.335193 read 0x00aece thru 0x0198f8 (0xea2b bytes) 000975 1790392434.346983 trunc from 0x02cba1 to 0x0347f9 (0x7c59 bytes) 000976 1790392434.361842 mapread 0x01c562 thru 0x02ba59 (0xf4f8 bytes) 000977 1790392434.363244 punch from 0x032214 to 0x0347f9 (0x25e6 bytes) 000979 1790392434.366924 write 0x018158 thru 0x024cb7 (0xcb60 bytes) 000984 1790392434.369453 trunc from 0x0347fa to 0x01b8c8 (0x18f31 bytes) 000985 1790392434.381807 MIRROR_RESYNC 11 mirrors 000986 1790392436.089102 MIRROR_RESYNC 11 mirrors 000987 1790392436.464706 trunc from 0x01b8c9 to 0x011317 (0xa5b1 bytes) 000988 1790392436.510926 MIRROR_RESYNC 11 mirrors 000989 1790392438.199149 mapwrite 0x01f5b5 thru 0x02d868 (0xe2b4 bytes) 000991 1790392438.340140 mapread 0x015152 thru 0x0245f2 (0xf4a1 bytes) 000992 1790392438.351307 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fa:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d869): Mirror 14: 0x7c28e64 (stale) Mirror 20: 0x7c28e64 (stale) Mirror 21: 0x7c28e64 (stale) Mirror 22: 0x7c28e64 (stale) Mirror 23: 0x7c28e64 (stale) Mirror 28: 0x7c28e64 (stale) Mirror 29: 0x7c28e64 (stale) Mirror 31: 0x7c28e64 (stale) Mirror 32: 0x7c28e64 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d869): Mirror 14: 0x7c28e64 (stale) Mirror 20: 0x7c28e64 (stale) Mirror 21: 0x7c28e64 (stale) Mirror 22: 0x7c28e64 (stale) Mirror 23: 0x7c28e64 (stale) Mirror 28: 0x7c28e64 (stale) Mirror 29: 0x7c28e64 (stale) Mirror 31: 0x7c28e64 (stale) Mirror 32: 0x7c28e64 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d869: skipped 000993 1790392438.763072 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2d869): Mirror 14: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d869): Mirror 14: 0x7c28e64 (stale) Mirror 20: 0x7c28e64 (stale) Mirror 21: 0x7c28e64 (stale) Mirror 22: 0x7c28e64 (stale) Mirror 23: 0x7c28e64 (stale) Mirror 28: 0x7c28e64 (stale) Mirror 29: 0x7c28e64 (stale) Mirror 31: 0x7c28e64 (stale) Mirror 32: 0x7c28e64 (stale) CRC-32 checksum value for chunk [0, 0x2d869): Mirror 1: 0xedf7face Mirror 33: 0xedf7face lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d869: skipped 000994 1790392439.757689 read 0x006a81 thru 0x00e8ab (0x7e2b bytes) 000995 1790392439.782909 read 0x01920e thru 0x0266e1 (0xd4d4 bytes) 000996 1790392439.827334 write 0x026c1e thru 0x02c25e (0x5641 bytes) 000997 1790392439.898437 read 0x003515 thru 0x00803b (0x4b27 bytes) 000999 1790392439.905318 mapwrite 0x001194 thru 0x010fd8 (0xfe45 bytes) 001000 1790392439.953651 write 0x003c2e thru 0x00627e (0x2651 bytes) All operations completed A-OK! PASS 70a (157s) == sanity-flr test 71: check mirror extend parent fid ==== 23:14:03 (1790392443) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (2s) == sanity-flr test 72: check append on FLR file ========== 23:14:06 (1790392446) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x342:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x366:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.306531 s, 6.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x342:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x343:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x366:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137229 s, 7.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x342:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x343:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x367:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x366:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x342:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x343:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x367:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x366:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x344:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (6s) == sanity-flr test 200a: stress test ===================== 23:14:12 (1790392452) Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre2 Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 8595872 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2786496 .. done Extending file size to 5005408 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9060960 .. Extending file size to 6978656 .. Extending file size to 7649440 .. Extending file size to 8477888 .. Extending file size to 8618784 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9088480 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8807840 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9124832 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9161824 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 57675 57677 57678 57679 57681 done 192.168.201.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg132-client.virtnet /mnt/lustre2 (opts:) 192.168.201.132@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg132-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (73s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:15:25 (1790392525) Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre2 Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6797184 .. Extending file size to 4641184 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8787328 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6877664 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9066400 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9149024 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9219520 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 59617 59618 59621 59622 59623 done 192.168.201.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg132-client.virtnet /mnt/lustre2 (opts:) 192.168.201.132@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg132-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 23:16:42 (1790392602) Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2944 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2944 2205440 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0499162 s, 42.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.241856 s, 8.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x2c968f77 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg132-client.virtnet /mnt/lustre2 (opts:) PASS 200c (12s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 23:16:55 (1790392615) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0628192 s, 33.4 MB/s PASS 202 (4s) == sanity-flr test 203: mirror file preserve mirror ID === 23:17:00 (1790392620) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x249:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x37d:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x37d:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x37e:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x359:0x0] } PASS 203 (5s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 23:17:05 (1790392625) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.105186 s, 19.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x37f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (5s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 23:17:11 (1790392631) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.184844 s, 11.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x380:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x381:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x380:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x381:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35d:0x0] } PASS 204b (5s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 23:17:16 (1790392636) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.145878 s, 14.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x382:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x382:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35f:0x0] } PASS 204c (27s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 23:17:43 (1790392663) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg132-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440725 s, 23.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x383:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x361:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x360:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x383:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x361:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x360:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x384:0x0] } PASS 204d (45s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 23:18:28 (1790392708) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg132-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x362:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x363:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0724178 s, 29.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x362:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x363:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x385:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x362:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x363:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x385:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (50s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 23:19:18 (1790392758) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg132-server mds-ost sync done. Creating new pool test_204f oleg132-server: Warning, pool lustre.test_204f not found Adding targets to pool oleg132-server: Warning, OST lustre-OST0000_UUID not found in pool lustre.test_204f oleg132-server: Warning, OST lustre-OST0001_UUID not found in pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.161283 s, 13.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x364:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x386:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x364:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x386:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg132-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg132-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg132-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (65s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 23:20:23 (1790392823) PASS 205a (4s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 23:20:27 (1790392827) PASS 205b (4s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 23:20:32 (1790392832) oleg132-server: Warning, pool lustre.test_206 not found Adding targets to pool oleg132-server: Warning, OST lustre-OST0000_UUID not found in pool lustre.test_206 oleg132-server: Warning, OST lustre-OST0001_UUID not found in pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg132-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg132-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg132-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (21s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 23:20:52 (1790392852) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318344 s, 32.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177724 s, 59.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x5445b0f5 Mirror 3: 0x5445b0f5 PASS 207 (6s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 23:20:58 (1790392858) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 23:21:00 (1790392860) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 23:21:02 (1790392862) Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0690916 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1530000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xa60000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xa60000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xa60000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xda0000 Read this (4 bytes): BBBB 192.168.201.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg132-client.virtnet /mnt/lustre2 (opts:) PASS 209a (22s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 23:21:24 (1790392884) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0116113 s, 706 kB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 23:21:27 (1790392887) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311301 s, 33.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x26e lmm_fid: [0x200000404:0x26e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x26e lmm_fid: [0x200000404:0x26e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x390:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834542 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-25 23:21:28.000000000 -0400 Modify: 2026-09-25 23:21:28.000000000 -0400 Change: 2026-09-25 23:21:29.000000000 -0400 Birth: 2026-09-25 23:21:29.000000000 -0400 PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 23:21:31 (1790392891) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 23:21:33 (1790392893) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.252276 s, 162 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0269916 s, 152 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 25 23:21 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x270:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 25 23:21 /mnt/lustre/f211.sanity-flr PASS 211 (5s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 23:21:38 (1790392898) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (4s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 23:21:42 (1790392902) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (4s) == sanity-flr test complete, duration 1668 sec =========== 23:21:46 (1790392906) === sanity-flr: start cleanup 23:21:47 (1790392907) === === sanity-flr: finish cleanup 23:21:50 (1790392910) ===