-----============= acceptance-small: sanity-flr ============----- Sun Sep 6 05:14:03 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg329-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 05:14:22 (1788686062) === oleg329-client.virtnet: executing check_config_client /mnt/lustre oleg329-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg329-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8bcc10f52000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8bcc10f52000.idle_timeout=debug disable quota as required oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 05:14:57 (1788686097) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8245] == sanity-flr test 0a: lfs mirror create with -N option == 05:14:59 (1788686099) PASS 0a (13s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 05:15:12 (1788686112) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 05:15:14 (1788686114) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 05:15:17 (1788686117) PASS 0d (17s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 05:15:34 (1788686134) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 05:15:37 (1788686137) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 05:15:39 (1788686139) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8bcc10f52000.stats=clear osc.lustre-OST0001-osc-ffff8bcc10f52000.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 05:15:45 (1788686145) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (8s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 05:15:53 (1788686153) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 05:16:00 (1788686160) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 05:16:03 (1788686163) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (6s) == sanity-flr test 1: create components with setstripe options ========================================================== 05:16:10 (1788686170) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (7s) == sanity-flr test 2: create components from existing files ========================================================== 05:16:17 (1788686177) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 05:16:23 (1788686183) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 05:16:30 (1788686190) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 05:16:37 (1788686197) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 05:16:47 (1788686207) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 05:16:53 (1788686213) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.250188 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854999 s, 12.3 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 05:17:01 (1788686221) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.115379 s, 45.4 MB/s PASS 22 (6s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 05:17:08 (1788686228) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 05:17:18 (1788686238) Stopping /mnt/lustre-ost1 (opts:) on oleg329-server oleg329-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8bcc10f52000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8bcc10f52000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg329-server: oleg329-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg329-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg329-server oleg329-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8bcc10f52000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8bcc10f52000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg329-server: oleg329-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg329-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (59s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 05:18:17 (1788686297) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.289174 s, 14.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg329-server oleg329-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8bcc10f52000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8bcc10f52000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg329-server: oleg329-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg329-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (36s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 05:18:53 (1788686333) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg329-server oleg329-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8bcc10f52000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8bcc10f52000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg329-server: oleg329-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg329-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg329-server oleg329-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8bcc10f52000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8bcc10f52000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg329-server: oleg329-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg329-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (66s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 05:19:59 (1788686399) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.46744 s, 21.2 MB/s -rw-r--r-- 1 root root 52428800 Sep 6 05:20 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.810s user 0m0.001s sys 0m0.170s Stopping /mnt/lustre-ost1 (opts:) on oleg329-server oleg329-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8bcc10f52000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8bcc10f52000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.828s user 0m0.001s sys 0m0.176s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg329-server: oleg329-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg329-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg329-server oleg329-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8bcc10f52000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8bcc10f52000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.055s user 0m0.001s sys 0m0.173s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg329-server: oleg329-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg329-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg329-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8bcc10f52000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8bcc10f52000.ost_server_uuid in FULL state after 0 sec PASS 33b (78s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 05:21:17 (1788686477) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 05:21:20 (1788686480) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 05:21:22 (1788686482) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 05:21:25 (1788686485) PASS 35 (5s) == sanity-flr test 36a: write to mirrored files ========== 05:21:30 (1788686490) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.523882 s, 24.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (10s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 05:21:40 (1788686500) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346904 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324787 s, 32.3 MB/s PASS 36b (5s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 05:21:45 (1788686505) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 05:21:52 (1788686512) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0010673 s, 959 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00080377 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000765568 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000853949 s, 1.2 MB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 05:22:02 (1788686522) create test files with size 15746942 .. 938c050f9e67c4c7aa098d421ecf3380 - 80d36cad392dbdb525f4bb6d186ec938 - 2be79ceff2c7715642822783b51b549e - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg329-client.virtnet /mnt/lustre (opts:) Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre PASS 37 (17s) == sanity-flr test 38: resync ============================ 05:22:19 (1788686539) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 05:22:31 (1788686551) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 05:22:38 (1788686558) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122723 s, 8.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327825 s, 32.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xba:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 05:22:48 (1788686568) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.127003 s, 16.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.388689 s, 10.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.117819 s, 17.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (13s) == sanity-flr test 42: lfs mirror verify ================= 05:23:01 (1788686581) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 05:23:04 (1788686584) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 05:23:07 (1788686587) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472252 s, 22.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 44a: lfs mirror split check =========== 05:23:13 (1788686593) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.256004 s, 12.3 MB/s PASS 44a (8s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 05:23:21 (1788686601) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc4:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc4:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 05:23:29 (1788686609) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.240911 s, 43.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg329-client.virtnet /mnt/lustre (opts:) Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 05:23:39 (1788686619) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8bcc09f72800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.254542 s, 41.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg329-client.virtnet /mnt/lustre (opts:) Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8bcc11653000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 05:23:52 (1788686632) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 05:24:31 (1788686671) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (33s) == sanity-flr test 47: Verify mirror obj alloc =========== 05:25:05 (1788686705) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 05:25:08 (1788686708) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.201058 s, 15.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.125538 s, 25.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x26e30c31 (nosync,stale) Mirror 2: 0x26e30c31 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 2e9fac5df7231742ebc6b4bae6f306e4 - mirror 1 checksum: 602f15f39cc64996e10cee721e79a527 - mirror 2 checksum: 2e9fac5df7231742ebc6b4bae6f306e4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-06T05:25:10-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-06T05:25:10-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 2e9fac5df7231742ebc6b4bae6f306e4 - mirror 1 checksum: 602f15f39cc64996e10cee721e79a527 - mirror 2 checksum: 602f15f39cc64996e10cee721e79a527 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-06T05:25:10-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-06T05:25:10-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } PASS 48 (9s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 05:25:16 (1788686716) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335948 s, 31.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 05:25:24 (1788686724) Failing mds1 on oleg329-server Stopping /mnt/lustre-mds1 (opts:) on oleg329-server 05:25:36 (1788686736) shut down facet: mds1 facet_host: oleg329-server facet_failover_host: oleg329-server Failover mds1 to oleg329-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-MDT0000 05:25:57 (1788686757) targets are mounted 05:25:57 (1788686757) facet_failover done oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438393 s, 23.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } PASS 50A (50s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 05:26:14 (1788686774) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0405742 s, 505 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0169369 s, 1.2 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 05:26:24 (1788686784) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0339342 s, 604 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0114783 s, 1.8 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 05:26:34 (1788686794) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00796554 s, 2.1 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 05:26:41 (1788686801) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0494447 s, 414 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (7s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 05:26:48 (1788686808) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00468279 s, 875 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0723469 s, 56.6 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 05:26:55 (1788686815) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 05:27:17 (1788686837) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 05:27:47 (1788686867) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 05:28:10 (1788686890) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 05:28:35 (1788686915) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 05:28:41 (1788686921) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57555 mirror split pid 57557 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57555 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57557 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 05:29:48 (1788686988) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 6990 skipping zero length punch hole skipping zero length zero range truncating to largest ever: 0x126b6 000003 1788686990.392716 trunc from 00000000 to 0x0126b5 (0x126b6 bytes) 000004 1788686990.409478 punch from 0x010cf9 to 0x0126b5 (0x19bd bytes) fallocating to largest ever: 0x32fed 000005 1788686990.414213 fallocate 0x024979 thru 0x032fec (0xe674 bytes) 000006 1788686990.418428 trunc from 0x008818 to 0x017c22 (0xf40b bytes) 000007 1788686990.423275 read 0x02c2ae thru 0x031135 (0x4e88 bytes) 000008 1788686990.431709 read 0x01c751 thru 0x02a4ba (0xdd6a bytes) 000009 1788686990.445858 mapread 0x014bcc thru 0x0169c5 (0x1dfa bytes) 000012 1788686990.452510 MIRROR_EXTEND to 2 mirrors 000013 1788686990.929139 read 0x030711 thru 0x032fec (0x28dc bytes) 000014 1788686990.935469 mapread 0x00d72e thru 0x00fba6 (0x2479 bytes) 000016 1788686990.958074 fallocate 0x011ebf thru 0x01aa6f (0x8bb1 bytes) 000017 1788686991.012774 trunc from 0x032fed to 0x018748 (0x1a8a4 bytes) 000018 1788686991.058860 read 0x002b53 thru 0x007af7 (0x4fa5 bytes) 000019 1788686991.074896 mapread 0x00805b thru 0x016d98 (0xed3e bytes) 000020 1788686991.075953 mapwrite 0x01acf6 thru 0x024594 (0x989f bytes) 000021 1788686991.115998 mapwrite 0x01176e thru 0x01f842 (0xe0d5 bytes) 000022 1788686991.154183 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x24595): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24595): Mirror 2: 0xda582eef (stale) CRC-32 checksum value for chunk [0, 0x24595): Mirror 1: 0xf75d97e7 Mirror 3: 0xf75d97e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24595: skipped 000023 1788686991.927091 punch from 0x000e32 to 0x002dfb (0x1fca bytes) 000024 1788686991.978038 trunc from 0x010dce to 0x01e58c (0xd7bf bytes) 000025 1788686991.985523 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x24595): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24595): Mirror 2: 0xda582eef (stale) Mirror 3: 0xf75d97e7 (stale) CRC-32 checksum value for chunk [0, 0x24595): Mirror 1: 0xc40e3f Mirror 4: 0xc40e3f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24595: skipped 000026 1788686992.684853 read 0x01e4b9 thru 0x0237be (0x5306 bytes) 000027 1788686992.696208 punch from 0x01a242 to 0x0204ae (0x626d bytes) 000028 1788686992.733165 trunc from 0x024595 to 0x01885c (0xbd38 bytes) 000029 1788686992.764541 fallocate 0x0060f2 thru 0x009a44 (0x3953 bytes) 000030 1788686992.771624 fallocate 0x027722 thru 0x02f731 (0x8010 bytes) 000031 1788686992.775603 write 0x029e6b thru 0x02a865 (0x9fb bytes) 000032 1788686992.789658 mapwrite 0x021937 thru 0x02e5b5 (0xcc7f bytes) 000033 1788686992.831491 fallocate 0x00e746 thru 0x01ce58 (0xe713 bytes) 000035 1788686992.841827 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2f732): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f732): Mirror 2: 0xf457184c (stale) Mirror 3: 0x91fdbc62 (stale) Mirror 4: 0xf8d07e6a (stale) CRC-32 checksum value for chunk [0, 0x2f732): Mirror 1: 0x7ae718cb Mirror 5: 0x7ae718cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f732: skipped 000036 1788686993.897733 punch from 0x011271 to 0x013272 (0x2002 bytes) 000037 1788686993.952012 punch from 0x018d2b to 0x019e60 (0x1136 bytes) 000038 1788686993.955493 trunc from 0x01824a to 0x0188f3 (0x6aa bytes) 000039 1788686993.972567 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f732): Mirror 2: 0xf457184c (stale) Mirror 3: 0x91fdbc62 (stale) Mirror 5: 0x7ae718cb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f732): Mirror 2: 0xf457184c (stale) Mirror 3: 0x91fdbc62 (stale) Mirror 5: 0x7ae718cb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f732: skipped 000040 1788686994.399253 trunc from 0x00d380 to 0x017f52 (0xabd3 bytes) 000041 1788686994.404312 mapread 0x016ded thru 0x0187af (0x19c3 bytes) 000043 1788686994.431833 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f732): Mirror 2: 0xf457184c (stale) Mirror 3: 0x91fdbc62 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f732): Mirror 2: 0xf457184c (stale) Mirror 3: 0x91fdbc62 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f732: skipped 000044 1788686994.711755 trunc from 0x02f732 to 0x001e65 (0x2d8cc bytes) 000045 1788686994.737106 mapread 0x00083b thru 0x001e65 (0x162b bytes) 000046 1788686994.749460 punch from 0x0011c9 to 0x001e65 (0xc9d bytes) 000047 1788686994.753854 mapwrite 0x03b1f9 thru 0x03ffff (0x4e07 bytes) 000048 1788686994.802560 trunc from 0x01b529 to 0x021881 (0x6359 bytes) 000049 1788686994.806177 punch from 0x001c3f to 0x007b6c (0x5f2e bytes) 000050 1788686994.812549 trunc from 0x005de5 to 0x007879 (0x1a95 bytes) 000051 1788686994.818866 write 0x03642e thru 0x03ffff (0x9bd2 bytes) 000052 1788686994.821762 write 0x009c3c thru 0x00cca9 (0x306e bytes) 000053 1788686994.831597 trunc from 0x040000 to 0x004f91 (0x3b06e bytes) 000054 1788686994.848845 write 0x01c19c thru 0x01c609 (0x46e bytes) 000055 1788686994.849798 trunc from 0x01c60a to 0x02bbbf (0xf5b6 bytes) 000056 1788686994.868955 write 0x01e942 thru 0x02d16c (0xe82b bytes) fallocating to largest ever: 0x337b6 000057 1788686994.878001 fallocate 0x0237dc thru 0x0337b5 (0xffda bytes) 000058 1788686994.886297 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x337b6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x337b6): Mirror 2: 0x103b14a6 (stale) Mirror 3: 0x8a61aeb3 (stale) CRC-32 checksum value for chunk [0, 0x337b6): Mirror 1: 0x4daa15bf Mirror 4: 0x4daa15bf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x337b6: skipped 000059 1788686996.022174 read 0x00dfb7 thru 0x013050 (0x509a bytes) 000060 1788686996.048055 read 0x021136 thru 0x02908e (0x7f59 bytes) 000062 1788686996.073774 trunc from 0x0337b6 to 0x02e7fe (0x4fb7 bytes) 000063 1788686996.167346 trunc from 0x01b384 to 0x02060a (0x5287 bytes) 000064 1788686996.174155 trunc from 0x01f6c6 to 0x022cfa (0x3635 bytes) 000065 1788686996.180290 trunc from 0x00ba4b to 0x019586 (0xdb3c bytes) 000066 1788686996.186882 read 0x00fee0 thru 0x01b7af (0xb8d0 bytes) 000067 1788686996.208424 read 0x028bb4 thru 0x02e7fe (0x5c4b bytes) 000068 1788686996.209008 punch from 0x01c921 to 0x025922 (0x9002 bytes) 000069 1788686996.214918 write 0x0171d0 thru 0x01d391 (0x61c2 bytes) 000070 1788686996.223495 mapread 0x00b050 thru 0x01038c (0x533d bytes) fallocating to largest ever: 0x3fb67 000071 1788686996.225006 fallocate 0x030591 thru 0x03fb66 (0xf5d6 bytes) 000072 1788686996.233098 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e7ff): Mirror 2: 0xec86d9d1 (stale) Mirror 4: 0x59d38e82 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e7ff): Mirror 2: 0xec86d9d1 (stale) Mirror 4: 0x59d38e82 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e7ff: skipped 000073 1788686996.698424 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e7ff): Mirror 4: 0x59d38e82 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e7ff): Mirror 4: 0x59d38e82 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e7ff: skipped 000074 1788686997.054873 read 0x000874 thru 0x00a39b (0x9b28 bytes) 000075 1788686997.071579 trunc from 0x022269 to 0x02c1f4 (0x9f8c bytes) 000076 1788686997.076895 trunc from 0x02e7ff to 0x024fc6 (0x9838 bytes) 000077 1788686997.098623 punch from 0x0048ec to 0x00cd80 (0x8495 bytes) 000078 1788686997.104222 mapwrite 0x02b748 thru 0x030756 (0x500f bytes) 000079 1788686997.150584 punch from 0x02cb43 to 0x02eff5 (0x24b3 bytes) 000080 1788686997.163247 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) 000081 1788686997.389889 write 0x0335ed thru 0x03937b (0x5d8f bytes) 000082 1788686997.398052 fallocate 0x01cdb5 thru 0x0276a9 (0xa8f5 bytes) 000083 1788686997.404959 punch from 0x00bfd7 to 0x013e24 (0x7e4e bytes) 000084 1788686997.414556 write 0x02558c thru 0x030739 (0xb1ae bytes) 000085 1788686997.429239 punch from 0x0283f7 to 0x02ffa9 (0x7bb3 bytes) 000086 1788686997.452828 trunc from 0x03937c to 0x00dd88 (0x2b5f3 bytes) 000087 1788686997.479862 mapwrite 0x036208 thru 0x03b479 (0x5272 bytes) 000090 1788686997.583108 punch from 0x008511 to 0x00aa26 (0x2516 bytes) 000091 1788686997.591207 fallocate 0x02cf1e thru 0x03c8b8 (0xf99b bytes) 000092 1788686997.602922 write 0x0231c8 thru 0x030e59 (0xdc92 bytes) 000093 1788686997.618083 mapwrite 0x01cb0b thru 0x0283e5 (0xb8db bytes) 000094 1788686997.663302 read 0x0214c7 thru 0x0303b5 (0xeeef bytes) 000096 1788686997.665308 write 0x0035a9 thru 0x004148 (0xba0 bytes) 000097 1788686997.685134 punch from 0x032006 to 0x035d00 (0x3cfb bytes) 000099 1788686997.698042 write 0x008a8e thru 0x01753f (0xeab2 bytes) fallocating to largest ever: 0x40000 000100 1788686997.707213 fallocate 0x037be1 thru 0x03ffff (0x841f bytes) 000101 1788686997.712366 mapwrite 0x01bd1d thru 0x026514 (0xa7f8 bytes) 000102 1788686997.732221 trunc from 0x029cd1 to 0x037e47 (0xe177 bytes) 000103 1788686997.738262 fallocate 0x00e4ac thru 0x012109 (0x3c5e bytes) 000104 1788686997.743001 punch from 0x016773 to 0x019912 (0x31a0 bytes) 000106 1788686997.749584 read 0x030119 thru 0x0345c7 (0x44af bytes) 000107 1788686997.760789 mapwrite 0x0111af thru 0x0177d4 (0x6626 bytes) 000108 1788686997.778240 write 0x008dd3 thru 0x015c0d (0xce3b bytes) 000109 1788686997.780265 mapwrite 0x03d87d thru 0x03efcc (0x1750 bytes) 000110 1788686997.799644 fallocate 0x019eb4 thru 0x02997a (0xfac7 bytes) 000111 1788686997.804295 mapwrite 0x014f15 thru 0x01832a (0x3416 bytes) 000112 1788686997.818541 trunc from 0x040000 to 0x030f4b (0xf0b4 bytes) 000114 1788686997.837848 mapread 0x017df8 thru 0x021e72 (0xa07b bytes) 000115 1788686997.838805 write 0x02cd40 thru 0x02d438 (0x6f9 bytes) 000116 1788686997.841253 trunc from 0x00231b to 0x008b64 (0x684a bytes) 000117 1788686997.864997 mapread 0x00ecf3 thru 0x013db9 (0x50c7 bytes) 000118 1788686997.866180 trunc from 0x026741 to 0x02f715 (0x8fd5 bytes) 000120 1788686997.881505 trunc from 0x030f4c to 0x015352 (0x1bbf9 bytes) 000121 1788686997.914195 fallocate 0x024d0a thru 0x025ec4 (0x11bb bytes) 000122 1788686997.926125 punch from 0x011a2f to 0x013971 (0x1f43 bytes) 000123 1788686997.935018 punch from 0x023e3a to 0x025adf (0x1ca6 bytes) 000124 1788686997.939262 mapwrite 0x03c84a thru 0x03ffff (0x37b6 bytes) 000125 1788686997.973847 fallocate 0x027a17 thru 0x037060 (0xf64a bytes) 000126 1788686997.978112 mapread 0x01beaf thru 0x01d49f (0x15f1 bytes) 000127 1788686998.000889 trunc from 0x040000 to 0x0046f7 (0x3b908 bytes) 000128 1788686998.030849 fallocate 0x01cc79 thru 0x01e21d (0x15a5 bytes) 000129 1788686998.042139 mapread 0x014a1d thru 0x019250 (0x4834 bytes) 000130 1788686998.070957 fallocate 0x026dfe thru 0x028b64 (0x1d67 bytes) 000131 1788686998.076189 punch from 0x01c1de to 0x01e21d (0x2040 bytes) 000132 1788686998.081126 trunc from 0x01e21e to 0x00c3d8 (0x11e45 bytes) 000133 1788686998.094056 write 0x00b6cc thru 0x0189d1 (0xd306 bytes) 000134 1788686998.100117 mapread 0x00c79b thru 0x01238a (0x5bf0 bytes) 000136 1788686998.102213 mapwrite 0x03a086 thru 0x03ab20 (0xa9b bytes) 000138 1788686998.179914 mapread 0x00f7f0 thru 0x01d2d1 (0xdae2 bytes) 000139 1788686998.208409 fallocate 0x03b642 thru 0x03ffff (0x49be bytes) 000140 1788686998.216611 read 0x03d874 thru 0x03ffff (0x278c bytes) 000141 1788686998.255222 trunc from 0x01e250 to 0x023350 (0x5101 bytes) 000142 1788686998.264422 mapwrite 0x001f9e thru 0x011896 (0xf8f9 bytes) 000143 1788686998.325609 write 0x005605 thru 0x00651e (0xf1a bytes) 000144 1788686998.336920 MIRROR_EXTEND to 2 mirrors 000145 1788686999.142500 mapread 0x0196e9 thru 0x0254cf (0xbde7 bytes) 000146 1788686999.177336 trunc from 0x00a62f to 0x015596 (0xaf68 bytes) 000147 1788686999.236532 fallocate 0x0121b2 thru 0x0203f0 (0xe23f bytes) 000148 1788686999.241279 read 0x02b534 thru 0x032144 (0x6c11 bytes) 000149 1788686999.263039 mapread 0x03ca3b thru 0x03ffff (0x35c5 bytes) 000150 1788686999.283258 trunc from 0x02738b to 0x0353d1 (0xe047 bytes) 000151 1788686999.291803 mapwrite 0x0163df thru 0x025d15 (0xf937 bytes) 000152 1788686999.339690 trunc from 0x026f9d to 0x02ac8c (0x3cf0 bytes) 000153 1788686999.357913 trunc from 0x014ff6 to 0x01a972 (0x597d bytes) 000154 1788686999.364283 mapread 0x0351d5 thru 0x03ffff (0xae2b bytes) 000155 1788686999.383994 trunc from 0x025787 to 0x02c8af (0x7129 bytes) 000157 1788686999.394891 read 0x003760 thru 0x005ceb (0x258c bytes) 000158 1788686999.419165 write 0x004167 thru 0x010fe6 (0xce80 bytes) 000159 1788686999.425552 mapread 0x015f1c thru 0x022ad5 (0xcbba bytes) 000160 1788686999.428809 fallocate 0x01bb37 thru 0x0219ec (0x5eb6 bytes) 000161 1788686999.437296 trunc from 0x040000 to 0x032189 (0xde76 bytes) 000162 1788686999.488273 trunc from 0x03218a to 0x028027 (0xa162 bytes) 000164 1788686999.504989 mapread 0x00dfa5 thru 0x0157b7 (0x7813 bytes) 000165 1788686999.506466 write 0x02bf2a thru 0x0379be (0xba95 bytes) 000166 1788686999.510776 trunc from 0x00383e to 0x00d6f6 (0x9eb9 bytes) 000168 1788686999.528992 MIRROR_RESYNC 2 mirrors 000169 1788687000.470206 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) 000170 1788687000.618604 read 0x0205ed thru 0x0216de (0x10f2 bytes) 000172 1788687000.624064 trunc from 0x02c21c to 0x031b60 (0x5945 bytes) 000175 1788687000.647546 trunc from 0x0379bf to 0x01272d (0x25291 bytes) 000176 1788687000.665275 fallocate 0x018270 thru 0x025f58 (0xdce9 bytes) 000177 1788687000.671796 trunc from 0x00dccc to 0x01272d (0x4a62 bytes) 000178 1788687000.676162 trunc from 0x01272e to 0x01685a (0x412d bytes) 000179 1788687000.691109 trunc from 0x00657d to 0x010de5 (0xa869 bytes) 000180 1788687000.696132 mapread 0x01020b thru 0x01064b (0x441 bytes) 000181 1788687000.702092 trunc from 0x0027a8 to 0x00d007 (0xa860 bytes) 000182 1788687000.707223 write 0x0215ff thru 0x02f328 (0xdd2a bytes) 000183 1788687000.709456 fallocate 0x00962b thru 0x018297 (0xec6d bytes) 000184 1788687000.714442 trunc from 0x02194c to 0x0237d8 (0x1e8d bytes) 000185 1788687000.731946 MIRROR_EXTEND to 2 mirrors 000186 1788687001.507638 mapwrite 0x00ab6a thru 0x01a22f (0xf6c6 bytes) 000187 1788687001.609545 write 0x017f37 thru 0x02133b (0x9405 bytes) 000188 1788687001.618446 trunc from 0x02b230 to 0x02ba9b (0x86c bytes) 000191 1788687001.626001 read 0x01459c thru 0x01fda6 (0xb80b bytes) 000192 1788687001.627132 mapread 0x02b75a thru 0x02c8f1 (0x1198 bytes) 000193 1788687001.637179 mapread 0x006d13 thru 0x00a73e (0x3a2c bytes) 000194 1788687001.638715 punch from 0x0219ca to 0x02f328 (0xd95f bytes) 000197 1788687001.656327 mapwrite 0x01ad9e thru 0x028dac (0xe00f bytes) 000198 1788687001.682270 write 0x023260 thru 0x02c933 (0x96d4 bytes) 000199 1788687001.689874 write 0x012dfd thru 0x01704d (0x4251 bytes) 000200 1788687001.691218 punch from 0x022010 to 0x0250dc (0x30cd bytes) 000201 1788687001.704119 read 0x02ae55 thru 0x02f328 (0x44d4 bytes) 000202 1788687001.712260 read 0x018d89 thru 0x02411c (0xb394 bytes) 000203 1788687001.713041 mapread 0x008763 thru 0x01136e (0x8c0c bytes) 000204 1788687001.717271 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) 000205 1788687001.854132 write 0x0358d7 thru 0x0368e6 (0x1010 bytes) 000206 1788687001.858884 trunc from 0x024ca9 to 0x02b7fe (0x6b56 bytes) 000207 1788687001.863808 mapwrite 0x01a015 thru 0x022852 (0x883e bytes) 000208 1788687001.897736 trunc from 0x0295d1 to 0x0310ce (0x7afe bytes) 000209 1788687001.901859 write 0x01ac9e thru 0x01eba8 (0x3f0b bytes) 000210 1788687001.902861 read 0x015fbf thru 0x01b11f (0x5161 bytes) 000211 1788687001.903408 mapread 0x02f08e thru 0x0368e6 (0x7859 bytes) 000212 1788687001.912436 mapwrite 0x01a7f9 thru 0x01b40f (0xc17 bytes) 000213 1788687001.927642 mapread 0x0353d2 thru 0x0368e6 (0x1515 bytes) 000214 1788687001.931216 mapwrite 0x028eaf thru 0x02ad3f (0x1e91 bytes) 000215 1788687001.946141 write 0x01bc76 thru 0x0248c8 (0x8c53 bytes) 000216 1788687001.948296 trunc from 0x0368e7 to 0x02538a (0x1155c bytes) 000217 1788687001.963334 fallocate 0x010964 thru 0x010f90 (0x62d bytes) 000218 1788687001.968298 trunc from 0x02538b to 0x006302 (0x1f088 bytes) 000221 1788687001.982813 read 0x00569e thru 0x006302 (0xc65 bytes) 000222 1788687002.006956 mapwrite 0x012aa8 thru 0x0226b9 (0xfc12 bytes) 000223 1788687002.074219 read 0x01ce52 thru 0x0226b9 (0x5868 bytes) 000224 1788687002.074846 read 0x01dc6d thru 0x0226b9 (0x4a4d bytes) 000225 1788687002.075376 trunc from 0x0226ba to 0x03c3db (0x19d22 bytes) 000227 1788687002.105926 write 0x03cf3a thru 0x03ffff (0x30c6 bytes) 000228 1788687002.119413 read 0x03defa thru 0x03f992 (0x1a99 bytes) 000229 1788687002.120015 fallocate 0x0006fa thru 0x00662c (0x5f33 bytes) 000231 1788687002.126630 fallocate 0x01b10f thru 0x022589 (0x747b bytes) 000232 1788687002.136482 trunc from 0x004762 to 0x00e514 (0x9db3 bytes) 000233 1788687002.148266 write 0x01e925 thru 0x02776b (0x8e47 bytes) 000234 1788687002.150257 write 0x0342b2 thru 0x03c17d (0x7ecc bytes) 000235 1788687002.155498 trunc from 0x040000 to 0x018e94 (0x2716b bytes) 000236 1788687002.176867 trunc from 0x007bc0 to 0x00f6b7 (0x7af8 bytes) 000237 1788687002.183637 trunc from 0x018e95 to 0x0398e4 (0x20a50 bytes) 000238 1788687002.203739 trunc from 0x0398e5 to 0x01a671 (0x1f273 bytes) 000240 1788687002.225257 read 0x008060 thru 0x0173bf (0xf360 bytes) 000241 1788687002.234621 fallocate 0x00c947 thru 0x00cc97 (0x351 bytes) 000242 1788687002.241860 write 0x0006ea thru 0x000feb (0x902 bytes) 000243 1788687002.243361 mapread 0x005b61 thru 0x00dc7f (0x811f bytes) 000245 1788687002.245555 punch from 0x00fe0c to 0x01a671 (0xa866 bytes) 000246 1788687002.255616 trunc from 0x01596e to 0x01a671 (0x4d04 bytes) 000247 1788687002.260871 trunc from 0x01a672 to 0x000c22 (0x19a4f bytes) 000248 1788687002.280045 mapread 0x000121 thru 0x000c22 (0xb02 bytes) 000249 1788687002.281466 punch from 0x00043f to 0x000c22 (0x7e4 bytes) 000250 1788687002.293707 write 0x00db57 thru 0x011ef7 (0x43a1 bytes) 000251 1788687002.295363 write 0x0267d8 thru 0x033b0a (0xd333 bytes) 000254 1788687002.300143 trunc from 0x01c219 to 0x029e0b (0xdbf3 bytes) 000255 1788687002.316494 mapread 0x0233a7 thru 0x0280b8 (0x4d12 bytes) 000256 1788687002.331050 trunc from 0x033b0b to 0x024083 (0xfa87 bytes) 000257 1788687002.342560 trunc from 0x01266f to 0x01e973 (0xc305 bytes) 000259 1788687002.356926 fallocate 0x0300a3 thru 0x035283 (0x51e1 bytes) 000260 1788687002.366537 mapwrite 0x012de4 thru 0x01366f (0x88c bytes) 000261 1788687002.416916 punch from 0x00add8 to 0x016257 (0xb480 bytes) 000263 1788687002.432300 punch from 0x032c76 to 0x035283 (0x260e bytes) 000264 1788687002.450267 fallocate 0x027ad6 thru 0x033ced (0xc218 bytes) 000266 1788687002.464963 mapread 0x00bcae thru 0x015d94 (0xa0e7 bytes) 000267 1788687002.545990 trunc from 0x035284 to 0x035091 (0x1f2 bytes) 000268 1788687002.558146 trunc from 0x00c7d4 to 0x019d5f (0xd58c bytes) 000269 1788687002.566072 trunc from 0x00ff6f to 0x01357a (0x360c bytes) 000270 1788687002.569782 trunc from 0x01c6c9 to 0x025a4b (0x9383 bytes) 000271 1788687002.574066 mapwrite 0x011fd4 thru 0x015686 (0x36b3 bytes) 000272 1788687002.606073 write 0x03be9e thru 0x03ffff (0x4162 bytes) 000273 1788687002.607201 write 0x004dcf thru 0x011692 (0xc8c4 bytes) 000274 1788687002.616648 fallocate 0x037b55 thru 0x03ed7c (0x7228 bytes) 000275 1788687002.624205 mapread 0x0337fe thru 0x0398a8 (0x60ab bytes) 000276 1788687002.644598 trunc from 0x040000 to 0x037708 (0x88f7 bytes) 000278 1788687002.663118 punch from 0x00f762 to 0x015a25 (0x62c4 bytes) 000279 1788687002.688094 mapread 0x01da5e thru 0x0226a6 (0x4c49 bytes) 000280 1788687002.738756 trunc from 0x01bea8 to 0x01e2c5 (0x241e bytes) 000281 1788687002.745261 mapwrite 0x018418 thru 0x021230 (0x8e19 bytes) 000283 1788687002.812076 fallocate 0x01966e thru 0x028c8a (0xf61d bytes) 000284 1788687002.822985 punch from 0x02bd34 to 0x037708 (0xb9d5 bytes) 000285 1788687002.834442 trunc from 0x037709 to 0x03912c (0x1a24 bytes) 000286 1788687002.863645 trunc from 0x03912d to 0x02d91b (0xb811 bytes) 000287 1788687002.881647 write 0x00b5d8 thru 0x014888 (0x92b1 bytes) 000288 1788687002.889049 read 0x02c365 thru 0x02d91b (0x15b7 bytes) 000289 1788687002.908421 trunc from 0x01dda0 to 0x01fd0c (0x1f6d bytes) 000290 1788687002.919487 mapwrite 0x00cf28 thru 0x01583d (0x8916 bytes) 000291 1788687002.937797 trunc from 0x00e476 to 0x013a8f (0x561a bytes) 000292 1788687002.949145 mapread 0x0067eb thru 0x01088a (0xa0a0 bytes) 000293 1788687002.970394 trunc from 0x00bf55 to 0x00e0e4 (0x2190 bytes) 000294 1788687002.982029 trunc from 0x011c9f to 0x021c96 (0xfff8 bytes) 000295 1788687002.996299 trunc from 0x02d91c to 0x024141 (0x97da bytes) 000297 1788687003.023247 MIRROR_EXTEND to 2 mirrors 000298 1788687003.918829 trunc from 0x024142 to 0x01951d (0xac24 bytes) 000299 1788687003.974228 trunc from 0x01951e to 0x00e6fb (0xae22 bytes) 000300 1788687003.996198 mapread 0x007305 thru 0x00d3c5 (0x60c1 bytes) 000301 1788687004.012027 mapwrite 0x01b005 thru 0x027ca6 (0xcca2 bytes) 000302 1788687004.083785 mapwrite 0x03a975 thru 0x03ffff (0x568b bytes) 000303 1788687004.153702 trunc from 0x015852 to 0x01db98 (0x8347 bytes) 000305 1788687004.159395 trunc from 0x040000 to 0x00917f (0x36e80 bytes) 000306 1788687004.174808 mapwrite 0x03dbc6 thru 0x03fe6e (0x22a9 bytes) 000307 1788687004.218480 mapwrite 0x02cfdc thru 0x02d15d (0x182 bytes) 000308 1788687004.240093 mapwrite 0x015519 thru 0x020c1b (0xb703 bytes) 000309 1788687004.280307 write 0x0268d7 thru 0x033989 (0xd0b3 bytes) 000310 1788687004.282270 fallocate 0x02cd55 thru 0x03588c (0x8b38 bytes) 000312 1788687004.288977 read 0x03c9ba thru 0x03e37d (0x19c4 bytes) 000313 1788687004.289917 mapwrite 0x01d41a thru 0x027946 (0xa52d bytes) 000315 1788687004.315147 mapwrite 0x03b411 thru 0x03ffff (0x4bef bytes) 000316 1788687004.370459 read 0x037e3c thru 0x03ffff (0x81c4 bytes) 000317 1788687004.371290 trunc from 0x040000 to 0x02520f (0x1adf0 bytes) 000318 1788687004.392076 read 0x016a54 thru 0x0243ad (0xd95a bytes) 000319 1788687004.392855 fallocate 0x01a76e thru 0x01e5e9 (0x3e7c bytes) 000320 1788687004.397564 punch from 0x007903 to 0x00957d (0x1c7b bytes) 000321 1788687004.405849 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) 000322 1788687004.599185 trunc from 0x0194e2 to 0x02182d (0x834c bytes) 000323 1788687004.605191 read 0x023021 thru 0x02520f (0x21ef bytes) 000324 1788687004.611922 fallocate 0x02528f thru 0x03019b (0xaf0d bytes) 000325 1788687004.618101 fallocate 0x003df6 thru 0x005501 (0x170c bytes) 000326 1788687004.624180 trunc from 0x03019c to 0x018b4a (0x17651 bytes) 000327 1788687004.651248 punch from 0x016606 to 0x018b4a (0x2545 bytes) 000328 1788687004.657894 punch from 0x00e711 to 0x018b4a (0xa43a bytes) 000329 1788687004.664856 punch from 0x017b28 to 0x018b4a (0x1023 bytes) 000330 1788687004.670802 mapwrite 0x035745 thru 0x03ffff (0xa8bb bytes) 000331 1788687004.764247 mapread 0x01848b thru 0x01b03d (0x2bb3 bytes) 000332 1788687004.780981 read 0x0025b3 thru 0x0065de (0x402c bytes) 000333 1788687004.834788 MIRROR_EXTEND to 2 mirrors 000334 1788687005.485488 MIRROR_RESYNC 2 mirrors 000335 1788687005.843432 trunc from 0x040000 to 0x0246c5 (0x1b93a bytes) 000336 1788687005.926130 trunc from 0x005e4f to 0x012fbe (0xd170 bytes) 000337 1788687005.940134 trunc from 0x009f9d to 0x00c587 (0x25eb bytes) 000338 1788687005.950310 trunc from 0x0246c6 to 0x007748 (0x1cf7d bytes) 000339 1788687005.989573 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x7749): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7749): Mirror 2: 0x22d3ba9d (stale) CRC-32 checksum value for chunk [0, 0x7749): Mirror 1: 0xd699bd0e Mirror 3: 0xd699bd0e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7749: skipped 000341 1788687007.070091 punch from 0x00358f to 0x007748 (0x41ba bytes) 000342 1788687007.159204 mapread 0x005d70 thru 0x007748 (0x19d9 bytes) 000343 1788687007.211091 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x7749): Mirror 2: 0x22d3ba9d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7749): Mirror 2: 0x22d3ba9d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7749: skipped 000344 1788687007.571418 mapread 0x007229 thru 0x007748 (0x520 bytes) 000345 1788687007.580178 trunc from 0x007749 to 0x01b41d (0x13cd5 bytes) 000346 1788687007.601205 trunc from 0x01b41e to 0x03d9f6 (0x225d9 bytes) 000347 1788687007.619742 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d9f7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d9f7): Mirror 2: 0xc3d1b110 (stale) CRC-32 checksum value for chunk [0, 0x3d9f7): Mirror 1: 0x9ef36de8 Mirror 3: 0x9ef36de8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d9f7: skipped 000348 1788687008.698469 write 0x0239de thru 0x02ae88 (0x74ab bytes) 000349 1788687008.782243 trunc from 0x03d943 to 0x03d9f6 (0xb4 bytes) 000350 1788687008.787160 mapwrite 0x0123e8 thru 0x01c3ca (0x9fe3 bytes) 000351 1788687008.831002 trunc from 0x037d5c to 0x03d172 (0x5417 bytes) 000352 1788687008.839701 read 0x014027 thru 0x017265 (0x323f bytes) 000354 1788687008.840935 read 0x00059c thru 0x006f9e (0x6a03 bytes) 000355 1788687008.863810 mapwrite 0x024f12 thru 0x0281af (0x329e bytes) 000356 1788687008.890093 punch from 0x02083c to 0x02458f (0x3d54 bytes) 000357 1788687008.913932 punch from 0x01dfba to 0x0284be (0xa505 bytes) 000358 1788687008.925278 read 0x01c31a thru 0x028536 (0xc21d bytes) 000361 1788687008.939004 write 0x02918b thru 0x02f625 (0x649b bytes) 000362 1788687008.943422 punch from 0x01df44 to 0x028594 (0xa651 bytes) 000363 1788687008.953715 punch from 0x03cff4 to 0x03d9f6 (0xa03 bytes) 000364 1788687008.962048 write 0x02259b thru 0x032315 (0xfd7b bytes) 000365 1788687008.972087 mapwrite 0x0174bd thru 0x01ba3b (0x457f bytes) 000366 1788687009.037742 read 0x006ef2 thru 0x014caa (0xddb9 bytes) 000367 1788687009.038422 read 0x02c821 thru 0x035dd0 (0x95b0 bytes) 000368 1788687009.038997 punch from 0x0232f2 to 0x0232f7 (0x6 bytes) 000369 1788687009.048208 read 0x00f1e1 thru 0x0124dc (0x32fc bytes) 000370 1788687009.049238 trunc from 0x00a318 to 0x00e9dc (0x46c5 bytes) 000371 1788687009.060765 trunc from 0x03d9f7 to 0x02148c (0x1c56a bytes) 000372 1788687009.116691 write 0x03f29d thru 0x03ffff (0xd63 bytes) 000373 1788687009.127934 punch from 0x013db6 to 0x020db0 (0xcffb bytes) 000374 1788687009.135469 mapwrite 0x014101 thru 0x01685f (0x275f bytes) 000376 1788687009.189844 fallocate 0x018c3f thru 0x01a4d2 (0x1894 bytes) 000377 1788687009.205419 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000378 1788687009.616283 MIRROR_RESYNC 2 mirrors 000379 1788687010.465334 mapread 0x00705f thru 0x015f03 (0xeea5 bytes) 000380 1788687010.492399 mapwrite 0x016fce thru 0x0244f2 (0xd525 bytes) 000381 1788687010.588243 mapwrite 0x018ea0 thru 0x023b97 (0xacf8 bytes) 000383 1788687010.613028 punch from 0x0396f5 to 0x03ffff (0x690b bytes) 000384 1788687010.617458 punch from 0x03359e to 0x036827 (0x328a bytes) 000385 1788687010.625558 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) 000386 1788687010.767007 punch from 0x03e840 to 0x03ffff (0x17c0 bytes) 000387 1788687010.772147 read 0x03e5f8 thru 0x03ffff (0x1a08 bytes) 000388 1788687010.778605 trunc from 0x00e1a1 to 0x019cd2 (0xbb32 bytes) 000390 1788687010.784765 punch from 0x0168b5 to 0x018e37 (0x2583 bytes) 000391 1788687010.789930 trunc from 0x014e13 to 0x01f072 (0xa260 bytes) 000393 1788687010.798233 fallocate 0x022d02 thru 0x03117c (0xe47b bytes) 000395 1788687010.807217 fallocate 0x01a602 thru 0x01b981 (0x1380 bytes) 000396 1788687010.811932 trunc from 0x040000 to 0x03bdc8 (0x4237 bytes) 000398 1788687010.835026 trunc from 0x002785 to 0x00502a (0x28a6 bytes) 000400 1788687010.840002 mapwrite 0x00536b thru 0x00892d (0x35c3 bytes) 000401 1788687010.896882 punch from 0x0211bf to 0x026dee (0x5c30 bytes) 000402 1788687010.912919 trunc from 0x000305 to 0x00249b (0x2197 bytes) 000403 1788687010.920634 read 0x01b449 thru 0x021b62 (0x671a bytes) 000404 1788687010.940442 trunc from 0x03bdc9 to 0x0300a1 (0xbd27 bytes) 000405 1788687010.979454 read 0x02cb51 thru 0x0300a1 (0x3551 bytes) 000406 1788687010.980595 read 0x0159c5 thru 0x0225c6 (0xcc02 bytes) 000407 1788687010.982163 read 0x023882 thru 0x02825b (0x49da bytes) 000408 1788687010.983084 fallocate 0x0288b5 thru 0x033f3f (0xb68b bytes) 000409 1788687010.994989 mapwrite 0x03d5b6 thru 0x03ffff (0x2a4a bytes) 000410 1788687011.036114 mapwrite 0x02fe8c thru 0x03dbf4 (0xdd69 bytes) 000411 1788687011.053187 mapread 0x03186e thru 0x035360 (0x3af3 bytes) 000412 1788687011.054036 mapwrite 0x00e30e thru 0x01c730 (0xe423 bytes) 000413 1788687011.084588 MIRROR_EXTEND to 2 mirrors 000414 1788687011.729354 trunc from 0x040000 to 0x01cb09 (0x234f6 bytes) 000415 1788687011.798285 punch from 0x00cea2 to 0x012d7c (0x5edb bytes) 000416 1788687011.803583 mapread 0x013688 thru 0x01a95f (0x72d8 bytes) 000417 1788687011.825658 read 0x008548 thru 0x009c12 (0x16cb bytes) 000418 1788687011.835628 punch from 0x002201 to 0x01096b (0xe76b bytes) 000419 1788687011.842968 trunc from 0x000580 to 0x0032cd (0x2d4e bytes) 000420 1788687011.850986 mapwrite 0x011884 thru 0x018686 (0x6e03 bytes) 000421 1788687011.874049 mapwrite 0x020354 thru 0x0203fd (0xaa bytes) 000422 1788687011.925145 trunc from 0x01ca73 to 0x0203fd (0x398b bytes) 000423 1788687011.930121 mapread 0x01c226 thru 0x0203fd (0x41d8 bytes) 000424 1788687011.946331 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x203fe): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x203fe): Mirror 2: 0x221900a5 (stale) CRC-32 checksum value for chunk [0, 0x203fe): Mirror 1: 0x644afc4b Mirror 3: 0x644afc4b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x203fe: skipped 000426 1788687012.871233 trunc from 0x0203fe to 0x01038e (0x1006f bytes) 000427 1788687012.955838 read 0x00f45f thru 0x01038e (0xf30 bytes) 000428 1788687012.975215 trunc from 0x0097ae to 0x01038e (0x6be1 bytes) 000429 1788687012.981953 write 0x016be2 thru 0x01d1e7 (0x6606 bytes) 000430 1788687012.983221 write 0x0201e1 thru 0x024554 (0x4374 bytes) 000431 1788687012.984241 write 0x0269fa thru 0x033abe (0xd0c5 bytes) 000432 1788687012.990873 read 0x01ca2f thru 0x021222 (0x47f4 bytes) 000433 1788687013.007273 write 0x011c06 thru 0x014c1a (0x3015 bytes) 000434 1788687013.014698 fallocate 0x0171f9 thru 0x01b888 (0x4690 bytes) 000435 1788687013.025248 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33abf): Mirror 2: 0xdab303ae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33abf): Mirror 2: 0xdab303ae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33abf: skipped 000436 1788687013.434556 mapwrite 0x037c61 thru 0x0399e2 (0x1d82 bytes) 000437 1788687013.513592 trunc from 0x0399e3 to 0x03f6cd (0x5ceb bytes) 000438 1788687013.535131 mapwrite 0x018c1e thru 0x01ff59 (0x733c bytes) 000439 1788687013.576480 MIRROR_RESYNC 2 mirrors 000440 1788687014.646279 mapwrite 0x00789a thru 0x00bd10 (0x4477 bytes) 000441 1788687014.769384 trunc from 0x03f6ce to 0x014296 (0x2b437 bytes) 000442 1788687014.798555 mapread 0x00c4db thru 0x00e8d4 (0x23fa bytes) 000443 1788687014.800200 fallocate 0x020f39 thru 0x0305ee (0xf6b6 bytes) 000444 1788687014.806022 trunc from 0x014297 to 0x0111d1 (0x30c5 bytes) 000445 1788687014.828033 punch from 0x009e22 to 0x0111d1 (0x73b0 bytes) 000446 1788687014.833241 trunc from 0x006aa6 to 0x00f410 (0x896b bytes) 000447 1788687014.838107 read 0x001756 thru 0x00af82 (0x982d bytes) 000448 1788687014.847092 mapread 0x004f53 thru 0x0111d1 (0xc27f bytes) 000449 1788687014.863390 fallocate 0x01e597 thru 0x01ed5b (0x7c5 bytes) 000452 1788687014.873268 write 0x00d94b thru 0x01c5ab (0xec61 bytes) 000456 1788687014.889597 write 0x037acc thru 0x03e06b (0x65a0 bytes) 000458 1788687014.892112 read 0x019fb9 thru 0x023b94 (0x9bdc bytes) 000459 1788687014.900220 read 0x00274f thru 0x007486 (0x4d38 bytes) 000460 1788687014.900858 mapwrite 0x02f910 thru 0x032989 (0x307a bytes) 000461 1788687014.930425 trunc from 0x02c2ca to 0x035521 (0x9258 bytes) 000462 1788687014.942488 mapwrite 0x010021 thru 0x01e0c4 (0xe0a4 bytes) 000463 1788687014.971219 trunc from 0x035333 to 0x037916 (0x25e4 bytes) 000464 1788687014.983606 write 0x030459 thru 0x0381b8 (0x7d60 bytes) 000465 1788687014.992955 write 0x029add thru 0x02f0ca (0x55ee bytes) 000466 1788687014.996493 trunc from 0x03e06c to 0x016ae8 (0x27583 bytes) 000467 1788687015.029599 mapread 0x007fe2 thru 0x00e732 (0x6751 bytes) 000468 1788687015.047920 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x16ae9): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16ae9): Mirror 2: 0x1c72afb7 (stale) CRC-32 checksum value for chunk [0, 0x16ae9): Mirror 1: 0x9d00747d Mirror 3: 0x9d00747d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16ae9: skipped 000469 1788687015.991098 write 0x02ef46 thru 0x039339 (0xa3f4 bytes) 000470 1788687016.048620 mapwrite 0x034663 thru 0x03e5b9 (0x9f57 bytes) 000471 1788687016.134602 mapread 0x001e5c thru 0x006587 (0x472c bytes) 000472 1788687016.147595 mapwrite 0x01afc7 thru 0x02608b (0xb0c5 bytes) 000473 1788687016.211860 fallocate 0x02eef7 thru 0x03a786 (0xb890 bytes) 000474 1788687016.221648 mapread 0x00a131 thru 0x00aa19 (0x8e9 bytes) 000475 1788687016.222463 mapwrite 0x03ff7a thru 0x03ffff (0x86 bytes) 000476 1788687016.267611 read 0x018ca6 thru 0x02189d (0x8bf8 bytes) 000477 1788687016.273450 MIRROR_RESYNC 3 mirrors 000478 1788687017.317113 mapread 0x02dc0b thru 0x032e36 (0x522c bytes) 000479 1788687017.345034 read 0x0064bc thru 0x00ff0d (0x9a52 bytes) 000480 1788687017.371611 mapwrite 0x03c8e1 thru 0x03ffff (0x371f bytes) 000481 1788687017.499785 write 0x01f746 thru 0x02a4b2 (0xad6d bytes) 000483 1788687017.524124 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x135a7d99 (stale) Mirror 3: 0x135a7d99 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfb6786c1 Mirror 4: 0xfb6786c1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000484 1788687018.621886 read 0x010263 thru 0x0196eb (0x9489 bytes) 000485 1788687018.659242 read 0x0260d3 thru 0x02aebf (0x4ded bytes) 000486 1788687018.688214 fallocate 0x03569a thru 0x03ffff (0xa966 bytes) 000487 1788687018.756711 punch from 0x0065bb to 0x00dfa2 (0x79e8 bytes) 000488 1788687018.766771 write 0x012e7d thru 0x01d790 (0xa914 bytes) 000490 1788687018.790238 mapread 0x03ec8a thru 0x03ffff (0x1376 bytes) 000491 1788687018.826122 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x135a7d99 (stale) Mirror 4: 0xfb6786c1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x135a7d99 (stale) Mirror 4: 0xfb6786c1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000493 1788687019.303707 trunc from 0x040000 to 0x02c29b (0x13d64 bytes) 000494 1788687019.346212 mapwrite 0x01b878 thru 0x02b636 (0xfdbf bytes) 000495 1788687019.446172 fallocate 0x01ab47 thru 0x01dd00 (0x31ba bytes) 000496 1788687019.455229 mapwrite 0x03af70 thru 0x03b6bb (0x74c bytes) 000497 1788687019.499211 fallocate 0x031072 thru 0x032255 (0x11e4 bytes) 000498 1788687019.506553 trunc from 0x008c77 to 0x00ed1b (0x60a5 bytes) 000500 1788687019.516070 trunc from 0x026ac9 to 0x02c3d2 (0x590a bytes) 000501 1788687019.527969 trunc from 0x03b6bc to 0x018387 (0x23334 bytes) 000503 1788687019.548380 fallocate 0x002580 thru 0x009744 (0x71c5 bytes) 000504 1788687019.560024 write 0x03e8a5 thru 0x03ffff (0x175b bytes) 000505 1788687019.562330 write 0x015d9f thru 0x021ed0 (0xc132 bytes) 000506 1788687019.570442 mapwrite 0x02bf20 thru 0x030d99 (0x4e7a bytes) 000507 1788687019.590986 fallocate 0x00b39e thru 0x00f116 (0x3d79 bytes) 000508 1788687019.596019 mapread 0x03150e thru 0x039aeb (0x85de bytes) 000509 1788687019.604605 mapread 0x027d74 thru 0x0284d2 (0x75f bytes) 000510 1788687019.609650 MIRROR_RESYNC 3 mirrors 000511 1788687020.331535 trunc from 0x040000 to 0x002ea2 (0x3d15d bytes) 000512 1788687020.381939 trunc from 0x002ea3 to 0x00f177 (0xc2d5 bytes) 000513 1788687020.416946 trunc from 0x00f178 to 0x01b6db (0xc564 bytes) 000515 1788687020.439071 trunc from 0x01b6dc to 0x00fa68 (0xbc73 bytes) 000516 1788687020.464609 fallocate 0x038234 thru 0x03ffff (0x7dcc bytes) 000517 1788687020.470203 trunc from 0x03db57 to 0x03e84f (0xcf9 bytes) 000518 1788687020.474558 fallocate 0x0129e2 thru 0x018907 (0x5f26 bytes) 000519 1788687020.478865 write 0x006d0b thru 0x008bdb (0x1ed1 bytes) 000520 1788687020.487870 trunc from 0x040000 to 0x005068 (0x3af97 bytes) 000521 1788687020.500600 fallocate 0x035b55 thru 0x03ffff (0xa4ab bytes) 000522 1788687020.508136 MIRROR_RESYNC 3 mirrors 000523 1788687021.076278 trunc from 0x0285ea to 0x028a42 (0x459 bytes) 000524 1788687021.145061 mapread 0x033cb6 thru 0x03e855 (0xaba0 bytes) 000525 1788687021.175210 read 0x018b46 thru 0x01fa08 (0x6ec3 bytes) 000526 1788687021.184989 trunc from 0x040000 to 0x019207 (0x26df8 bytes) 000527 1788687021.208579 mapwrite 0x02543d thru 0x03018d (0xad51 bytes) 000528 1788687021.262648 MIRROR_RESYNC 3 mirrors 000529 1788687022.006962 read 0x015577 thru 0x01f962 (0xa3ec bytes) 000530 1788687022.015951 write 0x00acc1 thru 0x01464c (0x998c bytes) 000531 1788687022.095139 fallocate 0x02569f thru 0x029fab (0x490d bytes) 000532 1788687022.099269 trunc from 0x00c195 to 0x0147d8 (0x8644 bytes) 000533 1788687022.132347 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3018e): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3018e): Mirror 3: 0x39723b8c (stale) Mirror 4: 0x39723b8c (stale) CRC-32 checksum value for chunk [0, 0x3018e): Mirror 1: 0xac742ad2 Mirror 5: 0xac742ad2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3018e: skipped 000534 1788687023.229118 write 0x02e3cc thru 0x03963c (0xb271 bytes) 000535 1788687023.302107 write 0x00a20c thru 0x013bb4 (0x99a9 bytes) 000536 1788687023.316931 read 0x003444 thru 0x00aebc (0x7a79 bytes) 000537 1788687023.346089 read 0x01c93a thru 0x01eba6 (0x226d bytes) 000538 1788687023.363223 trunc from 0x0329c1 to 0x03963c (0x6c7c bytes) 000540 1788687023.392064 read 0x015f99 thru 0x017733 (0x179b bytes) 000541 1788687023.393239 punch from 0x026691 to 0x02d1f4 (0x6b64 bytes) 000542 1788687023.401463 mapread 0x01b1e6 thru 0x02a16a (0xef85 bytes) 000543 1788687023.427159 mapread 0x007b99 thru 0x00a938 (0x2da0 bytes) 000544 1788687023.429460 mapread 0x00aee9 thru 0x0148f5 (0x9a0d bytes) 000545 1788687023.439061 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3963d): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3963d): Mirror 3: 0xd861175f (stale) Mirror 4: 0xd861175f (stale) Mirror 5: 0xe1c7e9d6 (stale) CRC-32 checksum value for chunk [0, 0x3963d): Mirror 1: 0xe18ddd69 Mirror 6: 0xe18ddd69 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3963d: skipped 000546 1788687024.460479 trunc from 0x03963d to 0x0029e8 (0x36c54 bytes) 000547 1788687024.559869 fallocate 0x03e88c thru 0x03ffff (0x1774 bytes) 000550 1788687024.575073 mapwrite 0x00e0b9 thru 0x010b0c (0x2a54 bytes) 000551 1788687024.646980 mapwrite 0x030e66 thru 0x035b98 (0x4d33 bytes) 000552 1788687024.725512 mapwrite 0x03e864 thru 0x03ffff (0x179c bytes) 000553 1788687024.792528 punch from 0x02139e to 0x0271d9 (0x5e3c bytes) 000554 1788687024.798192 trunc from 0x040000 to 0x01c7c8 (0x23837 bytes) 000555 1788687024.836837 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1c7c9): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c7c9): Mirror 3: 0x9a66ad3f (stale) Mirror 4: 0x9a66ad3f (stale) Mirror 5: 0xc3c279b2 (stale) Mirror 6: 0xd888e663 (stale) CRC-32 checksum value for chunk [0, 0x1c7c9): Mirror 1: 0x1d9fcb5f Mirror 7: 0x1d9fcb5f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c7c9: skipped 000556 1788687025.988514 read 0x01a500 thru 0x01c7c8 (0x22c9 bytes) 000557 1788687026.008300 punch from 0x00f951 to 0x0127b7 (0x2e67 bytes) 000558 1788687026.100452 mapwrite 0x03418d thru 0x03d2ed (0x9161 bytes) 000559 1788687026.232419 trunc from 0x03d2ee to 0x0385e3 (0x4d0a bytes) 000560 1788687026.245335 mapwrite 0x02df4b thru 0x02e64e (0x704 bytes) 000561 1788687026.275526 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x385e4): Mirror 4: 0x16102642 (stale) Mirror 5: 0x334c51e2 (stale) Mirror 6: 0x6816bf06 (stale) Mirror 7: 0xb4d0a431 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x385e4): Mirror 4: 0x16102642 (stale) Mirror 5: 0x334c51e2 (stale) Mirror 6: 0x6816bf06 (stale) Mirror 7: 0xb4d0a431 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x385e4: skipped 000562 1788687026.710425 trunc from 0x0385e4 to 0x0353c8 (0x321b bytes) 000563 1788687026.733435 punch from 0x002e34 to 0x003531 (0x6fe bytes) 000564 1788687026.737587 read 0x0021bc thru 0x0103e0 (0xe225 bytes) 000565 1788687026.746158 punch from 0x02cca3 to 0x034438 (0x7796 bytes) 000566 1788687026.751037 fallocate 0x021786 thru 0x027d1f (0x659a bytes) 000567 1788687026.755479 write 0x00a81f thru 0x017dd6 (0xd5b8 bytes) 000568 1788687026.774418 punch from 0x006e57 to 0x0092a0 (0x244a bytes) 000569 1788687026.779572 trunc from 0x01dcc2 to 0x0259b0 (0x7cef bytes) 000571 1788687026.783953 write 0x01cb21 thru 0x022a69 (0x5f49 bytes) 000572 1788687026.792635 fallocate 0x01cf8b thru 0x02920b (0xc281 bytes) 000573 1788687026.797371 mapwrite 0x03a75b thru 0x03fa8f (0x5335 bytes) 000575 1788687026.900393 trunc from 0x03fa90 to 0x0229c1 (0x1d0ce bytes) 000577 1788687026.924722 trunc from 0x0229c2 to 0x011aa8 (0x10f19 bytes) 000578 1788687026.942451 trunc from 0x011aa9 to 0x010539 (0x156f bytes) 000579 1788687026.962294 write 0x016758 thru 0x02086d (0xa116 bytes) 000580 1788687026.965441 mapread 0x000678 thru 0x00df75 (0xd8fe bytes) 000581 1788687026.976813 read 0x013d00 thru 0x01e2f5 (0xa5f6 bytes) 000582 1788687026.983552 trunc from 0x010941 to 0x01dde6 (0xd4a6 bytes) 000583 1788687026.996861 trunc from 0x02086e to 0x036b80 (0x16313 bytes) 000584 1788687027.014857 read 0x0100eb thru 0x011018 (0xf2e bytes) 000585 1788687027.028542 mapwrite 0x02f392 thru 0x039536 (0xa1a5 bytes) 000586 1788687027.062369 trunc from 0x039537 to 0x010f20 (0x28616 bytes) 000587 1788687027.082749 punch from 0x007c3f to 0x007f7d (0x33f bytes) 000588 1788687027.090437 fallocate 0x00e13c thru 0x013658 (0x551d bytes) 000589 1788687027.106542 punch from 0x000137 to 0x007b2d (0x79f7 bytes) 000590 1788687027.114098 write 0x000836 thru 0x00c5c2 (0xbd8d bytes) 000591 1788687027.121119 mapwrite 0x00ef75 thru 0x0145e4 (0x5670 bytes) 000592 1788687027.186749 fallocate 0x010878 thru 0x01e884 (0xe00d bytes) 000593 1788687027.193907 mapread 0x00d4d0 thru 0x0145e4 (0x7115 bytes) 000594 1788687027.197819 trunc from 0x0145e5 to 0x019f34 (0x5950 bytes) 000595 1788687027.223133 write 0x01b7c0 thru 0x01e434 (0x2c75 bytes) 000596 1788687027.226877 trunc from 0x01e435 to 0x01be12 (0x2622 bytes) 000597 1788687027.257172 write 0x005aab thru 0x00f9c2 (0x9f18 bytes) 000598 1788687027.260796 fallocate 0x03ed63 thru 0x03ffff (0x129d bytes) 000599 1788687027.267827 write 0x027237 thru 0x02f31b (0x80e5 bytes) 000600 1788687027.271017 trunc from 0x01eed8 to 0x021faf (0x30d8 bytes) 000601 1788687027.277129 write 0x0236ad thru 0x028738 (0x508c bytes) 000602 1788687027.285980 mapwrite 0x021563 thru 0x02b0b7 (0x9b55 bytes) 000603 1788687027.322555 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f31c): Mirror 4: 0x11e67f9f (stale) Mirror 5: 0x21afb359 (stale) Mirror 6: 0x53f1f4d2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f31c): Mirror 4: 0x11e67f9f (stale) Mirror 5: 0x21afb359 (stale) Mirror 6: 0x53f1f4d2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f31c: skipped 000605 1788687027.766296 punch from 0x0269e2 to 0x02f31b (0x893a bytes) 000606 1788687027.771882 punch from 0x009f2a to 0x00ccef (0x2dc6 bytes) 000607 1788687027.776785 write 0x023e60 thru 0x031873 (0xda14 bytes) 000608 1788687027.786445 fallocate 0x00c824 thru 0x00d28d (0xa6a bytes) 000609 1788687027.791245 trunc from 0x031874 to 0x03e13b (0xc8c8 bytes) 000610 1788687027.804639 punch from 0x02e97a to 0x037798 (0x8e1f bytes) 000611 1788687027.820650 write 0x00384d thru 0x00d7db (0x9f8f bytes) 000612 1788687027.829140 mapread 0x01b806 thru 0x021ba9 (0x63a4 bytes) 000613 1788687027.849338 fallocate 0x02f117 thru 0x0386e3 (0x95cd bytes) 000614 1788687027.855395 trunc from 0x039272 to 0x03c074 (0x2e03 bytes) 000615 1788687027.862828 mapwrite 0x017b3c thru 0x01edfa (0x72bf bytes) 000616 1788687027.889578 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e13c): Mirror 4: 0x29ba0b3 (stale) Mirror 5: 0x5a5c09c0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e13c): Mirror 4: 0x29ba0b3 (stale) Mirror 5: 0x5a5c09c0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e13c: skipped 000617 1788687028.175044 mapread 0x002bfb thru 0x012101 (0xf507 bytes) 000618 1788687028.197951 write 0x02dbb4 thru 0x038b9d (0xafea bytes) 000619 1788687028.208993 trunc from 0x03e13c to 0x004867 (0x398d4 bytes) 000620 1788687028.225430 punch from 0x002f95 to 0x004867 (0x18d3 bytes) 000621 1788687028.230358 mapread 0x001297 thru 0x004867 (0x35d1 bytes) 000622 1788687028.235845 punch from 0x00195e to 0x004867 (0x2f0a bytes) 000623 1788687028.240242 write 0x009c26 thru 0x016c4e (0xd029 bytes) 000624 1788687028.242647 mapread 0x001eda thru 0x0032ea (0x1411 bytes) 000625 1788687028.250881 trunc from 0x016c4f to 0x01e3fb (0x77ad bytes) 000626 1788687028.261099 mapread 0x018f99 thru 0x01de0b (0x4e73 bytes) 000627 1788687028.268645 mapwrite 0x029953 thru 0x037214 (0xd8c2 bytes) 000628 1788687028.325761 punch from 0x00180b to 0x009ee3 (0x86d9 bytes) 000629 1788687028.335013 mapwrite 0x00b24f thru 0x010c38 (0x59ea bytes) 000630 1788687028.351257 trunc from 0x037215 to 0x027ce3 (0xf531 bytes) 000631 1788687028.368636 read 0x002761 thru 0x011b77 (0xf417 bytes) 000632 1788687028.375332 punch from 0x01697e to 0x017233 (0x8b6 bytes) 000633 1788687028.379317 read 0x019d7a thru 0x0271a1 (0xd428 bytes) 000634 1788687028.387874 mapread 0x02746d thru 0x027ce3 (0x877 bytes) 000635 1788687028.388726 trunc from 0x027ce4 to 0x00acf3 (0x1cff0 bytes) 000636 1788687028.403439 punch from 0x009715 to 0x00acf3 (0x15df bytes) 000637 1788687028.408279 punch from 0x00ac62 to 0x00acf3 (0x92 bytes) 000638 1788687028.413321 read 0x0088e0 thru 0x0089f7 (0x118 bytes) 000639 1788687028.414084 punch from 0x003feb to 0x00acf3 (0x6d09 bytes) 000640 1788687028.418235 write 0x01dbba thru 0x02b9db (0xde22 bytes) 000641 1788687028.419908 trunc from 0x007a40 to 0x015e0f (0xe3d0 bytes) 000642 1788687028.425064 fallocate 0x019734 thru 0x02077a (0x7047 bytes) 000643 1788687028.429731 mapread 0x016a00 thru 0x0180d7 (0x16d8 bytes) 000644 1788687028.440471 fallocate 0x039f53 thru 0x03f398 (0x5446 bytes) 000645 1788687028.444822 punch from 0x027654 to 0x028862 (0x120f bytes) 000646 1788687028.459587 write 0x02f4d9 thru 0x03283f (0x3367 bytes) 000647 1788687028.469270 punch from 0x014ec0 to 0x01c666 (0x77a7 bytes) 000648 1788687028.473043 punch from 0x029645 to 0x02fe45 (0x6801 bytes) 000649 1788687028.483846 trunc from 0x032898 to 0x038ba3 (0x630c bytes) 000650 1788687028.488486 mapread 0x01d70a thru 0x01ebf2 (0x14e9 bytes) 000651 1788687028.489504 mapwrite 0x025a37 thru 0x03186f (0xbe39 bytes) 000652 1788687028.543089 fallocate 0x01e45d thru 0x0229cd (0x4571 bytes) 000653 1788687028.548064 mapwrite 0x000d4a thru 0x00f15b (0xe412 bytes) 000654 1788687028.573303 read 0x01a5d3 thru 0x020570 (0x5f9e bytes) 000655 1788687028.582148 trunc from 0x03f399 to 0x011bf5 (0x2d7a3 bytes) 000656 1788687028.603345 punch from 0x00f331 to 0x011bf5 (0x28c5 bytes) 000657 1788687028.608061 write 0x02ffc5 thru 0x03d5c7 (0xd603 bytes) 000658 1788687028.611010 fallocate 0x017222 thru 0x025615 (0xe3f4 bytes) 000660 1788687028.616451 punch from 0x034b51 to 0x03d5c7 (0x8a77 bytes) 000661 1788687028.629061 trunc from 0x006fa2 to 0x01450f (0xd56e bytes) 000662 1788687028.633926 punch from 0x007544 to 0x00cdf4 (0x58b1 bytes) 000663 1788687028.638927 fallocate 0x01420a thru 0x01c29c (0x8093 bytes) 000665 1788687028.644590 trunc from 0x0298dc to 0x037eff (0xe624 bytes) 000666 1788687028.652004 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3d5c8): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d5c8): Mirror 4: 0xf2c2375c (stale) Mirror 5: 0x975d035e (stale) CRC-32 checksum value for chunk [0, 0x3d5c8): Mirror 1: 0xf1315ae4 Mirror 6: 0xf1315ae4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d5c8: skipped 000667 1788687029.423655 trunc from 0x03d5c8 to 0x00ac94 (0x32933 bytes) 000668 1788687029.541851 trunc from 0x00ac95 to 0x000795 (0xa4ff bytes) 000669 1788687029.571470 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x796): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x796): Mirror 4: 0x4546eb1d (stale) Mirror 5: 0x4546eb1d (stale) Mirror 6: 0x4546eb1d (stale) CRC-32 checksum value for chunk [0, 0x796): Mirror 1: 0x4546eb1d Mirror 7: 0x4546eb1d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x796: skipped 000670 1788687030.383283 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x796): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x796): Mirror 4: 0x4546eb1d (stale) Mirror 5: 0x4546eb1d (stale) Mirror 6: 0x4546eb1d (stale) CRC-32 checksum value for chunk [0, 0x796): Mirror 1: 0x4546eb1d Mirror 7: 0x4546eb1d Mirror 8: 0x4546eb1d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x796: skipped 000671 1788687031.261632 write 0x03e146 thru 0x03ffff (0x1eba bytes) 000672 1788687031.305030 punch from 0x03d59e to 0x03e229 (0xc8c bytes) 000673 1788687031.323888 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x6c73b1ed (stale) Mirror 5: 0x69894ad2 (stale) Mirror 6: 0x980a6b88 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe929766e Mirror 9: 0xe929766e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000674 1788687032.309574 fallocate 0x02ab38 thru 0x035e81 (0xb34a bytes) 000675 1788687032.341928 mapwrite 0x03f9ad thru 0x03ffff (0x653 bytes) 000676 1788687032.359861 mapwrite 0x00e7a9 thru 0x01ce61 (0xe6b9 bytes) 000677 1788687032.410465 mapread 0x000e4e thru 0x007f4e (0x7101 bytes) 000678 1788687032.436843 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x69894ad2 (stale) Mirror 6: 0x980a6b88 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xe929766e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x69894ad2 (stale) Mirror 6: 0x980a6b88 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xe929766e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000679 1788687032.876294 mapwrite 0x0335b9 thru 0x03b643 (0x808b bytes) 000680 1788687032.913963 mapread 0x011bb1 thru 0x0124d2 (0x922 bytes) 000681 1788687032.932014 punch from 0x0081b9 to 0x0120bd (0x9f05 bytes) 000683 1788687032.944846 mapread 0x03e113 thru 0x03ffff (0x1eed bytes) 000684 1788687032.952571 mapwrite 0x03744e thru 0x03ffff (0x8bb2 bytes) 000685 1788687032.973054 trunc from 0x040000 to 0x0093a1 (0x36c5e bytes) 000686 1788687033.010442 mapread 0x006a42 thru 0x0093a1 (0x2960 bytes) 000687 1788687033.029659 read 0x0069b6 thru 0x0093a1 (0x29ec bytes) 000689 1788687033.031585 mapwrite 0x03c418 thru 0x03ffff (0x3be8 bytes) 000690 1788687033.068397 trunc from 0x0340bb to 0x0356c8 (0x160e bytes) 000691 1788687033.073652 fallocate 0x023484 thru 0x029bf1 (0x676e bytes) 000692 1788687033.079531 write 0x01f41f thru 0x02672d (0x730f bytes) 000693 1788687033.090272 trunc from 0x040000 to 0x0054de (0x3ab21 bytes) 000694 1788687033.110812 read 0x0038fb thru 0x0054de (0x1be4 bytes) 000695 1788687033.111442 trunc from 0x0054df to 0x0343bb (0x2eedd bytes) 000696 1788687033.128102 mapread 0x034361 thru 0x0343bb (0x5b bytes) 000697 1788687033.141061 read 0x00bc25 thru 0x017981 (0xbd5d bytes) 000698 1788687033.141985 write 0x02482a thru 0x02692f (0x2106 bytes) 000699 1788687033.143968 read 0x02a1c6 thru 0x02f965 (0x57a0 bytes) 000700 1788687033.144655 punch from 0x02be16 to 0x03358b (0x7776 bytes) 000701 1788687033.149479 fallocate 0x0149cd thru 0x0244c7 (0xfafb bytes) 000702 1788687033.154657 punch from 0x02c6b1 to 0x0343bb (0x7d0b bytes) 000703 1788687033.159471 mapwrite 0x0196d9 thru 0x023bbe (0xa4e6 bytes) 000704 1788687033.178013 trunc from 0x01b9f9 to 0x0203a7 (0x49af bytes) 000705 1788687033.183794 trunc from 0x012f07 to 0x0215a6 (0xe6a0 bytes) 000706 1788687033.188750 mapread 0x024a13 thru 0x025f3d (0x152b bytes) 000707 1788687033.194320 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x343bc): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x343bc): Mirror 5: 0x15f22668 (stale) Mirror 6: 0xb3ca20cc (stale) Mirror 7: 0xf351e2fe (stale) Mirror 8: 0xf351e2fe (stale) Mirror 9: 0xf351e2fe (stale) CRC-32 checksum value for chunk [0, 0x343bc): Mirror 1: 0x997fd9c1 Mirror 10: 0x997fd9c1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x343bc: skipped 000708 1788687034.294265 trunc from 0x00ce74 to 0x015310 (0x849d bytes) 000709 1788687034.351147 trunc from 0x0343bc to 0x00fccf (0x246ec bytes) 000710 1788687034.410416 fallocate 0x01bb00 thru 0x020bcf (0x50d0 bytes) 000711 1788687034.416106 punch from 0x01089e to 0x0162fb (0x5a5e bytes) 000712 1788687034.421130 trunc from 0x020bd0 to 0x00b274 (0x1595b bytes) 000713 1788687034.440299 mapread 0x002b40 thru 0x00b274 (0x8735 bytes) 000714 1788687034.459057 read 0x0015cf thru 0x00b274 (0x9ca6 bytes) 000716 1788687034.460297 mapwrite 0x029354 thru 0x035d95 (0xca42 bytes) 000717 1788687034.542182 trunc from 0x034e6d to 0x03560e (0x7a2 bytes) 000718 1788687034.549811 read 0x02fefa thru 0x0320e6 (0x21ed bytes) 000719 1788687034.550452 fallocate 0x034dea thru 0x03bd96 (0x6fad bytes) 000720 1788687034.555344 trunc from 0x035d96 to 0x016371 (0x1fa24 bytes) 000721 1788687034.586325 trunc from 0x004e32 to 0x011182 (0xc351 bytes) 000722 1788687034.593044 punch from 0x012466 to 0x016371 (0x3f0c bytes) 000723 1788687034.600791 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16372): Mirror 5: 0x5fb7406c (stale) Mirror 6: 0x6dc0b90 (stale) Mirror 7: 0xef8356f5 (stale) Mirror 8: 0xef8356f5 (stale) Mirror 9: 0xef8356f5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16372): Mirror 5: 0x5fb7406c (stale) Mirror 6: 0x6dc0b90 (stale) Mirror 7: 0xef8356f5 (stale) Mirror 8: 0xef8356f5 (stale) Mirror 9: 0xef8356f5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16372: skipped 000724 1788687035.046935 punch from 0x0074fe to 0x00c589 (0x508c bytes) 000725 1788687035.060557 mapwrite 0x03441d thru 0x03c7c1 (0x83a5 bytes) 000726 1788687035.128367 trunc from 0x039fce to 0x03c7c1 (0x27f4 bytes) 000727 1788687035.136018 mapread 0x025f1a thru 0x02ca2d (0x6b14 bytes) 000728 1788687035.147811 mapwrite 0x02b87e thru 0x031cac (0x642f bytes) 000729 1788687035.183315 trunc from 0x03c7c2 to 0x03c6e7 (0xda bytes) 000730 1788687035.195554 mapwrite 0x0020aa thru 0x010315 (0xe26c bytes) 000731 1788687035.224039 mapread 0x0289e0 thru 0x034275 (0xb896 bytes) 000732 1788687035.224998 trunc from 0x012e14 to 0x0159d5 (0x2bc2 bytes) 000733 1788687035.239523 trunc from 0x00af78 to 0x01405e (0x90e7 bytes) 000734 1788687035.256050 mapwrite 0x036bca thru 0x03ffff (0x9436 bytes) 000735 1788687035.342450 fallocate 0x026289 thru 0x027368 (0x10e0 bytes) 000736 1788687035.347996 trunc from 0x03b231 to 0x03ffff (0x4dcf bytes) 000737 1788687035.355456 fallocate 0x01579f thru 0x01f4dc (0x9d3e bytes) 000738 1788687035.360508 mapwrite 0x03b6a1 thru 0x03ffff (0x495f bytes) 000739 1788687035.393437 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x69894ad2 (stale) Mirror 6: 0x980a6b88 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xe929766e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x647756d4 Mirror 10: 0x647756d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000740 1788687036.458324 fallocate 0x03c494 thru 0x03ffff (0x3b6c bytes) 000741 1788687036.587317 MIRROR_RESYNC 7 mirrors 000742 1788687038.719016 trunc from 0x0292c8 to 0x03590e (0xc647 bytes) 000743 1788687038.758790 mapwrite 0x01b1a8 thru 0x02390d (0x8766 bytes) 000744 1788687038.804250 mapread 0x03c48d thru 0x03ffff (0x3b73 bytes) 000745 1788687038.815020 mapwrite 0x039c68 thru 0x03ffff (0x6398 bytes) 000746 1788687038.832025 MIRROR_RESYNC 7 mirrors 000749 1788687040.582438 mapread 0x019410 thru 0x01b274 (0x1e65 bytes) 000750 1788687040.612821 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) 000751 1788687040.969823 mapread 0x00ba99 thru 0x01b687 (0xfbef bytes) 000752 1788687040.992768 write 0x007bda thru 0x00f0c8 (0x74ef bytes) 000754 1788687041.043797 mapread 0x01ba37 thru 0x02b0d2 (0xf69c bytes) 000756 1788687041.067203 read 0x006b12 thru 0x00da25 (0x6f14 bytes) 000757 1788687041.071173 trunc from 0x0125d0 to 0x01822e (0x5c5f bytes) 000758 1788687041.075409 trunc from 0x040000 to 0x00cc58 (0x333a7 bytes) 000759 1788687041.094255 mapread 0x00caa4 thru 0x00cc58 (0x1b5 bytes) 000760 1788687041.095102 trunc from 0x002a48 to 0x00cc58 (0xa211 bytes) 000761 1788687041.114147 MIRROR_RESYNC 6 mirrors 000762 1788687042.296819 trunc from 0x00cc59 to 0x0384bf (0x2b867 bytes) 000763 1788687042.425578 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x384c0): Mirror 6: 0x904eef80 (stale) Mirror 8: 0x904eef80 (stale) Mirror 9: 0x904eef80 (stale) Mirror 10: 0x904eef80 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x384c0): Mirror 6: 0x904eef80 (stale) Mirror 8: 0x904eef80 (stale) Mirror 9: 0x904eef80 (stale) Mirror 10: 0x904eef80 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x384c0: skipped 000764 1788687042.800395 MIRROR_RESYNC 5 mirrors 000765 1788687044.142179 fallocate 0x0390c9 thru 0x03dee2 (0x4e1a bytes) 000767 1788687044.177631 mapread 0x02562b thru 0x027a2f (0x2405 bytes) 000768 1788687044.223744 write 0x0023a6 thru 0x010ed7 (0xeb32 bytes) 000770 1788687044.256463 mapwrite 0x003538 thru 0x00b2f6 (0x7dbf bytes) 000771 1788687044.280021 trunc from 0x0384c0 to 0x020879 (0x17c46 bytes) 000772 1788687044.310631 mapwrite 0x033337 thru 0x037369 (0x4033 bytes) 000773 1788687044.377017 trunc from 0x03736a to 0x031a20 (0x5949 bytes) 000774 1788687044.413831 trunc from 0x0286c9 to 0x02a962 (0x229a bytes) 000775 1788687044.418408 read 0x002555 thru 0x0076ff (0x51ab bytes) 000776 1788687044.419657 trunc from 0x031a21 to 0x023f72 (0xdaae bytes) 000777 1788687044.434052 fallocate 0x003762 thru 0x012308 (0xeba7 bytes) 000778 1788687044.440921 mapwrite 0x02193e thru 0x02a0cc (0x878f bytes) 000779 1788687044.564028 MIRROR_RESYNC 5 mirrors 000780 1788687045.976809 fallocate 0x0195af thru 0x023a28 (0xa47a bytes) 000783 1788687046.013162 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2a0cd): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a0cd): Mirror 6: 0xaff507d2 (stale) Mirror 8: 0xaff507d2 (stale) Mirror 9: 0xaff507d2 (stale) Mirror 10: 0xaff507d2 (stale) CRC-32 checksum value for chunk [0, 0x2a0cd): Mirror 1: 0xaff507d2 Mirror 11: 0xaff507d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a0cd: skipped 000784 1788687046.884878 read 0x00133a thru 0x00f362 (0xe029 bytes) 000785 1788687046.898865 fallocate 0x030d1b thru 0x03788c (0x6b72 bytes) 000786 1788687046.935056 trunc from 0x03788d to 0x00c63b (0x2b251 bytes) 000787 1788687046.964471 fallocate 0x0380b4 thru 0x0387ed (0x73a bytes) 000788 1788687046.969810 mapwrite 0x007d19 thru 0x010ae6 (0x8dce bytes) 000789 1788687047.036335 read 0x00dd4e thru 0x010ae6 (0x2d99 bytes) 000790 1788687047.037293 trunc from 0x00de7a to 0x010ae6 (0x2c6d bytes) 000791 1788687047.044388 write 0x028cfb thru 0x0386b1 (0xf9b7 bytes) 000792 1788687047.046744 trunc from 0x035ab8 to 0x0386b1 (0x2bfa bytes) 000793 1788687047.087335 write 0x02fb31 thru 0x0342ec (0x47bc bytes) 000794 1788687047.089183 fallocate 0x03e3b7 thru 0x03ffff (0x1c49 bytes) 000795 1788687047.098411 mapread 0x02a644 thru 0x02ee78 (0x4835 bytes) 000796 1788687047.101440 fallocate 0x029800 thru 0x02a3df (0xbe0 bytes) 000799 1788687047.111313 mapwrite 0x00460d thru 0x00d17f (0x8b73 bytes) 000800 1788687047.140846 trunc from 0x01b37d to 0x0291b2 (0xde36 bytes) 000801 1788687047.154122 punch from 0x02c97b to 0x036513 (0x9b99 bytes) 000802 1788687047.162353 fallocate 0x00e0bb thru 0x0100b4 (0x1ffa bytes) 000804 1788687047.166987 mapread 0x01206c thru 0x01e275 (0xc20a bytes) 000805 1788687047.189418 mapwrite 0x00bc99 thru 0x018268 (0xc5d0 bytes) 000806 1788687047.216500 punch from 0x019a84 to 0x01c0f3 (0x2670 bytes) 000807 1788687047.225841 punch from 0x0183b7 to 0x0277d2 (0xf41c bytes) 000808 1788687047.231946 trunc from 0x0386b2 to 0x03db7b (0x54ca bytes) 000809 1788687047.254855 mapread 0x000ac9 thru 0x007bd3 (0x710b bytes) 000810 1788687047.257680 write 0x03a6d8 thru 0x03ffff (0x5928 bytes) 000811 1788687047.267198 punch from 0x02e499 to 0x03de11 (0xf979 bytes) 000812 1788687047.282581 mapread 0x010bdd thru 0x01c027 (0xb44b bytes) 000813 1788687047.319338 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xbc54bdf4 (stale) Mirror 8: 0xbc54bdf4 (stale) Mirror 10: 0xbc54bdf4 (stale) Mirror 11: 0xbc54bdf4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xbc54bdf4 (stale) Mirror 8: 0xbc54bdf4 (stale) Mirror 10: 0xbc54bdf4 (stale) Mirror 11: 0xbc54bdf4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000814 1788687047.640821 trunc from 0x00114b to 0x00a673 (0x9529 bytes) 000815 1788687047.646936 write 0x00213f thru 0x00b790 (0x9652 bytes) 000816 1788687047.662464 punch from 0x018dd9 to 0x023cc0 (0xaee8 bytes) 000818 1788687047.671118 write 0x015a89 thru 0x0247af (0xed27 bytes) 000819 1788687047.681778 write 0x01acdc thru 0x02750c (0xc831 bytes) 000820 1788687047.687877 trunc from 0x014113 to 0x01f9d0 (0xb8be bytes) 000821 1788687047.715771 read 0x02149d thru 0x024a71 (0x35d5 bytes) 000822 1788687047.720480 MIRROR_RESYNC 5 mirrors 000823 1788687049.220328 read 0x03324a thru 0x03ffff (0xcdb6 bytes) 000825 1788687049.229097 read 0x03fde6 thru 0x03ffff (0x21a bytes) 000826 1788687049.230493 trunc from 0x040000 to 0x00da6b (0x32594 bytes) 000827 1788687049.279984 write 0x01edd8 thru 0x02b56f (0xc798 bytes) 000828 1788687049.286146 mapwrite 0x002242 thru 0x0080ae (0x5e6d bytes) 000830 1788687049.344793 read 0x026409 thru 0x026df3 (0x9eb bytes) 000832 1788687049.349817 MIRROR_RESYNC 5 mirrors 000833 1788687050.730959 fallocate 0x02d300 thru 0x038ba5 (0xb8a6 bytes) 000834 1788687050.766136 trunc from 0x01d2f7 to 0x02844f (0xb159 bytes) 000835 1788687050.774603 trunc from 0x038ba6 to 0x02ac2e (0xdf77 bytes) 000836 1788687050.795206 trunc from 0x02ac2f to 0x02f8fa (0x4ccc bytes) 000837 1788687050.814329 punch from 0x018400 to 0x022a45 (0xa646 bytes) 000838 1788687050.820302 mapread 0x01adab thru 0x0294c9 (0xe71f bytes) 000839 1788687050.848644 mapwrite 0x024a2c thru 0x02b4f2 (0x6ac7 bytes) 000840 1788687050.871802 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2f8fb): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f8fb): Mirror 6: 0x9fa55b5 (stale) Mirror 8: 0x9fa55b5 (stale) Mirror 10: 0x9fa55b5 (stale) Mirror 11: 0x9fa55b5 (stale) CRC-32 checksum value for chunk [0, 0x2f8fb): Mirror 1: 0x64e104da Mirror 12: 0x64e104da lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f8fb: skipped 000841 1788687051.844109 mapread 0x00179a thru 0x00dc08 (0xc46f bytes) 000842 1788687051.873798 mapread 0x005db8 thru 0x01036b (0xa5b4 bytes) 000843 1788687051.886717 mapwrite 0x0235e3 thru 0x030df5 (0xd813 bytes) 000844 1788687051.974187 punch from 0x0008b8 to 0x00bb9a (0xb2e3 bytes) 000845 1788687051.979830 write 0x02ea19 thru 0x0387ba (0x9da2 bytes) 000846 1788687051.982151 mapwrite 0x01b6fb thru 0x0203df (0x4ce5 bytes) 000847 1788687052.018306 write 0x02e19c thru 0x036669 (0x84ce bytes) 000848 1788687052.020258 mapread 0x003725 thru 0x008932 (0x520e bytes) 000849 1788687052.076459 mapread 0x01fa48 thru 0x027a74 (0x802d bytes) 000850 1788687052.078143 read 0x030a5d thru 0x037fd5 (0x7579 bytes) 000851 1788687052.079132 punch from 0x029f60 to 0x031680 (0x7721 bytes) 000852 1788687052.140078 trunc from 0x024612 to 0x026cd0 (0x26bf bytes) 000853 1788687052.151920 mapwrite 0x02efe7 thru 0x038c0c (0x9c26 bytes) 000854 1788687052.225821 mapread 0x01b049 thru 0x01f055 (0x400d bytes) 000855 1788687052.231481 fallocate 0x001950 thru 0x008494 (0x6b45 bytes) 000856 1788687052.249195 fallocate 0x038973 thru 0x03ffff (0x768d bytes) 000857 1788687052.260092 trunc from 0x038c0d to 0x0248e8 (0x14324 bytes) 000859 1788687052.314001 mapwrite 0x00b168 thru 0x018111 (0xcfaa bytes) 000860 1788687052.373519 fallocate 0x030210 thru 0x03ef8a (0xed7b bytes) 000861 1788687052.386946 write 0x037ffb thru 0x03ffff (0x8005 bytes) 000863 1788687052.397593 mapwrite 0x036f90 thru 0x03ffff (0x9070 bytes) 000864 1788687052.431107 fallocate 0x0016e0 thru 0x00aec6 (0x97e7 bytes) 000866 1788687052.436452 mapwrite 0x0166e3 thru 0x021377 (0xac95 bytes) 000867 1788687052.478475 read 0x03b7cf thru 0x03bfa4 (0x7d6 bytes) 000868 1788687052.479533 mapread 0x02a63c thru 0x038a4e (0xe413 bytes) 000869 1788687052.491544 mapread 0x0187fa thru 0x01997b (0x1182 bytes) 000870 1788687052.492519 mapread 0x003b0c thru 0x00ef70 (0xb465 bytes) 000871 1788687052.493756 mapwrite 0x030a5e thru 0x030d1b (0x2be bytes) 000872 1788687052.513478 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x5e0d4259 (stale) Mirror 8: 0x5e0d4259 (stale) Mirror 10: 0x5e0d4259 (stale) Mirror 11: 0x5e0d4259 (stale) Mirror 12: 0xf38a1b5a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x44a8c32e Mirror 13: 0x44a8c32e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000873 1788687053.692065 mapwrite 0x0344bb thru 0x037d8f (0x38d5 bytes) 000874 1788687053.858309 trunc from 0x040000 to 0x02fec3 (0x1013c bytes) 000875 1788687053.902754 mapread 0x00b571 thru 0x0193fe (0xde8e bytes) 000876 1788687053.924202 trunc from 0x027e6d to 0x02ec5e (0x6df2 bytes) 000877 1788687053.939562 mapread 0x0026dc thru 0x00bbf7 (0x951c bytes) 000878 1788687053.953435 punch from 0x0249b7 to 0x02be62 (0x74ac bytes) 000879 1788687053.957365 mapread 0x00733f thru 0x00a63d (0x32ff bytes) 000880 1788687053.960312 mapwrite 0x032dbf thru 0x03b7b4 (0x89f6 bytes) 000881 1788687054.044425 fallocate 0x010d4b thru 0x01af02 (0xa1b8 bytes) 000882 1788687054.049110 mapwrite 0x01caad thru 0x02b001 (0xe555 bytes) 000883 1788687054.101274 trunc from 0x01e861 to 0x02bb86 (0xd326 bytes) 000884 1788687054.107773 mapread 0x03034a thru 0x039592 (0x9249 bytes) 000885 1788687054.111283 trunc from 0x01499e to 0x01cd77 (0x83da bytes) 000886 1788687054.116758 mapwrite 0x03ef1c thru 0x03ffff (0x10e4 bytes) 000887 1788687054.167795 trunc from 0x0364ab to 0x03ffff (0x9b55 bytes) 000888 1788687054.173920 mapread 0x005262 thru 0x01353d (0xe2dc bytes) 000889 1788687054.176431 write 0x005bc4 thru 0x015139 (0xf576 bytes) 000890 1788687054.184023 mapwrite 0x004ddf thru 0x010434 (0xb656 bytes) 000891 1788687054.221242 punch from 0x0231fc to 0x0287ca (0x55cf bytes) 000892 1788687054.235619 MIRROR_RESYNC 7 mirrors 000893 1788687057.068973 write 0x02a390 thru 0x03104f (0x6cc0 bytes) 000894 1788687057.136791 write 0x02ff54 thru 0x0380b2 (0x815f bytes) 000895 1788687057.142259 trunc from 0x037dc1 to 0x03ffff (0x823f bytes) 000896 1788687057.176440 fallocate 0x03922b thru 0x03ffff (0x6dd5 bytes) 000897 1788687057.199514 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x42358c96 (stale) Mirror 8: 0x42358c96 (stale) Mirror 10: 0x42358c96 (stale) Mirror 11: 0x42358c96 (stale) Mirror 12: 0x42358c96 (stale) Mirror 13: 0x42358c96 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc889cc46 Mirror 14: 0xc889cc46 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000898 1788687058.458831 mapwrite 0x02238d thru 0x031c90 (0xf904 bytes) 000899 1788687058.715329 read 0x01eb9f thru 0x029f8a (0xb3ec bytes) 000900 1788687058.718603 mapread 0x017f08 thru 0x027eda (0xffd3 bytes) 000901 1788687058.741372 trunc from 0x031b5c to 0x035acf (0x3f74 bytes) 000903 1788687058.753746 write 0x0059e6 thru 0x00f12b (0x9746 bytes) 000904 1788687058.797364 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x42358c96 (stale) Mirror 8: 0x42358c96 (stale) Mirror 10: 0x42358c96 (stale) Mirror 11: 0x42358c96 (stale) Mirror 12: 0x42358c96 (stale) Mirror 13: 0x42358c96 (stale) Mirror 14: 0xc889cc46 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd95dc99c Mirror 15: 0xd95dc99c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000905 1788687060.193048 write 0x03ab14 thru 0x03ffff (0x54ec bytes) 000906 1788687060.247633 mapread 0x036a83 thru 0x03e3d5 (0x7953 bytes) 000907 1788687060.271476 punch from 0x02261e to 0x0240fd (0x1ae0 bytes) 000908 1788687060.284437 mapread 0x015530 thru 0x015a28 (0x4f9 bytes) 000909 1788687060.304579 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x42358c96 (stale) Mirror 8: 0x42358c96 (stale) Mirror 10: 0x42358c96 (stale) Mirror 11: 0x42358c96 (stale) Mirror 12: 0x42358c96 (stale) Mirror 13: 0x42358c96 (stale) Mirror 14: 0xc889cc46 (stale) Mirror 15: 0xd95dc99c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf12cae63 Mirror 16: 0xf12cae63 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000910 1788687061.818530 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x42358c96 (stale) Mirror 8: 0x42358c96 (stale) Mirror 10: 0x42358c96 (stale) Mirror 12: 0x42358c96 (stale) Mirror 13: 0x42358c96 (stale) Mirror 14: 0xc889cc46 (stale) Mirror 15: 0xd95dc99c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x42358c96 (stale) Mirror 8: 0x42358c96 (stale) Mirror 10: 0x42358c96 (stale) Mirror 12: 0x42358c96 (stale) Mirror 13: 0x42358c96 (stale) Mirror 14: 0xc889cc46 (stale) Mirror 15: 0xd95dc99c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf12cae63 Mirror 16: 0xf12cae63 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000911 1788687062.852265 mapread 0x01eff7 thru 0x0238d6 (0x48e0 bytes) 000912 1788687062.895626 write 0x00c1f8 thru 0x015930 (0x9739 bytes) 000913 1788687062.946850 write 0x036fc8 thru 0x03ffff (0x9038 bytes) 000914 1788687062.953883 read 0x0362a4 thru 0x03826a (0x1fc7 bytes) 000915 1788687062.964995 MIRROR_RESYNC 9 mirrors 000917 1788687066.053826 trunc from 0x040000 to 0x033530 (0xcacf bytes) 000918 1788687066.108351 mapread 0x005466 thru 0x00fe8b (0xaa26 bytes) 000919 1788687066.130846 mapwrite 0x038bb1 thru 0x03f38b (0x67db bytes) 000920 1788687066.214310 read 0x0361fe thru 0x03a9a2 (0x47a5 bytes) 000921 1788687066.214937 trunc from 0x03f38c to 0x00a8a8 (0x34ae3 bytes) 000922 1788687066.239340 trunc from 0x00a8a9 to 0x02a059 (0x1f7b1 bytes) 000923 1788687066.262217 read 0x017e1d thru 0x01e48d (0x6671 bytes) 000924 1788687066.280596 MIRROR_RESYNC 9 mirrors 000925 1788687068.172734 write 0x0188d0 thru 0x01d23a (0x496b bytes) 000926 1788687068.220735 fallocate 0x001a99 thru 0x004965 (0x2ecd bytes) 000927 1788687068.226763 fallocate 0x01bef8 thru 0x01d4be (0x15c7 bytes) 000928 1788687068.233103 mapread 0x00686d thru 0x00bcbb (0x544f bytes) 000929 1788687068.253088 trunc from 0x02a05a to 0x003bf3 (0x26466 bytes) 000930 1788687068.276310 read 0x0025b9 thru 0x003bf3 (0x163b bytes) 000931 1788687068.278035 punch from 0x001497 to 0x003bf3 (0x275d bytes) 000932 1788687068.284609 trunc from 0x000d93 to 0x003bf3 (0x2e61 bytes) 000933 1788687068.292429 punch from 0x000383 to 0x0004f1 (0x16f bytes) 000934 1788687068.301560 fallocate 0x01ec9f thru 0x0240be (0x5420 bytes) 000935 1788687068.311884 punch from 0x012919 to 0x0218b8 (0xefa0 bytes) 000936 1788687068.319150 trunc from 0x015632 to 0x020be7 (0xb5b6 bytes) 000937 1788687068.325289 read 0x00ee03 thru 0x00f91a (0xb18 bytes) 000938 1788687068.341172 trunc from 0x0240bf to 0x015058 (0xf066 bytes) 000939 1788687068.356314 mapwrite 0x009ce1 thru 0x00be4e (0x216e bytes) 000940 1788687068.370811 trunc from 0x005ab0 to 0x015058 (0xf5a9 bytes) 000941 1788687068.380493 write 0x03ed68 thru 0x03ffff (0x1298 bytes) 000942 1788687068.382385 trunc from 0x040000 to 0x019c6c (0x26393 bytes) 000943 1788687068.401751 write 0x02da51 thru 0x02e815 (0xdc5 bytes) 000944 1788687068.404230 mapwrite 0x01c506 thru 0x02852b (0xc026 bytes) 000945 1788687068.502515 read 0x001650 thru 0x00c93f (0xb2f0 bytes) 000946 1788687068.518830 trunc from 0x02e816 to 0x023c82 (0xab93 bytes) 000947 1788687068.555798 punch from 0x00c356 to 0x01292b (0x65d6 bytes) 000948 1788687068.561136 trunc from 0x023c83 to 0x03f805 (0x1bb83 bytes) 000949 1788687068.573844 punch from 0x02b102 to 0x02fe4e (0x4d4d bytes) 000951 1788687068.579659 mapread 0x01592f thru 0x021122 (0xb7f4 bytes) 000952 1788687068.583863 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3f806): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f806): Mirror 6: 0xba541e29 (stale) Mirror 8: 0xba541e29 (stale) Mirror 10: 0xba541e29 (stale) Mirror 12: 0xba541e29 (stale) Mirror 13: 0xba541e29 (stale) Mirror 14: 0xba541e29 (stale) Mirror 15: 0xba541e29 (stale) Mirror 16: 0xba541e29 (stale) CRC-32 checksum value for chunk [0, 0x3f806): Mirror 1: 0xc400f6c2 Mirror 17: 0xc400f6c2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f806: skipped 000953 1788687069.620119 trunc from 0x03f806 to 0x00af7c (0x34889 bytes) 000954 1788687069.775734 trunc from 0x0030f8 to 0x0094e2 (0x63eb bytes) 000955 1788687069.786220 mapwrite 0x0147d3 thru 0x02262d (0xde5b bytes) 000956 1788687069.878548 mapread 0x011d6f thru 0x0156d4 (0x3966 bytes) 000957 1788687069.880214 read 0x017b22 thru 0x01c14e (0x462d bytes) 000958 1788687069.880853 mapread 0x008541 thru 0x00ec4c (0x670c bytes) 000959 1788687069.892931 fallocate 0x035241 thru 0x03ffff (0xadbf bytes) 000960 1788687069.902849 mapwrite 0x03573b thru 0x03ffff (0xa8c5 bytes) 000962 1788687069.996752 fallocate 0x00e6d4 thru 0x01dda0 (0xf6cd bytes) 000963 1788687070.003263 write 0x021457 thru 0x02c29b (0xae45 bytes) 000964 1788687070.008617 mapread 0x01e792 thru 0x028f98 (0xa807 bytes) 000965 1788687070.009652 trunc from 0x040000 to 0x00598f (0x3a670 bytes) 000966 1788687070.047168 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x5990): Mirror 6: 0x3c3886a2 (stale) Mirror 8: 0x3c3886a2 (stale) Mirror 10: 0x3c3886a2 (stale) Mirror 12: 0x3c3886a2 (stale) Mirror 13: 0x3c3886a2 (stale) Mirror 14: 0x3c3886a2 (stale) Mirror 15: 0x3c3886a2 (stale) Mirror 16: 0x3c3886a2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5990): Mirror 6: 0x3c3886a2 (stale) Mirror 8: 0x3c3886a2 (stale) Mirror 10: 0x3c3886a2 (stale) Mirror 12: 0x3c3886a2 (stale) Mirror 13: 0x3c3886a2 (stale) Mirror 14: 0x3c3886a2 (stale) Mirror 15: 0x3c3886a2 (stale) Mirror 16: 0x3c3886a2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5990: skipped 000967 1788687070.566150 read 0x00334a thru 0x00598f (0x2646 bytes) 000968 1788687070.579635 write 0x03e709 thru 0x03ffff (0x18f7 bytes) 000969 1788687070.593305 punch from 0x03bac8 to 0x03ffff (0x4538 bytes) 000970 1788687070.625436 mapwrite 0x00ae6e thru 0x019193 (0xe326 bytes) 000971 1788687070.725906 trunc from 0x040000 to 0x02f3ba (0x10c45 bytes) 000972 1788687070.755406 fallocate 0x0044b8 thru 0x014057 (0xfba0 bytes) 000973 1788687070.778901 mapread 0x003b74 thru 0x00b0cf (0x755c bytes) 000974 1788687070.779804 trunc from 0x02f3bb to 0x026885 (0x8b35 bytes) 000975 1788687070.796732 punch from 0x00e252 to 0x01c981 (0xe730 bytes) 000976 1788687070.802516 trunc from 0x006144 to 0x007e9f (0x1d5c bytes) 000977 1788687070.807351 write 0x00b72f thru 0x013505 (0x7dd7 bytes) 000978 1788687070.818948 mapread 0x0012b3 thru 0x00bb69 (0xa8b7 bytes) 000979 1788687070.831194 trunc from 0x026886 to 0x03531b (0xea96 bytes) 000980 1788687070.844589 write 0x024673 thru 0x02f3a0 (0xad2e bytes) 000981 1788687070.858336 mapwrite 0x01b08e thru 0x01fb24 (0x4a97 bytes) 000982 1788687070.882824 read 0x015f80 thru 0x01e40b (0x848c bytes) 000984 1788687070.883904 mapread 0x02a74f thru 0x03531b (0xabcd bytes) 000988 1788687070.894976 write 0x036765 thru 0x03ffff (0x989b bytes) 000989 1788687070.897027 punch from 0x000297 to 0x0061c0 (0x5f2a bytes) 000990 1788687070.903107 write 0x001b51 thru 0x00e145 (0xc5f5 bytes) 000991 1788687070.910526 write 0x035262 thru 0x03ffff (0xad9e bytes) 000992 1788687070.919114 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x66b8517d (stale) Mirror 8: 0x66b8517d (stale) Mirror 10: 0x66b8517d (stale) Mirror 12: 0x66b8517d (stale) Mirror 13: 0x66b8517d (stale) Mirror 14: 0x66b8517d (stale) Mirror 15: 0x66b8517d (stale) Mirror 16: 0x66b8517d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x538b35e1 Mirror 17: 0x538b35e1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000993 1788687072.053504 mapread 0x035e37 thru 0x03ffff (0xa1c9 bytes) 000994 1788687072.104472 mapwrite 0x02c76a thru 0x037b2a (0xb3c1 bytes) 000995 1788687072.268218 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x66b8517d (stale) Mirror 8: 0x66b8517d (stale) Mirror 10: 0x66b8517d (stale) Mirror 12: 0x66b8517d (stale) Mirror 13: 0x66b8517d (stale) Mirror 14: 0x66b8517d (stale) Mirror 15: 0x66b8517d (stale) Mirror 16: 0x66b8517d (stale) Mirror 17: 0x538b35e1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6854caf6 Mirror 18: 0x6854caf6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000996 1788687073.666425 trunc from 0x00c5a7 to 0x01aa1d (0xe477 bytes) 000997 1788687073.798829 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x66b8517d (stale) Mirror 8: 0x66b8517d (stale) Mirror 10: 0x66b8517d (stale) Mirror 13: 0x66b8517d (stale) Mirror 14: 0x66b8517d (stale) Mirror 15: 0x66b8517d (stale) Mirror 16: 0x66b8517d (stale) Mirror 17: 0x538b35e1 (stale) Mirror 18: 0x6854caf6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x66b8517d (stale) Mirror 8: 0x66b8517d (stale) Mirror 10: 0x66b8517d (stale) Mirror 13: 0x66b8517d (stale) Mirror 14: 0x66b8517d (stale) Mirror 15: 0x66b8517d (stale) Mirror 16: 0x66b8517d (stale) Mirror 17: 0x538b35e1 (stale) Mirror 18: 0x6854caf6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000998 1788687074.379925 punch from 0x0273a4 to 0x032f7a (0xbbd7 bytes) 000999 1788687074.385801 mapwrite 0x01be7b thru 0x02a6b9 (0xe83f bytes) 001000 1788687074.435558 write 0x008c7d thru 0x00c82d (0x3bb1 bytes) All operations completed A-OK! PASS 70a (92s) == sanity-flr test 71: check mirror extend parent fid ==== 05:31:20 (1788687080) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1cf:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (12s) == sanity-flr test 72: check append on FLR file ========== 05:31:32 (1788687092) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.224563 s, 9.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170822 s, 6.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x117:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 05:31:40 (1788687100) Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre2 Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 925696 .. Extending file size to 1437312 .. failed Extending file size to 1070016 .. Extending file size to 2822208 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3620704 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5802752 .. Extending file size to 3749728 .. Extending file size to 4744064 .. failed Extending file size to 4748224 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6733504 .. Extending file size to 6409536 .. Extending file size to 7865504 .. Extending file size to 7268352 .. Extending file size to 7415520 .. Extending file size to 8761760 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8830848 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 61447 61448 61451 61452 61453 192.168.203.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg329-client.virtnet /mnt/lustre2 (opts:) 192.168.203.129@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg329-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (81s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 05:33:01 (1788687181) Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre2 Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1711104 .. Extending file size to 3425376 .. Extending file size to 6514688 .. Extending file size to 5271872 .. Extending file size to 8188128 .. Extending file size to 7721568 .. Extending file size to 7778624 .. Extending file size to 8925664 .. Extending file size to 8043744 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8088704 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8850848 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9229664 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62957 62958 62960 62962 62963 done 192.168.203.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg329-client.virtnet /mnt/lustre2 (opts:) 192.168.203.129@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg329-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (83s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 05:34:24 (1788687264) Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2276 1269028 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2192 1269112 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25688 3581332 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17868 3589152 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43556 7170484 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2276 1269028 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2192 1269112 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25688 3581332 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17868 3589152 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43556 7170484 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.110522 s, 19.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.159927 s, 13.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x3f5fb799 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg329-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 05:34:41 (1788687281) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.190535 s, 11.0 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 05:34:49 (1788687289) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x63:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 05:34:57 (1788687297) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0929731 s, 22.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 05:35:05 (1788687305) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123662 s, 17.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 05:35:14 (1788687314) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104216 s, 20.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } PASS 204c (31s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 05:35:45 (1788687345) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg329-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797077 s, 13.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x250:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x251:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x250:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x250:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x251:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x250:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x251:0x0] } PASS 204d (48s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 05:36:33 (1788687393) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg329-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.120129 s, 17.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (49s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 05:37:22 (1788687442) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg329-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg329-server: Pool lustre.test_204f created Adding targets to pool oleg329-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg329-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0922133 s, 22.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg329-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg329-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg329-server: Pool lustre.test_204f destroyed PASS 204f (73s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 05:38:35 (1788687515) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 05:38:43 (1788687523) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 05:38:50 (1788687530) oleg329-server: Pool lustre.test_206 created Adding targets to pool oleg329-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg329-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg329-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg329-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg329-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 05:39:17 (1788687557) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730339 s, 14.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601084 s, 17.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xfa302585 Mirror 3: 0xfa302585 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 05:39:28 (1788687568) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 05:39:32 (1788687572) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 05:39:36 (1788687576) Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.051585 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x16b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1e90000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1e90000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1e90000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1e30000 Read this (4 bytes): BBBB 192.168.203.129@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg329-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 05:40:14 (1788687614) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.01886 s, 434 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 05:40:23 (1788687623) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482034 s, 21.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x84 lmm_fid: [0x200000405:0x84:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x84 lmm_fid: [0x200000405:0x84:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13f:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611268 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 05:40:24.000000000 -0400 Modify: 2026-09-06 05:40:24.000000000 -0400 Change: 2026-09-06 05:40:26.000000000 -0400 Birth: 2026-09-06 05:40:24.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 05:40:32 (1788687632) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470696 s, 22.3 MB/s oleg329-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg329-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg329-server: O/280000401/d29/13d: File not found by ext2_lookup PASS 210b (20s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 05:40:53 (1788687653) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.216701 s, 189 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0272441 s, 150 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 6 05:40 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x89:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 6 05:40 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 05:41:00 (1788687660) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 05:41:07 (1788687667) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (7s) == sanity-flr test complete, duration 1629 sec =========== 05:41:14 (1788687674) === sanity-flr: start cleanup 05:41:16 (1788687676) === === sanity-flr: finish cleanup 05:41:24 (1788687684) ===