-----============= acceptance-small: sanity-flr ============----- Sat Sep 5 19:59:09 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg413-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 19:59:30 (1788652770) === oleg413-client.virtnet: executing check_config_client /mnt/lustre oleg413-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg413-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff892f493af000.idle_timeout=debug osc.lustre-OST0001-osc-ffff892f493af000.idle_timeout=debug disable quota as required oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 20:00:12 (1788652812) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8206] == sanity-flr test 0a: lfs mirror create with -N option == 20:00:15 (1788652815) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:00:34 (1788652834) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:00:38 (1788652838) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 20:00:41 (1788652841) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:01:06 (1788652866) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:01:09 (1788652869) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 20:01:14 (1788652874) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff892f493af000.stats=clear osc.lustre-OST0001-osc-ffff892f493af000.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:01:21 (1788652881) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:01:31 (1788652891) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:01:38 (1788652898) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 20:01:42 (1788652902) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 20:01:51 (1788652911) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 20:02:01 (1788652921) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:02:08 (1788652928) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:02:18 (1788652938) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:02:28 (1788652948) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 20:02:40 (1788652960) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:02:47 (1788652967) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 1.35462 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056724 s, 18.5 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:02:57 (1788652977) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.315834 s, 29.9 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 20:03:06 (1788652986) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:03:18 (1788652998) Stopping /mnt/lustre-ost1 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff892f493af000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff892f493af000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg413-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff892f493af000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff892f493af000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg413-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (79s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:04:37 (1788653077) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.647726 s, 17.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff892f493af000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff892f493af000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg413-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (45s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:05:22 (1788653122) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff892f493af000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff892f493af000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg413-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff892f493af000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff892f493af000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg413-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (86s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:06:49 (1788653209) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.72151 s, 19.3 MB/s -rw-r--r-- 1 root root 52428800 Sep 5 20:06 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.972s user 0m0.000s sys 0m0.197s Stopping /mnt/lustre-ost1 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff892f493af000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff892f493af000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.254s user 0m0.000s sys 0m0.168s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg413-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff892f493af000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff892f493af000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.213s user 0m0.000s sys 0m0.180s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg413-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg413-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff892f493af000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff892f493af000.ost_server_uuid in FULL state after 0 sec PASS 33b (96s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:08:25 (1788653305) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:08:28 (1788653308) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:08:32 (1788653312) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 20:08:36 (1788653316) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 20:08:43 (1788653323) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.578363 s, 21.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:08:55 (1788653335) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284479 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220865 s, 47.5 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:09:02 (1788653342) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:09:10 (1788653350) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00176451 s, 580 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00118657 s, 863 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000777243 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00179869 s, 569 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 20:09:19 (1788653359) create test files with size 15733343 .. 8c83b9b3058abc7d142f4a5367787e57 - f383cf9e5133526000def57eb87ffc78 - bb106c5c4511d22cc47db42d38c05c41 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg413-client.virtnet /mnt/lustre (opts:) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 20:09:44 (1788653384) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:09:58 (1788653398) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:10:04 (1788653404) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709095 s, 14.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277505 s, 37.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 20:10:14 (1788653414) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.112122 s, 18.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.312744 s, 13.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.203297 s, 10.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 20:10:32 (1788653432) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 20:10:35 (1788653435) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:10:39 (1788653439) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301981 s, 34.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 20:10:48 (1788653448) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.447987 s, 7.0 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:10:59 (1788653459) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xca:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:11:12 (1788653472) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.30191 s, 34.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg413-client.virtnet /mnt/lustre (opts:) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:11:23 (1788653483) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff892f49532000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.456525 s, 23.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg413-client.virtnet /mnt/lustre (opts:) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff892f50410000.xattr_cache=1 PASS 44e (18s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:11:41 (1788653501) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (40s) == sanity-flr test 46: Verify setstripe --copy option ==== 20:12:21 (1788653541) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 20:13:02 (1788653582) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 20:13:06 (1788653586) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.262128 s, 12.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.15356 s, 20.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xa74a52 (nosync,stale) Mirror 2: 0xa74a52 (nosync,stale) ** verify mirror 2 doesn't change original checksum: e8cd5f0e56bc036c2b4a2e561c69c30e - mirror 1 checksum: 68cd46ed5303e339c8fe548a9fdbea56 - mirror 2 checksum: e8cd5f0e56bc036c2b4a2e561c69c30e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-05T20:13:10-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-05T20:13:10-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: e8cd5f0e56bc036c2b4a2e561c69c30e - mirror 1 checksum: 68cd46ed5303e339c8fe548a9fdbea56 - mirror 2 checksum: 68cd46ed5303e339c8fe548a9fdbea56 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-05T20:13:10-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-05T20:13:10-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 20:13:17 (1788653597) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055907 s, 18.8 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:13:24 (1788653604) Failing mds1 on oleg413-server Stopping /mnt/lustre-mds1 (opts:) on oleg413-server 20:13:36 (1788653616) shut down facet: mds1 facet_host: oleg413-server facet_failover_host: oleg413-server Failover mds1 to oleg413-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-MDT0000 20:14:09 (1788653649) targets are mounted 20:14:09 (1788653649) facet_failover done oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214536 s, 48.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd8:0x0] } PASS 50A (61s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:14:25 (1788653665) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0521669 s, 393 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0140077 s, 1.5 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:14:36 (1788653676) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0700575 s, 292 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00897538 s, 2.3 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:14:46 (1788653686) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0211836 s, 773 kB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:14:56 (1788653696) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0317654 s, 645 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:15:04 (1788653704) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0111397 s, 368 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0220891 s, 185 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:15:13 (1788653713) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:15:34 (1788653734) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:16:03 (1788653763) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x48:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:16:28 (1788653788) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:16:54 (1788653814) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 20:17:02 (1788653822) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 57576 mirror split pid 57577 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57576 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57577 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 20:18:11 (1788653891) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 3893 skipping zero size read 000004 1788653894.296284 write 0x01e897 thru 0x022893 (0x3ffd bytes) 000006 1788653894.307398 read 0x01070b thru 0x01bd32 (0xb628 bytes) zero_range to largest ever: 0x1d657 000007 1788653894.325137 trunc from 0x00f320 to 0x01d656 (0xe337 bytes) 000008 1788653894.347893 mapwrite 0x0061df thru 0x0102ad (0xa0cf bytes) 000009 1788653894.409918 write 0x039f34 thru 0x03ffff (0x60cc bytes) 000010 1788653894.412144 mapwrite 0x0020b7 thru 0x00ccf7 (0xac41 bytes) fallocating to largest ever: 0x32888 000012 1788653894.492036 fallocate 0x022956 thru 0x032887 (0xff32 bytes) 000013 1788653894.501877 mapread 0x00eefc thru 0x0192d3 (0xa3d8 bytes) 000014 1788653894.519636 punch from 0x00fec8 to 0x014cf8 (0x4e31 bytes) 000015 1788653894.529803 write 0x0114c1 thru 0x0181a3 (0x6ce3 bytes) 000016 1788653894.536923 mapread 0x03af06 thru 0x03ffff (0x50fa bytes) 000019 1788653894.550867 MIRROR_EXTEND to 2 mirrors 000020 1788653895.645398 trunc from 0x040000 to 0x005a95 (0x3a56a bytes) 000021 1788653895.704243 fallocate 0x003ade thru 0x00c86b (0x8d8e bytes) 000022 1788653895.711870 write 0x006c13 thru 0x012223 (0xb611 bytes) 000024 1788653895.723880 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) 000025 1788653895.975481 read 0x01008e thru 0x012223 (0x2196 bytes) fallocating to largest ever: 0x3fecd 000026 1788653895.984026 fallocate 0x032c04 thru 0x03fecc (0xd2c9 bytes) 000027 1788653895.992029 write 0x022b31 thru 0x02b6c6 (0x8b96 bytes) 000028 1788653896.014643 trunc from 0x01f7f6 to 0x02c793 (0xcf9e bytes) 000029 1788653896.040339 mapread 0x039a5d thru 0x03f134 (0x56d8 bytes) 000030 1788653896.073951 mapread 0x01264a thru 0x01920c (0x6bc3 bytes) 000031 1788653896.096169 write 0x01080c thru 0x011f7c (0x1771 bytes) 000032 1788653896.097083 mapwrite 0x032117 thru 0x03c84e (0xa738 bytes) 000033 1788653896.153563 fallocate 0x037242 thru 0x03f69c (0x845b bytes) 000034 1788653896.163927 mapread 0x01159c thru 0x012d29 (0x178e bytes) 000035 1788653896.164930 trunc from 0x02621b to 0x029fc5 (0x3dab bytes) 000036 1788653896.180647 punch from 0x005b51 to 0x0105d4 (0xaa84 bytes) 000037 1788653896.193572 mapread 0x0024d6 thru 0x004836 (0x2361 bytes) 000038 1788653896.206440 read 0x03508a thru 0x03fecc (0xae43 bytes) 000039 1788653896.207098 mapread 0x03e82c thru 0x03fecc (0x16a1 bytes) 000040 1788653896.207831 mapread 0x0154ba thru 0x0228a5 (0xd3ec bytes) 000041 1788653896.236753 read 0x0372ac thru 0x03d7c2 (0x6517 bytes) 000042 1788653896.238107 mapwrite 0x01c428 thru 0x02071d (0x42f6 bytes) 000043 1788653896.274487 punch from 0x035081 to 0x03fecc (0xae4c bytes) 000044 1788653896.286389 trunc from 0x03fecd to 0x0202d5 (0x1fbf7 bytes) 000045 1788653896.313326 fallocate 0x017905 thru 0x01d659 (0x5d55 bytes) 000046 1788653896.324578 trunc from 0x018548 to 0x019d0b (0x17c4 bytes) 000047 1788653896.334631 write 0x025abe thru 0x027e5a (0x239d bytes) 000048 1788653896.335708 write 0x020640 thru 0x022019 (0x19da bytes) 000049 1788653896.351472 punch from 0x01c802 to 0x027e5a (0xb659 bytes) 000050 1788653896.379430 write 0x020b1c thru 0x02ae72 (0xa357 bytes) 000051 1788653896.386951 punch from 0x01170a to 0x0160a8 (0x499f bytes) fallocating to largest ever: 0x40000 000052 1788653896.398977 fallocate 0x03d858 thru 0x03ffff (0x27a8 bytes) 000054 1788653896.413645 punch from 0x01e45c to 0x0205ef (0x2194 bytes) 000055 1788653896.444697 read 0x019210 thru 0x027c54 (0xea45 bytes) 000056 1788653896.469870 mapread 0x014d52 thru 0x0221f7 (0xd4a6 bytes) 000057 1788653896.472321 write 0x0372e6 thru 0x03ffff (0x8d1a bytes) 000058 1788653896.474780 mapwrite 0x02e1a0 thru 0x02eba3 (0xa04 bytes) 000059 1788653896.519466 trunc from 0x040000 to 0x03890f (0x76f0 bytes) 000060 1788653896.545001 mapwrite 0x03ce33 thru 0x03ffff (0x31cd bytes) 000061 1788653896.675340 write 0x02724d thru 0x028819 (0x15cd bytes) 000062 1788653896.677907 read 0x00f4b0 thru 0x010b2f (0x1680 bytes) 000063 1788653896.678669 write 0x0164c1 thru 0x016e62 (0x9a2 bytes) 000064 1788653896.690209 mapread 0x00450a thru 0x00be90 (0x7987 bytes) 000065 1788653896.691496 mapwrite 0x000f1a thru 0x00c3d5 (0xb4bc bytes) 000067 1788653896.737857 MIRROR_EXTEND to 2 mirrors 000068 1788653898.028010 trunc from 0x00916d to 0x01528f (0xc123 bytes) 000069 1788653898.086310 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) 000070 1788653898.248355 mapwrite 0x0205c2 thru 0x02539d (0x4ddc bytes) 000071 1788653898.312667 punch from 0x009b1f to 0x013d25 (0xa207 bytes) 000072 1788653898.321905 write 0x00a0d1 thru 0x015a03 (0xb933 bytes) 000073 1788653898.345217 mapread 0x002438 thru 0x006702 (0x42cb bytes) 000074 1788653898.363017 read 0x01841f thru 0x022519 (0xa0fb bytes) 000075 1788653898.372319 mapwrite 0x00718a thru 0x009679 (0x24f0 bytes) 000076 1788653898.401822 trunc from 0x00789b to 0x00fe1c (0x8582 bytes) 000077 1788653898.414540 punch from 0x03eb0f to 0x03ffff (0x14f1 bytes) 000078 1788653898.424478 punch from 0x00106c to 0x00a1a4 (0x9139 bytes) 000079 1788653898.435058 trunc from 0x005698 to 0x00658e (0xef7 bytes) 000080 1788653898.446769 trunc from 0x01834e to 0x01e603 (0x62b6 bytes) 000081 1788653898.454860 trunc from 0x040000 to 0x0014d0 (0x3eb2f bytes) 000082 1788653898.488748 mapread 0x0000a2 thru 0x0014d0 (0x142f bytes) 000083 1788653898.498769 mapwrite 0x0141f8 thru 0x0237cd (0xf5d6 bytes) 000084 1788653898.576171 trunc from 0x0005c5 to 0x009d83 (0x97bf bytes) 000085 1788653898.587304 mapwrite 0x012252 thru 0x01db6f (0xb91e bytes) 000086 1788653898.620565 mapread 0x01a5d1 thru 0x022545 (0x7f75 bytes) 000087 1788653898.621518 trunc from 0x0237ce to 0x010b18 (0x12cb5 bytes) 000088 1788653898.640621 read 0x00bc7a thru 0x010b18 (0x4e9f bytes) 000089 1788653898.656409 read 0x0059e5 thru 0x010b18 (0xb134 bytes) 000090 1788653898.659608 read 0x000d29 thru 0x001961 (0xc39 bytes) 000092 1788653898.661746 punch from 0x0055f3 to 0x0059ea (0x3f8 bytes) 000093 1788653898.669725 punch from 0x00f840 to 0x010b18 (0x12d9 bytes) 000094 1788653898.674749 mapwrite 0x03d959 thru 0x03ffff (0x26a7 bytes) 000095 1788653898.729329 mapread 0x03fb21 thru 0x03ffff (0x4df bytes) 000096 1788653898.731624 mapwrite 0x03f290 thru 0x03ffff (0xd70 bytes) 000097 1788653898.755098 fallocate 0x01be27 thru 0x021880 (0x5a5a bytes) 000098 1788653898.762413 read 0x03beb3 thru 0x03ffff (0x414d bytes) 000100 1788653898.763449 punch from 0x023166 to 0x0277fa (0x4695 bytes) 000102 1788653898.773658 write 0x012116 thru 0x02191c (0xf807 bytes) 000103 1788653898.778644 mapwrite 0x018c07 thru 0x0215a9 (0x89a3 bytes) 000105 1788653898.817369 mapread 0x020644 thru 0x0219f2 (0x13af bytes) 000106 1788653898.820131 trunc from 0x040000 to 0x00aa25 (0x355da bytes) 000107 1788653898.882574 MIRROR_EXTEND to 2 mirrors 000108 1788653899.961628 punch from 0x00071b to 0x00aa25 (0xa30b bytes) 000109 1788653900.043021 write 0x016153 thru 0x016b38 (0x9e6 bytes) 000110 1788653900.065904 MIRROR_RESYNC 2 mirrors 000111 1788653900.942219 MIRROR_EXTEND to 3 mirrors 000112 1788653901.852338 trunc from 0x00e156 to 0x016b38 (0x89e3 bytes) 000113 1788653901.878954 read 0x000a75 thru 0x00a86c (0x9df8 bytes) 000114 1788653901.893189 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16b39): Mirror 3: 0xc583acb3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16b39): Mirror 3: 0xc583acb3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16b39: skipped 000115 1788653902.177432 mapwrite 0x0221b2 thru 0x02d657 (0xb4a6 bytes) 000116 1788653902.257544 mapread 0x0064d7 thru 0x0129e9 (0xc513 bytes) 000117 1788653902.284458 mapread 0x017726 thru 0x0275d8 (0xfeb3 bytes) 000118 1788653902.293895 punch from 0x02c423 to 0x02d657 (0x1235 bytes) 000119 1788653902.299954 read 0x0220f4 thru 0x028879 (0x6786 bytes) 000120 1788653902.302969 trunc from 0x012dd1 to 0x013d7a (0xfaa bytes) 000121 1788653902.309587 fallocate 0x02da5c thru 0x03d874 (0xfe19 bytes) 000122 1788653902.316919 mapwrite 0x00d1f0 thru 0x01b27c (0xe08d bytes) 000123 1788653902.359738 write 0x002920 thru 0x00357a (0xc5b bytes) 000124 1788653902.368633 fallocate 0x008af3 thru 0x00eca0 (0x61ae bytes) 000125 1788653902.388792 write 0x0220cb thru 0x03091d (0xe853 bytes) 000126 1788653902.400864 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) 000127 1788653902.625989 trunc from 0x014114 to 0x01607a (0x1f67 bytes) 000128 1788653902.634770 write 0x0026e8 thru 0x00b9da (0x92f3 bytes) 000129 1788653902.662746 trunc from 0x003342 to 0x012d6d (0xfa2c bytes) 000130 1788653902.693190 mapread 0x01d96c thru 0x022b40 (0x51d5 bytes) 000131 1788653902.736524 trunc from 0x02afd7 to 0x03091d (0x5947 bytes) 000132 1788653902.742752 fallocate 0x021640 thru 0x024e3f (0x3800 bytes) 000134 1788653902.752024 trunc from 0x03091e to 0x0050a8 (0x2b875 bytes) 000135 1788653902.781602 punch from 0x0030bd to 0x003c80 (0xbc4 bytes) 000136 1788653902.789253 trunc from 0x001249 to 0x0050a8 (0x3e60 bytes) 000137 1788653902.798108 punch from 0x000b3f to 0x0050a8 (0x456a bytes) 000138 1788653902.811473 fallocate 0x0031bf thru 0x00fd77 (0xcbb9 bytes) 000139 1788653902.817097 trunc from 0x003eb2 to 0x0050a8 (0x11f7 bytes) 000140 1788653902.821417 mapwrite 0x0135dc thru 0x01899a (0x53bf bytes) 000141 1788653902.865318 read 0x001d7b thru 0x0113fc (0xf682 bytes) 000142 1788653902.878236 fallocate 0x03389d thru 0x03b4be (0x7c22 bytes) 000143 1788653902.882745 write 0x009886 thru 0x00a0fb (0x876 bytes) 000144 1788653902.883786 trunc from 0x007e7c to 0x00db97 (0x5d1c bytes) 000145 1788653902.893789 mapwrite 0x013bf1 thru 0x0224e3 (0xe8f3 bytes) 000146 1788653902.939699 punch from 0x00edfc to 0x013a85 (0x4c8a bytes) 000147 1788653902.945419 mapwrite 0x038b8f thru 0x03ffff (0x7471 bytes) 000149 1788653903.037144 fallocate 0x018608 thru 0x01c257 (0x3c50 bytes) 000150 1788653903.044333 fallocate 0x009b29 thru 0x00b1b4 (0x168c bytes) 000151 1788653903.065686 MIRROR_EXTEND to 2 mirrors 000152 1788653904.375517 mapwrite 0x01b200 thru 0x02abe0 (0xf9e1 bytes) 000154 1788653904.524235 trunc from 0x040000 to 0x01e698 (0x21967 bytes) 000155 1788653904.562435 fallocate 0x03a9b8 thru 0x03eb66 (0x41af bytes) 000156 1788653904.576465 write 0x013df4 thru 0x022434 (0xe641 bytes) 000157 1788653904.586135 trunc from 0x0205a0 to 0x022434 (0x1e95 bytes) 000158 1788653904.603040 punch from 0x01b931 to 0x022434 (0x6b04 bytes) 000159 1788653904.609675 write 0x00e759 thru 0x01a34e (0xbbf6 bytes) 000160 1788653904.616478 trunc from 0x0101e8 to 0x0170c3 (0x6edc bytes) 000161 1788653904.638518 mapread 0x0149ce thru 0x01e997 (0x9fca bytes) 000162 1788653904.661228 mapwrite 0x00aa05 thru 0x00e8f8 (0x3ef4 bytes) 000163 1788653904.690747 read 0x009b8e thru 0x014c0d (0xb080 bytes) 000164 1788653904.692185 read 0x011c13 thru 0x01e15e (0xc54c bytes) 000165 1788653904.698974 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x22435): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22435): Mirror 2: 0x35f33512 (stale) CRC-32 checksum value for chunk [0, 0x22435): Mirror 1: 0xf1a76246 Mirror 3: 0xf1a76246 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22435: skipped 000168 1788653905.914164 trunc from 0x022435 to 0x02f3ff (0xcfcb bytes) 000169 1788653906.038971 mapread 0x01a235 thru 0x022203 (0x7fcf bytes) 000170 1788653906.074857 fallocate 0x028a59 thru 0x029fe3 (0x158b bytes) 000171 1788653906.085406 punch from 0x007f0c to 0x00d5ed (0x56e2 bytes) 000172 1788653906.103543 write 0x017105 thru 0x023165 (0xc061 bytes) 000174 1788653906.105452 punch from 0x010d8a to 0x01c735 (0xb9ac bytes) 000175 1788653906.141964 mapread 0x02111d thru 0x028ff0 (0x7ed4 bytes) 000176 1788653906.176559 MIRROR_RESYNC 3 mirrors 000178 1788653907.303519 MIRROR_EXTEND to 4 mirrors 000179 1788653908.325651 punch from 0x01e2b6 to 0x02017b (0x1ec6 bytes) 000180 1788653908.398600 fallocate 0x005009 thru 0x012c26 (0xdc1e bytes) 000181 1788653908.416512 punch from 0x0076b3 to 0x012e7e (0xb7cc bytes) 000182 1788653908.432569 mapwrite 0x02c687 thru 0x03538e (0x8d08 bytes) 000183 1788653908.544186 punch from 0x0341dd to 0x03538e (0x11b2 bytes) 000184 1788653908.550245 read 0x0211d4 thru 0x022836 (0x1663 bytes) 000185 1788653908.560155 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3538f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3538f): Mirror 2: 0xe367a963 (stale) Mirror 3: 0xe367a963 (stale) Mirror 4: 0xe367a963 (stale) CRC-32 checksum value for chunk [0, 0x3538f): Mirror 1: 0x2cb7d99 Mirror 5: 0x2cb7d99 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3538f: skipped 000186 1788653909.621476 read 0x01997f thru 0x01fcbd (0x633f bytes) 000187 1788653909.640123 read 0x01c737 thru 0x01e8a4 (0x216e bytes) 000188 1788653909.653030 write 0x019de1 thru 0x0273b1 (0xd5d1 bytes) 000189 1788653909.748515 read 0x025440 thru 0x031640 (0xc201 bytes) 000190 1788653909.750234 trunc from 0x01f762 to 0x022ee9 (0x3788 bytes) 000191 1788653909.776962 mapread 0x01e715 thru 0x01eba8 (0x494 bytes) 000192 1788653909.778404 read 0x02497d thru 0x025f4c (0x15d0 bytes) 000193 1788653909.779510 fallocate 0x03ac99 thru 0x03ffff (0x5367 bytes) 000194 1788653909.789602 write 0x0064d5 thru 0x011cf5 (0xb821 bytes) 000195 1788653909.791451 trunc from 0x01db7d to 0x02a0ac (0xc530 bytes) 000197 1788653909.798524 mapwrite 0x036ed6 thru 0x03ffff (0x912a bytes) 000198 1788653909.882806 write 0x01b24a thru 0x01f2ec (0x40a3 bytes) 000199 1788653909.889796 MIRROR_RESYNC 5 mirrors 000200 1788653911.710499 mapread 0x016e14 thru 0x0227be (0xb9ab bytes) 000201 1788653911.749959 mapwrite 0x00250e thru 0x00e84d (0xc340 bytes) 000202 1788653911.862169 read 0x000087 thru 0x002971 (0x28eb bytes) 000203 1788653911.868818 mapread 0x0136c4 thru 0x01a608 (0x6f45 bytes) 000204 1788653911.882733 read 0x01f977 thru 0x024ba1 (0x522b bytes) 000205 1788653911.896611 read 0x03f958 thru 0x03fa5d (0x106 bytes) 000206 1788653911.901742 MIRROR_RESYNC 5 mirrors 000208 1788653913.945555 trunc from 0x040000 to 0x02ea8c (0x11573 bytes) 000209 1788653914.005426 mapread 0x02787d thru 0x02ea8c (0x7210 bytes) 000210 1788653914.031123 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ea8d): Mirror 3: 0x71a5945b (stale) Mirror 4: 0x71a5945b (stale) Mirror 5: 0x71a5945b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ea8d): Mirror 3: 0x71a5945b (stale) Mirror 4: 0x71a5945b (stale) Mirror 5: 0x71a5945b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ea8d: skipped 000211 1788653914.550987 punch from 0x00a910 to 0x01a3a2 (0xfa93 bytes) 000212 1788653914.560552 mapread 0x019b6a thru 0x01d208 (0x369f bytes) 000213 1788653914.617870 mapwrite 0x00b254 thru 0x01905b (0xde08 bytes) 000215 1788653914.673673 read 0x015953 thru 0x0209cf (0xb07d bytes) 000216 1788653914.674336 punch from 0x0215e6 to 0x02541c (0x3e37 bytes) 000217 1788653914.689569 fallocate 0x0059e4 thru 0x0126d4 (0xccf1 bytes) 000218 1788653914.699671 read 0x015598 thru 0x01a16e (0x4bd7 bytes) 000219 1788653914.700266 trunc from 0x02ea8d to 0x01fcf0 (0xed9c bytes) 000220 1788653914.736681 trunc from 0x01fcf1 to 0x02f9e2 (0xfcf2 bytes) 000222 1788653914.790437 mapread 0x00748e thru 0x015cfe (0xe871 bytes) 000223 1788653914.794943 trunc from 0x02f9e3 to 0x0097e7 (0x261fb bytes) 000224 1788653914.826999 punch from 0x007136 to 0x009040 (0x1f0b bytes) 000225 1788653914.838860 punch from 0x0058c3 to 0x005d50 (0x48e bytes) 000226 1788653914.848526 mapread 0x001bc3 thru 0x0097e7 (0x7c25 bytes) 000227 1788653914.864380 read 0x004792 thru 0x0097e7 (0x5056 bytes) 000228 1788653914.872276 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x97e8): Mirror 3: 0xfbd195ce (stale) Mirror 5: 0xfbd195ce (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x97e8): Mirror 3: 0xfbd195ce (stale) Mirror 5: 0xfbd195ce (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x97e8: skipped 000230 1788653915.219914 trunc from 0x000681 to 0x0097e7 (0x9167 bytes) 000231 1788653915.229184 trunc from 0x007b10 to 0x0097e7 (0x1cd8 bytes) 000232 1788653915.245200 punch from 0x003488 to 0x003928 (0x4a1 bytes) 000233 1788653915.251731 read 0x001244 thru 0x007c46 (0x6a03 bytes) 000234 1788653915.268798 punch from 0x0011b8 to 0x001820 (0x669 bytes) 000236 1788653915.276507 mapread 0x00417b thru 0x0097e7 (0x566d bytes) 000237 1788653915.294549 mapwrite 0x00f246 thru 0x01e2f0 (0xf0ab bytes) 000238 1788653915.393744 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e2f1): Mirror 5: 0xe4babe31 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e2f1): Mirror 5: 0xe4babe31 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e2f1: skipped 000239 1788653915.753924 trunc from 0x00b271 to 0x0192dc (0xe06c bytes) 000240 1788653915.763890 write 0x025d69 thru 0x028767 (0x29ff bytes) 000241 1788653915.779947 read 0x0166fd thru 0x019381 (0x2c85 bytes) 000242 1788653915.804194 write 0x028bb1 thru 0x02d809 (0x4c59 bytes) 000243 1788653915.805382 trunc from 0x02d80a to 0x02c815 (0xff4 bytes) 000244 1788653915.836888 mapread 0x02099c thru 0x02c815 (0xbe7a bytes) 000245 1788653915.886157 trunc from 0x01b61f to 0x01d31f (0x1d01 bytes) 000246 1788653915.900747 write 0x01a25a thru 0x027bef (0xd996 bytes) 000247 1788653915.917199 fallocate 0x03f2d3 thru 0x03ffff (0xd2d bytes) 000249 1788653915.938461 punch from 0x0302d3 to 0x03120b (0xf39 bytes) 000251 1788653915.956374 fallocate 0x014495 thru 0x01b11a (0x6c86 bytes) 000252 1788653915.971823 trunc from 0x01bed1 to 0x01cb9f (0xccf bytes) 000253 1788653915.999956 punch from 0x0317e6 to 0x03a2a1 (0x8abc bytes) 000254 1788653916.010986 read 0x008170 thru 0x017038 (0xeec9 bytes) 000255 1788653916.033082 trunc from 0x040000 to 0x018553 (0x27aac bytes) 000256 1788653916.079469 read 0x013475 thru 0x018553 (0x50df bytes) 000258 1788653916.081353 punch from 0x0112fb to 0x018553 (0x7259 bytes) 000259 1788653916.088858 mapwrite 0x03b56c thru 0x03ffff (0x4a94 bytes) 000260 1788653916.146902 read 0x005327 thru 0x0086ce (0x33a8 bytes) 000261 1788653916.147968 trunc from 0x040000 to 0x00414d (0x3beb2 bytes) 000262 1788653916.184511 mapread 0x001afa thru 0x00414d (0x2654 bytes) 000264 1788653916.186804 trunc from 0x002636 to 0x00414d (0x1b18 bytes) 000265 1788653916.196405 read 0x00034e thru 0x002b36 (0x27e9 bytes) 000268 1788653916.213024 write 0x00ea71 thru 0x015921 (0x6eb1 bytes) 000271 1788653916.217276 trunc from 0x015922 to 0x003671 (0x122b0 bytes) 000272 1788653916.244352 fallocate 0x0075e8 thru 0x011083 (0x9a9c bytes) 000273 1788653916.252401 fallocate 0x031343 thru 0x033503 (0x21c1 bytes) 000274 1788653916.263879 trunc from 0x011084 to 0x03b517 (0x2a494 bytes) 000275 1788653916.312248 mapread 0x004497 thru 0x01104f (0xcbb9 bytes) 000276 1788653916.334970 trunc from 0x02b456 to 0x0384f0 (0xd09b bytes) 000278 1788653916.351963 mapwrite 0x00a75f thru 0x01a080 (0xf922 bytes) 000279 1788653916.406614 read 0x03a509 thru 0x03b517 (0x100f bytes) 000280 1788653916.413833 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b518): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b518): Mirror 5: 0x37294a45 (stale) CRC-32 checksum value for chunk [0, 0x3b518): Mirror 1: 0xc6a14da3 Mirror 6: 0xc6a14da3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b518: skipped 000281 1788653917.704225 punch from 0x00b78a to 0x0136f5 (0x7f6c bytes) 000282 1788653917.770409 fallocate 0x0362bf thru 0x03ed78 (0x8aba bytes) 000283 1788653917.785168 mapwrite 0x02f692 thru 0x035610 (0x5f7f bytes) 000284 1788653917.834851 trunc from 0x03b48f to 0x03d483 (0x1ff5 bytes) 000285 1788653917.841321 trunc from 0x036781 to 0x03c3e6 (0x5c66 bytes) 000286 1788653917.854635 mapread 0x015586 thru 0x01ed9a (0x9815 bytes) 000287 1788653917.873401 read 0x026d57 thru 0x02ca6b (0x5d15 bytes) 000288 1788653917.890626 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3ed79): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed79): Mirror 5: 0x3908279a (stale) Mirror 6: 0x7ba6e6db (stale) CRC-32 checksum value for chunk [0, 0x3ed79): Mirror 1: 0x4ed166ae Mirror 7: 0x4ed166ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed79: skipped 000289 1788653919.354771 mapread 0x0239da thru 0x0320ef (0xe716 bytes) 000290 1788653919.444599 punch from 0x0130bb to 0x0223a7 (0xf2ed bytes) 000291 1788653919.521124 mapread 0x01ac1f thru 0x0290d7 (0xe4b9 bytes) 000292 1788653919.581269 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ed79): Mirror 6: 0x7ba6e6db (stale) Mirror 7: 0x4ed166ae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed79): Mirror 6: 0x7ba6e6db (stale) Mirror 7: 0x4ed166ae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed79: skipped 000293 1788653920.029209 mapwrite 0x03aae7 thru 0x03ffff (0x5519 bytes) 000294 1788653920.112111 punch from 0x014297 to 0x01d294 (0x8ffe bytes) 000295 1788653920.118915 trunc from 0x0011d0 to 0x006162 (0x4f93 bytes) 000296 1788653920.123874 fallocate 0x038600 thru 0x03ffff (0x7a00 bytes) 000297 1788653920.134922 mapread 0x034ca9 thru 0x03ffff (0xb357 bytes) 000298 1788653920.154162 punch from 0x00c556 to 0x0125d6 (0x6081 bytes) 000299 1788653920.162234 fallocate 0x02e557 thru 0x03e066 (0xfb10 bytes) 000300 1788653920.169102 read 0x002ff7 thru 0x00bfa0 (0x8faa bytes) 000301 1788653920.179559 trunc from 0x040000 to 0x037c5e (0x83a1 bytes) 000302 1788653920.203435 punch from 0x036905 to 0x037c5e (0x135a bytes) 000304 1788653920.213661 write 0x01afe4 thru 0x01b185 (0x1a2 bytes) 000305 1788653920.227879 trunc from 0x00ca1e to 0x012335 (0x5918 bytes) 000307 1788653920.234796 trunc from 0x037c5f to 0x01d9d2 (0x1a28c bytes) 000308 1788653920.275933 mapwrite 0x0323a7 thru 0x03775c (0x53b6 bytes) 000309 1788653920.386213 punch from 0x02ec6c to 0x03775c (0x8af1 bytes) 000310 1788653920.400372 trunc from 0x0117c5 to 0x01393e (0x217a bytes) 000311 1788653920.414516 read 0x0287dc thru 0x02a1d5 (0x19fa bytes) 000312 1788653920.457102 write 0x02b3dc thru 0x02f3fc (0x4021 bytes) 000313 1788653920.458468 mapwrite 0x00d0ad thru 0x01b001 (0xdf55 bytes) 000314 1788653920.492029 punch from 0x01b08f to 0x023801 (0x8773 bytes) 000315 1788653920.498444 mapread 0x018a6a thru 0x01985e (0xdf5 bytes) 000316 1788653920.500727 punch from 0x004ea1 to 0x013dc3 (0xef23 bytes) 000317 1788653920.507841 mapwrite 0x0287d0 thru 0x0295ca (0xdfb bytes) 000318 1788653920.546823 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3775d): Mirror 7: 0xffa1992b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3775d): Mirror 7: 0xffa1992b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3775d: skipped 000319 1788653920.897419 mapwrite 0x009a01 thru 0x010616 (0x6c16 bytes) 000320 1788653920.937148 write 0x0268bb thru 0x02d5b4 (0x6cfa bytes) 000321 1788653920.951866 mapwrite 0x02fffb thru 0x03c622 (0xc628 bytes) 000322 1788653921.005730 mapwrite 0x0359c7 thru 0x03ccd3 (0x730d bytes) 000324 1788653921.046230 MIRROR_RESYNC 2 mirrors 000326 1788653921.829886 fallocate 0x01ca53 thru 0x0244c2 (0x7a70 bytes) 000328 1788653921.905411 fallocate 0x03f4f1 thru 0x03ffff (0xb0f bytes) 000329 1788653921.916712 mapread 0x001d71 thru 0x00ca70 (0xad00 bytes) 000330 1788653921.957302 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) 000331 1788653922.164408 mapread 0x00dfc2 thru 0x01adaf (0xcdee bytes) 000332 1788653922.202266 fallocate 0x01fedb thru 0x020a7b (0xba1 bytes) 000333 1788653922.207940 trunc from 0x03ccd4 to 0x03ddcf (0x10fc bytes) 000335 1788653922.263729 mapread 0x01a10a thru 0x0260df (0xbfd6 bytes) 000336 1788653922.275898 read 0x012e25 thru 0x018d99 (0x5f75 bytes) 000339 1788653922.279126 fallocate 0x00cb06 thru 0x010dcb (0x42c6 bytes) 000340 1788653922.284202 punch from 0x0276d8 to 0x02d1ae (0x5ad7 bytes) 000342 1788653922.290599 trunc from 0x03ddd0 to 0x0143fe (0x299d1 bytes) 000343 1788653922.304683 write 0x009260 thru 0x015582 (0xc323 bytes) 000344 1788653922.306199 punch from 0x00431c to 0x0109b4 (0xc699 bytes) 000345 1788653922.327203 fallocate 0x02e450 thru 0x0344cb (0x607c bytes) 000346 1788653922.332788 mapread 0x012cb2 thru 0x01f9c2 (0xcd11 bytes) 000347 1788653922.355862 read 0x0158a7 thru 0x01df4a (0x86a4 bytes) 000349 1788653922.358378 fallocate 0x038b4e thru 0x03bfc5 (0x3478 bytes) 000351 1788653922.365166 fallocate 0x02361b thru 0x0250d2 (0x1ab8 bytes) 000352 1788653922.370865 mapread 0x01d1f1 thru 0x02bbe8 (0xe9f8 bytes) 000353 1788653922.381780 mapread 0x011d18 thru 0x01a9e8 (0x8cd1 bytes) 000355 1788653922.383860 write 0x0113a9 thru 0x011f67 (0xbbf bytes) 000357 1788653922.386292 read 0x00870b thru 0x00e9b3 (0x62a9 bytes) 000358 1788653922.387447 fallocate 0x030d5f thru 0x036920 (0x5bc2 bytes) 000359 1788653922.393214 mapread 0x012995 thru 0x01e6fe (0xbd6a bytes) 000360 1788653922.394266 mapread 0x002b1b thru 0x005106 (0x25ec bytes) 000361 1788653922.395802 fallocate 0x00de36 thru 0x01c9e9 (0xebb4 bytes) 000362 1788653922.401405 trunc from 0x036921 to 0x025dd0 (0x10b50 bytes) 000363 1788653922.419715 mapread 0x01abc1 thru 0x01f6b9 (0x4af9 bytes) 000364 1788653922.421311 trunc from 0x024351 to 0x025dd0 (0x1a80 bytes) 000365 1788653922.426568 fallocate 0x022131 thru 0x0249c3 (0x2893 bytes) 000367 1788653922.433282 write 0x021497 thru 0x0221e4 (0xd4e bytes) 000368 1788653922.434480 mapwrite 0x00d200 thru 0x0102c3 (0x30c4 bytes) 000369 1788653922.473039 MIRROR_EXTEND to 2 mirrors 000370 1788653923.381395 mapread 0x00418d thru 0x0123e5 (0xe259 bytes) 000372 1788653923.419424 MIRROR_RESYNC 2 mirrors 000375 1788653923.888552 punch from 0x01f14a to 0x025dd0 (0x6c87 bytes) 000376 1788653923.952545 fallocate 0x01df22 thru 0x02b601 (0xd6e0 bytes) 000377 1788653923.958021 read 0x01c524 thru 0x02b5c2 (0xf09f bytes) 000378 1788653923.988588 punch from 0x00a3fb to 0x015cd8 (0xb8de bytes) 000379 1788653924.009744 read 0x008fbb thru 0x013a1d (0xaa63 bytes) 000380 1788653924.039556 write 0x031c6a thru 0x03d0c4 (0xb45b bytes) 000381 1788653924.046319 write 0x030a3f thru 0x038aef (0x80b1 bytes) 000382 1788653924.054664 read 0x023c93 thru 0x027b58 (0x3ec6 bytes) 000383 1788653924.055365 trunc from 0x02391c to 0x02dfd0 (0xa6b5 bytes) 000384 1788653924.060533 fallocate 0x0240ee thru 0x024ee1 (0xdf4 bytes) 000385 1788653924.065018 punch from 0x036cec to 0x03d0c4 (0x63d9 bytes) 000386 1788653924.077830 mapread 0x00fc10 thru 0x016669 (0x6a5a bytes) 000387 1788653924.079847 trunc from 0x039ce1 to 0x03d0c4 (0x33e4 bytes) 000388 1788653924.085114 trunc from 0x00084c to 0x00e190 (0xd945 bytes) 000389 1788653924.090556 trunc from 0x03d0c5 to 0x023eb7 (0x1920d bytes) 000390 1788653924.118662 trunc from 0x023eb8 to 0x015a5d (0xe45a bytes) 000391 1788653924.133678 punch from 0x00f267 to 0x015a5d (0x67f7 bytes) 000392 1788653924.140318 trunc from 0x004aea to 0x012280 (0xd797 bytes) 000393 1788653924.149739 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x15a5e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15a5e): Mirror 2: 0xdcf09be3 (stale) CRC-32 checksum value for chunk [0, 0x15a5e): Mirror 1: 0xa2bc5ca0 Mirror 3: 0xa2bc5ca0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15a5e: skipped 000394 1788653924.847014 write 0x0255fd thru 0x02815a (0x2b5e bytes) 000395 1788653924.905146 punch from 0x004ed2 to 0x00f354 (0xa483 bytes) 000396 1788653924.909639 read 0x0224e8 thru 0x02815a (0x5c73 bytes) 000397 1788653924.926026 punch from 0x006647 to 0x00fe8b (0x9845 bytes) 000398 1788653924.936694 trunc from 0x02815b to 0x024529 (0x3c31 bytes) 000399 1788653924.995615 trunc from 0x02452a to 0x015847 (0xece2 bytes) 000400 1788653925.025202 MIRROR_RESYNC 3 mirrors 000401 1788653925.932379 MIRROR_RESYNC 3 mirrors 000402 1788653926.288565 mapwrite 0x03bac4 thru 0x03cf55 (0x1492 bytes) 000403 1788653926.379624 trunc from 0x03cf56 to 0x00601c (0x36f39 bytes) 000404 1788653926.408157 punch from 0x0011ce to 0x00601c (0x4e4f bytes) 000405 1788653926.415312 mapwrite 0x00091f thru 0x004208 (0x38ea bytes) 000406 1788653926.449539 trunc from 0x003e9f to 0x00601c (0x217e bytes) 000407 1788653926.457318 trunc from 0x00601d to 0x027e8e (0x21e72 bytes) 000408 1788653926.483890 fallocate 0x0321c4 thru 0x032981 (0x7be bytes) 000409 1788653926.497693 MIRROR_RESYNC 3 mirrors 000410 1788653927.578754 read 0x01d290 thru 0x02a4b7 (0xd228 bytes) 000411 1788653927.603463 punch from 0x01b261 to 0x01be10 (0xbb0 bytes) 000412 1788653927.659642 trunc from 0x032982 to 0x00be8a (0x26af7 bytes) 000413 1788653927.708549 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xbe8b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbe8b): Mirror 2: 0x3d9ca816 (stale) Mirror 3: 0x3d9ca816 (stale) CRC-32 checksum value for chunk [0, 0xbe8b): Mirror 1: 0x3d9ca816 Mirror 4: 0x3d9ca816 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbe8b: skipped 000414 1788653928.670392 write 0x00fd67 thru 0x0191bd (0x9457 bytes) 000415 1788653928.718405 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x191be): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x191be): Mirror 2: 0xd0eec6d1 (stale) Mirror 3: 0xd0eec6d1 (stale) Mirror 4: 0xd0eec6d1 (stale) CRC-32 checksum value for chunk [0, 0x191be): Mirror 1: 0xfbc942f8 Mirror 5: 0xfbc942f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x191be: skipped 000416 1788653929.837563 punch from 0x00a946 to 0x00f1b0 (0x486b bytes) 000417 1788653929.882571 mapwrite 0x01be50 thru 0x023e60 (0x8011 bytes) 000418 1788653929.944142 write 0x011521 thru 0x01d4fc (0xbfdc bytes) 000419 1788653929.950371 trunc from 0x01b08e to 0x023e60 (0x8dd3 bytes) 000420 1788653929.969521 fallocate 0x00699b thru 0x00caab (0x6111 bytes) 000421 1788653929.974577 mapwrite 0x02acf5 thru 0x036411 (0xb71d bytes) 000422 1788653930.040491 trunc from 0x00767f to 0x00f5be (0x7f40 bytes) 000423 1788653930.049828 trunc from 0x002f2e to 0x006024 (0x30f7 bytes) 000427 1788653930.056566 fallocate 0x006750 thru 0x015e6b (0xf71c bytes) 000428 1788653930.066623 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36412): Mirror 2: 0x98640411 (stale) Mirror 3: 0x98640411 (stale) Mirror 5: 0x77c113fc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36412): Mirror 2: 0x98640411 (stale) Mirror 3: 0x98640411 (stale) Mirror 5: 0x77c113fc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36412: skipped 000431 1788653930.463674 fallocate 0x018c38 thru 0x0248b6 (0xbc7f bytes) 000432 1788653930.469235 write 0x00c9bd thru 0x015cfb (0x933f bytes) 000434 1788653930.484686 trunc from 0x036412 to 0x004e0e (0x31603 bytes) 000435 1788653930.507675 punch from 0x00238e to 0x004e0e (0x2a81 bytes) 000436 1788653930.515995 trunc from 0x002725 to 0x002af8 (0x3d4 bytes) 000437 1788653930.527483 MIRROR_RESYNC 4 mirrors 000438 1788653931.359823 write 0x00ad8e thru 0x013645 (0x88b8 bytes) 000439 1788653931.393992 fallocate 0x03a743 thru 0x03e5a2 (0x3e60 bytes) 000440 1788653931.399618 trunc from 0x013646 to 0x01013f (0x3506 bytes) 000441 1788653931.424697 mapread 0x00fd4f thru 0x01013f (0x3f1 bytes) 000442 1788653931.431496 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10140): Mirror 2: 0xc21540b4 (stale) Mirror 5: 0xc21540b4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10140): Mirror 2: 0xc21540b4 (stale) Mirror 5: 0xc21540b4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10140: skipped 000443 1788653931.761421 mapwrite 0x0019a3 thru 0x011026 (0xf684 bytes) 000444 1788653931.904095 write 0x01dc80 thru 0x01e8c2 (0xc43 bytes) 000445 1788653931.905199 punch from 0x013eb7 to 0x01e8c2 (0xaa0c bytes) 000446 1788653931.928214 fallocate 0x035bfa thru 0x03eed4 (0x92db bytes) 000448 1788653931.937381 read 0x0025e6 thru 0x00f62b (0xd046 bytes) 000450 1788653931.939559 punch from 0x0181e7 to 0x027926 (0xf740 bytes) 000451 1788653931.949900 MIRROR_RESYNC 3 mirrors 000452 1788653932.784783 write 0x005d96 thru 0x0134df (0xd74a bytes) 000454 1788653932.883554 write 0x032326 thru 0x0340b9 (0x1d94 bytes) 000456 1788653932.908948 mapread 0x010dd9 thru 0x01c749 (0xb971 bytes) 000458 1788653932.980819 punch from 0x01b936 to 0x0273cf (0xba9a bytes) 000459 1788653932.986558 write 0x01076d thru 0x01ea72 (0xe306 bytes) 000460 1788653932.993616 mapwrite 0x00c690 thru 0x01b409 (0xed7a bytes) 000461 1788653933.022199 read 0x0346d9 thru 0x03eed4 (0xa7fc bytes) 000462 1788653933.037760 fallocate 0x00c02e thru 0x011791 (0x5764 bytes) 000463 1788653933.066483 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3eed5): Mirror 5: 0x36fb533f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eed5): Mirror 5: 0x36fb533f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eed5: skipped 000464 1788653933.540068 punch from 0x02465d to 0x03053c (0xbee0 bytes) 000465 1788653933.546297 read 0x00043e thru 0x00cba1 (0xc764 bytes) 000467 1788653933.560012 mapread 0x034e38 thru 0x03eed4 (0xa09d bytes) 000468 1788653933.579782 fallocate 0x00ba46 thru 0x011967 (0x5f22 bytes) 000469 1788653933.585350 write 0x0195e3 thru 0x02691f (0xd33d bytes) 000470 1788653933.602880 trunc from 0x01c62e to 0x01e57d (0x1f50 bytes) 000471 1788653933.620101 write 0x00751e thru 0x012a91 (0xb574 bytes) 000472 1788653933.627134 fallocate 0x01a226 thru 0x0254e4 (0xb2bf bytes) 000473 1788653933.646313 MIRROR_RESYNC 2 mirrors 000474 1788653934.395641 punch from 0x008ff4 to 0x00ffd0 (0x6fdd bytes) 000475 1788653934.432369 punch from 0x0328c0 to 0x03eed4 (0xc615 bytes) 000476 1788653934.440390 write 0x012cb0 thru 0x01ca72 (0x9dc3 bytes) 000477 1788653934.451859 read 0x02f281 thru 0x0361ea (0x6f6a bytes) 000479 1788653934.461288 mapread 0x00f68c thru 0x019dca (0xa73f bytes) 000480 1788653934.480783 punch from 0x03a07d to 0x03eed4 (0x4e58 bytes) 000481 1788653934.486359 mapwrite 0x027d98 thru 0x028a2b (0xc94 bytes) 000482 1788653934.512889 trunc from 0x03eed5 to 0x026d52 (0x18182 bytes) 000484 1788653934.536150 trunc from 0x005b55 to 0x00fb48 (0x9ff4 bytes) 000485 1788653934.543537 mapwrite 0x02dfda thru 0x03de71 (0xfe98 bytes) 000486 1788653934.601421 trunc from 0x0047f6 to 0x00917c (0x4987 bytes) 000487 1788653934.607837 trunc from 0x02f122 to 0x03c0d4 (0xcfb3 bytes) 000488 1788653934.615848 mapread 0x032284 thru 0x034757 (0x24d4 bytes) 000489 1788653934.635487 trunc from 0x03de72 to 0x00fd33 (0x2e13e bytes) 000491 1788653934.705038 MIRROR_RESYNC 2 mirrors 000492 1788653935.330037 trunc from 0x00fd34 to 0x006c39 (0x90fa bytes) 000493 1788653935.395584 read 0x006807 thru 0x006c39 (0x433 bytes) 000494 1788653935.418111 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) 000495 1788653935.676954 trunc from 0x000521 to 0x006c39 (0x6719 bytes) 000496 1788653935.689622 mapread 0x006244 thru 0x006c39 (0x9f6 bytes) 000497 1788653935.708908 trunc from 0x006c3a to 0x01a466 (0x1382d bytes) 000498 1788653935.746021 punch from 0x005725 to 0x014b16 (0xf3f2 bytes) 000499 1788653935.757567 trunc from 0x01a467 to 0x007056 (0x13410 bytes) 000501 1788653935.816402 fallocate 0x023240 thru 0x0308e2 (0xd6a3 bytes) 000502 1788653935.829467 MIRROR_EXTEND to 2 mirrors 000503 1788653936.556901 mapwrite 0x03d3e3 thru 0x03e8bc (0x14da bytes) 000504 1788653936.622577 trunc from 0x03e8bd to 0x0263c3 (0x184f9 bytes) 000505 1788653936.644270 trunc from 0x022fe5 to 0x0263c3 (0x33df bytes) 000506 1788653936.657383 write 0x01d138 thru 0x02a345 (0xd20e bytes) 000507 1788653936.667764 fallocate 0x00b291 thru 0x012c11 (0x7981 bytes) 000508 1788653936.675741 punch from 0x016bdc to 0x01938d (0x27b2 bytes) 000509 1788653936.685129 trunc from 0x02a346 to 0x0197a0 (0x10ba5 bytes) 000510 1788653936.708340 trunc from 0x018fc0 to 0x0197a0 (0x7e1 bytes) 000511 1788653936.717155 fallocate 0x012bcd thru 0x01ab32 (0x7f66 bytes) 000512 1788653936.723418 mapread 0x0123a8 thru 0x015ba3 (0x37fc bytes) 000513 1788653936.741333 write 0x01a300 thru 0x01fe6e (0x5b6f bytes) 000514 1788653936.742725 mapwrite 0x033e9a thru 0x03ffff (0xc166 bytes) 000515 1788653936.851856 mapwrite 0x008f8a thru 0x015829 (0xc8a0 bytes) 000516 1788653936.910486 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) 000517 1788653937.106571 write 0x039fbe thru 0x03ffff (0x6042 bytes) 000518 1788653937.131527 MIRROR_EXTEND to 2 mirrors 000519 1788653938.322141 punch from 0x033ddc to 0x03b7a8 (0x79cd bytes) 000520 1788653938.391446 mapwrite 0x02aab4 thru 0x03a50a (0xfa57 bytes) 000521 1788653938.512407 read 0x01b57c thru 0x027a70 (0xc4f5 bytes) 000522 1788653938.527949 read 0x00567c thru 0x00ab73 (0x54f8 bytes) 000523 1788653938.545934 fallocate 0x0293e0 thru 0x0358b1 (0xc4d2 bytes) 000524 1788653938.556741 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x76ab91a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x88b413de Mirror 3: 0x88b413de lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000525 1788653939.842730 MIRROR_RESYNC 3 mirrors 000526 1788653940.706944 MIRROR_RESYNC 3 mirrors 000527 1788653941.232035 mapwrite 0x026dd3 thru 0x0309e7 (0x9c15 bytes) 000528 1788653941.383948 write 0x03aee4 thru 0x03ffff (0x511c bytes) 000529 1788653941.402807 MIRROR_RESYNC 3 mirrors 000531 1788653942.464332 punch from 0x034ff0 to 0x03f74b (0xa75c bytes) 000532 1788653942.494953 fallocate 0x02087e thru 0x02fbd5 (0xf358 bytes) 000533 1788653942.501620 mapwrite 0x02850d thru 0x02c73a (0x422e bytes) 000534 1788653942.538940 fallocate 0x031321 thru 0x032c0f (0x18ef bytes) 000535 1788653942.543942 write 0x01c4ac thru 0x02c212 (0xfd67 bytes) 000536 1788653942.549314 fallocate 0x039994 thru 0x03ffff (0x666c bytes) 000537 1788653942.554650 fallocate 0x01899e thru 0x022e37 (0xa49a bytes) 000538 1788653942.560936 punch from 0x02425c to 0x033ee9 (0xfc8e bytes) 000539 1788653942.579685 read 0x010ef6 thru 0x01e558 (0xd663 bytes) 000540 1788653942.593349 read 0x02a482 thru 0x02a8ac (0x42b bytes) 000541 1788653942.605336 fallocate 0x00ee60 thru 0x019dd4 (0xaf75 bytes) 000542 1788653942.613472 MIRROR_RESYNC 3 mirrors 000543 1788653944.141779 read 0x02b318 thru 0x03583c (0xa525 bytes) 000544 1788653944.161749 trunc from 0x03136e to 0x03ffff (0xec92 bytes) 000545 1788653944.257676 mapwrite 0x0130e7 thru 0x01ac05 (0x7b1f bytes) 000547 1788653944.313088 punch from 0x00f19b to 0x016638 (0x749e bytes) 000548 1788653944.322993 read 0x028d3f thru 0x02d208 (0x44ca bytes) 000549 1788653944.342767 fallocate 0x0365e0 thru 0x03ffff (0x9a20 bytes) 000550 1788653944.380215 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x83cdb56e (stale) Mirror 3: 0x83cdb56e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8704b825 Mirror 4: 0x8704b825 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000551 1788653945.832165 mapread 0x014a65 thru 0x01f00b (0xa5a7 bytes) 000552 1788653945.910523 punch from 0x002afe to 0x003e29 (0x132c bytes) 000553 1788653946.005439 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x83cdb56e (stale) Mirror 4: 0x8704b825 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x83cdb56e (stale) Mirror 4: 0x8704b825 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000555 1788653946.619379 punch from 0x00c89c to 0x00d097 (0x7fc bytes) 000556 1788653946.626265 read 0x007daf thru 0x008059 (0x2ab bytes) 000557 1788653946.641701 trunc from 0x040000 to 0x003806 (0x3c7f9 bytes) 000558 1788653946.664432 write 0x028e17 thru 0x036d2b (0xdf15 bytes) 000559 1788653946.675802 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x36d2c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36d2c): Mirror 3: 0xb80d10f1 (stale) Mirror 4: 0x863e142d (stale) CRC-32 checksum value for chunk [0, 0x36d2c): Mirror 1: 0xda5c6c0f Mirror 5: 0xda5c6c0f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36d2c: skipped 000560 1788653947.613590 trunc from 0x01b484 to 0x02b27c (0xfdf9 bytes) 000561 1788653947.699989 mapread 0x02f758 thru 0x030a7a (0x1323 bytes) 000562 1788653947.715853 trunc from 0x036d2c to 0x01bd0d (0x1b01e bytes) 000563 1788653947.762493 read 0x010916 thru 0x01bd0d (0xb3f8 bytes) 000564 1788653947.782237 write 0x037f65 thru 0x039dcd (0x1e69 bytes) 000565 1788653947.787574 mapwrite 0x02b7b0 thru 0x031273 (0x5ac4 bytes) 000566 1788653947.865323 write 0x02058f thru 0x0239dc (0x344e bytes) 000567 1788653947.928867 trunc from 0x032bee to 0x0357e0 (0x2bf3 bytes) 000568 1788653947.942497 mapwrite 0x000cb6 thru 0x00139e (0x6e9 bytes) 000569 1788653948.011678 fallocate 0x0288fe thru 0x03473c (0xbe3f bytes) 000570 1788653948.023792 read 0x01cf65 thru 0x0214f0 (0x458c bytes) 000571 1788653948.044572 punch from 0x0038ad to 0x00d68a (0x9dde bytes) 000572 1788653948.052226 trunc from 0x039dce to 0x034853 (0x557a bytes) 000573 1788653948.076891 mapwrite 0x035bbe thru 0x03898b (0x2dce bytes) 000574 1788653948.159451 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3898c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3898c): Mirror 3: 0x363d6369 (stale) Mirror 4: 0x86f6b6bc (stale) Mirror 5: 0xe054c7a9 (stale) CRC-32 checksum value for chunk [0, 0x3898c): Mirror 1: 0x7b1244ad Mirror 6: 0x7b1244ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3898c: skipped 000575 1788653949.458759 trunc from 0x00bd95 to 0x00f32b (0x3597 bytes) 000576 1788653949.507895 trunc from 0x01afe0 to 0x0236d8 (0x86f9 bytes) 000577 1788653949.514373 fallocate 0x035252 thru 0x03ffff (0xadae bytes) 000578 1788653949.520363 trunc from 0x025020 to 0x0314b5 (0xc496 bytes) 000579 1788653949.526149 write 0x031c5c thru 0x03b3fb (0x97a0 bytes) 000580 1788653949.547022 write 0x019d7f thru 0x01a902 (0xb84 bytes) 000581 1788653949.555884 trunc from 0x01648e to 0x01be8d (0x5a00 bytes) 000582 1788653949.595810 trunc from 0x040000 to 0x03bd61 (0x429e bytes) 000583 1788653949.689996 read 0x027860 thru 0x0289cc (0x116d bytes) 000584 1788653949.701626 mapwrite 0x01160d thru 0x01c24d (0xac41 bytes) 000585 1788653949.797084 mapwrite 0x01ff9a thru 0x028414 (0x847b bytes) 000586 1788653949.833684 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3bd62): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd62): Mirror 3: 0xf887f951 (stale) Mirror 4: 0x66797f85 (stale) Mirror 5: 0x4d960bd1 (stale) Mirror 6: 0xe8846269 (stale) CRC-32 checksum value for chunk [0, 0x3bd62): Mirror 1: 0x6ba25c03 Mirror 7: 0x6ba25c03 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd62: skipped 000587 1788653950.926639 read 0x03a569 thru 0x03bd61 (0x17f9 bytes) 000588 1788653950.939311 trunc from 0x00e5e8 to 0x01d2da (0xecf3 bytes) 000589 1788653950.979149 write 0x038644 thru 0x03ffff (0x79bc bytes) 000590 1788653950.999335 write 0x031aff thru 0x03ffff (0xe501 bytes) 000591 1788653951.007225 fallocate 0x028357 thru 0x033d39 (0xb9e3 bytes) 000593 1788653951.012815 write 0x00c338 thru 0x01067a (0x4343 bytes) 000594 1788653951.022961 read 0x010e03 thru 0x014d3c (0x3f3a bytes) 000595 1788653951.032825 punch from 0x0000e6 to 0x00766e (0x7589 bytes) 000596 1788653951.038575 read 0x02e63d thru 0x0331f5 (0x4bb9 bytes) 000597 1788653951.039289 fallocate 0x03852b thru 0x03ffff (0x7ad5 bytes) 000598 1788653951.044948 mapwrite 0x02c423 thru 0x02f9b2 (0x3590 bytes) 000600 1788653951.064304 write 0x02c777 thru 0x039a1b (0xd2a5 bytes) 000601 1788653951.065802 read 0x006c14 thru 0x008548 (0x1935 bytes) 000602 1788653951.077148 trunc from 0x040000 to 0x01f842 (0x207bd bytes) 000603 1788653951.103097 fallocate 0x00f932 thru 0x010736 (0xe05 bytes) 000604 1788653951.108831 read 0x010aa4 thru 0x01567a (0x4bd7 bytes) 000606 1788653951.111306 trunc from 0x01f843 to 0x00973f (0x16103 bytes) 000607 1788653951.128428 trunc from 0x009740 to 0x0097ac (0x6d bytes) 000609 1788653951.144284 write 0x02d1f2 thru 0x03aab7 (0xd8c6 bytes) 000610 1788653951.146861 write 0x02e037 thru 0x037911 (0x98db bytes) 000611 1788653951.148368 write 0x00fe71 thru 0x0117dd (0x196d bytes) 000612 1788653951.156150 read 0x023f45 thru 0x026a1c (0x2ad8 bytes) 000613 1788653951.156810 mapread 0x0059ca thru 0x010ba1 (0xb1d8 bytes) 000614 1788653951.157806 trunc from 0x03aab8 to 0x03c9cc (0x1f15 bytes) 000615 1788653951.184868 fallocate 0x027adf thru 0x032dd8 (0xb2fa bytes) 000616 1788653951.196736 trunc from 0x03c9cd to 0x020dc5 (0x1bc07 bytes) 000617 1788653951.229123 write 0x0130b2 thru 0x0164dd (0x342c bytes) 000618 1788653951.230219 mapwrite 0x01bd01 thru 0x028d38 (0xd038 bytes) 000619 1788653951.317341 mapread 0x000b0f thru 0x009b34 (0x9026 bytes) 000620 1788653951.320179 read 0x02845b thru 0x028d38 (0x8de bytes) 000621 1788653951.326646 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x28d39): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28d39): Mirror 3: 0x9e9a9a9e (stale) Mirror 4: 0x349c1c83 (stale) Mirror 5: 0x6a948daf (stale) Mirror 6: 0x6f97f07 (stale) Mirror 7: 0x2f5b5066 (stale) CRC-32 checksum value for chunk [0, 0x28d39): Mirror 1: 0x82c61662 Mirror 8: 0x82c61662 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28d39: skipped 000622 1788653952.450919 mapwrite 0x023f71 thru 0x032d29 (0xedb9 bytes) 000623 1788653952.644129 fallocate 0x0134c8 thru 0x01a089 (0x6bc2 bytes) 000624 1788653952.649667 read 0x00ee2b thru 0x019710 (0xa8e6 bytes) 000625 1788653952.658390 mapread 0x016ac9 thru 0x016fd8 (0x510 bytes) 000626 1788653952.659907 punch from 0x023c38 to 0x025c64 (0x202d bytes) 000628 1788653952.666154 mapwrite 0x015432 thru 0x020efb (0xbaca bytes) 000629 1788653952.700117 fallocate 0x01b84b thru 0x01fdb4 (0x456a bytes) 000630 1788653952.713082 trunc from 0x032d2a to 0x003cda (0x2f04f bytes) 000631 1788653952.755786 mapwrite 0x0081af thru 0x008ebf (0xd11 bytes) 000632 1788653952.824175 MIRROR_RESYNC 7 mirrors 000635 1788653954.519021 write 0x012d4a thru 0x01661a (0x38d1 bytes) 000636 1788653954.563340 mapread 0x007bfb thru 0x00b7d9 (0x3bdf bytes) 000637 1788653954.588869 trunc from 0x01661b to 0x0275a1 (0x10f87 bytes) 000638 1788653954.652520 punch from 0x02438e to 0x025203 (0xe76 bytes) 000639 1788653954.661865 mapread 0x026d60 thru 0x0275a1 (0x842 bytes) 000640 1788653954.683409 mapread 0x011b39 thru 0x01bed7 (0xa39f bytes) 000641 1788653954.699598 mapwrite 0x013b25 thru 0x018233 (0x470f bytes) 000642 1788653954.735615 fallocate 0x011c8f thru 0x0149cf (0x2d41 bytes) 000643 1788653954.742753 write 0x01099d thru 0x017e2b (0x748f bytes) 000644 1788653954.759559 mapread 0x015648 thru 0x024cd8 (0xf691 bytes) 000645 1788653954.760577 read 0x0196a8 thru 0x02232a (0x8c83 bytes) 000646 1788653954.769692 fallocate 0x02c57d thru 0x036a69 (0xa4ed bytes) 000647 1788653954.796648 trunc from 0x0275a2 to 0x02bcd0 (0x472f bytes) 000648 1788653954.824412 punch from 0x010019 to 0x018da6 (0x8d8e bytes) 000649 1788653954.896524 mapread 0x000123 thru 0x0079e9 (0x78c7 bytes) 000650 1788653954.909323 fallocate 0x02ef54 thru 0x035223 (0x62d0 bytes) 000651 1788653954.918678 trunc from 0x0144e7 to 0x020dd7 (0xc8f1 bytes) 000652 1788653954.925494 fallocate 0x034356 thru 0x03dba1 (0x984c bytes) 000653 1788653954.938434 mapread 0x01aab4 thru 0x01ff1f (0x546c bytes) 000654 1788653954.966507 mapwrite 0x004ebe thru 0x00e710 (0x9853 bytes) 000655 1788653954.991393 mapread 0x010086 thru 0x01d1ea (0xd165 bytes) 000656 1788653954.998693 write 0x011b47 thru 0x016bd9 (0x5093 bytes) 000657 1788653955.000189 mapread 0x017a54 thru 0x023a7c (0xc029 bytes) 000658 1788653955.002721 write 0x01a8e1 thru 0x02528e (0xa9ae bytes) 000659 1788653955.005535 mapwrite 0x006122 thru 0x007495 (0x1374 bytes) 000660 1788653955.028233 trunc from 0x035224 to 0x00ca9b (0x28788 bytes) 000661 1788653955.049077 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xca9c): Mirror 3: 0xfdb0d70 (stale) Mirror 4: 0xfdb0d70 (stale) Mirror 5: 0xfdb0d70 (stale) Mirror 7: 0xfdb0d70 (stale) Mirror 8: 0xfdb0d70 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xca9c): Mirror 3: 0xfdb0d70 (stale) Mirror 4: 0xfdb0d70 (stale) Mirror 5: 0xfdb0d70 (stale) Mirror 7: 0xfdb0d70 (stale) Mirror 8: 0xfdb0d70 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xca9c: skipped 000662 1788653955.537572 mapread 0x0072e7 thru 0x00ca9b (0x57b5 bytes) 000663 1788653955.564225 MIRROR_RESYNC 6 mirrors 000664 1788653956.816695 write 0x03e42e thru 0x03ffff (0x1bd2 bytes) 000666 1788653956.892643 punch from 0x021cff to 0x0299a0 (0x7ca2 bytes) 000667 1788653956.914414 mapread 0x02501a thru 0x027530 (0x2517 bytes) 000668 1788653956.948689 write 0x01371f thru 0x020a33 (0xd315 bytes) 000669 1788653956.974457 trunc from 0x040000 to 0x017abb (0x28544 bytes) 000670 1788653957.034719 mapwrite 0x01044e thru 0x01c6a3 (0xc256 bytes) 000671 1788653957.145711 read 0x000ed8 thru 0x00f395 (0xe4be bytes) 000672 1788653957.169533 MIRROR_RESYNC 6 mirrors 000673 1788653959.577074 fallocate 0x0374c1 thru 0x03ae7c (0x39bc bytes) 000674 1788653959.639474 read 0x02605a thru 0x02df32 (0x7ed9 bytes) 000675 1788653959.657653 punch from 0x018eb8 to 0x01c823 (0x396c bytes) 000676 1788653959.667645 trunc from 0x007b9e to 0x00df6c (0x63cf bytes) 000677 1788653959.674167 mapread 0x014aa3 thru 0x01b6bf (0x6c1d bytes) 000678 1788653959.718240 read 0x00072e thru 0x009d9f (0x9672 bytes) 000680 1788653959.742945 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3ae7d): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ae7d): Mirror 3: 0x6a62d569 (stale) Mirror 4: 0x6a62d569 (stale) Mirror 5: 0x6a62d569 (stale) Mirror 7: 0x6a62d569 (stale) Mirror 8: 0x6a62d569 (stale) CRC-32 checksum value for chunk [0, 0x3ae7d): Mirror 1: 0xa6cbe1e8 Mirror 9: 0xa6cbe1e8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ae7d: skipped 000681 1788653961.356718 mapwrite 0x008012 thru 0x013692 (0xb681 bytes) 000682 1788653961.558957 write 0x02f581 thru 0x03051f (0xf9f bytes) 000684 1788653961.595801 MIRROR_RESYNC 7 mirrors 000685 1788653964.704107 trunc from 0x03ae7d to 0x0221c9 (0x18cb3 bytes) 000686 1788653964.785345 MIRROR_RESYNC 7 mirrors 000687 1788653966.800018 read 0x002e00 thru 0x011ab5 (0xecb6 bytes) 000688 1788653966.811879 mapread 0x004e12 thru 0x00d42b (0x861a bytes) 000689 1788653966.812707 trunc from 0x00c114 to 0x01a29d (0xe18a bytes) 000690 1788653966.868416 MIRROR_RESYNC 7 mirrors 000691 1788653968.955613 read 0x01f87f thru 0x0221c9 (0x294b bytes) 000692 1788653968.960859 mapwrite 0x01fd2d thru 0x024b1b (0x4def bytes) 000693 1788653969.038208 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x24b1c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24b1c): Mirror 3: 0x309d56af (stale) Mirror 4: 0x309d56af (stale) Mirror 5: 0x309d56af (stale) Mirror 7: 0x309d56af (stale) Mirror 8: 0x309d56af (stale) Mirror 9: 0x309d56af (stale) CRC-32 checksum value for chunk [0, 0x24b1c): Mirror 1: 0xa36b007 Mirror 10: 0xa36b007 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24b1c: skipped 000694 1788653970.217687 read 0x01084b thru 0x01725c (0x6a12 bytes) 000695 1788653970.235274 write 0x034c7b thru 0x0373ae (0x2734 bytes) 000696 1788653970.287627 trunc from 0x01284b to 0x01b771 (0x8f27 bytes) 000697 1788653970.293584 write 0x03e6de thru 0x03ffff (0x1922 bytes) 000698 1788653970.296518 trunc from 0x03a432 to 0x03ffff (0x5bce bytes) 000699 1788653970.310263 mapread 0x02cee8 thru 0x0307ae (0x38c7 bytes) 000701 1788653970.330863 write 0x01075e thru 0x013771 (0x3014 bytes) 000702 1788653970.341375 mapwrite 0x0079da thru 0x009e6e (0x2495 bytes) 000703 1788653970.367811 trunc from 0x040000 to 0x00d3fe (0x32c01 bytes) 000704 1788653970.386694 fallocate 0x0177d9 thru 0x0258d5 (0xe0fd bytes) 000705 1788653970.392904 mapread 0x0196ee thru 0x0258d5 (0xc1e8 bytes) 000706 1788653970.410355 fallocate 0x02d9c7 thru 0x03d671 (0xfcab bytes) 000707 1788653970.416663 mapwrite 0x0284c9 thru 0x02ed92 (0x68ca bytes) 000708 1788653970.444234 trunc from 0x03d672 to 0x024eb1 (0x187c0 bytes) 000709 1788653970.457826 fallocate 0x030a49 thru 0x032a00 (0x1fb8 bytes) 000710 1788653970.463731 punch from 0x02d7b6 to 0x02fa51 (0x229c bytes) 000711 1788653970.474484 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x32a01): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32a01): Mirror 3: 0x41342f87 (stale) Mirror 4: 0x41342f87 (stale) Mirror 5: 0x41342f87 (stale) Mirror 7: 0x41342f87 (stale) Mirror 8: 0x41342f87 (stale) Mirror 9: 0x41342f87 (stale) Mirror 10: 0x7fd6871b (stale) CRC-32 checksum value for chunk [0, 0x32a01): Mirror 1: 0x2169014d Mirror 11: 0x2169014d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32a01: skipped 000712 1788653971.734703 write 0x030e37 thru 0x03fe4d (0xf017 bytes) 000713 1788653971.791215 punch from 0x02f273 to 0x03802f (0x8dbd bytes) 000714 1788653971.807696 punch from 0x039a71 to 0x03fe4d (0x63dd bytes) 000715 1788653971.812793 write 0x00a0e9 thru 0x014f35 (0xae4d bytes) 000716 1788653971.823036 mapwrite 0x012020 thru 0x01c4eb (0xa4cc bytes) 000717 1788653971.859174 read 0x0100e0 thru 0x016166 (0x6087 bytes) 000718 1788653971.859842 trunc from 0x03fe4e to 0x0161f8 (0x29c55 bytes) 000719 1788653971.883114 trunc from 0x011c29 to 0x0161f8 (0x45d0 bytes) 000720 1788653971.888013 mapwrite 0x018844 thru 0x021680 (0x8e3d bytes) 000721 1788653971.939619 MIRROR_RESYNC 9 mirrors 000722 1788653974.169213 mapread 0x01a4f3 thru 0x01d936 (0x3444 bytes) 000723 1788653974.210355 MIRROR_RESYNC 9 mirrors 000724 1788653974.893242 mapwrite 0x00a33a thru 0x01076b (0x6432 bytes) 000725 1788653975.010015 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x21681): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21681): Mirror 3: 0xdfa230a9 (stale) Mirror 4: 0xdfa230a9 (stale) Mirror 5: 0xdfa230a9 (stale) Mirror 7: 0xdfa230a9 (stale) Mirror 8: 0xdfa230a9 (stale) Mirror 9: 0xdfa230a9 (stale) Mirror 10: 0xdfa230a9 (stale) Mirror 11: 0xdfa230a9 (stale) CRC-32 checksum value for chunk [0, 0x21681): Mirror 1: 0x46db33f7 Mirror 12: 0x46db33f7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21681: skipped 000726 1788653976.317115 write 0x02d01d thru 0x03c0e1 (0xf0c5 bytes) 000728 1788653976.404492 write 0x0212b8 thru 0x02aedc (0x9c25 bytes) 000729 1788653976.424861 fallocate 0x03c351 thru 0x03fe6e (0x3b1e bytes) 000730 1788653976.430208 fallocate 0x02bbb7 thru 0x032a5d (0x6ea7 bytes) 000731 1788653976.435740 mapread 0x007432 thru 0x00bc03 (0x47d2 bytes) 000732 1788653976.453160 mapread 0x0201e7 thru 0x02346e (0x3288 bytes) 000733 1788653976.462623 read 0x0116a9 thru 0x0180c2 (0x6a1a bytes) 000734 1788653976.471762 fallocate 0x0311da thru 0x037997 (0x67be bytes) 000736 1788653976.477990 mapread 0x03eec1 thru 0x03fe6e (0xfae bytes) 000737 1788653976.486569 punch from 0x0244d1 to 0x024fea (0xb1a bytes) 000741 1788653976.513522 punch from 0x024b61 to 0x025a4e (0xeee bytes) 000742 1788653976.518791 trunc from 0x03fe6f to 0x03bb2d (0x4341 bytes) 000743 1788653976.549298 trunc from 0x03bb2e to 0x00fc1e (0x2bf0f bytes) 000744 1788653976.576531 punch from 0x004618 to 0x0078fa (0x32e3 bytes) 000745 1788653976.588920 fallocate 0x00a320 thru 0x00c1f5 (0x1ed6 bytes) 000746 1788653976.595793 write 0x00a68d thru 0x00ec19 (0x458d bytes) 000748 1788653976.597448 mapwrite 0x03e4f8 thru 0x03ffff (0x1b08 bytes) 000749 1788653976.645767 trunc from 0x040000 to 0x00aff3 (0x3500c bytes) 000750 1788653976.709791 trunc from 0x00aff4 to 0x034f2d (0x29f3a bytes) 000751 1788653976.733739 read 0x02c6d3 thru 0x034f2d (0x885b bytes) 000753 1788653976.777326 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34f2e): Mirror 3: 0x75347aa7 (stale) Mirror 4: 0x75347aa7 (stale) Mirror 5: 0x75347aa7 (stale) Mirror 7: 0x75347aa7 (stale) Mirror 9: 0x75347aa7 (stale) Mirror 10: 0x75347aa7 (stale) Mirror 11: 0x75347aa7 (stale) Mirror 12: 0x5e02800 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34f2e): Mirror 3: 0x75347aa7 (stale) Mirror 4: 0x75347aa7 (stale) Mirror 5: 0x75347aa7 (stale) Mirror 7: 0x75347aa7 (stale) Mirror 9: 0x75347aa7 (stale) Mirror 10: 0x75347aa7 (stale) Mirror 11: 0x75347aa7 (stale) Mirror 12: 0x5e02800 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34f2e: skipped 000755 1788653977.517654 mapread 0x01455a thru 0x01f74c (0xb1f3 bytes) 000756 1788653977.611274 trunc from 0x034f2e to 0x03594a (0xa1d bytes) 000757 1788653977.678719 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3594b): Mirror 3: 0xa214a4f6 (stale) Mirror 4: 0xa214a4f6 (stale) Mirror 5: 0xa214a4f6 (stale) Mirror 7: 0xa214a4f6 (stale) Mirror 10: 0xa214a4f6 (stale) Mirror 11: 0xa214a4f6 (stale) Mirror 12: 0x801e57cd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3594b): Mirror 3: 0xa214a4f6 (stale) Mirror 4: 0xa214a4f6 (stale) Mirror 5: 0xa214a4f6 (stale) Mirror 7: 0xa214a4f6 (stale) Mirror 10: 0xa214a4f6 (stale) Mirror 11: 0xa214a4f6 (stale) Mirror 12: 0x801e57cd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3594b: skipped 000758 1788653978.315235 read 0x02f2a0 thru 0x03594a (0x66ab bytes) 000759 1788653978.328740 trunc from 0x03594b to 0x017e2c (0x1db1e bytes) 000760 1788653978.354769 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x17e2d): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17e2d): Mirror 3: 0x85e9bf25 (stale) Mirror 4: 0x85e9bf25 (stale) Mirror 5: 0x85e9bf25 (stale) Mirror 7: 0x85e9bf25 (stale) Mirror 10: 0x85e9bf25 (stale) Mirror 11: 0x85e9bf25 (stale) Mirror 12: 0xe7c72915 (stale) CRC-32 checksum value for chunk [0, 0x17e2d): Mirror 1: 0x17f80e94 Mirror 13: 0x17f80e94 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17e2d: skipped 000762 1788653979.916690 mapread 0x0042cd thru 0x006cbf (0x29f3 bytes) 000763 1788653980.005583 punch from 0x0033b1 to 0x013272 (0xfec2 bytes) 000764 1788653980.088640 mapwrite 0x022eb1 thru 0x02a40c (0x755c bytes) 000765 1788653980.209199 trunc from 0x02a40d to 0x015434 (0x14fd8 bytes) 000766 1788653980.256515 write 0x00832f thru 0x012117 (0x9de9 bytes) 000767 1788653980.273890 punch from 0x00d75a to 0x00f4cf (0x1d76 bytes) 000768 1788653980.333011 read 0x004157 thru 0x00b495 (0x733f bytes) 000769 1788653980.349575 trunc from 0x015435 to 0x02cde7 (0x179b3 bytes) 000770 1788653980.378009 write 0x03f703 thru 0x03ffff (0x8fd bytes) 000772 1788653980.381007 read 0x0356a8 thru 0x03ffff (0xa958 bytes) 000774 1788653980.419922 read 0x01f775 thru 0x028dfc (0x9688 bytes) 000775 1788653980.421124 punch from 0x03fe4a to 0x03ffff (0x1b6 bytes) 000777 1788653980.432257 trunc from 0x040000 to 0x023515 (0x1caea bytes) 000778 1788653980.459664 read 0x012c0b thru 0x017863 (0x4c59 bytes) 000779 1788653980.461463 write 0x036f08 thru 0x03ffff (0x90f8 bytes) 000780 1788653980.468862 write 0x010556 thru 0x01b282 (0xad2d bytes) 000781 1788653980.479693 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa2233db4 (stale) Mirror 4: 0xa2233db4 (stale) Mirror 5: 0xa2233db4 (stale) Mirror 7: 0xa2233db4 (stale) Mirror 10: 0xa2233db4 (stale) Mirror 11: 0xa2233db4 (stale) Mirror 12: 0xd7387936 (stale) Mirror 13: 0x67775733 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x66d8f384 Mirror 14: 0x66d8f384 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000782 1788653981.813390 read 0x039abf thru 0x03dbbc (0x40fe bytes) 000783 1788653981.838181 write 0x02a2e9 thru 0x035bf2 (0xb90a bytes) 000784 1788653981.935117 punch from 0x00f698 to 0x019969 (0xa2d2 bytes) 000786 1788653981.950466 mapwrite 0x02464e thru 0x032216 (0xdbc9 bytes) 000787 1788653982.049235 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa2233db4 (stale) Mirror 4: 0xa2233db4 (stale) Mirror 5: 0xa2233db4 (stale) Mirror 7: 0xa2233db4 (stale) Mirror 11: 0xa2233db4 (stale) Mirror 12: 0xd7387936 (stale) Mirror 13: 0x67775733 (stale) Mirror 14: 0x66d8f384 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa2233db4 (stale) Mirror 4: 0xa2233db4 (stale) Mirror 5: 0xa2233db4 (stale) Mirror 7: 0xa2233db4 (stale) Mirror 11: 0xa2233db4 (stale) Mirror 12: 0xd7387936 (stale) Mirror 13: 0x67775733 (stale) Mirror 14: 0x66d8f384 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000788 1788653982.717037 write 0x013689 thru 0x01c66a (0x8fe2 bytes) 000789 1788653982.736600 trunc from 0x027630 to 0x027b1c (0x4ed bytes) 000791 1788653982.745359 trunc from 0x02de40 to 0x03ca8e (0xec4f bytes) 000793 1788653982.751945 mapwrite 0x00c2ea thru 0x01029e (0x3fb5 bytes) 000794 1788653982.822748 mapread 0x029f88 thru 0x037a1f (0xda98 bytes) 000796 1788653982.866605 mapread 0x0282b1 thru 0x02a0af (0x1dff bytes) 000797 1788653982.868422 mapwrite 0x01f893 thru 0x028a7a (0x91e8 bytes) 000798 1788653982.912846 mapwrite 0x010c0a thru 0x0126d3 (0x1aca bytes) 000799 1788653982.935869 trunc from 0x040000 to 0x0243e5 (0x1bc1a bytes) 000800 1788653982.953657 write 0x031025 thru 0x03e919 (0xd8f5 bytes) 000801 1788653982.961751 MIRROR_RESYNC 9 mirrors 000802 1788653986.520997 mapwrite 0x028cf7 thru 0x030b64 (0x7e6e bytes) 000804 1788653986.685016 punch from 0x02c467 to 0x035b96 (0x9730 bytes) 000805 1788653986.708303 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e91a): Mirror 3: 0x1f2ea5a1 (stale) Mirror 4: 0x1f2ea5a1 (stale) Mirror 5: 0x1f2ea5a1 (stale) Mirror 7: 0x1f2ea5a1 (stale) Mirror 12: 0x1f2ea5a1 (stale) Mirror 13: 0x1f2ea5a1 (stale) Mirror 14: 0x1f2ea5a1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e91a): Mirror 3: 0x1f2ea5a1 (stale) Mirror 4: 0x1f2ea5a1 (stale) Mirror 5: 0x1f2ea5a1 (stale) Mirror 7: 0x1f2ea5a1 (stale) Mirror 12: 0x1f2ea5a1 (stale) Mirror 13: 0x1f2ea5a1 (stale) Mirror 14: 0x1f2ea5a1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e91a: skipped 000806 1788653987.319627 mapread 0x03685c thru 0x03e919 (0x80be bytes) 000808 1788653987.340903 trunc from 0x03e91a to 0x01baf0 (0x22e29 bytes) 000809 1788653987.368396 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1baf1): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1baf1): Mirror 3: 0x6bf76e71 (stale) Mirror 4: 0x6bf76e71 (stale) Mirror 5: 0x6bf76e71 (stale) Mirror 7: 0x6bf76e71 (stale) Mirror 12: 0x6bf76e71 (stale) Mirror 13: 0x6bf76e71 (stale) Mirror 14: 0x6bf76e71 (stale) CRC-32 checksum value for chunk [0, 0x1baf1): Mirror 1: 0x6bf76e71 Mirror 15: 0x6bf76e71 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1baf1: skipped 000810 1788653988.783288 write 0x03375f thru 0x03596f (0x2211 bytes) 000811 1788653988.847229 mapread 0x013fc1 thru 0x017a2f (0x3a6f bytes) 000812 1788653988.875481 write 0x00f52b thru 0x013ab2 (0x4588 bytes) 000813 1788653988.877993 trunc from 0x004577 to 0x00e526 (0x9fb0 bytes) 000815 1788653988.884239 punch from 0x00c003 to 0x0175f6 (0xb5f4 bytes) 000816 1788653988.903026 mapwrite 0x01c9da thru 0x02361e (0x6c45 bytes) 000818 1788653988.939532 fallocate 0x02857f thru 0x02ff97 (0x7a19 bytes) 000819 1788653988.949282 punch from 0x027ced to 0x02e6b1 (0x69c5 bytes) 000820 1788653988.958489 mapread 0x021dc3 thru 0x0247e8 (0x2a26 bytes) 000821 1788653988.959548 fallocate 0x010980 thru 0x018df4 (0x8475 bytes) 000822 1788653988.968977 punch from 0x02f8ba to 0x03596f (0x60b6 bytes) 000823 1788653988.979479 punch from 0x02f8de to 0x033b26 (0x4249 bytes) 000824 1788653988.993297 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x35970): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35970): Mirror 3: 0x7af2b9a8 (stale) Mirror 4: 0x7af2b9a8 (stale) Mirror 5: 0x7af2b9a8 (stale) Mirror 7: 0x7af2b9a8 (stale) Mirror 12: 0x7af2b9a8 (stale) Mirror 13: 0x7af2b9a8 (stale) Mirror 14: 0x7af2b9a8 (stale) Mirror 15: 0xce04b997 (stale) CRC-32 checksum value for chunk [0, 0x35970): Mirror 1: 0x5b749443 Mirror 16: 0x5b749443 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35970: skipped 000826 1788653990.565006 mapwrite 0x01e960 thru 0x02221a (0x38bb bytes) 000827 1788653990.726835 read 0x00d7c0 thru 0x019e8a (0xc6cb bytes) 000828 1788653990.739844 write 0x02c066 thru 0x037757 (0xb6f2 bytes) 000829 1788653990.760018 mapread 0x033a12 thru 0x037757 (0x3d46 bytes) 000830 1788653990.767193 write 0x012390 thru 0x014990 (0x2601 bytes) 000831 1788653990.769533 trunc from 0x037758 to 0x01fbd2 (0x17b85 bytes) 000832 1788653990.805576 read 0x00c6a6 thru 0x0137d9 (0x7134 bytes) 000833 1788653990.824828 fallocate 0x00c66f thru 0x01c589 (0xff1b bytes) 000834 1788653990.832446 trunc from 0x006d2a to 0x00a18b (0x3462 bytes) 000835 1788653990.838197 read 0x012c5c thru 0x018d15 (0x60ba bytes) 000838 1788653990.841503 trunc from 0x01fbd3 to 0x016a61 (0x9171 bytes) 000839 1788653990.867953 trunc from 0x016a62 to 0x033808 (0x1cda7 bytes) 000841 1788653990.901256 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x33809): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33809): Mirror 3: 0x63bf78e0 (stale) Mirror 4: 0x63bf78e0 (stale) Mirror 5: 0x63bf78e0 (stale) Mirror 7: 0x63bf78e0 (stale) Mirror 12: 0x63bf78e0 (stale) Mirror 13: 0x63bf78e0 (stale) Mirror 14: 0x63bf78e0 (stale) Mirror 15: 0xfaf6aff5 (stale) Mirror 16: 0xdf78ceda (stale) CRC-32 checksum value for chunk [0, 0x33809): Mirror 1: 0xa67e125a Mirror 17: 0xa67e125a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33809: skipped 000842 1788653992.153685 trunc from 0x01de30 to 0x026305 (0x84d6 bytes) 000843 1788653992.215241 punch from 0x000620 to 0x00b8b0 (0xb291 bytes) 000845 1788653992.225812 punch from 0x02f35f to 0x033808 (0x44aa bytes) 000846 1788653992.230960 punch from 0x031b94 to 0x033808 (0x1c75 bytes) 000847 1788653992.235262 mapwrite 0x0090e7 thru 0x015624 (0xc53e bytes) 000848 1788653992.280481 read 0x01f2b9 thru 0x020878 (0x15c0 bytes) 000849 1788653992.286972 mapwrite 0x03269f thru 0x036cc9 (0x462b bytes) 000850 1788653992.332428 mapread 0x02d76c thru 0x036cc9 (0x955e bytes) 000852 1788653992.334465 fallocate 0x00b1cc thru 0x00f0a5 (0x3eda bytes) 000853 1788653992.339838 punch from 0x030eb0 to 0x036cc9 (0x5e1a bytes) 000854 1788653992.343996 mapread 0x0278cb thru 0x029e1a (0x2550 bytes) 000855 1788653992.352702 fallocate 0x00a209 thru 0x01144d (0x7245 bytes) 000856 1788653992.361582 MIRROR_RESYNC 11 mirrors 000857 1788653995.368953 trunc from 0x036cca to 0x003157 (0x33b72 bytes) 000858 1788653995.423839 read 0x001964 thru 0x003157 (0x17f4 bytes) 000860 1788653995.432728 trunc from 0x000119 to 0x003157 (0x303f bytes) 000862 1788653995.437633 read 0x00313c thru 0x003157 (0x1c bytes) 000863 1788653995.450668 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3158): Mirror 4: 0xd2631588 (stale) Mirror 5: 0xd2631588 (stale) Mirror 7: 0xd2631588 (stale) Mirror 12: 0xd2631588 (stale) Mirror 13: 0xd2631588 (stale) Mirror 14: 0xd2631588 (stale) Mirror 15: 0xd2631588 (stale) Mirror 16: 0xd2631588 (stale) Mirror 17: 0xd2631588 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3158): Mirror 4: 0xd2631588 (stale) Mirror 5: 0xd2631588 (stale) Mirror 7: 0xd2631588 (stale) Mirror 12: 0xd2631588 (stale) Mirror 13: 0xd2631588 (stale) Mirror 14: 0xd2631588 (stale) Mirror 15: 0xd2631588 (stale) Mirror 16: 0xd2631588 (stale) Mirror 17: 0xd2631588 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3158: skipped 000864 1788653996.123303 trunc from 0x0002c1 to 0x0017f0 (0x1530 bytes) 000865 1788653996.129536 mapwrite 0x0249f4 thru 0x02b940 (0x6f4d bytes) 000866 1788653996.249896 MIRROR_RESYNC 10 mirrors 000867 1788653998.758671 trunc from 0x02b941 to 0x025664 (0x62dc bytes) 000868 1788653998.856859 MIRROR_RESYNC 10 mirrors 000869 1788654000.369577 mapread 0x01190f thru 0x012640 (0xd32 bytes) 000870 1788654000.381100 trunc from 0x025665 to 0x01ef79 (0x66eb bytes) 000871 1788654000.426308 trunc from 0x01ef7a to 0x01173c (0xd83d bytes) 000872 1788654000.451303 trunc from 0x01173d to 0x009bb4 (0x7b88 bytes) 000874 1788654000.464812 trunc from 0x004ede to 0x009bb4 (0x4cd7 bytes) 000875 1788654000.472348 read 0x008d08 thru 0x009bb4 (0xead bytes) 000876 1788654000.478501 write 0x0195cb thru 0x026049 (0xca7f bytes) 000877 1788654000.480250 fallocate 0x02abe6 thru 0x02d2b7 (0x26d2 bytes) 000878 1788654000.494015 punch from 0x00d117 to 0x015953 (0x883d bytes) 000879 1788654000.504035 read 0x011bbc thru 0x01827e (0x66c3 bytes) 000880 1788654000.517096 trunc from 0x0075c3 to 0x00dcf1 (0x672f bytes) 000881 1788654000.527775 mapread 0x00c30b thru 0x012fcd (0x6cc3 bytes) 000882 1788654000.545696 mapwrite 0x0374f8 thru 0x03e814 (0x731d bytes) 000883 1788654000.590720 trunc from 0x03e815 to 0x007da0 (0x36a74 bytes) 000885 1788654000.611146 mapwrite 0x02b5ef thru 0x02fed6 (0x48e8 bytes) 000886 1788654000.680139 read 0x020e4d thru 0x028dfa (0x7fae bytes) 000887 1788654000.702875 punch from 0x02bd66 to 0x02dc21 (0x1ebc bytes) 000888 1788654000.712253 mapread 0x0068b3 thru 0x00ae1e (0x456c bytes) 000890 1788654000.713642 trunc from 0x00a802 to 0x019f83 (0xf782 bytes) 000892 1788654000.723108 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fed7): Mirror 4: 0x490175a2 (stale) Mirror 5: 0x490175a2 (stale) Mirror 7: 0x490175a2 (stale) Mirror 12: 0x490175a2 (stale) Mirror 13: 0x490175a2 (stale) Mirror 14: 0x490175a2 (stale) Mirror 15: 0x490175a2 (stale) Mirror 17: 0x490175a2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fed7): Mirror 4: 0x490175a2 (stale) Mirror 5: 0x490175a2 (stale) Mirror 7: 0x490175a2 (stale) Mirror 12: 0x490175a2 (stale) Mirror 13: 0x490175a2 (stale) Mirror 14: 0x490175a2 (stale) Mirror 15: 0x490175a2 (stale) Mirror 17: 0x490175a2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fed7: skipped 000893 1788654001.482237 mapread 0x026a33 thru 0x02a49d (0x3a6b bytes) 000894 1788654001.541569 mapwrite 0x007c6c thru 0x0150d4 (0xd469 bytes) 000895 1788654001.654615 mapwrite 0x01d108 thru 0x02aab4 (0xd9ad bytes) 000897 1788654001.691783 write 0x03fe74 thru 0x03ffff (0x18c bytes) 000898 1788654001.698838 mapread 0x025d7c thru 0x02869d (0x2922 bytes) 000899 1788654001.708309 mapread 0x037fe4 thru 0x03ffff (0x801c bytes) 000900 1788654001.732354 trunc from 0x040000 to 0x0284e7 (0x17b18 bytes) 000901 1788654001.761587 read 0x00753a thru 0x0104e4 (0x8fab bytes) 000902 1788654001.763754 mapwrite 0x03e90f thru 0x03ffff (0x16f1 bytes) 000903 1788654001.847791 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x3c26a85f (stale) Mirror 5: 0x3c26a85f (stale) Mirror 7: 0x3c26a85f (stale) Mirror 13: 0x3c26a85f (stale) Mirror 14: 0x3c26a85f (stale) Mirror 15: 0x3c26a85f (stale) Mirror 17: 0x3c26a85f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x3c26a85f (stale) Mirror 5: 0x3c26a85f (stale) Mirror 7: 0x3c26a85f (stale) Mirror 13: 0x3c26a85f (stale) Mirror 14: 0x3c26a85f (stale) Mirror 15: 0x3c26a85f (stale) Mirror 17: 0x3c26a85f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000905 1788654002.547590 mapwrite 0x00f431 thru 0x018cae (0x987e bytes) 000906 1788654002.597537 trunc from 0x040000 to 0x009e8b (0x36174 bytes) 000907 1788654002.625296 mapwrite 0x0159c0 thru 0x015cee (0x32f bytes) 000908 1788654002.674403 mapwrite 0x019d7e thru 0x019e55 (0xd8 bytes) 000909 1788654002.747558 trunc from 0x019e56 to 0x027db1 (0xdf5c bytes) 000910 1788654002.770964 read 0x01f299 thru 0x027db1 (0x8b19 bytes) 000911 1788654002.802123 mapread 0x008c1e thru 0x010364 (0x7747 bytes) 000912 1788654002.846784 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27db2): Mirror 5: 0x4e7bf547 (stale) Mirror 7: 0x4e7bf547 (stale) Mirror 13: 0x4e7bf547 (stale) Mirror 14: 0x4e7bf547 (stale) Mirror 15: 0x4e7bf547 (stale) Mirror 17: 0x4e7bf547 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27db2): Mirror 5: 0x4e7bf547 (stale) Mirror 7: 0x4e7bf547 (stale) Mirror 13: 0x4e7bf547 (stale) Mirror 14: 0x4e7bf547 (stale) Mirror 15: 0x4e7bf547 (stale) Mirror 17: 0x4e7bf547 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27db2: skipped 000913 1788654003.590654 punch from 0x016e59 to 0x019da9 (0x2f51 bytes) 000914 1788654003.595822 fallocate 0x010add thru 0x018aa7 (0x7fcb bytes) 000915 1788654003.603571 punch from 0x00105b to 0x004116 (0x30bc bytes) 000916 1788654003.612266 mapread 0x00a740 thru 0x00f0c7 (0x4988 bytes) 000917 1788654003.632739 punch from 0x01a109 to 0x01c423 (0x231b bytes) 000919 1788654003.646301 write 0x03aefb thru 0x03ffff (0x5105 bytes) 000920 1788654003.656142 mapread 0x039404 thru 0x03ffff (0x6bfc bytes) 000921 1788654003.681230 trunc from 0x019927 to 0x024421 (0xaafb bytes) 000922 1788654003.687635 mapread 0x03817c thru 0x03ffff (0x7e84 bytes) 000923 1788654003.691720 mapread 0x017d08 thru 0x021577 (0x9870 bytes) 000924 1788654003.726609 trunc from 0x040000 to 0x03ca20 (0x35df bytes) 000925 1788654003.746164 mapread 0x037839 thru 0x037d1a (0x4e2 bytes) 000926 1788654003.747087 fallocate 0x01315d thru 0x015268 (0x210c bytes) 000927 1788654003.757467 trunc from 0x00027f to 0x00b2f2 (0xb074 bytes) 000928 1788654003.763827 trunc from 0x0395fe to 0x03aa6b (0x146e bytes) 000929 1788654003.779664 mapwrite 0x001ffc thru 0x0116d4 (0xf6d9 bytes) 000930 1788654003.825191 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3ca21): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ca21): Mirror 5: 0x26ceb8ab (stale) Mirror 7: 0x26ceb8ab (stale) Mirror 13: 0x26ceb8ab (stale) Mirror 14: 0x26ceb8ab (stale) Mirror 15: 0x26ceb8ab (stale) Mirror 17: 0x26ceb8ab (stale) CRC-32 checksum value for chunk [0, 0x3ca21): Mirror 1: 0xf659c328 Mirror 18: 0xf659c328 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ca21: skipped 000932 1788654005.186246 MIRROR_RESYNC 8 mirrors 000933 1788654007.370024 mapread 0x02aebf thru 0x0342b6 (0x93f8 bytes) 000934 1788654007.432634 MIRROR_EXTEND to 9 mirrors 000936 1788654008.589481 write 0x0114e3 thru 0x016472 (0x4f90 bytes) 000937 1788654008.648340 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ca21): Mirror 5: 0xf659c328 (stale) Mirror 7: 0xf659c328 (stale) Mirror 13: 0xf659c328 (stale) Mirror 14: 0xf659c328 (stale) Mirror 17: 0xf659c328 (stale) Mirror 18: 0xf659c328 (stale) Mirror 19: 0xf659c328 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ca21): Mirror 5: 0xf659c328 (stale) Mirror 7: 0xf659c328 (stale) Mirror 13: 0xf659c328 (stale) Mirror 14: 0xf659c328 (stale) Mirror 17: 0xf659c328 (stale) Mirror 18: 0xf659c328 (stale) Mirror 19: 0xf659c328 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ca21: skipped 000938 1788654009.479923 fallocate 0x009d55 thru 0x013e30 (0xa0dc bytes) 000939 1788654009.499110 fallocate 0x003c8b thru 0x0059b9 (0x1d2f bytes) 000940 1788654009.536194 punch from 0x03186d to 0x033bb4 (0x2348 bytes) 000941 1788654009.576698 MIRROR_RESYNC 8 mirrors 000942 1788654012.095832 trunc from 0x01e00f to 0x029bac (0xbb9e bytes) 000943 1788654012.139674 mapwrite 0x0334d8 thru 0x03e2e8 (0xae11 bytes) 000944 1788654012.240014 fallocate 0x01beec thru 0x01d1f1 (0x1306 bytes) 000945 1788654012.252409 MIRROR_RESYNC 8 mirrors 000946 1788654014.098948 fallocate 0x03ac98 thru 0x03ffff (0x5368 bytes) 000947 1788654014.141222 trunc from 0x027c42 to 0x033689 (0xba48 bytes) 000948 1788654014.148628 fallocate 0x027e71 thru 0x02966a (0x17fa bytes) 000949 1788654014.155274 fallocate 0x023d52 thru 0x02895a (0x4c09 bytes) 000950 1788654014.162120 trunc from 0x040000 to 0x03c99c (0x3663 bytes) 000951 1788654014.191108 trunc from 0x03c99d to 0x00dced (0x2ecaf bytes) 000953 1788654014.218506 fallocate 0x00fef6 thru 0x015d5c (0x5e67 bytes) 000954 1788654014.229010 mapread 0x00d24e thru 0x015d5c (0x8b0f bytes) 000955 1788654014.256008 fallocate 0x036bad thru 0x039b48 (0x2f9c bytes) 000956 1788654014.262678 trunc from 0x039b49 to 0x0204d5 (0x19673 bytes) 000957 1788654014.285487 read 0x01cd7a thru 0x0204d5 (0x375c bytes) 000958 1788654014.301485 fallocate 0x01eaec thru 0x023d5b (0x5270 bytes) 000959 1788654014.307981 trunc from 0x0204d6 to 0x014a47 (0xba8e bytes) 000961 1788654014.345191 fallocate 0x0295d3 thru 0x035d45 (0xc773 bytes) 000962 1788654014.360755 mapread 0x006a80 thru 0x00ede3 (0x8364 bytes) 000963 1788654014.373618 write 0x011be9 thru 0x018872 (0x6c8a bytes) 000965 1788654014.377006 punch from 0x00168c to 0x00401c (0x2991 bytes) 000966 1788654014.383663 trunc from 0x001526 to 0x00a4ca (0x8fa5 bytes) 000968 1788654014.390395 punch from 0x01209e to 0x014b89 (0x2aec bytes) 000969 1788654014.424918 mapwrite 0x02b942 thru 0x035f0a (0xa5c9 bytes) 000970 1788654014.500947 trunc from 0x014192 to 0x016403 (0x2272 bytes) 000971 1788654014.510927 mapread 0x00d836 thru 0x011fe3 (0x47ae bytes) 000972 1788654014.516931 mapwrite 0x00a289 thru 0x01a048 (0xfdc0 bytes) 000973 1788654014.681146 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35f0b): Mirror 5: 0x7fcccdd7 (stale) Mirror 13: 0x7fcccdd7 (stale) Mirror 14: 0x7fcccdd7 (stale) Mirror 17: 0x7fcccdd7 (stale) Mirror 18: 0x7fcccdd7 (stale) Mirror 19: 0x7fcccdd7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35f0b): Mirror 5: 0x7fcccdd7 (stale) Mirror 13: 0x7fcccdd7 (stale) Mirror 14: 0x7fcccdd7 (stale) Mirror 17: 0x7fcccdd7 (stale) Mirror 18: 0x7fcccdd7 (stale) Mirror 19: 0x7fcccdd7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35f0b: skipped 000974 1788654015.389437 trunc from 0x01d419 to 0x028028 (0xac10 bytes) 000975 1788654015.408703 punch from 0x018399 to 0x01e738 (0x63a0 bytes) 000979 1788654015.423478 trunc from 0x00c4ad to 0x00ee48 (0x299c bytes) 000980 1788654015.432359 fallocate 0x03cb22 thru 0x03ffff (0x34de bytes) 000981 1788654015.439588 mapwrite 0x027d51 thru 0x02f8df (0x7b8f bytes) 000982 1788654015.524585 trunc from 0x035f0b to 0x025e4f (0x100bb bytes) 000984 1788654015.558323 punch from 0x013637 to 0x01b84e (0x8218 bytes) 000985 1788654015.569356 trunc from 0x0149ee to 0x021c48 (0xd25b bytes) 000986 1788654015.595569 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25e50): Mirror 5: 0x14f4fa05 (stale) Mirror 14: 0x14f4fa05 (stale) Mirror 17: 0x14f4fa05 (stale) Mirror 18: 0x14f4fa05 (stale) Mirror 19: 0x14f4fa05 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25e50): Mirror 5: 0x14f4fa05 (stale) Mirror 14: 0x14f4fa05 (stale) Mirror 17: 0x14f4fa05 (stale) Mirror 18: 0x14f4fa05 (stale) Mirror 19: 0x14f4fa05 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25e50: skipped 000987 1788654016.367991 write 0x00129c thru 0x005a20 (0x4785 bytes) 000988 1788654016.384390 mapwrite 0x019042 thru 0x01b849 (0x2808 bytes) 000989 1788654016.500919 trunc from 0x01bf41 to 0x0243ac (0x846c bytes) 000990 1788654016.511920 read 0x02505c thru 0x025e4f (0xdf4 bytes) 000991 1788654016.524239 punch from 0x004df2 to 0x012fdd (0xe1ec bytes) 000992 1788654016.532304 read 0x0219ef thru 0x0256aa (0x3cbc bytes) 000993 1788654016.547973 fallocate 0x03dfca thru 0x03ffff (0x2036 bytes) 000994 1788654016.556917 trunc from 0x025e50 to 0x0313b3 (0xb564 bytes) 000995 1788654016.572330 mapread 0x0019b2 thru 0x00609d (0x46ec bytes) 000996 1788654016.573252 fallocate 0x03f116 thru 0x03ffff (0xeea bytes) 000997 1788654016.581014 fallocate 0x011e44 thru 0x011fa1 (0x15e bytes) 000998 1788654016.587593 write 0x022f75 thru 0x02a1a1 (0x722d bytes) 001000 1788654016.597476 punch from 0x009bda to 0x00e2e8 (0x470f bytes) All operations completed A-OK! PASS 70a (132s) == sanity-flr test 71: check mirror extend parent fid ==== 20:20:23 (1788654023) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x199:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (12s) == sanity-flr test 72: check append on FLR file ========== 20:20:36 (1788654036) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.32833 s, 6.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x138:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196655 s, 5.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x138:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x138:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x138:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x138:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x139:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 20:20:47 (1788654047) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre2 Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4317600 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6588960 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6144928 .. Extending file size to 8378624 .. Extending file size to 8113248 .. Extending file size to 8424832 .. Extending file size to 8368320 .. Extending file size to 8830624 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9082208 .. Extending file size to 9159392 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 61397 61399 61400 61402 61404 done 192.168.204.113@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre2 (opts:) 192.168.204.113@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (93s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:22:20 (1788654140) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre2 Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 1244128 .. Extending file size to 2607840 .. failed Extending file size to 7931552 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8843328 .. Extending file size to 7162240 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8800256 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9050528 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9067616 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 62740 62741 62744 62746 62747 192.168.204.113@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre2 (opts:) 192.168.204.113@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:23:40 (1788654220) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2460 1268844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1269244 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24588 3582432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18764 3588256 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43352 7170688 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2460 1268844 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2060 1269244 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24588 3582432 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18764 3588256 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43352 7170688 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.117827 s, 17.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.32796 s, 6.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x10a6a8dd (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.113@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre2 (opts:) PASS 200c (24s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:24:07 (1788654247) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.108834 s, 19.3 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 20:24:15 (1788654255) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa0:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:24:24 (1788654264) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.067134 s, 31.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:24:33 (1788654273) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.137927 s, 15.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20d:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:24:42 (1788654282) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.133381 s, 15.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x150:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x150:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } PASS 204c (37s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:25:19 (1788654319) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg413-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776348 s, 13.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20d:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20d:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20e:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:26:15 (1788654375) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg413-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x152:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100755 s, 20.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x152:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x151:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x152:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x151:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (48s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:27:04 (1788654424) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg413-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg413-server: Pool lustre.test_204f created Adding targets to pool oleg413-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg413-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.197117 s, 10.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x153:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x152:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x153:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x152:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg413-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg413-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg413-server: Pool lustre.test_204f destroyed PASS 204f (75s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:28:18 (1788654498) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:28:28 (1788654508) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:28:36 (1788654516) oleg413-server: Pool lustre.test_206 created Adding targets to pool oleg413-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg413-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg413-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg413-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg413-server: Pool lustre.test_206 destroyed PASS 206 (33s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:29:09 (1788654549) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136471 s, 7.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318875 s, 32.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x29caa9d4 Mirror 3: 0x29caa9d4 PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:29:22 (1788654562) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:29:26 (1788654566) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:29:31 (1788654571) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0557054 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xf60000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1fe0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1fe0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1fe0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x17f0000 Read this (4 bytes): BBBB 192.168.204.113@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre2 (opts:) PASS 209a (44s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:30:15 (1788654615) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0187972 s, 436 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:30:23 (1788654623) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692551 s, 15.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xbf lmm_fid: [0x200000405:0xbf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xbf lmm_fid: [0x200000405:0xbf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15d:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611327 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 20:30:25.000000000 -0400 Modify: 2026-09-05 20:30:26.000000000 -0400 Change: 2026-09-05 20:30:27.000000000 -0400 Birth: 2026-09-05 20:30:26.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:30:34 (1788654634) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382476 s, 27.4 MB/s oleg413-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg413-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg413-server: O/280000401/d29/15d: File not found by ext2_lookup PASS 210b (26s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:31:00 (1788654660) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.285455 s, 143 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0217082 s, 189 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 5 20:31 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xc4:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 5 20:31 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:31:10 (1788654670) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:31:18 (1788654678) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1936 sec =========== 20:31:27 (1788654687) === sanity-flr: start cleanup 20:31:29 (1788654689) === === sanity-flr: finish cleanup 20:31:38 (1788654698) ===