-----============= acceptance-small: sanity-flr ============----- Thu Apr 2 17:52:49 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg218-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 17:52:56 (1775166776) === oleg218-client.virtnet: executing check_config_client /mnt/lustre oleg218-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg218-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b5244dd0800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b5244dd0800.idle_timeout=debug disable quota as required oleg218-server: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 17:53:11 (1775166791) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8181] == sanity-flr test 0a: lfs mirror create with -N option == 17:53:12 (1775166792) PASS 0a (12s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 17:53:24 (1775166804) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 17:53:27 (1775166807) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 17:53:29 (1775166809) PASS 0d (22s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 17:53:51 (1775166831) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 17:53:55 (1775166835) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 17:53:59 (1775166839) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9b5244dd0800.stats=clear osc.lustre-OST0001-osc-ffff9b5244dd0800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 17:54:07 (1775166847) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (7s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 17:54:14 (1775166854) PASS 0j (5s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 17:54:19 (1775166859) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 17:54:21 (1775166861) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (6s) == sanity-flr test 2: create components from existing files ========================================================== 17:54:27 (1775166867) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 17:54:31 (1775166871) PASS 3 (5s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 17:54:36 (1775166876) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (5s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 17:54:41 (1775166881) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (4s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 17:54:47 (1775166887) PASS 7 (4s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 17:54:51 (1775166891) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 0.956944 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062234 s, 16.8 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 17:54:58 (1775166898) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.10906 s, 57.7 MB/s PASS 22 (5s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 17:55:03 (1775166903) Stopping /mnt/lustre-ost1 (opts:) on oleg218-server oleg218-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b5244dd0800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b5244dd0800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg218-server: oleg218-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg218-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg218-server oleg218-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b5244dd0800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b5244dd0800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg218-server: oleg218-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg218-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (44s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 17:55:48 (1775166948) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0935463 s, 33.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg218-server oleg218-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b5244dd0800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b5244dd0800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg218-server: oleg218-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg218-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (18s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 17:56:07 (1775166967) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg218-server oleg218-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b5244dd0800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b5244dd0800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg218-server: oleg218-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg218-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg218-server oleg218-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b5244dd0800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b5244dd0800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg218-server: oleg218-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg218-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (45s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 17:56:51 (1775167011) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.8252 s, 28.7 MB/s -rw-r--r-- 1 root root 52428800 Apr 2 17:56 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.491s user 0m0.000s sys 0m0.162s Stopping /mnt/lustre-ost1 (opts:) on oleg218-server oleg218-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b5244dd0800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b5244dd0800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.543s user 0m0.001s sys 0m0.161s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg218-server: oleg218-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg218-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg218-server oleg218-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b5244dd0800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b5244dd0800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.733s user 0m0.001s sys 0m0.182s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg218-server: oleg218-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg218-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg218-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b5244dd0800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b5244dd0800.ost_server_uuid in FULL state after 0 sec PASS 33b (51s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 17:57:42 (1775167062) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 17:57:44 (1775167064) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 17:57:46 (1775167066) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 17:57:48 (1775167068) PASS 35 (4s) == sanity-flr test 36a: write to mirrored files ========== 17:57:52 (1775167072) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.34729 s, 36.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (8s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 17:58:00 (1775167080) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033151 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234159 s, 44.8 MB/s PASS 36b (5s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 17:58:05 (1775167085) PASS 36c (5s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 17:58:10 (1775167090) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00070316 s, 1.5 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000815932 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00093388 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00086551 s, 1.2 MB/s flv=10 olv=10 PASS 36d (6s) == sanity-flr test 37: mirror I/O API verification ======= 17:58:17 (1775167097) create test files with size 15750795 .. 449b2e97822c91fb3717fcd44356545c - 74754f133bb6f1d8d15788c1448456ab - 5bc809bdc20e0523ec2dcfbb8f3431ea - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg218-client.virtnet /mnt/lustre (opts:) Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre PASS 37 (13s) == sanity-flr test 38: resync ============================ 17:58:29 (1775167109) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (7s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 17:58:37 (1775167117) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 17:58:40 (1775167120) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504028 s, 20.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3e:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018152 s, 57.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (5s) == sanity-flr test 41: lfs mirror resync check =========== 17:58:45 (1775167125) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0521557 s, 40.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.158949 s, 26.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.105148 s, 19.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (8s) == sanity-flr test 42: lfs mirror verify ================= 17:58:53 (1775167133) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 17:58:55 (1775167135) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 17:58:56 (1775167136) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309174 s, 33.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (5s) == sanity-flr test 44a: lfs mirror split check =========== 17:59:01 (1775167141) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.219847 s, 14.3 MB/s PASS 44a (6s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 17:59:07 (1775167147) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x48:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4a:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4a:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (5s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 17:59:13 (1775167153) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.160272 s, 65.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg218-client.virtnet /mnt/lustre (opts:) Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (8s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 17:59:21 (1775167161) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9b52587fd000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.149818 s, 70.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg218-client.virtnet /mnt/lustre (opts:) Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9b5260053000.xattr_cache=1 PASS 44e (7s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 17:59:28 (1775167168) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (18s) == sanity-flr test 46: Verify setstripe --copy option ==== 17:59:46 (1775167186) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (16s) == sanity-flr test 47: Verify mirror obj alloc =========== 18:00:02 (1775167202) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 18:00:03 (1775167203) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0772752 s, 40.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0530962 s, 59.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x69440c8d (stale) Mirror 2: 0x69440c8d (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 34c33290658337375f2e3a1eb845405d - mirror 1 checksum: ad5f37c8c445da667021fb04a35090dd - mirror 2 checksum: 34c33290658337375f2e3a1eb845405d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Apr 2 18:00:05 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Apr 2 18:00:05 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 34c33290658337375f2e3a1eb845405d - mirror 1 checksum: ad5f37c8c445da667021fb04a35090dd - mirror 2 checksum: ad5f37c8c445da667021fb04a35090dd - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Apr 2 18:00:05 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Apr 2 18:00:05 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } PASS 48 (4s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 18:00:07 (1775167207) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193297 s, 54.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (3s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 18:00:10 (1775167210) Failing mds1 on oleg218-server Stopping /mnt/lustre-mds1 (opts:) on oleg218-server 18:00:18 (1775167218) shut down facet: mds1 facet_host: oleg218-server facet_failover_host: oleg218-server Failover mds1 to oleg218-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg218-server: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-MDT0000 18:00:33 (1775167233) targets are mounted 18:00:33 (1775167233) facet_failover done oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139158 s, 75.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } PASS 50A (31s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 18:00:41 (1775167241) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.010742 s, 1.9 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0080866 s, 2.5 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 18:00:45 (1775167245) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0131737 s, 1.6 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00448572 s, 4.6 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 18:00:48 (1775167248) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00298121 s, 5.5 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 18:00:51 (1775167251) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0184525 s, 1.1 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 18:00:54 (1775167254) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00198423 s, 2.1 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0165943 s, 247 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 18:00:56 (1775167256) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) PASS 60b (7s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 18:01:03 (1775167263) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 18:01:23 (1775167283) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x48:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 18:01:41 (1775167301) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 18:01:59 (1775167319) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 18:02:02 (1775167322) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 55859 mirror split pid 55860 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 55859 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 55860 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 18:03:05 (1775167385) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 7386 truncating to largest ever: 0x75e2 000001 1775167386.123328 trunc from 00000000 to 0x0075e1 (0x75e2 bytes) 000002 1775167386.133914 trunc from 0x005fb9 to 0x0075e1 (0x1629 bytes) 000003 1775167386.135850 write 0x02190e thru 0x02376b (0x1e5e bytes) 000004 1775167386.136844 mapread 0x015dd4 thru 0x02376b (0xd998 bytes) fallocating to largest ever: 0x40000 000005 1775167386.148256 fallocate 0x03d3c7 thru 0x03ffff (0x2c39 bytes) 000006 1775167386.154229 write 0x00546b thru 0x0132b0 (0xde46 bytes) 000008 1775167386.163662 trunc from 0x022681 to 0x02296e (0x2ee bytes) 000009 1775167386.172520 mapread 0x015445 thru 0x015d60 (0x91c bytes) 000011 1775167386.174092 trunc from 0x02376c to 0x018329 (0xb442 bytes) 000012 1775167386.193869 trunc from 0x01832a to 0x00ea3e (0x98eb bytes) 000013 1775167386.203981 punch from 0x001ef5 to 0x00a74f (0x885b bytes) 000014 1775167386.207012 trunc from 0x005b8d to 0x00923b (0x36af bytes) 000015 1775167386.209611 write 0x00a0fa thru 0x01986c (0xf773 bytes) 000016 1775167386.213961 write 0x0186c1 thru 0x01f97a (0x72ba bytes) 000017 1775167386.214977 punch from 0x0039c5 to 0x00ef3a (0xb576 bytes) 000018 1775167386.225146 fallocate 0x0377a9 thru 0x03a34b (0x2ba3 bytes) 000020 1775167386.228087 write 0x00242d thru 0x004f54 (0x2b28 bytes) 000021 1775167386.232059 punch from 0x029d1c to 0x02b709 (0x19ee bytes) 000022 1775167386.235434 mapwrite 0x027647 thru 0x036d62 (0xf71c bytes) 000023 1775167386.262022 mapwrite 0x022a35 thru 0x026362 (0x392e bytes) 000024 1775167386.275670 fallocate 0x02bee2 thru 0x031247 (0x5366 bytes) 000025 1775167386.282514 trunc from 0x035248 to 0x03a34b (0x5104 bytes) 000026 1775167386.285261 write 0x00045e thru 0x00c860 (0xc403 bytes) 000027 1775167386.288435 trunc from 0x0077fd to 0x0165cf (0xedd3 bytes) 000029 1775167386.300557 trunc from 0x01084a to 0x01e225 (0xd9dc bytes) 000030 1775167386.303632 punch from 0x008534 to 0x01542b (0xcef8 bytes) 000031 1775167386.306085 trunc from 0x03a34c to 0x038722 (0x1c29 bytes) 000032 1775167386.311809 trunc from 0x002814 to 0x00495a (0x2147 bytes) 000033 1775167386.314390 fallocate 0x026bbe thru 0x0345e4 (0xda27 bytes) 000034 1775167386.316873 trunc from 0x038723 to 0x024fc6 (0x1375c bytes) 000035 1775167386.323000 fallocate 0x02c1b6 thru 0x035072 (0x8ebd bytes) 000036 1775167386.327486 mapwrite 0x0224b5 thru 0x0250a3 (0x2bef bytes) 000037 1775167386.337478 trunc from 0x035073 to 0x03d2cf (0x825d bytes) 000039 1775167386.347454 trunc from 0x03d2d0 to 0x010067 (0x2d268 bytes) 000040 1775167386.354529 mapwrite 0x015c7a thru 0x019479 (0x3800 bytes) 000041 1775167386.375981 trunc from 0x003140 to 0x009b34 (0x69f5 bytes) 000042 1775167386.378592 read 0x00295f thru 0x00ec5b (0xc2fd bytes) 000043 1775167386.382133 trunc from 0x007240 to 0x009f2d (0x2cee bytes) 000044 1775167386.384549 fallocate 0x0312ba thru 0x03adc8 (0x9b0f bytes) 000045 1775167386.387327 mapwrite 0x0022ed thru 0x0083bf (0x60d3 bytes) 000047 1775167386.399501 fallocate 0x03eddd thru 0x03ffff (0x1223 bytes) 000048 1775167386.402427 read 0x028592 thru 0x02942d (0xe9c bytes) 000050 1775167386.407639 mapread 0x037675 thru 0x03adc8 (0x3754 bytes) 000051 1775167386.408602 mapread 0x032b5e thru 0x0356f6 (0x2b99 bytes) 000052 1775167386.409591 punch from 0x038a5c to 0x03adc8 (0x236d bytes) 000053 1775167386.412477 punch from 0x028d9c to 0x029370 (0x5d5 bytes) 000055 1775167386.414872 trunc from 0x026264 to 0x032907 (0xc6a4 bytes) 000056 1775167386.417120 fallocate 0x002a3d thru 0x004967 (0x1f2b bytes) 000057 1775167386.419544 read 0x031a8f thru 0x03adc8 (0x933a bytes) 000058 1775167386.424970 punch from 0x0056f1 to 0x0071a5 (0x1ab5 bytes) 000059 1775167386.427620 mapwrite 0x017147 thru 0x01abb1 (0x3a6b bytes) 000060 1775167386.434831 fallocate 0x0073f5 thru 0x011fc4 (0xabd0 bytes) 000062 1775167386.439434 mapwrite 0x019793 thru 0x027314 (0xdb82 bytes) 000063 1775167386.448922 fallocate 0x023471 thru 0x0235fe (0x18e bytes) 000064 1775167386.451295 punch from 0x02380a to 0x030433 (0xcc2a bytes) 000065 1775167386.453427 mapread 0x01c54b thru 0x026602 (0xa0b8 bytes) 000066 1775167386.461538 read 0x02a087 thru 0x039fef (0xff69 bytes) 000067 1775167386.465130 fallocate 0x02bf6a thru 0x03b647 (0xf6de bytes) 000068 1775167386.467860 trunc from 0x03b648 to 0x0004ff (0x3b148 bytes) 000069 1775167386.477587 fallocate 0x01d0dd thru 0x01de1e (0xd42 bytes) 000070 1775167386.481794 read 0x0069b9 thru 0x014afa (0xe142 bytes) 000071 1775167386.486876 trunc from 0x015b8e to 0x01de1e (0x8291 bytes) 000072 1775167386.489882 trunc from 0x00ff7a to 0x01b0c1 (0xb148 bytes) 000073 1775167386.492764 mapwrite 0x009792 thru 0x00eeba (0x5729 bytes) 000074 1775167386.508741 write 0x010762 thru 0x01e618 (0xdeb7 bytes) 000076 1775167386.513762 read 0x01d6b1 thru 0x01e618 (0xf68 bytes) 000077 1775167386.514439 trunc from 0x01b0ae to 0x01e275 (0x31c8 bytes) 000078 1775167386.529304 mapwrite 0x0222aa thru 0x02ada2 (0x8af9 bytes) 000079 1775167386.555919 punch from 0x005558 to 0x011c1e (0xc6c7 bytes) 000081 1775167386.563162 MIRROR_EXTEND to 2 mirrors 000083 1775167386.809234 fallocate 0x00415e thru 0x00d80f (0x96b2 bytes) 000084 1775167386.824908 fallocate 0x032a25 thru 0x03ffff (0xd5db bytes) 000085 1775167386.827445 mapwrite 0x0164f9 thru 0x01caa3 (0x65ab bytes) 000086 1775167386.843132 mapread 0x02484b thru 0x024a14 (0x1ca bytes) 000087 1775167386.847145 fallocate 0x025054 thru 0x028a3b (0x39e8 bytes) 000088 1775167386.849721 trunc from 0x011d78 to 0x01811b (0x63a4 bytes) 000089 1775167386.852516 punch from 0x02f819 to 0x03163b (0x1e23 bytes) 000091 1775167386.855375 fallocate 0x02a48c thru 0x0371cf (0xcd44 bytes) 000092 1775167386.858038 read 0x017082 thru 0x0196c8 (0x2647 bytes) 000093 1775167386.861856 mapread 0x03c47b thru 0x03ee84 (0x2a0a bytes) 000094 1775167386.867652 MIRROR_RESYNC 2 mirrors 000095 1775167387.183896 trunc from 0x021c84 to 0x023901 (0x1c7e bytes) 000097 1775167387.197942 read 0x00e056 thru 0x018ec2 (0xae6d bytes) 000099 1775167387.203626 mapread 0x00ae29 thru 0x01526c (0xa444 bytes) 000100 1775167387.212403 mapwrite 0x02fee3 thru 0x03fe5d (0xff7b bytes) 000101 1775167387.233182 mapwrite 0x0176dc thru 0x026df1 (0xf716 bytes) 000102 1775167387.256518 trunc from 0x03213f to 0x03c32a (0xa1ec bytes) 000103 1775167387.262033 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x24c7511c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc54030b1 Mirror 3: 0xc54030b1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000104 1775167387.645026 trunc from 0x01401f to 0x021f33 (0xdf15 bytes) 000105 1775167387.662475 trunc from 0x040000 to 0x0126bf (0x2d940 bytes) 000106 1775167387.682767 MIRROR_RESYNC 3 mirrors 000107 1775167387.969498 trunc from 0x010f71 to 0x0126bf (0x174f bytes) 000108 1775167387.985848 punch from 0x0022a4 to 0x00582c (0x3589 bytes) 000109 1775167387.988421 write 0x03d70f thru 0x03ffff (0x28f1 bytes) 000110 1775167387.989632 punch from 0x033578 to 0x03b811 (0x829a bytes) 000111 1775167387.993697 MIRROR_RESYNC 3 mirrors 000112 1775167388.250671 mapread 0x01e79e thru 0x02e195 (0xf9f8 bytes) 000113 1775167388.260819 write 0x027df7 thru 0x030fad (0x91b7 bytes) 000114 1775167388.279661 punch from 0x00c3b2 to 0x0173fb (0xb04a bytes) 000115 1775167388.284567 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9bec6628 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9bec6628 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000116 1775167388.418947 write 0x025e41 thru 0x027b8f (0x1d4f bytes) 000117 1775167388.431993 trunc from 0x02daaa to 0x030cb4 (0x320b bytes) 000118 1775167388.436457 trunc from 0x040000 to 0x010a16 (0x2f5e9 bytes) 000119 1775167388.451711 mapread 0x0101a5 thru 0x010a16 (0x872 bytes) 000120 1775167388.456763 fallocate 0x02bdfd thru 0x02e74e (0x2952 bytes) 000121 1775167388.462331 mapwrite 0x007887 thru 0x01080d (0x8f87 bytes) 000122 1775167388.475842 mapread 0x003eb4 thru 0x009048 (0x5195 bytes) 000123 1775167388.476810 read 0x00564c thru 0x010a16 (0xb3cb bytes) 000124 1775167388.477398 fallocate 0x015b0c thru 0x0222f6 (0xc7eb bytes) 000125 1775167388.480482 read 0x0078f7 thru 0x00c705 (0x4e0f bytes) 000126 1775167388.484985 write 0x024a5a thru 0x033d6c (0xf313 bytes) 000127 1775167388.486608 mapwrite 0x03d1c1 thru 0x03ffff (0x2e3f bytes) 000128 1775167388.506923 mapwrite 0x03ee80 thru 0x03ffff (0x1180 bytes) 000129 1775167388.515234 punch from 0x02964a to 0x035ba9 (0xc560 bytes) 000130 1775167388.518146 trunc from 0x001d11 to 0x006f43 (0x5233 bytes) 000131 1775167388.524689 write 0x024fe2 thru 0x02af73 (0x5f92 bytes) 000132 1775167388.528168 trunc from 0x01cbf6 to 0x029272 (0xc67d bytes) 000133 1775167388.534463 fallocate 0x013c02 thru 0x01dede (0xa2dd bytes) 000134 1775167388.536994 mapwrite 0x01bca6 thru 0x02a104 (0xe45f bytes) 000135 1775167388.577928 trunc from 0x00a7cc to 0x0137a6 (0x8fdb bytes) 000137 1775167388.583908 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) 000138 1775167388.631793 punch from 0x022824 to 0x02fbc8 (0xd3a5 bytes) 000139 1775167388.634945 trunc from 0x040000 to 0x03d5ed (0x2a12 bytes) 000140 1775167388.643142 trunc from 0x02f95f to 0x033fda (0x467c bytes) 000141 1775167388.645618 write 0x00321d thru 0x003635 (0x419 bytes) 000142 1775167388.650441 MIRROR_EXTEND to 2 mirrors 000143 1775167388.923639 trunc from 0x03d5ee to 0x03f402 (0x1e15 bytes) 000144 1775167388.949364 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) 000146 1775167388.996593 mapwrite 0x0399f6 thru 0x03f55d (0x5b68 bytes) 000147 1775167389.018158 MIRROR_EXTEND to 2 mirrors 000148 1775167389.232030 mapread 0x0367f8 thru 0x03c995 (0x619e bytes) 000149 1775167389.238682 fallocate 0x0155ff thru 0x0213c3 (0xbdc5 bytes) 000150 1775167389.250035 fallocate 0x032ee3 thru 0x039292 (0x63b0 bytes) 000151 1775167389.252815 trunc from 0x01a3cd to 0x02a1ec (0xfe20 bytes) 000152 1775167389.254516 write 0x00365a thru 0x006dc7 (0x376e bytes) 000153 1775167389.260317 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) 000154 1775167389.304479 trunc from 0x03f55e to 0x038b62 (0x69fb bytes) 000155 1775167389.319962 mapread 0x01cc80 thru 0x029fc2 (0xd343 bytes) 000156 1775167389.336183 trunc from 0x038b63 to 0x00cd7b (0x2bde7 bytes) 000157 1775167389.341274 read 0x00449e thru 0x00cd7b (0x88de bytes) 000158 1775167389.344052 read 0x00cd0a thru 0x00cd7b (0x72 bytes) 000159 1775167389.344520 mapread 0x0042f0 thru 0x00cd7b (0x8a8c bytes) 000160 1775167389.345272 fallocate 0x0243fc thru 0x02b82e (0x7433 bytes) 000161 1775167389.347383 trunc from 0x0093e1 to 0x00cd7b (0x399b bytes) 000162 1775167389.349270 mapread 0x00ca97 thru 0x00cd7b (0x2e5 bytes) 000163 1775167389.352918 trunc from 0x000508 to 0x00cd7b (0xc874 bytes) 000164 1775167389.355041 fallocate 0x0164f4 thru 0x022ad6 (0xc5e3 bytes) 000165 1775167389.356891 punch from 0x0067f4 to 0x00c512 (0x5d1f bytes) 000166 1775167389.358633 trunc from 0x004a38 to 0x00c33c (0x7905 bytes) 000167 1775167389.360717 mapwrite 0x00ed9c thru 0x00f68f (0x8f4 bytes) 000168 1775167389.373684 trunc from 0x0052cf to 0x00a099 (0x4dcb bytes) 000169 1775167389.375492 write 0x033ca1 thru 0x03ec37 (0xaf97 bytes) 000171 1775167389.377379 fallocate 0x00145e thru 0x00a642 (0x91e5 bytes) 000172 1775167389.379694 trunc from 0x03ac65 to 0x03ec37 (0x3fd3 bytes) 000173 1775167389.384907 punch from 0x0396c6 to 0x03a49e (0xdd9 bytes) 000174 1775167389.386940 trunc from 0x0252c7 to 0x0275a4 (0x22de bytes) 000175 1775167389.388828 punch from 0x01a7a2 to 0x028c12 (0xe471 bytes) 000176 1775167389.390540 punch from 0x02aa78 to 0x03556f (0xaaf8 bytes) 000177 1775167389.392310 mapread 0x01e312 thru 0x022605 (0x42f4 bytes) 000178 1775167389.396651 read 0x01ede8 thru 0x02179c (0x29b5 bytes) 000179 1775167389.397129 mapread 0x02b481 thru 0x03a794 (0xf314 bytes) 000180 1775167389.398231 read 0x00f826 thru 0x017071 (0x784c bytes) 000181 1775167389.398775 write 0x008013 thru 0x01490b (0xc8f9 bytes) 000182 1775167389.402795 trunc from 0x03ec38 to 0x031c23 (0xd014 bytes) 000186 1775167389.410616 mapwrite 0x02c334 thru 0x037099 (0xad66 bytes) 000187 1775167389.439235 mapwrite 0x016730 thru 0x0221e6 (0xbab7 bytes) 000191 1775167389.450493 punch from 0x022fb8 to 0x02ed32 (0xbd7b bytes) 000193 1775167389.454958 MIRROR_EXTEND to 2 mirrors 000194 1775167389.735501 fallocate 0x027697 thru 0x02a1bc (0x2b26 bytes) 000195 1775167389.751832 punch from 0x034b6c to 0x037099 (0x252e bytes) 000196 1775167389.754278 fallocate 0x010ea3 thru 0x0170f1 (0x624f bytes) 000197 1775167389.756867 mapwrite 0x02b791 thru 0x033ea4 (0x8714 bytes) 000198 1775167389.776884 fallocate 0x01ca5c thru 0x01caa1 (0x46 bytes) 000199 1775167389.780246 trunc from 0x03709a to 0x01ec82 (0x18417 bytes) 000200 1775167389.802926 read 0x010f7d thru 0x019aa4 (0x8b28 bytes) 000201 1775167389.811274 mapwrite 0x008b0b thru 0x008e05 (0x2fb bytes) 000202 1775167389.825108 read 0x01b656 thru 0x01ec82 (0x362d bytes) 000203 1775167389.829327 write 0x019d41 thru 0x021aa9 (0x7d69 bytes) 000204 1775167389.830457 punch from 0x00dab2 to 0x01abbe (0xd10d bytes) 000205 1775167389.836947 mapread 0x01ae16 thru 0x01b141 (0x32c bytes) 000206 1775167389.843311 MIRROR_RESYNC 2 mirrors 000207 1775167390.038638 trunc from 0x011703 to 0x017707 (0x6005 bytes) 000208 1775167390.053799 trunc from 0x0065bf to 0x011a63 (0xb4a5 bytes) 000209 1775167390.056166 mapwrite 0x014b7e thru 0x01b604 (0x6a87 bytes) 000210 1775167390.073144 read 0x00bd72 thru 0x014b65 (0x8df4 bytes) 000211 1775167390.076467 trunc from 0x021aaa to 0x03c865 (0x1adbc bytes) 000212 1775167390.095771 trunc from 0x002822 to 0x00b881 (0x9060 bytes) 000213 1775167390.100912 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) 000215 1775167390.174159 mapread 0x030f57 thru 0x033071 (0x211b bytes) 000216 1775167390.184618 mapwrite 0x01ae55 thru 0x022dfe (0x7faa bytes) 000217 1775167390.198558 punch from 0x02cb16 to 0x02cdac (0x297 bytes) 000218 1775167390.200963 mapread 0x03bd38 thru 0x03c865 (0xb2e bytes) 000219 1775167390.204388 punch from 0x01e99e to 0x028100 (0x9763 bytes) 000220 1775167390.206830 fallocate 0x008f05 thru 0x01753c (0xe638 bytes) 000221 1775167390.209142 trunc from 0x03c866 to 0x03f4db (0x2c76 bytes) 000222 1775167390.215974 mapread 0x03d7d7 thru 0x03f4db (0x1d05 bytes) 000224 1775167390.220069 fallocate 0x02ac99 thru 0x03a56d (0xf8d5 bytes) 000225 1775167390.222568 fallocate 0x000d60 thru 0x008a68 (0x7d09 bytes) 000226 1775167390.225047 fallocate 0x018fc4 thru 0x01e1e6 (0x5223 bytes) 000227 1775167390.227147 trunc from 0x023311 to 0x023a58 (0x748 bytes) 000229 1775167390.230062 write 0x020b66 thru 0x02b39a (0xa835 bytes) 000230 1775167390.236229 fallocate 0x0203cd thru 0x021978 (0x15ac bytes) 000231 1775167390.238993 write 0x02ca49 thru 0x031bcb (0x5183 bytes) 000232 1775167390.241848 trunc from 0x02cb13 to 0x02d9e7 (0xed5 bytes) 000233 1775167390.247182 read 0x02444e thru 0x02dfb8 (0x9b6b bytes) 000234 1775167390.249923 write 0x03397e thru 0x03c627 (0x8caa bytes) 000235 1775167390.251117 write 0x01c257 thru 0x0286d7 (0xc481 bytes) 000236 1775167390.252454 punch from 0x02a4a8 to 0x02eb03 (0x465c bytes) 000237 1775167390.255750 mapread 0x024f77 thru 0x02ea5f (0x9ae9 bytes) 000238 1775167390.260506 trunc from 0x008947 to 0x00f75d (0x6e17 bytes) 000239 1775167390.262697 mapwrite 0x00e60a thru 0x01b91b (0xd312 bytes) 000240 1775167390.277370 trunc from 0x03f4dc to 0x0363ea (0x90f1 bytes) 000241 1775167390.288602 trunc from 0x0363eb to 0x039e78 (0x3a8e bytes) 000242 1775167390.298730 trunc from 0x039e79 to 0x0116a8 (0x287d0 bytes) 000243 1775167390.307771 fallocate 0x01f4b6 thru 0x028dfe (0x9949 bytes) 000246 1775167390.311785 read 0x01c658 thru 0x01fc26 (0x35cf bytes) 000247 1775167390.321175 trunc from 0x007132 to 0x014957 (0xd826 bytes) 000248 1775167390.323957 trunc from 0x01a9de to 0x01f735 (0x4d58 bytes) 000249 1775167390.326417 read 0x025ff9 thru 0x0287e3 (0x27eb bytes) 000250 1775167390.327063 read 0x01bc6a thru 0x01e732 (0x2ac9 bytes) 000252 1775167390.334515 MIRROR_EXTEND to 2 mirrors 000253 1775167390.564879 read 0x001cdf thru 0x00b0b2 (0x93d4 bytes) 000254 1775167390.570121 MIRROR_EXTEND to 3 mirrors 000255 1775167390.796751 mapread 0x0121a0 thru 0x01847a (0x62db bytes) 000256 1775167390.811758 write 0x00142c thru 0x00fce2 (0xe8b7 bytes) 000257 1775167390.839078 write 0x002928 thru 0x0113a2 (0xea7b bytes) 000258 1775167390.843005 punch from 0x0284a9 to 0x028dfe (0x956 bytes) 000259 1775167390.845128 read 0x01082b thru 0x010854 (0x2a bytes) 000260 1775167390.846841 trunc from 0x01914d to 0x01b3b8 (0x226c bytes) 000261 1775167390.849150 trunc from 0x028dff to 0x01d989 (0xb475 bytes) 000262 1775167390.870444 punch from 0x00b528 to 0x00e959 (0x3432 bytes) 000263 1775167390.876720 write 0x00aee6 thru 0x00f10d (0x4228 bytes) 000264 1775167390.877946 trunc from 0x013091 to 0x0131db (0x14b bytes) 000265 1775167390.881016 fallocate 0x02e18b thru 0x032755 (0x45cb bytes) 000267 1775167390.884174 trunc from 0x017621 to 0x01d989 (0x6369 bytes) 000269 1775167390.887881 mapwrite 0x006c2e thru 0x009f6c (0x333f bytes) 000270 1775167390.898217 trunc from 0x011177 to 0x011750 (0x5da bytes) 000271 1775167390.901020 read 0x010875 thru 0x01aacc (0xa258 bytes) 000272 1775167390.904688 trunc from 0x01d98a to 0x030d70 (0x133e7 bytes) 000273 1775167390.916519 fallocate 0x036a5c thru 0x037a50 (0xff5 bytes) 000274 1775167390.922124 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30d71): Mirror 3: 0x4d0b3e2c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30d71): Mirror 3: 0x4d0b3e2c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30d71: skipped 000275 1775167391.030444 trunc from 0x01f1b6 to 0x02b39f (0xc1ea bytes) 000277 1775167391.032786 read 0x025469 thru 0x030d70 (0xb908 bytes) 000278 1775167391.036015 read 0x014507 thru 0x022e04 (0xe8fe bytes) 000279 1775167391.041228 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) 000280 1775167391.102607 MIRROR_EXTEND to 2 mirrors 000281 1775167391.354622 trunc from 0x00f629 to 0x010983 (0x135b bytes) 000282 1775167391.367447 trunc from 0x01957f to 0x026cdd (0xd75f bytes) 000283 1775167391.369780 punch from 0x02839b to 0x030d70 (0x89d6 bytes) 000284 1775167391.372124 trunc from 0x030d71 to 0x03bd09 (0xaf99 bytes) 000285 1775167391.381077 mapwrite 0x028b70 thru 0x031392 (0x8823 bytes) 000286 1775167391.397606 mapwrite 0x0238eb thru 0x02bcc6 (0x83dc bytes) 000287 1775167391.406698 write 0x02df58 thru 0x03c78f (0xe838 bytes) 000288 1775167391.408118 mapread 0x00b8df thru 0x015bfb (0xa31d bytes) 000289 1775167391.418600 trunc from 0x03c790 to 0x020d08 (0x1ba87 bytes) 000290 1775167391.424477 read 0x00a519 thru 0x010115 (0x5bfd bytes) 000292 1775167391.425369 trunc from 0x013ad7 to 0x020d08 (0xd232 bytes) 000293 1775167391.427620 write 0x0393ef thru 0x03c66c (0x327e bytes) 000294 1775167391.428394 fallocate 0x01290d thru 0x0136bd (0xdb1 bytes) 000295 1775167391.430946 mapread 0x037865 thru 0x03c66c (0x4e08 bytes) 000296 1775167391.434819 mapwrite 0x03b146 thru 0x03ffff (0x4eba bytes) 000297 1775167391.455228 write 0x03888c thru 0x03e8b2 (0x6027 bytes) 000298 1775167391.456425 fallocate 0x030ae4 thru 0x038d36 (0x8253 bytes) 000300 1775167391.460054 fallocate 0x00cc6e thru 0x019ac8 (0xce5b bytes) 000301 1775167391.462524 mapread 0x00a7ab thru 0x011691 (0x6ee7 bytes) 000302 1775167391.463628 read 0x01e7e3 thru 0x02b2d4 (0xcaf2 bytes) 000303 1775167391.468137 trunc from 0x040000 to 0x01ce3a (0x231c5 bytes) 000304 1775167391.474290 read 0x00fd9e thru 0x01cad6 (0xcd39 bytes) 000305 1775167391.475020 mapwrite 0x003cfc thru 0x00b5a8 (0x78ad bytes) 000306 1775167391.486947 fallocate 0x0072bb thru 0x00ca54 (0x579a bytes) 000307 1775167391.489495 trunc from 0x0100a9 to 0x017d70 (0x7cc8 bytes) 000308 1775167391.491516 write 0x00e8fc thru 0x016d70 (0x8475 bytes) 000310 1775167391.494906 punch from 0x01bf37 to 0x01ce3a (0xf04 bytes) 000311 1775167391.497059 fallocate 0x01ea79 thru 0x02418e (0x5716 bytes) 000312 1775167391.499609 mapwrite 0x0062b1 thru 0x0160bf (0xfe0f bytes) 000313 1775167391.509876 fallocate 0x0002a6 thru 0x003e68 (0x3bc3 bytes) 000314 1775167391.511903 mapwrite 0x0305ee thru 0x03a4d6 (0x9ee9 bytes) 000315 1775167391.533074 mapwrite 0x0012cf thru 0x004d98 (0x3aca bytes) 000316 1775167391.541773 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3a4d7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a4d7): Mirror 2: 0x525be7b3 (stale) CRC-32 checksum value for chunk [0, 0x3a4d7): Mirror 1: 0xfc5a9ae0 Mirror 3: 0xfc5a9ae0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a4d7: skipped 000317 1775167391.798631 read 0x013ca7 thru 0x016706 (0x2a60 bytes) 000318 1775167391.803442 read 0x0320dd thru 0x0385c0 (0x64e4 bytes) 000319 1775167391.810653 fallocate 0x00655c thru 0x00a548 (0x3fed bytes) 000320 1775167391.824877 mapread 0x021b39 thru 0x02c02b (0xa4f3 bytes) 000321 1775167391.847437 mapwrite 0x02e243 thru 0x03d26f (0xf02d bytes) 000322 1775167391.910847 read 0x03339c thru 0x03d26f (0x9ed4 bytes) 000323 1775167391.911472 trunc from 0x02d678 to 0x031db5 (0x473e bytes) 000324 1775167391.914130 read 0x027294 thru 0x035887 (0xe5f4 bytes) 000325 1775167391.918378 punch from 0x00c72a to 0x018d07 (0xc5de bytes) 000326 1775167391.921582 trunc from 0x032686 to 0x0334b4 (0xe2f bytes) 000327 1775167391.927084 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3d270): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d270): Mirror 2: 0xc71bb228 (stale) Mirror 3: 0xc38d4619 (stale) CRC-32 checksum value for chunk [0, 0x3d270): Mirror 1: 0x549cd311 Mirror 4: 0x549cd311 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d270: skipped 000328 1775167392.226779 mapwrite 0x02da76 thru 0x03b9ac (0xdf37 bytes) 000330 1775167392.275973 fallocate 0x02f5c3 thru 0x02fe7a (0x8b8 bytes) 000331 1775167392.277940 fallocate 0x021b1f thru 0x025e75 (0x4357 bytes) 000332 1775167392.281972 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3d270): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d270): Mirror 2: 0xc71bb228 (stale) Mirror 3: 0xc38d4619 (stale) Mirror 4: 0x549cd311 (stale) CRC-32 checksum value for chunk [0, 0x3d270): Mirror 1: 0x3aa660a6 Mirror 5: 0x3aa660a6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d270: skipped 000333 1775167392.563247 punch from 0x02610f to 0x029209 (0x30fb bytes) 000334 1775167392.577057 trunc from 0x03d270 to 0x03821c (0x5053 bytes) 000335 1775167392.586503 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3821d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3821d): Mirror 2: 0xb528ef0b (stale) Mirror 3: 0x64b37b36 (stale) Mirror 4: 0x8e45a1bb (stale) Mirror 5: 0x7b3a2582 (stale) CRC-32 checksum value for chunk [0, 0x3821d): Mirror 1: 0x7b3a2582 Mirror 6: 0x7b3a2582 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3821d: skipped 000336 1775167392.887802 trunc from 0x03821d to 0x03a10f (0x1ef3 bytes) 000337 1775167392.905070 trunc from 0x03a110 to 0x03d079 (0x2f6a bytes) 000340 1775167392.915232 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3d07a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d07a): Mirror 2: 0x280cfa85 (stale) Mirror 3: 0xf0987064 (stale) Mirror 4: 0x8e0abd6b (stale) Mirror 5: 0x6e668a1d (stale) Mirror 6: 0x614eb67c (stale) CRC-32 checksum value for chunk [0, 0x3d07a): Mirror 1: 0x614eb67c Mirror 7: 0x614eb67c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d07a: skipped 000341 1775167393.239365 write 0x010c39 thru 0x0176de (0x6aa6 bytes) 000343 1775167393.256610 mapread 0x02519c thru 0x029d59 (0x4bbe bytes) 000344 1775167393.261034 write 0x02384d thru 0x02e84c (0xb000 bytes) 000345 1775167393.264234 punch from 0x036473 to 0x0384ba (0x2048 bytes) 000346 1775167393.266722 trunc from 0x03d07a to 0x020c5b (0x1c41e bytes) 000347 1775167393.276486 read 0x016a55 thru 0x01acd6 (0x4282 bytes) 000348 1775167393.279374 punch from 0x00af4e to 0x01ae4b (0xfefe bytes) 000349 1775167393.285348 mapwrite 0x032992 thru 0x03ffff (0xd66e bytes) 000350 1775167393.310044 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7a7159fb (stale) Mirror 3: 0x86bead88 (stale) Mirror 4: 0xae4ba97b (stale) Mirror 5: 0x441c80c1 (stale) Mirror 6: 0x6b618680 (stale) Mirror 7: 0x6b618680 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x975596cd Mirror 8: 0x975596cd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000351 1775167393.596845 fallocate 0x002700 thru 0x011d00 (0xf601 bytes) 000352 1775167393.611553 trunc from 0x019ad7 to 0x01a092 (0x5bc bytes) 000353 1775167393.613844 write 0x02023f thru 0x0276e2 (0x74a4 bytes) 000355 1775167393.619731 mapread 0x01bf19 thru 0x02bd06 (0xfdee bytes) 000357 1775167393.628319 mapread 0x000c58 thru 0x00c62f (0xb9d8 bytes) 000358 1775167393.634786 mapwrite 0x03f74f thru 0x03ffff (0x8b1 bytes) 000359 1775167393.649330 write 0x02b948 thru 0x035ea4 (0xa55d bytes) 000361 1775167393.653995 punch from 0x00779c to 0x00a9eb (0x3250 bytes) 000362 1775167393.659351 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7a7159fb (stale) Mirror 3: 0x86bead88 (stale) Mirror 4: 0xae4ba97b (stale) Mirror 5: 0x441c80c1 (stale) Mirror 6: 0x6b618680 (stale) Mirror 8: 0x975596cd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7a7159fb (stale) Mirror 3: 0x86bead88 (stale) Mirror 4: 0xae4ba97b (stale) Mirror 5: 0x441c80c1 (stale) Mirror 6: 0x6b618680 (stale) Mirror 8: 0x975596cd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000363 1775167393.812249 punch from 0x022a0b to 0x02e66c (0xbc62 bytes) 000364 1775167393.814992 mapwrite 0x001d0d thru 0x002b0d (0xe01 bytes) 000365 1775167393.825885 mapread 0x012a2a thru 0x016d4e (0x4325 bytes) 000366 1775167393.829862 fallocate 0x022e11 thru 0x0269a9 (0x3b99 bytes) 000367 1775167393.832565 punch from 0x01d122 to 0x027557 (0xa436 bytes) 000368 1775167393.834736 mapread 0x00a038 thru 0x015e0f (0xbdd8 bytes) 000369 1775167393.838740 punch from 0x017e81 to 0x021acc (0x9c4c bytes) 000370 1775167393.841136 read 0x000828 thru 0x010548 (0xfd21 bytes) 000371 1775167393.843454 write 0x017609 thru 0x024ca2 (0xd69a bytes) 000372 1775167393.850891 mapwrite 0x011f6b thru 0x01cb57 (0xabed bytes) 000373 1775167393.859432 trunc from 0x023e24 to 0x0327c9 (0xe9a6 bytes) 000375 1775167393.862305 fallocate 0x007d17 thru 0x01127e (0x9568 bytes) 000377 1775167393.865063 punch from 0x0128cf to 0x021c2c (0xf35e bytes) 000378 1775167393.867595 punch from 0x0122cd to 0x01deca (0xbbfe bytes) 000379 1775167393.869539 mapwrite 0x0190fb thru 0x01d01f (0x3f25 bytes) 000380 1775167393.879087 write 0x031f27 thru 0x034ace (0x2ba8 bytes) 000381 1775167393.882611 write 0x00f1db thru 0x01065c (0x1482 bytes) 000382 1775167393.883332 mapread 0x03c6a7 thru 0x03ffff (0x3959 bytes) 000384 1775167393.887062 punch from 0x035dd7 to 0x03c3cf (0x65f9 bytes) 000385 1775167393.891581 MIRROR_RESYNC 7 mirrors 000386 1775167394.742007 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) 000387 1775167394.878813 read 0x036c52 thru 0x03ffff (0x93ae bytes) 000388 1775167394.883905 write 0x02ca3d thru 0x03a043 (0xd607 bytes) 000390 1775167394.906501 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfb676499 (stale) Mirror 5: 0xfb676499 (stale) Mirror 6: 0xfb676499 (stale) Mirror 8: 0xfb676499 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfb676499 (stale) Mirror 5: 0xfb676499 (stale) Mirror 6: 0xfb676499 (stale) Mirror 8: 0xfb676499 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000391 1775167395.042935 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfb676499 (stale) Mirror 6: 0xfb676499 (stale) Mirror 8: 0xfb676499 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfb676499 (stale) Mirror 6: 0xfb676499 (stale) Mirror 8: 0xfb676499 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000392 1775167395.192165 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xfb676499 (stale) Mirror 8: 0xfb676499 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xfb676499 (stale) Mirror 8: 0xfb676499 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000393 1775167395.338381 read 0x02d945 thru 0x034a05 (0x70c1 bytes) 000394 1775167395.343662 trunc from 0x01969f to 0x01dbde (0x4540 bytes) 000395 1775167395.346391 trunc from 0x040000 to 0x02398a (0x1c675 bytes) 000396 1775167395.354472 fallocate 0x00c896 thru 0x01af19 (0xe684 bytes) 000397 1775167395.356570 write 0x007c97 thru 0x0143a9 (0xc713 bytes) 000398 1775167395.361387 punch from 0x01f376 to 0x02398a (0x4615 bytes) 000399 1775167395.363931 mapwrite 0x016fe7 thru 0x02688e (0xf8a8 bytes) 000400 1775167395.395057 MIRROR_RESYNC 3 mirrors 000402 1775167395.636283 trunc from 0x02688f to 0x020290 (0x65fe bytes) 000403 1775167395.675880 MIRROR_RESYNC 3 mirrors 000404 1775167395.888490 read 0x011bf1 thru 0x01a0f2 (0x8502 bytes) 000405 1775167395.892022 write 0x0183e6 thru 0x021c64 (0x987f bytes) 000406 1775167395.911675 mapwrite 0x03e02c thru 0x03ffff (0x1fd4 bytes) 000407 1775167395.953845 punch from 0x00fc27 to 0x019f25 (0xa2ff bytes) 000408 1775167395.957251 read 0x0289ee thru 0x02a61f (0x1c32 bytes) 000409 1775167395.966334 MIRROR_RESYNC 3 mirrors 000410 1775167396.288749 trunc from 0x040000 to 0x020e57 (0x1f1a8 bytes) 000411 1775167396.309376 trunc from 0x00a139 to 0x019b16 (0xf9de bytes) 000412 1775167396.314010 trunc from 0x020e58 to 0x006c7f (0x1a1d8 bytes) 000413 1775167396.331347 write 0x00c737 thru 0x0153a7 (0x8c71 bytes) 000414 1775167396.332847 write 0x022d22 thru 0x02c137 (0x9416 bytes) 000415 1775167396.334265 mapwrite 0x024fca thru 0x02ce87 (0x7ebe bytes) 000416 1775167396.353171 mapread 0x0188a3 thru 0x018ee7 (0x645 bytes) 000417 1775167396.356837 mapread 0x002a47 thru 0x00911b (0x66d5 bytes) 000420 1775167396.365723 punch from 0x00014d to 0x00f99c (0xf850 bytes) 000422 1775167396.374723 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2ce88): Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ce88): Mirror 6: 0x56f6865d (stale) Mirror 8: 0x56f6865d (stale) CRC-32 checksum value for chunk [0, 0x2ce88): Mirror 1: 0x43f6d027 Mirror 9: 0x43f6d027 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ce88: skipped 000423 1775167396.741284 trunc from 0x009755 to 0x018220 (0xeacc bytes) 000424 1775167396.756668 trunc from 0x02ce88 to 0x0214b3 (0xb9d4 bytes) 000426 1775167396.773620 read 0x01fd09 thru 0x0214b3 (0x17ab bytes) 000427 1775167396.778328 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x214b4): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x214b4): Mirror 6: 0x73aa4942 (stale) Mirror 8: 0x73aa4942 (stale) Mirror 9: 0x6434ff42 (stale) CRC-32 checksum value for chunk [0, 0x214b4): Mirror 1: 0x939a66d3 Mirror 10: 0x939a66d3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x214b4: skipped 000428 1775167397.125325 mapread 0x01a8fe thru 0x0214b3 (0x6bb6 bytes) 000429 1775167397.132451 read 0x00a571 thru 0x019142 (0xebd2 bytes) 000430 1775167397.138390 fallocate 0x021c50 thru 0x0313eb (0xf79c bytes) 000431 1775167397.155310 punch from 0x008183 to 0x017b5a (0xf9d8 bytes) 000432 1775167397.159870 mapwrite 0x0185c8 thru 0x025c91 (0xd6ca bytes) 000433 1775167397.181356 write 0x004277 thru 0x00a8dd (0x6667 bytes) 000434 1775167397.187299 mapread 0x029557 thru 0x0313eb (0x7e95 bytes) 000435 1775167397.192526 fallocate 0x00db6f thru 0x00eae3 (0xf75 bytes) 000436 1775167397.197345 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x313ec): Mirror 8: 0x3d2feb8b (stale) Mirror 9: 0x61903a04 (stale) Mirror 10: 0xb60e6c30 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x313ec): Mirror 8: 0x3d2feb8b (stale) Mirror 9: 0x61903a04 (stale) Mirror 10: 0xb60e6c30 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x313ec: skipped 000437 1775167397.363303 trunc from 0x02f2e5 to 0x0313eb (0x2107 bytes) 000438 1775167397.366979 trunc from 0x0313ec to 0x03b379 (0x9f8e bytes) 000439 1775167397.391008 mapwrite 0x024c9b thru 0x0266d7 (0x1a3d bytes) 000440 1775167397.409088 fallocate 0x03be1e thru 0x03f99c (0x3b7f bytes) 000441 1775167397.411948 mapread 0x02b364 thru 0x0353e6 (0xa083 bytes) 000442 1775167397.416600 trunc from 0x03b37a to 0x02847a (0x12eff bytes) 000443 1775167397.425559 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2847b): Mirror 9: 0xc1ba4725 (stale) Mirror 10: 0xa39268d4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2847b): Mirror 9: 0xc1ba4725 (stale) Mirror 10: 0xa39268d4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2847b: skipped 000444 1775167397.576808 mapread 0x00c3d4 thru 0x01a980 (0xe5ad bytes) 000445 1775167397.588913 read 0x01a1aa thru 0x021d39 (0x7b90 bytes) 000446 1775167397.591931 trunc from 0x02847b to 0x01ed6d (0x970d bytes) 000447 1775167397.603022 mapwrite 0x00a652 thru 0x01461e (0x9fcd bytes) 000449 1775167397.614810 read 0x008bec thru 0x0151e8 (0xc5fd bytes) 000450 1775167397.615463 mapwrite 0x00d189 thru 0x01c146 (0xefbe bytes) 000451 1775167397.626436 fallocate 0x03ae8f thru 0x03ffff (0x5171 bytes) 000453 1775167397.630000 fallocate 0x010d82 thru 0x011f4d (0x11cc bytes) 000454 1775167397.633595 fallocate 0x002f7b thru 0x0054c0 (0x2546 bytes) 000456 1775167397.636901 trunc from 0x01348a to 0x018a7c (0x55f3 bytes) 000457 1775167397.639364 fallocate 0x01ac49 thru 0x027e51 (0xd209 bytes) 000458 1775167397.644048 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xcbcf9ea7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xcbcf9ea7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000460 1775167397.745770 mapread 0x0295b3 thru 0x03479f (0xb1ed bytes) 000461 1775167397.757118 trunc from 0x03432d to 0x03a4a1 (0x6175 bytes) 000462 1775167397.759423 punch from 0x025e62 to 0x028b29 (0x2cc8 bytes) 000463 1775167397.761748 trunc from 0x025783 to 0x0321a7 (0xca25 bytes) 000464 1775167397.764121 write 0x0155e2 thru 0x02357f (0xdf9e bytes) 000465 1775167397.772730 punch from 0x01f016 to 0x025d52 (0x6d3d bytes) 000466 1775167397.778962 read 0x02b5e2 thru 0x0383c7 (0xcde6 bytes) 000467 1775167397.783736 trunc from 0x0207c9 to 0x02bf5d (0xb795 bytes) 000468 1775167397.788243 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xcbcf9ea7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcc8d5b55 Mirror 10: 0xcc8d5b55 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000469 1775167398.128356 fallocate 0x0122fe thru 0x01e9fc (0xc6ff bytes) 000470 1775167398.143842 mapwrite 0x013adc thru 0x014f5c (0x1481 bytes) 000471 1775167398.155706 write 0x039c00 thru 0x03ffff (0x6400 bytes) 000473 1775167398.163048 fallocate 0x003da7 thru 0x011b6c (0xddc6 bytes) 000474 1775167398.167504 MIRROR_RESYNC 3 mirrors 000475 1775167398.516609 mapread 0x00294a thru 0x007c9a (0x5351 bytes) 000476 1775167398.524256 trunc from 0x036042 to 0x03c32f (0x62ee bytes) 000477 1775167398.539375 trunc from 0x000251 to 0x00e6ce (0xe47e bytes) 000478 1775167398.542349 fallocate 0x026619 thru 0x02a2c7 (0x3caf bytes) 000479 1775167398.544774 trunc from 0x025761 to 0x0315de (0xbe7e bytes) 000481 1775167398.547237 mapread 0x00dfe5 thru 0x019a07 (0xba23 bytes) 000483 1775167398.555329 write 0x010e5d thru 0x0179f6 (0x6b9a bytes) 000484 1775167398.556457 mapwrite 0x005e5d thru 0x009ac2 (0x3c66 bytes) 000485 1775167398.570442 write 0x00fce1 thru 0x018545 (0x8865 bytes) 000486 1775167398.571756 trunc from 0x040000 to 0x03dc4a (0x23b5 bytes) 000487 1775167398.587181 mapread 0x01d62b thru 0x024a33 (0x7409 bytes) 000488 1775167398.591971 trunc from 0x003533 to 0x00f713 (0xc1e1 bytes) 000489 1775167398.597795 write 0x03f47c thru 0x03ffff (0xb84 bytes) 000490 1775167398.598839 mapwrite 0x03c491 thru 0x03ffff (0x3b6f bytes) 000491 1775167398.609072 mapread 0x0195f2 thru 0x024889 (0xb298 bytes) 000492 1775167398.612330 MIRROR_RESYNC 3 mirrors 000494 1775167398.894171 mapwrite 0x0152d5 thru 0x01665a (0x1386 bytes) 000495 1775167398.937686 read 0x03f0b2 thru 0x03f2ea (0x239 bytes) 000496 1775167398.943023 MIRROR_RESYNC 3 mirrors 000497 1775167399.226486 punch from 0x01c329 to 0x01ca7e (0x756 bytes) 000498 1775167399.238525 fallocate 0x019b82 thru 0x0243b9 (0xa838 bytes) 000499 1775167399.241651 read 0x01579a thru 0x018568 (0x2dcf bytes) 000500 1775167399.247397 read 0x02ae06 thru 0x031162 (0x635d bytes) 000503 1775167399.253264 write 0x008d56 thru 0x01762d (0xe8d8 bytes) 000504 1775167399.254500 trunc from 0x040000 to 0x03512c (0xaed3 bytes) 000505 1775167399.262021 mapwrite 0x00cacb thru 0x0166c4 (0x9bfa bytes) 000506 1775167399.273376 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3512d): Mirror 9: 0x56b67f78 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3512d): Mirror 9: 0x56b67f78 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3512d: skipped 000507 1775167399.368771 mapread 0x033852 thru 0x03512c (0x18db bytes) 000508 1775167399.374729 read 0x01a360 thru 0x01cb82 (0x2823 bytes) 000509 1775167399.376485 punch from 0x034aa8 to 0x034dc4 (0x31d bytes) 000510 1775167399.378165 mapread 0x01a854 thru 0x025668 (0xae15 bytes) 000511 1775167399.381476 trunc from 0x03512d to 0x0191d1 (0x1bf5b bytes) 000512 1775167399.387638 trunc from 0x013ff6 to 0x016287 (0x2292 bytes) 000513 1775167399.389601 mapread 0x010692 thru 0x010c17 (0x586 bytes) 000514 1775167399.393166 mapread 0x00079f thru 0x00a334 (0x9b96 bytes) 000515 1775167399.397110 mapwrite 0x00911c thru 0x01436f (0xb254 bytes) 000516 1775167399.407233 read 0x006f11 thru 0x00b347 (0x4437 bytes) 000517 1775167399.407896 trunc from 0x0191d2 to 0x01d971 (0x47a0 bytes) 000518 1775167399.413072 read 0x001469 thru 0x005018 (0x3bb0 bytes) 000519 1775167399.415184 MIRROR_RESYNC 2 mirrors 000520 1775167399.614168 mapwrite 0x018ed0 thru 0x01e075 (0x51a6 bytes) 000521 1775167399.647563 fallocate 0x03bf1b thru 0x03ffff (0x40e5 bytes) 000522 1775167399.650064 trunc from 0x0099fe to 0x01413e (0xa741 bytes) 000523 1775167399.652773 trunc from 0x01e076 to 0x022215 (0x41a0 bytes) 000524 1775167399.659882 fallocate 0x021ba4 thru 0x02ab3f (0x8f9c bytes) 000526 1775167399.665672 MIRROR_RESYNC 2 mirrors 000527 1775167399.942614 mapwrite 0x00c4f3 thru 0x01545a (0x8f68 bytes) 000528 1775167399.999957 mapread 0x014306 thru 0x01798e (0x3689 bytes) 000529 1775167400.005122 trunc from 0x003211 to 0x00c433 (0x9223 bytes) 000533 1775167400.010062 trunc from 0x022216 to 0x02e10a (0xbef5 bytes) 000534 1775167400.025279 fallocate 0x0399e4 thru 0x03ffff (0x661c bytes) 000535 1775167400.029095 fallocate 0x02f610 thru 0x038da2 (0x9793 bytes) 000536 1775167400.032105 punch from 0x0085ac to 0x00c261 (0x3cb6 bytes) 000537 1775167400.036542 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) 000538 1775167400.100811 write 0x021526 thru 0x028161 (0x6c3c bytes) 000539 1775167400.108858 read 0x02454c thru 0x02de2b (0x98e0 bytes) 000541 1775167400.117250 punch from 0x01a8bc to 0x028cbf (0xe404 bytes) 000544 1775167400.125813 mapread 0x006de5 thru 0x00a52f (0x374b bytes) 000545 1775167400.132021 fallocate 0x01f1f4 thru 0x024be4 (0x59f1 bytes) 000547 1775167400.135154 fallocate 0x02a18b thru 0x02d6e5 (0x355b bytes) 000549 1775167400.138529 mapread 0x010248 thru 0x0107a3 (0x55c bytes) 000550 1775167400.139413 fallocate 0x010a53 thru 0x01435e (0x390c bytes) 000551 1775167400.142306 mapread 0x01e018 thru 0x02678f (0x8778 bytes) 000552 1775167400.148866 mapwrite 0x009fc2 thru 0x00f81e (0x585d bytes) 000553 1775167400.164102 fallocate 0x0227dc thru 0x0313c0 (0xebe5 bytes) 000554 1775167400.167097 mapwrite 0x0178d8 thru 0x022271 (0xa99a bytes) 000555 1775167400.180396 write 0x010ec9 thru 0x01ee49 (0xdf81 bytes) 000556 1775167400.181909 trunc from 0x0313c1 to 0x00f874 (0x21b4c bytes) 000557 1775167400.191524 trunc from 0x00be9c to 0x00e307 (0x246c bytes) 000558 1775167400.194605 trunc from 0x00ac72 to 0x00f874 (0x4c03 bytes) 000559 1775167400.197405 mapwrite 0x018d19 thru 0x01dc23 (0x4f0b bytes) 000560 1775167400.224114 trunc from 0x016890 to 0x01dc23 (0x7394 bytes) 000563 1775167400.228247 read 0x01973a thru 0x01dc23 (0x44ea bytes) 000564 1775167400.233490 trunc from 0x01dc24 to 0x0247d2 (0x6baf bytes) 000565 1775167400.240455 mapread 0x0005cd thru 0x00e0eb (0xdb1f bytes) 000566 1775167400.241737 fallocate 0x032739 thru 0x03f18a (0xca52 bytes) 000569 1775167400.245734 trunc from 0x03d546 to 0x03ea01 (0x14bc bytes) 000570 1775167400.248366 trunc from 0x024db9 to 0x02a106 (0x534e bytes) 000571 1775167400.252342 read 0x03d902 thru 0x03f18a (0x1889 bytes) 000572 1775167400.258336 read 0x02ce8c thru 0x02fdaa (0x2f1f bytes) 000573 1775167400.258996 read 0x0004f6 thru 0x00510b (0x4c16 bytes) 000575 1775167400.260140 mapwrite 0x0083d3 thru 0x00d029 (0x4c57 bytes) 000576 1775167400.276444 MIRROR_EXTEND to 2 mirrors 000578 1775167400.577083 trunc from 0x01cb14 to 0x02c01d (0xf50a bytes) 000579 1775167400.595117 trunc from 0x0008ee to 0x00c23a (0xb94d bytes) 000581 1775167400.598807 write 0x01d2be thru 0x028e02 (0xbb45 bytes) 000582 1775167400.607435 trunc from 0x03f18b to 0x03dc5b (0x152f bytes) 000583 1775167400.628248 fallocate 0x0243e2 thru 0x02fe26 (0xba45 bytes) 000585 1775167400.631850 read 0x028c7c thru 0x035236 (0xc5bb bytes) 000586 1775167400.635728 mapread 0x00124d thru 0x0038fa (0x26ae bytes) 000587 1775167400.645918 mapwrite 0x01a605 thru 0x029292 (0xec8e bytes) 000588 1775167400.667493 punch from 0x02c615 to 0x035a40 (0x942c bytes) 000589 1775167400.670895 mapread 0x029335 thru 0x02f4ee (0x61ba bytes) 000590 1775167400.680449 write 0x029c55 thru 0x02c068 (0x2414 bytes) 000591 1775167400.681753 trunc from 0x03d7f1 to 0x03dc5b (0x46b bytes) 000592 1775167400.685700 trunc from 0x03dc5c to 0x00d272 (0x309e9 bytes) 000593 1775167400.693740 mapread 0x005e6b thru 0x00d272 (0x7408 bytes) 000594 1775167400.695022 trunc from 0x006cd7 to 0x00d272 (0x659c bytes) 000595 1775167400.700442 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xd273): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd273): Mirror 2: 0xd6a5e559 (stale) CRC-32 checksum value for chunk [0, 0xd273): Mirror 1: 0x59f1575b Mirror 3: 0x59f1575b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd273: skipped 000596 1775167401.005930 trunc from 0x000cf2 to 0x00d272 (0xc581 bytes) 000597 1775167401.022049 mapwrite 0x022e0c thru 0x02382c (0xa21 bytes) 000598 1775167401.051584 write 0x01d1ce thru 0x021f6c (0x4d9f bytes) 000599 1775167401.056058 trunc from 0x02382d to 0x02685c (0x3030 bytes) 000600 1775167401.062464 trunc from 0x02190c to 0x02685c (0x4f51 bytes) 000601 1775167401.069452 mapread 0x00fff1 thru 0x01bca6 (0xbcb6 bytes) 000602 1775167401.074278 fallocate 0x022e33 thru 0x02405a (0x1228 bytes) 000603 1775167401.077070 trunc from 0x02685d to 0x0043c2 (0x2249a bytes) 000604 1775167401.084058 mapwrite 0x027acf thru 0x02ba15 (0x3f47 bytes) 000605 1775167401.105477 write 0x0322db thru 0x0350b7 (0x2ddd bytes) 000606 1775167401.106531 trunc from 0x0350b8 to 0x0022ea (0x32dcd bytes) 000607 1775167401.117583 trunc from 0x0022eb to 0x02fa3a (0x2d750 bytes) 000608 1775167401.124239 write 0x018254 thru 0x021635 (0x93e2 bytes) 000609 1775167401.130830 trunc from 0x02fa3b to 0x020a0f (0xf02b bytes) 000610 1775167401.139784 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20a10): Mirror 3: 0x66108898 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20a10): Mirror 3: 0x66108898 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20a10: skipped 000611 1775167401.256979 trunc from 0x020a10 to 0x01ba70 (0x4f9f bytes) 000612 1775167401.266861 write 0x03d0d5 thru 0x03ffff (0x2f2b bytes) 000613 1775167401.268259 write 0x0321aa thru 0x03d227 (0xb07e bytes) 000614 1775167401.272215 punch from 0x032a18 to 0x03cda4 (0xa38d bytes) 000615 1775167401.279627 fallocate 0x034627 thru 0x039ccc (0x56a6 bytes) 000616 1775167401.283383 fallocate 0x00fcdd thru 0x01ae78 (0xb19c bytes) 000617 1775167401.286068 fallocate 0x00766b thru 0x013af7 (0xc48d bytes) 000618 1775167401.288458 punch from 0x01d2be to 0x01fb9e (0x28e1 bytes) 000619 1775167401.293806 fallocate 0x013aa4 thru 0x016e34 (0x3391 bytes) 000620 1775167401.296612 read 0x038e96 thru 0x0392e9 (0x454 bytes) 000621 1775167401.301544 MIRROR_RESYNC 2 mirrors 000622 1775167401.572966 trunc from 0x03f609 to 0x03ffff (0x9f7 bytes) 000623 1775167401.591733 mapread 0x017a6c thru 0x01aeba (0x344f bytes) 000626 1775167401.600834 trunc from 0x040000 to 0x0198bd (0x26742 bytes) 000627 1775167401.623661 trunc from 0x008dfc to 0x014bb2 (0xbdb7 bytes) 000628 1775167401.626409 write 0x00a7e0 thru 0x01693a (0xc15b bytes) 000630 1775167401.632468 write 0x000996 thru 0x00c161 (0xb7cc bytes) 000631 1775167401.636314 mapwrite 0x035898 thru 0x0366bb (0xe24 bytes) 000632 1775167401.658849 mapwrite 0x0220f6 thru 0x030347 (0xe252 bytes) 000634 1775167401.677384 trunc from 0x0366bc to 0x00475c (0x31f5f bytes) 000635 1775167401.688645 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x475d): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x475d): Mirror 3: 0x4b4d248b (stale) CRC-32 checksum value for chunk [0, 0x475d): Mirror 1: 0x7c09c7d3 Mirror 4: 0x7c09c7d3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x475d: skipped 000636 1775167401.939864 trunc from 0x003771 to 0x00475c (0xfec bytes) 000637 1775167401.952563 write 0x03b1ba thru 0x03ffff (0x4e46 bytes) 000638 1775167401.955527 trunc from 0x032706 to 0x035b7c (0x3477 bytes) 000639 1775167401.957987 read 0x03c349 thru 0x03ffff (0x3cb7 bytes) 000640 1775167401.958606 read 0x025657 thru 0x032518 (0xcec2 bytes) 000641 1775167401.965612 MIRROR_RESYNC 3 mirrors 000642 1775167402.195928 mapread 0x02bd00 thru 0x0323de (0x66df bytes) 000643 1775167402.202805 write 0x01541b thru 0x01dce0 (0x88c6 bytes) 000644 1775167402.219266 mapwrite 0x023ed1 thru 0x027b4f (0x3c7f bytes) 000645 1775167402.234655 trunc from 0x008a01 to 0x015cba (0xd2ba bytes) 000646 1775167402.237306 fallocate 0x0258f2 thru 0x02e9f5 (0x9104 bytes) 000647 1775167402.239211 punch from 0x00564a to 0x00ac34 (0x55eb bytes) 000648 1775167402.241225 punch from 0x01ab54 to 0x01b3fa (0x8a7 bytes) 000649 1775167402.243747 read 0x011ec6 thru 0x01a25d (0x8398 bytes) 000650 1775167402.247504 trunc from 0x039cf7 to 0x03f838 (0x5b42 bytes) 000651 1775167402.249870 trunc from 0x018c6d to 0x023d29 (0xb0bd bytes) 000652 1775167402.252357 punch from 0x02a80f to 0x035c5c (0xb44e bytes) 000653 1775167402.255024 trunc from 0x040000 to 0x01aa0e (0x255f1 bytes) 000654 1775167402.265815 write 0x03b372 thru 0x03da2d (0x26bc bytes) 000655 1775167402.266699 trunc from 0x0080ab to 0x00ffca (0x7f20 bytes) 000656 1775167402.272853 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3da2e): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3da2e): Mirror 3: 0xc82979e7 (stale) Mirror 4: 0xc82979e7 (stale) CRC-32 checksum value for chunk [0, 0x3da2e): Mirror 1: 0x4f05e1ee Mirror 5: 0x4f05e1ee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3da2e: skipped 000657 1775167402.543872 trunc from 0x03da2e to 0x03573d (0x82f0 bytes) 000658 1775167402.563509 read 0x026ad1 thru 0x027097 (0x5c7 bytes) 000659 1775167402.565663 mapwrite 0x00ba1e thru 0x017065 (0xb648 bytes) 000660 1775167402.581644 read 0x009d44 thru 0x015965 (0xbc22 bytes) 000661 1775167402.583818 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3573e): Mirror 4: 0x35ba7167 (stale) Mirror 5: 0x9fe82966 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3573e): Mirror 4: 0x35ba7167 (stale) Mirror 5: 0x9fe82966 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3573e: skipped 000662 1775167402.694442 trunc from 0x005b86 to 0x00e670 (0x8aeb bytes) 000663 1775167402.696990 mapwrite 0x0037ce thru 0x0099ed (0x6220 bytes) 000664 1775167402.717924 read 0x015bca thru 0x01a921 (0x4d58 bytes) 000665 1775167402.722666 mapread 0x01b87a thru 0x028645 (0xcdcc bytes) 000666 1775167402.734331 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3573e): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3573e): Mirror 4: 0x35ba7167 (stale) Mirror 5: 0x9fe82966 (stale) CRC-32 checksum value for chunk [0, 0x3573e): Mirror 1: 0x66f13428 Mirror 6: 0x66f13428 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3573e: skipped 000667 1775167403.056424 mapread 0x01c869 thru 0x024732 (0x7eca bytes) 000668 1775167403.083803 mapwrite 0x004bde thru 0x007c20 (0x3043 bytes) 000669 1775167403.145332 trunc from 0x02e3a8 to 0x03573d (0x7396 bytes) 000670 1775167403.149737 fallocate 0x03a1ca thru 0x03e5b9 (0x43f0 bytes) 000671 1775167403.153154 trunc from 0x03573e to 0x02d6f4 (0x8049 bytes) 000672 1775167403.171274 mapread 0x018734 thru 0x023ecf (0xb79c bytes) 000674 1775167403.178430 trunc from 0x02d6f5 to 0x0082a4 (0x25450 bytes) 000675 1775167403.184370 trunc from 0x0082a5 to 0x03e1f3 (0x35f4f bytes) 000676 1775167403.191403 punch from 0x01906b to 0x025122 (0xc0b8 bytes) 000678 1775167403.194523 punch from 0x02a28f to 0x02eccc (0x4a3e bytes) 000679 1775167403.197798 trunc from 0x03e1f4 to 0x02b46f (0x12d84 bytes) 000680 1775167403.203921 punch from 0x009e41 to 0x00d370 (0x3530 bytes) 000681 1775167403.206300 punch from 0x02b3c0 to 0x02b46f (0xb0 bytes) 000682 1775167403.208304 punch from 0x02b337 to 0x02b46f (0x139 bytes) 000683 1775167403.210548 mapread 0x024c10 thru 0x02b46f (0x6860 bytes) 000684 1775167403.218515 punch from 0x017684 to 0x01fe9f (0x881c bytes) 000685 1775167403.221131 mapwrite 0x037aa4 thru 0x03ffff (0x855c bytes) 000686 1775167403.239736 punch from 0x03f4c9 to 0x03ffff (0xb37 bytes) 000687 1775167403.242999 mapwrite 0x019677 thru 0x022190 (0x8b1a bytes) 000688 1775167403.258641 punch from 0x025dd2 to 0x02c1bf (0x63ee bytes) 000689 1775167403.263292 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf23db6a6 (stale) Mirror 5: 0x869c137f (stale) Mirror 6: 0x1b15e56b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x748cc1c1 Mirror 7: 0x748cc1c1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000690 1775167403.598576 trunc from 0x040000 to 0x0355c3 (0xaa3c bytes) 000692 1775167403.631652 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x355c4): Mirror 4: 0xb67f9446 (stale) Mirror 6: 0xf007cfef (stale) Mirror 7: 0x2af82b3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x355c4): Mirror 4: 0xb67f9446 (stale) Mirror 6: 0xf007cfef (stale) Mirror 7: 0x2af82b3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x355c4: skipped 000693 1775167403.792038 trunc from 0x000a5b to 0x00e212 (0xd7b8 bytes) 000694 1775167403.798405 MIRROR_RESYNC 4 mirrors 000695 1775167404.218575 read 0x031f51 thru 0x0355c3 (0x3673 bytes) 000696 1775167404.222404 write 0x01607c thru 0x021ebe (0xbe43 bytes) 000697 1775167404.236634 fallocate 0x00ded9 thru 0x0162b3 (0x83db bytes) 000698 1775167404.239416 write 0x03043b thru 0x033eba (0x3a80 bytes) 000699 1775167404.244774 read 0x01b6c1 thru 0x0213c5 (0x5d05 bytes) 000700 1775167404.245504 write 0x0010ad thru 0x009758 (0x86ac bytes) 000701 1775167404.250473 punch from 0x027be2 to 0x02adc4 (0x31e3 bytes) 000703 1775167404.254227 punch from 0x0211ae to 0x028bdc (0x7a2f bytes) 000704 1775167404.260216 trunc from 0x02a3a6 to 0x03265b (0x82b6 bytes) 000705 1775167404.262384 write 0x02ff7c thru 0x0353ef (0x5474 bytes) 000706 1775167404.265302 trunc from 0x02a5e5 to 0x0355c3 (0xafdf bytes) 000707 1775167404.271346 punch from 0x008536 to 0x01700d (0xead8 bytes) 000708 1775167404.275097 MIRROR_RESYNC 4 mirrors 000709 1775167404.624690 mapread 0x01ca5c thru 0x026c88 (0xa22d bytes) 000711 1775167404.635186 read 0x0231b0 thru 0x02c103 (0x8f54 bytes) 000712 1775167404.640907 MIRROR_EXTEND to 5 mirrors 000713 1775167404.909750 mapread 0x034c21 thru 0x0355c3 (0x9a3 bytes) 000714 1775167404.914350 trunc from 0x0355c4 to 0x011fc5 (0x235fe bytes) 000715 1775167404.935348 write 0x00e955 thru 0x01dbd0 (0xf27c bytes) 000716 1775167404.940666 write 0x012b32 thru 0x020482 (0xd951 bytes) 000717 1775167404.942010 mapread 0x00b66c thru 0x00c015 (0x9aa bytes) 000718 1775167404.942919 punch from 0x00f8e5 to 0x01c1b5 (0xc8d1 bytes) 000719 1775167404.948292 mapread 0x00c811 thru 0x0133ef (0x6bdf bytes) 000720 1775167404.952043 write 0x01feba thru 0x026e3e (0x6f85 bytes) 000721 1775167404.953313 trunc from 0x026e3f to 0x017d41 (0xf0fd bytes) 000722 1775167404.960679 read 0x003c60 thru 0x00c06c (0x840d bytes) 000723 1775167404.963634 write 0x03e9c1 thru 0x03ffff (0x163f bytes) 000724 1775167404.964604 write 0x037137 thru 0x03b8d3 (0x479d bytes) 000725 1775167404.968103 fallocate 0x017262 thru 0x0224d2 (0xb271 bytes) 000726 1775167404.970888 trunc from 0x040000 to 0x02367d (0x1c982 bytes) 000727 1775167404.977390 mapwrite 0x0201f9 thru 0x02811f (0x7f27 bytes) 000728 1775167404.995077 trunc from 0x028120 to 0x02d583 (0x5464 bytes) 000729 1775167405.000172 read 0x008d32 thru 0x011b57 (0x8e26 bytes) 000730 1775167405.002250 MIRROR_RESYNC 5 mirrors 000731 1775167405.411808 punch from 0x0183e0 to 0x01f465 (0x7086 bytes) 000732 1775167405.425747 mapwrite 0x00958d thru 0x014d54 (0xb7c8 bytes) 000733 1775167405.441059 mapread 0x016b38 thru 0x01b0c1 (0x458a bytes) 000734 1775167405.442181 write 0x02334b thru 0x023d47 (0x9fd bytes) 000736 1775167405.446153 trunc from 0x02d584 to 0x03f71a (0x12197 bytes) 000737 1775167405.452307 mapwrite 0x02e794 thru 0x03bd96 (0xd603 bytes) 000738 1775167405.467763 mapwrite 0x00f8fe thru 0x0141e6 (0x48e9 bytes) 000739 1775167405.475632 read 0x02fca6 thru 0x02fef0 (0x24b bytes) 000740 1775167405.476302 mapread 0x01eaa5 thru 0x0244e2 (0x5a3e bytes) 000741 1775167405.477372 punch from 0x02b4ca to 0x03283c (0x7373 bytes) 000742 1775167405.482360 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f71b): Mirror 6: 0x6ec12540 (stale) Mirror 7: 0x6ec12540 (stale) Mirror 8: 0x6ec12540 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f71b): Mirror 6: 0x6ec12540 (stale) Mirror 7: 0x6ec12540 (stale) Mirror 8: 0x6ec12540 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f71b: skipped 000743 1775167405.622706 mapwrite 0x0153ce thru 0x024ab0 (0xf6e3 bytes) 000744 1775167405.644486 trunc from 0x03f71b to 0x002cb2 (0x3ca68 bytes) 000745 1775167405.653379 read 0x002718 thru 0x002cb2 (0x59b bytes) 000746 1775167405.656113 mapwrite 0x002410 thru 0x0111df (0xedd0 bytes) 000747 1775167405.684830 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x111e0): Mirror 6: 0xc77fd1e7 (stale) Mirror 8: 0xc77fd1e7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x111e0): Mirror 6: 0xc77fd1e7 (stale) Mirror 8: 0xc77fd1e7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x111e0: skipped 000748 1775167405.789446 mapwrite 0x01d357 thru 0x01e130 (0xdda bytes) 000749 1775167405.805988 mapwrite 0x03e248 thru 0x03f7c8 (0x1581 bytes) 000750 1775167405.821960 punch from 0x016b1a to 0x0209d8 (0x9ebf bytes) 000751 1775167405.824518 mapread 0x0017c4 thru 0x002904 (0x1141 bytes) 000753 1775167405.829244 write 0x03c14c thru 0x03ffff (0x3eb4 bytes) 000754 1775167405.830370 write 0x03ae17 thru 0x03ffff (0x51e9 bytes) 000755 1775167405.831414 mapread 0x030d65 thru 0x035784 (0x4a20 bytes) 000756 1775167405.835716 mapread 0x029f79 thru 0x02f539 (0x55c1 bytes) 000757 1775167405.836735 write 0x03fd1e thru 0x03ffff (0x2e2 bytes) 000758 1775167405.837325 mapwrite 0x02ef36 thru 0x030afb (0x1bc6 bytes) 000759 1775167405.846799 fallocate 0x02cb35 thru 0x02eafc (0x1fc8 bytes) 000760 1775167405.851823 MIRROR_RESYNC 3 mirrors 000761 1775167406.177473 punch from 0x00db75 to 0x014844 (0x6cd0 bytes) 000762 1775167406.191818 punch from 0x00e605 to 0x01060d (0x2009 bytes) 000763 1775167406.195589 fallocate 0x006122 thru 0x008f7a (0x2e59 bytes) 000765 1775167406.200689 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x62323f36 (stale) Mirror 8: 0x62323f36 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4586826f Mirror 9: 0x4586826f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000766 1775167406.534426 mapwrite 0x01c87a thru 0x02a598 (0xdd1f bytes) 000767 1775167406.585554 trunc from 0x040000 to 0x004f88 (0x3b077 bytes) 000768 1775167406.596550 write 0x02a693 thru 0x030f7b (0x68e9 bytes) 000769 1775167406.597863 mapread 0x00b834 thru 0x013574 (0x7d41 bytes) 000770 1775167406.602074 mapread 0x00d95e thru 0x019224 (0xb8c7 bytes) 000772 1775167406.606386 fallocate 0x00eae4 thru 0x01528c (0x67a9 bytes) 000773 1775167406.610682 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30f7c): Mirror 8: 0x5914528a (stale) Mirror 9: 0x1f103cc5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30f7c): Mirror 8: 0x5914528a (stale) Mirror 9: 0x1f103cc5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30f7c: skipped 000774 1775167406.723036 write 0x00df25 thru 0x014e2c (0x6f08 bytes) 000775 1775167406.729017 write 0x016b44 thru 0x018368 (0x1825 bytes) 000776 1775167406.733304 mapread 0x01a35e thru 0x0216b1 (0x7354 bytes) 000777 1775167406.737759 read 0x0217ad thru 0x02869a (0x6eee bytes) 000778 1775167406.740861 mapwrite 0x017513 thru 0x023108 (0xbbf6 bytes) 000779 1775167406.754843 mapread 0x02f024 thru 0x030f7b (0x1f58 bytes) 000780 1775167406.761394 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x30f7c): Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30f7c): Mirror 8: 0x5914528a (stale) Mirror 9: 0x1f103cc5 (stale) CRC-32 checksum value for chunk [0, 0x30f7c): Mirror 1: 0xe617bde0 Mirror 10: 0xe617bde0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30f7c: skipped 000781 1775167407.068544 write 0x02099d thru 0x0234ee (0x2b52 bytes) 000783 1775167407.088253 read 0x000883 thru 0x001695 (0xe13 bytes) 000784 1775167407.091054 write 0x03992b thru 0x03ffff (0x66d5 bytes) 000785 1775167407.092244 punch from 0x027d04 to 0x029355 (0x1652 bytes) 000786 1775167407.094681 read 0x0013d8 thru 0x0037b2 (0x23db bytes) 000788 1775167407.099734 fallocate 0x0015c7 thru 0x008870 (0x72aa bytes) 000789 1775167407.104149 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x62323f36 (stale) Mirror 9: 0x4586826f (stale) Mirror 10: 0xe19727ac (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x432384 Mirror 11: 0x432384 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000790 1775167407.480594 read 0x0204a1 thru 0x030303 (0xfe63 bytes) 000791 1775167407.487277 MIRROR_RESYNC 5 mirrors 000792 1775167407.889159 mapread 0x03fe09 thru 0x03ffff (0x1f7 bytes) 000793 1775167407.893643 MIRROR_EXTEND to 6 mirrors 000794 1775167408.193880 mapread 0x015a27 thru 0x01f737 (0x9d11 bytes) 000795 1775167408.201029 trunc from 0x028c4b to 0x02c89b (0x3c51 bytes) 000796 1775167408.214125 mapread 0x014349 thru 0x021e1e (0xdad6 bytes) 000797 1775167408.225512 write 0x02f6cb thru 0x03a504 (0xae3a bytes) 000798 1775167408.234046 trunc from 0x040000 to 0x02438e (0x1bc71 bytes) 000799 1775167408.244671 mapread 0x007ea3 thru 0x00cc35 (0x4d93 bytes) 000800 1775167408.249156 read 0x019346 thru 0x0214ef (0x81aa bytes) 000801 1775167408.249897 punch from 0x00ab09 to 0x01a95d (0xfe55 bytes) 000802 1775167408.254076 trunc from 0x02438f to 0x00dfd5 (0x163b9 bytes) 000803 1775167408.260733 punch from 0x003b7a to 0x007629 (0x3ab0 bytes) 000804 1775167408.268004 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xdfd6): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdfd6): Mirror 8: 0xc3448854 (stale) Mirror 9: 0xc3448854 (stale) Mirror 10: 0xc3448854 (stale) Mirror 11: 0xc3448854 (stale) Mirror 12: 0xc3448854 (stale) CRC-32 checksum value for chunk [0, 0xdfd6): Mirror 1: 0x51b00727 Mirror 13: 0x51b00727 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdfd6: skipped 000805 1775167408.654146 trunc from 0x00bb3d to 0x00dfd5 (0x2499 bytes) 000807 1775167408.669784 trunc from 0x00dfd6 to 0x01e820 (0x1084b bytes) 000808 1775167408.678123 MIRROR_RESYNC 7 mirrors 000809 1775167409.039140 mapwrite 0x01fcb3 thru 0x023afc (0x3e4a bytes) 000810 1775167409.081813 read 0x01feb7 thru 0x023afc (0x3c46 bytes) 000811 1775167409.082595 mapwrite 0x006a54 thru 0x00f0f7 (0x86a4 bytes) 000813 1775167409.096892 read 0x01e36e thru 0x021b8d (0x3820 bytes) 000814 1775167409.099782 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23afd): Mirror 8: 0x646ac0d1 (stale) Mirror 9: 0x646ac0d1 (stale) Mirror 11: 0x646ac0d1 (stale) Mirror 12: 0x646ac0d1 (stale) Mirror 13: 0x646ac0d1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23afd): Mirror 8: 0x646ac0d1 (stale) Mirror 9: 0x646ac0d1 (stale) Mirror 11: 0x646ac0d1 (stale) Mirror 12: 0x646ac0d1 (stale) Mirror 13: 0x646ac0d1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23afd: skipped 000815 1775167409.251065 trunc from 0x023afd to 0x010009 (0x13af3 bytes) 000816 1775167409.258340 read 0x006164 thru 0x010009 (0x9ea6 bytes) 000817 1775167409.262158 punch from 0x0048f2 to 0x00770e (0x2e1d bytes) 000819 1775167409.265592 read 0x00321a thru 0x010009 (0xcdf0 bytes) 000821 1775167409.269892 mapwrite 0x00dda6 thru 0x01c87d (0xead8 bytes) 000822 1775167409.292187 read 0x015361 thru 0x01c87d (0x751d bytes) 000823 1775167409.292791 fallocate 0x02bcc8 thru 0x0303bf (0x46f8 bytes) 000824 1775167409.296919 read 0x012cad thru 0x014dec (0x2140 bytes) 000826 1775167409.297995 trunc from 0x01c87e to 0x001927 (0x1af56 bytes) 000827 1775167409.310946 fallocate 0x02e80d thru 0x03978c (0xaf80 bytes) 000828 1775167409.313989 mapwrite 0x005604 thru 0x00e996 (0x9393 bytes) 000830 1775167409.330368 punch from 0x015347 to 0x01ba0a (0x66c4 bytes) 000831 1775167409.335401 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3978d): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3978d): Mirror 8: 0x4b825fd1 (stale) Mirror 9: 0x4b825fd1 (stale) Mirror 11: 0x4b825fd1 (stale) Mirror 12: 0x4b825fd1 (stale) Mirror 13: 0x4b825fd1 (stale) CRC-32 checksum value for chunk [0, 0x3978d): Mirror 1: 0x9cd1c647 Mirror 14: 0x9cd1c647 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3978d: skipped 000832 1775167409.729803 trunc from 0x03978d to 0x007c6b (0x31b21 bytes) 000833 1775167409.765941 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x7c6c): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7c6c): Mirror 8: 0x18e6bd2 (stale) Mirror 9: 0x18e6bd2 (stale) Mirror 11: 0x18e6bd2 (stale) Mirror 12: 0x18e6bd2 (stale) Mirror 13: 0x18e6bd2 (stale) Mirror 14: 0x2da53e7b (stale) CRC-32 checksum value for chunk [0, 0x7c6c): Mirror 1: 0x2da53e7b Mirror 15: 0x2da53e7b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7c6c: skipped 000834 1775167410.098443 punch from 0x0062d1 to 0x007c6b (0x199b bytes) 000835 1775167410.112706 read 0x005f74 thru 0x007c6b (0x1cf8 bytes) 000836 1775167410.115394 trunc from 0x007c6c to 0x0121b0 (0xa545 bytes) 000837 1775167410.132516 punch from 0x001fd3 to 0x005ab5 (0x3ae3 bytes) 000838 1775167410.134770 trunc from 0x0121b1 to 0x02d0db (0x1af2b bytes) 000839 1775167410.145829 trunc from 0x02d0dc to 0x029c25 (0x34b6 bytes) 000840 1775167410.153624 punch from 0x018382 to 0x024e81 (0xcb00 bytes) 000841 1775167410.156780 mapread 0x01dff6 thru 0x025bde (0x7be9 bytes) 000842 1775167410.165492 fallocate 0x0086ab thru 0x011d8d (0x96e3 bytes) 000843 1775167410.168268 fallocate 0x005b7b thru 0x01400e (0xe494 bytes) 000844 1775167410.170871 write 0x00ea95 thru 0x0182e1 (0x984d bytes) 000845 1775167410.175624 fallocate 0x01fa4c thru 0x0292d9 (0x988e bytes) 000846 1775167410.180257 MIRROR_RESYNC 8 mirrors 000847 1775167410.782310 mapwrite 0x0014c4 thru 0x00f4c4 (0xe001 bytes) 000848 1775167410.822172 mapwrite 0x023076 thru 0x0296fe (0x6689 bytes) 000849 1775167410.832954 fallocate 0x0305cc thru 0x03788c (0x72c1 bytes) 000850 1775167410.835230 mapwrite 0x0231b0 thru 0x0274ce (0x431f bytes) 000851 1775167410.843141 mapread 0x025283 thru 0x029c25 (0x49a3 bytes) 000852 1775167410.844126 mapwrite 0x0275d9 thru 0x02848b (0xeb3 bytes) 000853 1775167410.853866 write 0x018ee3 thru 0x0202ee (0x740c bytes) 000854 1775167410.860786 fallocate 0x00fed5 thru 0x01e2be (0xe3ea bytes) 000855 1775167410.863773 trunc from 0x01d165 to 0x029786 (0xc622 bytes) 000856 1775167410.870469 fallocate 0x0063e7 thru 0x00a86b (0x4485 bytes) 000859 1775167410.877226 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x29c26): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29c26): Mirror 8: 0x2113ca5c (stale) Mirror 9: 0x2113ca5c (stale) Mirror 11: 0x2113ca5c (stale) Mirror 12: 0x2113ca5c (stale) Mirror 13: 0x2113ca5c (stale) Mirror 14: 0x2113ca5c (stale) Mirror 15: 0x2113ca5c (stale) CRC-32 checksum value for chunk [0, 0x29c26): Mirror 1: 0x5229b91b Mirror 16: 0x5229b91b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29c26: skipped 000860 1775167411.187395 trunc from 0x0152ea to 0x0166d4 (0x13eb bytes) 000861 1775167411.206983 trunc from 0x029c26 to 0x0356c4 (0xba9f bytes) 000862 1775167411.222795 mapread 0x00cf11 thru 0x01c152 (0xf242 bytes) 000863 1775167411.230901 read 0x019666 thru 0x02551a (0xbeb5 bytes) 000864 1775167411.233811 write 0x00eac6 thru 0x013d52 (0x528d bytes) 000865 1775167411.234835 write 0x0359fc thru 0x036168 (0x76d bytes) 000866 1775167411.238324 punch from 0x00198d to 0x00b497 (0x9b0b bytes) 000867 1775167411.240776 mapread 0x026b58 thru 0x030b92 (0xa03b bytes) 000868 1775167411.244348 trunc from 0x036169 to 0x00fb16 (0x26652 bytes) 000869 1775167411.250612 fallocate 0x01dae4 thru 0x02a743 (0xcc60 bytes) 000870 1775167411.253588 trunc from 0x00a3ae to 0x0165bb (0xc20e bytes) 000872 1775167411.258360 trunc from 0x02118b to 0x02a743 (0x95b9 bytes) 000873 1775167411.260553 mapwrite 0x03f1e8 thru 0x03ffff (0xe18 bytes) 000874 1775167411.276483 read 0x0227da thru 0x0253b0 (0x2bd7 bytes) 000875 1775167411.282409 fallocate 0x025e71 thru 0x030c56 (0xade6 bytes) 000877 1775167411.287307 MIRROR_RESYNC 9 mirrors 000878 1775167411.716953 read 0x02ae73 thru 0x02aee2 (0x70 bytes) 000879 1775167411.718630 trunc from 0x040000 to 0x03f8b7 (0x748 bytes) 000880 1775167411.745382 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f8b8): Mirror 8: 0xc57d94c0 (stale) Mirror 9: 0xc57d94c0 (stale) Mirror 11: 0xc57d94c0 (stale) Mirror 12: 0xc57d94c0 (stale) Mirror 13: 0xc57d94c0 (stale) Mirror 15: 0xc57d94c0 (stale) Mirror 16: 0xc57d94c0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f8b8): Mirror 8: 0xc57d94c0 (stale) Mirror 9: 0xc57d94c0 (stale) Mirror 11: 0xc57d94c0 (stale) Mirror 12: 0xc57d94c0 (stale) Mirror 13: 0xc57d94c0 (stale) Mirror 15: 0xc57d94c0 (stale) Mirror 16: 0xc57d94c0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f8b8: skipped 000881 1775167411.927982 trunc from 0x01d791 to 0x024dcf (0x763f bytes) 000882 1775167411.930079 write 0x028ef5 thru 0x02a6da (0x17e6 bytes) 000883 1775167411.935427 punch from 0x033fe8 to 0x03f8b7 (0xb8d0 bytes) 000884 1775167411.940021 punch from 0x016d3e to 0x01ea24 (0x7ce7 bytes) 000885 1775167411.942476 mapwrite 0x00636b thru 0x010b91 (0xa827 bytes) 000886 1775167411.955226 mapread 0x0246f4 thru 0x024cc1 (0x5ce bytes) 000888 1775167411.959689 trunc from 0x01c2f4 to 0x0227e8 (0x64f5 bytes) 000890 1775167411.962275 fallocate 0x03714a thru 0x038cbd (0x1b74 bytes) 000891 1775167411.965159 fallocate 0x00820c thru 0x0181bd (0xffb2 bytes) 000892 1775167411.967311 write 0x03ed53 thru 0x03ffff (0x12ad bytes) 000893 1775167411.972312 trunc from 0x03e430 to 0x03ffff (0x1bd0 bytes) 000894 1775167411.977566 trunc from 0x040000 to 0x0021ab (0x3de54 bytes) 000895 1775167411.992987 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x21ac): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21ac): Mirror 8: 0xb0c67a4b (stale) Mirror 9: 0xb0c67a4b (stale) Mirror 11: 0xb0c67a4b (stale) Mirror 12: 0xb0c67a4b (stale) Mirror 13: 0xb0c67a4b (stale) Mirror 15: 0xb0c67a4b (stale) Mirror 16: 0xb0c67a4b (stale) CRC-32 checksum value for chunk [0, 0x21ac): Mirror 1: 0xb0c67a4b Mirror 17: 0xb0c67a4b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21ac: skipped 000897 1775167412.299830 mapread 0x0002f2 thru 0x0021ab (0x1eba bytes) 000898 1775167412.313311 trunc from 0x0006f1 to 0x0021ab (0x1abb bytes) 000899 1775167412.332698 MIRROR_RESYNC 9 mirrors 000900 1775167412.792978 write 0x01d2b8 thru 0x0216b4 (0x43fd bytes) 000901 1775167412.804343 trunc from 0x01dd1d to 0x0216b4 (0x3998 bytes) 000902 1775167412.810976 MIRROR_RESYNC 9 mirrors 000903 1775167413.421779 MIRROR_RESYNC 9 mirrors 000904 1775167413.617581 mapwrite 0x03932c thru 0x03ffff (0x6cd4 bytes) 000905 1775167413.646891 trunc from 0x040000 to 0x0154a2 (0x2ab5d bytes) 000907 1775167413.656438 write 0x021cc0 thru 0x0280bc (0x63fd bytes) 000908 1775167413.659307 punch from 0x00ac54 to 0x0192f8 (0xe6a5 bytes) 000909 1775167413.661857 mapread 0x013b3b thru 0x016711 (0x2bd7 bytes) 000911 1775167413.669190 MIRROR_RESYNC 9 mirrors 000913 1775167414.106084 trunc from 0x0280bd to 0x026e7c (0x1240 bytes) 000914 1775167414.128490 trunc from 0x026e7d to 0x02570c (0x1770 bytes) 000915 1775167414.137060 trunc from 0x00adbb to 0x00ea81 (0x3cc7 bytes) 000916 1775167414.139879 fallocate 0x03fb5d thru 0x03ffff (0x4a3 bytes) 000917 1775167414.144420 mapread 0x02a40f thru 0x0374c5 (0xd0b7 bytes) 000918 1775167414.156839 punch from 0x00d112 to 0x01c515 (0xf404 bytes) 000919 1775167414.159585 trunc from 0x027875 to 0x02cf99 (0x5725 bytes) 000920 1775167414.162075 mapwrite 0x02a2de thru 0x03836d (0xe090 bytes) 000921 1775167414.176686 punch from 0x037108 to 0x03ffff (0x8ef8 bytes) 000922 1775167414.179190 fallocate 0x005acb thru 0x0128ed (0xce23 bytes) 000924 1775167414.185085 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x183e1a8f (stale) Mirror 11: 0x183e1a8f (stale) Mirror 12: 0x183e1a8f (stale) Mirror 13: 0x183e1a8f (stale) Mirror 15: 0x183e1a8f (stale) Mirror 16: 0x183e1a8f (stale) Mirror 17: 0x183e1a8f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x183e1a8f (stale) Mirror 11: 0x183e1a8f (stale) Mirror 12: 0x183e1a8f (stale) Mirror 13: 0x183e1a8f (stale) Mirror 15: 0x183e1a8f (stale) Mirror 16: 0x183e1a8f (stale) Mirror 17: 0x183e1a8f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000926 1775167414.335013 trunc from 0x03237b to 0x03faa8 (0xd72e bytes) 000927 1775167414.337117 mapread 0x03d30d thru 0x03ffff (0x2cf3 bytes) 000928 1775167414.344343 fallocate 0x007451 thru 0x008e33 (0x19e3 bytes) 000929 1775167414.346860 trunc from 0x007217 to 0x00e2df (0x70c9 bytes) 000930 1775167414.348763 trunc from 0x0074f1 to 0x00bbe6 (0x46f6 bytes) 000931 1775167414.350825 mapwrite 0x0064dd thru 0x00c8dc (0x6400 bytes) 000933 1775167414.363052 mapwrite 0x03a8ba thru 0x03ffff (0x5746 bytes) 000934 1775167414.372849 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x183e1a8f (stale) Mirror 11: 0x183e1a8f (stale) Mirror 12: 0x183e1a8f (stale) Mirror 13: 0x183e1a8f (stale) Mirror 16: 0x183e1a8f (stale) Mirror 17: 0x183e1a8f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x183e1a8f (stale) Mirror 11: 0x183e1a8f (stale) Mirror 12: 0x183e1a8f (stale) Mirror 13: 0x183e1a8f (stale) Mirror 16: 0x183e1a8f (stale) Mirror 17: 0x183e1a8f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000935 1775167414.519735 trunc from 0x00d508 to 0x018cf2 (0xb7eb bytes) 000936 1775167414.526748 read 0x03ce6c thru 0x03e8ba (0x1a4f bytes) 000937 1775167414.530890 trunc from 0x0163e7 to 0x01aca7 (0x48c1 bytes) 000938 1775167414.533647 read 0x0126ff thru 0x01b2a5 (0x8ba7 bytes) 000939 1775167414.540059 fallocate 0x017066 thru 0x01f46e (0x8409 bytes) 000940 1775167414.542967 read 0x00dd5c thru 0x0141f6 (0x649b bytes) 000941 1775167414.543561 fallocate 0x01321e thru 0x01cf54 (0x9d37 bytes) 000942 1775167414.546447 trunc from 0x040000 to 0x02e3ed (0x11c12 bytes) 000943 1775167414.558254 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e3ee): Mirror 9: 0x3bdf8c02 (stale) Mirror 11: 0x3bdf8c02 (stale) Mirror 12: 0x3bdf8c02 (stale) Mirror 13: 0x3bdf8c02 (stale) Mirror 17: 0x3bdf8c02 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e3ee): Mirror 9: 0x3bdf8c02 (stale) Mirror 11: 0x3bdf8c02 (stale) Mirror 12: 0x3bdf8c02 (stale) Mirror 13: 0x3bdf8c02 (stale) Mirror 17: 0x3bdf8c02 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e3ee: skipped 000944 1775167414.702012 trunc from 0x02e3ee to 0x00ba9e (0x2294f bytes) 000945 1775167414.713024 punch from 0x0083c2 to 0x00ba9e (0x36dd bytes) 000946 1775167414.716365 mapwrite 0x00ccfa thru 0x019899 (0xcba0 bytes) 000947 1775167414.749215 trunc from 0x0101e8 to 0x019899 (0x96b2 bytes) 000948 1775167414.755600 fallocate 0x01c44c thru 0x01e816 (0x23cb bytes) 000949 1775167414.758453 mapwrite 0x03361b thru 0x03ffff (0xc9e5 bytes) 000950 1775167414.780312 trunc from 0x040000 to 0x03ada6 (0x5259 bytes) 000951 1775167414.785782 mapread 0x01dcc2 thru 0x023b30 (0x5e6f bytes) 000952 1775167414.789745 mapwrite 0x022e12 thru 0x02ba85 (0x8c74 bytes) 000953 1775167414.806590 punch from 0x03a951 to 0x03ada6 (0x456 bytes) 000954 1775167414.809221 write 0x036773 thru 0x03a74d (0x3fdb bytes) 000955 1775167414.813804 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ada7): Mirror 11: 0x53e93cff (stale) Mirror 12: 0x53e93cff (stale) Mirror 13: 0x53e93cff (stale) Mirror 17: 0x53e93cff (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ada7): Mirror 11: 0x53e93cff (stale) Mirror 12: 0x53e93cff (stale) Mirror 13: 0x53e93cff (stale) Mirror 17: 0x53e93cff (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ada7: skipped 000957 1775167414.977826 read 0x0066f6 thru 0x00b944 (0x524f bytes) 000958 1775167414.982596 mapwrite 0x02d500 thru 0x03180a (0x430b bytes) 000959 1775167414.995913 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3ada7): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ada7): Mirror 11: 0x53e93cff (stale) Mirror 12: 0x53e93cff (stale) Mirror 13: 0x53e93cff (stale) Mirror 17: 0x53e93cff (stale) CRC-32 checksum value for chunk [0, 0x3ada7): Mirror 1: 0xc740d15f Mirror 18: 0xc740d15f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ada7: skipped 000960 1775167415.377048 trunc from 0x020d36 to 0x024d46 (0x4011 bytes) 000961 1775167415.390364 mapwrite 0x004c29 thru 0x00843c (0x3814 bytes) 000963 1775167415.402124 trunc from 0x015aa8 to 0x016aca (0x1023 bytes) 000964 1775167415.403934 fallocate 0x026bfc thru 0x0351d4 (0xe5d9 bytes) 000965 1775167415.405906 mapread 0x036457 thru 0x03ada6 (0x4950 bytes) 000966 1775167415.408783 punch from 0x00184c to 0x003f91 (0x2746 bytes) 000967 1775167415.410492 fallocate 0x003c0c thru 0x006461 (0x2856 bytes) 000968 1775167415.412148 trunc from 0x03ada7 to 0x01e44b (0x1c95b bytes) 000970 1775167415.418825 mapwrite 0x03d0df thru 0x03ffff (0x2f21 bytes) 000971 1775167415.429242 read 0x03d888 thru 0x03ffff (0x2778 bytes) 000972 1775167415.431389 MIRROR_RESYNC 6 mirrors 000973 1775167415.987997 MIRROR_EXTEND to 7 mirrors 000974 1775167416.273535 fallocate 0x0321c7 thru 0x0327e2 (0x61c bytes) 000975 1775167416.292743 mapread 0x008bd6 thru 0x014632 (0xba5d bytes) 000976 1775167416.306543 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xb9ae507e (stale) Mirror 12: 0xb9ae507e (stale) Mirror 17: 0xb9ae507e (stale) Mirror 18: 0xb9ae507e (stale) Mirror 19: 0xb9ae507e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xb9ae507e (stale) Mirror 12: 0xb9ae507e (stale) Mirror 17: 0xb9ae507e (stale) Mirror 18: 0xb9ae507e (stale) Mirror 19: 0xb9ae507e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000977 1775167416.447147 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xb9ae507e (stale) Mirror 12: 0xb9ae507e (stale) Mirror 17: 0xb9ae507e (stale) Mirror 18: 0xb9ae507e (stale) Mirror 19: 0xb9ae507e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb9ae507e Mirror 20: 0xb9ae507e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000978 1775167416.815514 mapread 0x03fbc3 thru 0x03ffff (0x43d bytes) 000979 1775167416.827599 read 0x00fa1d thru 0x015ef1 (0x64d5 bytes) 000980 1775167416.834679 trunc from 0x040000 to 0x00d9c5 (0x3263a bytes) 000981 1775167416.859686 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xd9c6): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd9c6): Mirror 11: 0x1fee7aa5 (stale) Mirror 12: 0x1fee7aa5 (stale) Mirror 17: 0x1fee7aa5 (stale) Mirror 18: 0x1fee7aa5 (stale) Mirror 19: 0x1fee7aa5 (stale) Mirror 20: 0x1fee7aa5 (stale) CRC-32 checksum value for chunk [0, 0xd9c6): Mirror 1: 0x1fee7aa5 Mirror 21: 0x1fee7aa5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd9c6: skipped 000982 1775167417.177278 trunc from 0x00d9c6 to 0x02aedd (0x1d518 bytes) 000983 1775167417.200253 mapwrite 0x03d5e3 thru 0x03ffff (0x2a1d bytes) 000985 1775167417.221872 mapwrite 0x03c79b thru 0x03ffff (0x3865 bytes) 000986 1775167417.230827 punch from 0x03974e to 0x03b3d7 (0x1c8a bytes) 000987 1775167417.232844 fallocate 0x027173 thru 0x03374e (0xc5dc bytes) 000988 1775167417.237336 trunc from 0x01126f to 0x020818 (0xf5aa bytes) 000989 1775167417.239233 trunc from 0x01e61c to 0x0276b8 (0x909d bytes) 000990 1775167417.240965 mapread 0x0127d7 thru 0x0210dc (0xe906 bytes) 000991 1775167417.247751 write 0x02930d thru 0x038a27 (0xf71b bytes) 000992 1775167417.252990 fallocate 0x0281cd thru 0x0355d9 (0xd40d bytes) 000993 1775167417.255285 write 0x00dd55 thru 0x01b04c (0xd2f8 bytes) 000994 1775167417.256298 fallocate 0x034ee9 thru 0x03ffff (0xb117 bytes) 000995 1775167417.258848 trunc from 0x009858 to 0x015ce7 (0xc490 bytes) 000997 1775167417.267931 MIRROR_RESYNC 8 mirrors 000998 1775167418.017707 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x4d4:0x0]: Invalid argument (22) 000999 1775167418.140527 read 0x00946f thru 0x010c6a (0x77fc bytes) 001000 1775167418.143507 mapread 0x023727 thru 0x02b628 (0x7f02 bytes) All operations completed A-OK! PASS 70a (35s) == sanity-flr test 71: check mirror extend parent fid ==== 18:03:40 (1775167420) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x51d:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (3s) == sanity-flr test 72: check append on FLR file ========== 18:03:44 (1775167424) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x92:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x95:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0695488 s, 30.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x92:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x93:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x95:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505369 s, 20.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x92:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x93:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x96:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x95:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x92:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x93:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x96:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x95:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x94:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (4s) == sanity-flr test 200a: stress test ===================== 18:03:47 (1775167427) Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre2 Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6310400 .. Extending file size to 2897344 .. failed Extending file size to 8205184 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7080896 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7349216 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8240384 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8365856 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9368480 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8520992 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8596192 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8861056 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8879360 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8912768 .. failed Extending file size to 9126784 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61783 61784 61785 61789 61790 done 192.168.202.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg218-client.virtnet /mnt/lustre2 (opts:) 192.168.202.118@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg218-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (65s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 18:04:52 (1775167492) Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre2 Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4968608 .. Extending file size to 6462624 .. Extending file size to 8365920 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7874496 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8362080 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8393184 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8580128 .. failed Extending file size to 8439168 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8478944 .. Extending file size to 9135552 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9023296 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9075936 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9323872 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 64838 64839 64841 64843 64845 192.168.202.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg218-client.virtnet /mnt/lustre2 (opts:) 192.168.202.118@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg218-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 18:05:56 (1775167556) Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2504 1268800 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2628 1268676 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26056 3580964 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18068 3588952 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 44124 7169916 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2504 1268800 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2628 1268676 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 26056 3580964 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18068 3588952 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 44124 7169916 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0429161 s, 48.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0487952 s, 43.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xd1efbd8e (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg218-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 18:06:03 (1775167563) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0344815 s, 60.8 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 18:06:05 (1775167565) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x60:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 18:06:07 (1775167567) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.031383 s, 66.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x6b7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x6b9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x6ba:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 18:06:10 (1775167570) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0546056 s, 38.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 18:06:13 (1775167573) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0367337 s, 57.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 18:06:38 (1775167598) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg218-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188242 s, 55.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x6b8:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x6bc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x6bb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x6b8:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x6bc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x6bb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x6b9:0x0] } PASS 204d (34s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 18:07:12 (1775167632) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg218-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0342587 s, 61.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (33s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 18:07:45 (1775167665) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg218-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg218-server: Pool lustre.test_204f created Adding targets to pool oleg218-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg218-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0282109 s, 74.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg218-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg218-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg218-server: Pool lustre.test_204f destroyed PASS 204f (48s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 18:08:33 (1775167713) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 18:08:35 (1775167715) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 18:08:38 (1775167718) oleg218-server: Pool lustre.test_206 created Adding targets to pool oleg218-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg218-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg218-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg218-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg218-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (15s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 18:08:53 (1775167733) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317906 s, 33.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124754 s, 84.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x749ea752 Mirror 3: 0x749ea752 PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 18:08:56 (1775167736) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 18:08:57 (1775167737) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 18:08:59 (1775167739) Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.015922 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x6f0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x6f0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x6f0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x6f0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1150000 Read this (4 bytes): BBBB 192.168.202.118@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg218-client.virtnet /mnt/lustre2 (opts:) PASS 209a (17s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 18:09:15 (1775167755) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00340041 s, 2.4 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 18:09:17 (1775167757) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148625 s, 70.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x81 lmm_fid: [0x200000405:0x81:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x81 lmm_fid: [0x200000405:0x81:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbc:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611265 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-02 18:09:18.000000000 -0400 Modify: 2026-04-02 18:09:18.000000000 -0400 Change: 2026-04-02 18:09:18.000000000 -0400 Birth: 2026-04-02 18:09:18.000000000 -0400 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 18:09:20 (1775167760) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132165 s, 79.3 MB/s oleg218-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg218-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg218-server: O/280000401/d27/bb: File not found by ext2_lookup PASS 210b (11s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 18:09:31 (1775167771) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0359276 s, 1.1 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00493071 s, 831 kB/s size after second write -rw-r--r-- 1 root root 4096 Apr 2 18:09 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x86:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Apr 2 18:09 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 18:09:33 (1775167773) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (3s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 18:09:36 (1775167776) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (2s) == sanity-flr test complete, duration 1008 sec =========== 18:09:38 (1775167778) === sanity-flr: start cleanup 18:09:39 (1775167779) === === sanity-flr: finish cleanup 18:09:41 (1775167781) ===