-----============= acceptance-small: sanity-flr ============----- Wed Feb 18 02:02:01 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg340-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 02:02:21 (1771398141) === oleg340-client.virtnet: executing check_config_client /mnt/lustre oleg340-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg340-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff990482d54000.idle_timeout=debug osc.lustre-OST0001-osc-ffff990482d54000.idle_timeout=debug disable quota as required oleg340-server: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 02:02:59 (1771398179) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8213] == sanity-flr test 0a: lfs mirror create with -N option == 02:03:01 (1771398181) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 02:03:15 (1771398195) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 02:03:18 (1771398198) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 02:03:22 (1771398202) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 02:03:45 (1771398225) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 02:03:49 (1771398229) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 02:03:52 (1771398232) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff990482d54000.stats=clear osc.lustre-OST0001-osc-ffff990482d54000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 02:04:00 (1771398240) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 02:04:10 (1771398250) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 02:04:18 (1771398258) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 02:04:22 (1771398262) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (7s) == sanity-flr test 2: create components from existing files ========================================================== 02:04:30 (1771398270) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 02:04:37 (1771398277) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 02:04:44 (1771398284) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 02:04:52 (1771398292) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 02:05:03 (1771398303) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 02:05:10 (1771398310) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.261656 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873573 s, 12.0 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 02:05:18 (1771398318) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.152374 s, 41.3 MB/s PASS 22 (8s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 02:05:26 (1771398326) Stopping /mnt/lustre-ost1 (opts:) on oleg340-server oleg340-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff990482d54000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff990482d54000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg340-server: oleg340-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg340-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg340-server oleg340-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff990482d54000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff990482d54000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg340-server: oleg340-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg340-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (75s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 02:06:41 (1771398401) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.597373 s, 19.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg340-server oleg340-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff990482d54000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff990482d54000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg340-server: oleg340-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg340-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (41s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 02:07:22 (1771398442) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg340-server oleg340-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff990482d54000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff990482d54000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg340-server: oleg340-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg340-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg340-server oleg340-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff990482d54000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff990482d54000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg340-server: oleg340-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg340-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (85s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 02:08:47 (1771398527) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.68225 s, 14.2 MB/s -rw-r--r-- 1 root root 52428800 Feb 18 02:08 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.408s user 0m0.002s sys 0m0.186s Stopping /mnt/lustre-ost1 (opts:) on oleg340-server oleg340-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff990482d54000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff990482d54000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.671s user 0m0.001s sys 0m0.220s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg340-server: oleg340-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg340-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg340-server oleg340-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff990482d54000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff990482d54000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.640s user 0m0.002s sys 0m0.169s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg340-server: oleg340-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg340-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg340-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff990482d54000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff990482d54000.ost_server_uuid in FULL state after 0 sec PASS 33b (92s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 02:10:20 (1771398620) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 02:10:23 (1771398623) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 02:10:27 (1771398627) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 02:10:31 (1771398631) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 02:10:39 (1771398639) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.368068 s, 34.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 02:10:53 (1771398653) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332501 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516859 s, 20.3 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 02:11:00 (1771398660) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 02:11:10 (1771398670) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000785656 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00119026 s, 860 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00539556 s, 190 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0107585 s, 95.2 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 02:11:22 (1771398682) create test files with size 15748061 .. f969d3942ecad04ed38c9ee3aca675d7 - 5af1e1f493380f238d8ae65690d3c835 - 46a9a08736e07071104381e0452b2525 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Starting client: oleg340-client.virtnet: -o user_xattr,flock 192.168.203.140@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 02:11:47 (1771398707) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 02:12:01 (1771398721) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 02:12:08 (1771398728) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695413 s, 15.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207612 s, 50.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x41:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 02:12:17 (1771398737) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0513823 s, 40.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.414982 s, 10.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.185725 s, 11.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 02:12:31 (1771398751) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 02:12:34 (1771398754) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 02:12:38 (1771398758) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339709 s, 30.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 02:12:47 (1771398767) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.300065 s, 10.5 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 02:12:59 (1771398779) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4a:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x48:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x48:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 02:13:10 (1771398790) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.426066 s, 24.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Starting client: oleg340-client.virtnet: -o user_xattr,flock 192.168.203.140@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 02:13:23 (1771398803) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff990490dbb800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.31056 s, 33.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Starting client: oleg340-client.virtnet: -o user_xattr,flock 192.168.203.140@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9904b3055800.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 02:13:40 (1771398820) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (44s) == sanity-flr test 46: Verify setstripe --copy option ==== 02:14:24 (1771398864) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (39s) == sanity-flr test 47: Verify mirror obj alloc =========== 02:15:04 (1771398904) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 02:15:08 (1771398908) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.153293 s, 20.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.280266 s, 11.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x53bec289 (stale) Mirror 2: 0x53bec289 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 6b796eeb5905fe16724f51b279c6f002 - mirror 1 checksum: 0201e21f2c26d5f1532f66914cc357db - mirror 2 checksum: 6b796eeb5905fe16724f51b279c6f002 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Feb 18 02:15:11 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x57:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Feb 18 02:15:11 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x58:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 6b796eeb5905fe16724f51b279c6f002 - mirror 1 checksum: 0201e21f2c26d5f1532f66914cc357db - mirror 2 checksum: 0201e21f2c26d5f1532f66914cc357db - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Feb 18 02:15:11 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x57:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Feb 18 02:15:11 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x58:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 02:15:20 (1771398920) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614699 s, 17.1 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 308224.. 309247: 1024: 0001: net,eof 1: 0.. 1023: 312320.. 313343: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (10s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 02:15:30 (1771398930) Failing mds1 on oleg340-server Stopping /mnt/lustre-mds1 (opts:) on oleg340-server 02:15:43 (1771398943) shut down facet: mds1 facet_host: oleg340-server facet_failover_host: oleg340-server Failover mds1 to oleg340-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg340-server: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-MDT0000 02:16:15 (1771398975) targets are mounted 02:16:15 (1771398975) facet_failover done oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328395 s, 31.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } PASS 50A (64s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 02:16:34 (1771398994) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0513992 s, 398 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0272666 s, 751 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (12s) == sanity-flr test 50b: mirror rsync handles sparseness == 02:16:46 (1771399006) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0430484 s, 476 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0228047 s, 898 kB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (13s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 02:16:59 (1771399019) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00693987 s, 2.4 MB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 02:17:09 (1771399029) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0372462 s, 550 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 02:17:17 (1771399037) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00568036 s, 721 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0333622 s, 123 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 02:17:26 (1771399046) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 02:17:48 (1771399068) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 02:18:17 (1771399097) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 02:18:42 (1771399122) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 02:19:08 (1771399148) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 02:19:18 (1771399158) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 55952 mirror split pid 55953 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4987: 55952 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4987: 55953 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 02:20:27 (1771399227) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 9229 skipping zero length punch hole truncating to largest ever: 0x29b2a 000002 1771399229.813632 trunc from 00000000 to 0x029b29 (0x29b2a bytes) 000003 1771399229.844588 mapwrite 0x033e5f thru 0x03ffff (0xc1a1 bytes) 000004 1771399229.944314 read 0x03689f thru 0x03ffff (0x9761 bytes) 000005 1771399229.945865 mapwrite 0x01432b thru 0x01bbf6 (0x78cc bytes) 000006 1771399229.978219 write 0x02577d thru 0x029c99 (0x451d bytes) 000007 1771399229.991935 write 0x01bc48 thru 0x02754c (0xb905 bytes) 000008 1771399229.994840 trunc from 0x040000 to 0x009507 (0x36af8 bytes) 000009 1771399230.011115 fallocate 0x008700 thru 0x01727e (0xeb7f bytes) 000010 1771399230.016156 write 0x028c98 thru 0x02934c (0x6b5 bytes) 000011 1771399230.018939 fallocate 0x007280 thru 0x0164a4 (0xf225 bytes) 000012 1771399230.027010 mapwrite 0x0278ee thru 0x0294a1 (0x1bb4 bytes) 000013 1771399230.067615 mapwrite 0x0358a6 thru 0x03c2dd (0x6a38 bytes) 000014 1771399230.126227 fallocate 0x014a28 thru 0x01ab7a (0x6153 bytes) 000015 1771399230.134108 mapwrite 0x01e819 thru 0x0276f2 (0x8eda bytes) 000016 1771399230.185896 MIRROR_EXTEND to 2 mirrors 000017 1771399230.780079 trunc from 0x03c2de to 0x010667 (0x2bc76 bytes) 000018 1771399230.881551 mapread 0x00de3f thru 0x010667 (0x2829 bytes) 000019 1771399230.905552 MIRROR_RESYNC 2 mirrors 000020 1771399231.368315 mapread 0x00fa72 thru 0x010667 (0xbf6 bytes) 000021 1771399231.386885 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000022 1771399231.538612 trunc from 0x004d9e to 0x006683 (0x18e6 bytes) 000023 1771399231.554819 write 0x031b85 thru 0x03a7d7 (0x8c53 bytes) 000024 1771399231.566101 trunc from 0x03a7d8 to 0x013f9f (0x26838 bytes) 000026 1771399231.598950 write 0x023b0f thru 0x028ebb (0x53ad bytes) fallocating to largest ever: 0x40000 000027 1771399231.602509 fallocate 0x03f62f thru 0x03ffff (0x9d1 bytes) 000028 1771399231.608224 trunc from 0x013882 to 0x02071d (0xce9c bytes) 000030 1771399231.614621 trunc from 0x01aebd to 0x021133 (0x6277 bytes) 000031 1771399231.622156 mapread 0x024595 thru 0x026cc2 (0x272e bytes) 000032 1771399231.624593 write 0x023149 thru 0x02a92f (0x77e7 bytes) 000034 1771399231.634081 read 0x02edc9 thru 0x0381f1 (0x9429 bytes) 000035 1771399231.644318 punch from 0x0207d0 to 0x02ed90 (0xe5c1 bytes) 000036 1771399231.658182 punch from 0x006f9c to 0x008365 (0x13ca bytes) 000038 1771399231.664196 mapread 0x00acec thru 0x012e7c (0x8191 bytes) 000039 1771399231.686782 write 0x02b484 thru 0x0376cc (0xc249 bytes) 000040 1771399231.712995 punch from 0x0082eb to 0x009e45 (0x1b5b bytes) 000041 1771399231.718669 punch from 0x01ec17 to 0x02bf37 (0xd321 bytes) 000042 1771399231.742396 trunc from 0x040000 to 0x038ee5 (0x711a bytes) 000043 1771399231.759074 trunc from 0x02c207 to 0x037abe (0xb8b8 bytes) 000044 1771399231.765025 trunc from 0x03315f to 0x038ee5 (0x5d87 bytes) 000045 1771399231.779792 MIRROR_EXTEND to 2 mirrors 000046 1771399232.590769 punch from 0x0013b3 to 0x00e9bb (0xd609 bytes) 000047 1771399232.637786 mapwrite 0x0014f9 thru 0x00f952 (0xe45a bytes) 000048 1771399232.691641 read 0x011c05 thru 0x019cc8 (0x80c4 bytes) 000049 1771399232.706098 trunc from 0x038ee6 to 0x009a36 (0x2f4af bytes) 000050 1771399232.772642 trunc from 0x009a37 to 0x03e495 (0x34a5f bytes) 000051 1771399232.864020 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000052 1771399233.130243 fallocate 0x005057 thru 0x011cb4 (0xcc5e bytes) 000053 1771399233.138782 write 0x03b916 thru 0x03ffff (0x46ea bytes) 000054 1771399233.163217 read 0x01b016 thru 0x027999 (0xc984 bytes) 000055 1771399233.187233 trunc from 0x040000 to 0x030cbc (0xf343 bytes) 000056 1771399233.221803 mapwrite 0x03490a thru 0x03965c (0x4d53 bytes) 000057 1771399233.347086 fallocate 0x012239 thru 0x0127c2 (0x58a bytes) 000058 1771399233.365899 write 0x02e231 thru 0x038341 (0xa111 bytes) 000060 1771399233.369566 mapwrite 0x020e94 thru 0x029fb3 (0x9120 bytes) 000062 1771399233.421172 punch from 0x029d64 to 0x030445 (0x66e2 bytes) 000063 1771399233.431178 trunc from 0x029759 to 0x02dd34 (0x45dc bytes) 000064 1771399233.439408 write 0x030b17 thru 0x038637 (0x7b21 bytes) 000065 1771399233.469112 trunc from 0x03965d to 0x005819 (0x33e43 bytes) 000066 1771399233.503847 write 0x004381 thru 0x00a7fe (0x647e bytes) 000067 1771399233.533270 trunc from 0x00a7ff to 0x02e6e8 (0x23eea bytes) 000068 1771399233.593881 trunc from 0x02e6e9 to 0x017bc0 (0x16b28 bytes) 000069 1771399233.636874 mapwrite 0x015c26 thru 0x01dbdd (0x7fb8 bytes) 000070 1771399233.691724 write 0x02109a thru 0x02348a (0x23f1 bytes) 000071 1771399233.693183 trunc from 0x021a3c to 0x02348a (0x1a4f bytes) 000072 1771399233.709135 write 0x02d09f thru 0x036aee (0x9a50 bytes) 000073 1771399233.724153 MIRROR_EXTEND to 2 mirrors 000074 1771399235.027788 read 0x009def thru 0x00f06a (0x527c bytes) 000075 1771399235.047054 trunc from 0x036aef to 0x02ba6d (0xb081 bytes) 000076 1771399235.102075 trunc from 0x01dc21 to 0x0214ca (0x38aa bytes) 000079 1771399235.113375 punch from 0x000c06 to 0x001bdd (0xfd8 bytes) 000080 1771399235.119138 read 0x026421 thru 0x02ba6d (0x564d bytes) 000081 1771399235.132629 read 0x012871 thru 0x012bdc (0x36c bytes) 000082 1771399235.133746 mapread 0x015a9a thru 0x01e360 (0x88c7 bytes) 000083 1771399235.141779 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2ba6e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ba6e): Mirror 2: 0xe5053d86 (stale) CRC-32 checksum value for chunk [0, 0x2ba6e): Mirror 1: 0x6ef32c2a Mirror 3: 0x6ef32c2a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ba6e: skipped 000085 1771399236.359835 trunc from 0x02ba6e to 0x012a43 (0x1902a bytes) 000086 1771399236.477979 mapread 0x00a673 thru 0x012a43 (0x83d1 bytes) 000087 1771399236.515323 write 0x027a57 thru 0x034e5c (0xd406 bytes) 000088 1771399236.524959 mapread 0x02498d thru 0x02737a (0x29ee bytes) 000089 1771399236.540756 write 0x00379f thru 0x00b47f (0x7ce1 bytes) 000090 1771399236.551901 mapwrite 0x002fde thru 0x003fc4 (0xfe7 bytes) 000091 1771399236.577769 read 0x013efb thru 0x01c8ba (0x89c0 bytes) 000092 1771399236.594064 read 0x02accf thru 0x0317e1 (0x6b13 bytes) 000094 1771399236.596198 mapwrite 0x013838 thru 0x020322 (0xcaeb bytes) 000096 1771399236.643701 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34e5d): Mirror 3: 0xaa3dfd4a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34e5d): Mirror 3: 0xaa3dfd4a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34e5d: skipped 000097 1771399236.971866 punch from 0x01735d to 0x01e6b8 (0x735c bytes) 000098 1771399236.979124 trunc from 0x034e5d to 0x031225 (0x3c37 bytes) 000099 1771399237.002828 write 0x009f73 thru 0x00d2b0 (0x333e bytes) 000100 1771399237.020110 trunc from 0x024b7d to 0x02e5bb (0x9a3f bytes) 000101 1771399237.026744 mapread 0x01fac8 thru 0x0280b6 (0x85ef bytes) 000102 1771399237.048787 trunc from 0x031226 to 0x01b71c (0x15b09 bytes) 000103 1771399237.066769 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1b71d): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b71d): Mirror 3: 0xde2b6fff (stale) CRC-32 checksum value for chunk [0, 0x1b71d): Mirror 1: 0xcad541d0 Mirror 4: 0xcad541d0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b71d: skipped 000104 1771399237.987066 punch from 0x00422b to 0x008a50 (0x4826 bytes) 000105 1771399238.047588 mapwrite 0x00930d thru 0x00d147 (0x3e3b bytes) 000106 1771399238.093191 trunc from 0x01b71d to 0x029e68 (0xe74c bytes) 000107 1771399238.131309 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29e69): Mirror 4: 0x49a3765d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29e69): Mirror 4: 0x49a3765d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29e69: skipped 000109 1771399238.406613 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000110 1771399238.569149 fallocate 0x03c8ad thru 0x03ffff (0x3753 bytes) 000111 1771399238.580135 read 0x0258e0 thru 0x029e68 (0x4589 bytes) 000112 1771399238.588885 mapwrite 0x01a183 thru 0x01c5e0 (0x245e bytes) 000113 1771399238.634647 write 0x012b53 thru 0x015a53 (0x2f01 bytes) 000114 1771399238.650529 write 0x0355a7 thru 0x039c93 (0x46ed bytes) 000116 1771399238.652524 mapwrite 0x01cb3d thru 0x027018 (0xa4dc bytes) 000117 1771399238.702392 write 0x00f8fe thru 0x011209 (0x190c bytes) 000118 1771399238.703587 trunc from 0x039c94 to 0x009c63 (0x30030 bytes) 000119 1771399238.757956 punch from 0x007985 to 0x009c63 (0x22df bytes) 000120 1771399238.767741 write 0x0386c3 thru 0x03ffff (0x793d bytes) 000122 1771399238.781610 write 0x02a286 thru 0x02a748 (0x4c3 bytes) 000123 1771399238.796384 fallocate 0x0182cd thru 0x024602 (0xc336 bytes) 000125 1771399238.821704 mapread 0x01f69d thru 0x02bb52 (0xc4b6 bytes) 000126 1771399238.837994 write 0x016f08 thru 0x01a188 (0x3281 bytes) 000127 1771399238.867608 mapread 0x03524d thru 0x03f21f (0x9fd3 bytes) 000128 1771399238.886654 punch from 0x017395 to 0x024c6a (0xd8d6 bytes) 000129 1771399238.925071 fallocate 0x028621 thru 0x031826 (0x9206 bytes) 000131 1771399238.937296 punch from 0x006f65 to 0x00aa6b (0x3b07 bytes) 000132 1771399238.943816 mapwrite 0x03d115 thru 0x03ffff (0x2eeb bytes) 000133 1771399238.989876 trunc from 0x039f7f to 0x03ffff (0x6081 bytes) 000134 1771399238.998143 write 0x0398a0 thru 0x03f3e2 (0x5b43 bytes) 000135 1771399239.040687 trunc from 0x000c04 to 0x002e9f (0x229c bytes) 000136 1771399239.053656 mapwrite 0x008e52 thru 0x011524 (0x86d3 bytes) 000137 1771399239.123650 trunc from 0x021c5a to 0x02fce9 (0xe090 bytes) 000138 1771399239.134534 mapwrite 0x03dd3a thru 0x03ffff (0x22c6 bytes) 000139 1771399239.160804 punch from 0x0193be to 0x022fe2 (0x9c25 bytes) 000141 1771399239.185721 mapwrite 0x014d24 thru 0x01c6a1 (0x797e bytes) 000142 1771399239.242937 trunc from 0x040000 to 0x00f3fa (0x30c05 bytes) 000143 1771399239.270609 write 0x01dffa thru 0x026493 (0x849a bytes) 000144 1771399239.275622 mapread 0x013c3a thru 0x0224cd (0xe894 bytes) 000145 1771399239.306773 read 0x00c3fc thru 0x017bfb (0xb800 bytes) 000146 1771399239.308548 mapread 0x0232f3 thru 0x026493 (0x31a1 bytes) 000147 1771399239.328710 mapread 0x006ea9 thru 0x00bf25 (0x507d bytes) 000148 1771399239.332371 fallocate 0x01d68e thru 0x01f4cc (0x1e3f bytes) 000150 1771399239.349929 mapwrite 0x034e02 thru 0x03ed7f (0x9f7e bytes) 000152 1771399239.482122 trunc from 0x023b3c to 0x026ad6 (0x2f9b bytes) 000153 1771399239.504077 fallocate 0x00c9dd thru 0x00e6eb (0x1d0f bytes) 000154 1771399239.525709 mapread 0x01d30e thru 0x02831d (0xb010 bytes) 000155 1771399239.570801 trunc from 0x03ed80 to 0x02e21b (0x10b64 bytes) 000156 1771399239.618797 read 0x028d46 thru 0x02e21b (0x54d6 bytes) 000157 1771399239.619591 mapread 0x001b2e thru 0x00ae6c (0x933f bytes) 000158 1771399239.620778 trunc from 0x02e21c to 0x004a3f (0x297dc bytes) 000159 1771399239.667105 trunc from 0x004a40 to 0x009326 (0x48e7 bytes) 000160 1771399239.689797 fallocate 0x0289e1 thru 0x03880f (0xfe2f bytes) 000161 1771399239.695496 trunc from 0x038810 to 0x028863 (0xffac bytes) 000162 1771399239.774377 punch from 0x0108d5 to 0x015614 (0x4d40 bytes) 000163 1771399239.790751 trunc from 0x028864 to 0x03e189 (0x15926 bytes) 000164 1771399239.824707 write 0x0266a4 thru 0x033629 (0xcf86 bytes) 000165 1771399239.855332 punch from 0x012853 to 0x0221be (0xf96c bytes) 000166 1771399239.872171 mapread 0x034ac6 thru 0x03e189 (0x96c4 bytes) 000167 1771399239.873539 mapwrite 0x00b4f7 thru 0x012b4e (0x7658 bytes) 000168 1771399239.917560 trunc from 0x00d721 to 0x018cb8 (0xb598 bytes) 000169 1771399239.922942 mapread 0x00bb49 thru 0x00c876 (0xd2e bytes) 000170 1771399239.924810 write 0x003f1f thru 0x0100e4 (0xc1c6 bytes) 000171 1771399239.932574 read 0x00f9cd thru 0x01da62 (0xe096 bytes) 000172 1771399239.943804 mapwrite 0x03431f thru 0x03ffff (0xbce1 bytes) 000173 1771399240.017949 fallocate 0x008e86 thru 0x00c05d (0x31d8 bytes) 000175 1771399240.028330 punch from 0x02e0ca to 0x0317c7 (0x36fe bytes) 000176 1771399240.034499 read 0x01bc8a thru 0x026387 (0xa6fe bytes) 000178 1771399240.037109 read 0x006ee7 thru 0x016c38 (0xfd52 bytes) 000179 1771399240.042602 MIRROR_EXTEND to 2 mirrors 000180 1771399241.003867 fallocate 0x03147e thru 0x036f36 (0x5ab9 bytes) 000181 1771399241.071955 fallocate 0x030fce thru 0x03636c (0x539f bytes) 000182 1771399241.085746 write 0x004dda thru 0x0055b9 (0x7e0 bytes) 000183 1771399241.128292 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000184 1771399241.386738 mapread 0x007afa thru 0x014e69 (0xd370 bytes) 000185 1771399241.443809 mapwrite 0x015ab4 thru 0x02578e (0xfcdb bytes) 000186 1771399241.511191 mapread 0x029efe thru 0x02e789 (0x488c bytes) 000187 1771399241.535496 trunc from 0x040000 to 0x00dc86 (0x32379 bytes) 000188 1771399241.564096 mapread 0x0059f6 thru 0x00c18b (0x6796 bytes) 000189 1771399241.565832 write 0x01e561 thru 0x02af01 (0xc9a1 bytes) 000190 1771399241.570827 fallocate 0x01cfc4 thru 0x0297b2 (0xc7ef bytes) 000192 1771399241.586083 mapwrite 0x01b6b0 thru 0x02876d (0xd0be bytes) 000193 1771399241.658093 mapread 0x00865f thru 0x012c7b (0xa61d bytes) 000194 1771399241.675746 read 0x02ab93 thru 0x02af01 (0x36f bytes) 000195 1771399241.676830 trunc from 0x02af02 to 0x00e146 (0x1cdbb bytes) 000197 1771399241.703323 write 0x038ed6 thru 0x03ffff (0x712a bytes) 000198 1771399241.710412 MIRROR_EXTEND to 2 mirrors 000199 1771399242.539751 mapread 0x001f63 thru 0x00ed22 (0xcdc0 bytes) 000200 1771399242.597787 trunc from 0x023e2d to 0x02863d (0x4811 bytes) 000201 1771399242.697092 fallocate 0x01935a thru 0x022ab2 (0x9759 bytes) 000203 1771399242.717468 trunc from 0x00fdde to 0x01dff9 (0xe21c bytes) 000204 1771399242.730578 mapread 0x00266e thru 0x008e4f (0x67e2 bytes) 000205 1771399242.785597 mapwrite 0x03f36c thru 0x03ffff (0xc94 bytes) 000206 1771399242.854034 write 0x03bf24 thru 0x03ffff (0x40dc bytes) 000207 1771399242.860340 read 0x002815 thru 0x008f88 (0x6774 bytes) 000209 1771399242.861710 mapwrite 0x0160a6 thru 0x019b4c (0x3aa7 bytes) 000210 1771399242.952671 fallocate 0x033b17 thru 0x0342ac (0x796 bytes) 000211 1771399242.964512 trunc from 0x040000 to 0x001e0d (0x3e1f2 bytes) 000212 1771399242.991590 trunc from 0x000015 to 0x001e0d (0x1df9 bytes) 000214 1771399242.999470 read 0x000bdd thru 0x001e0d (0x1231 bytes) 000216 1771399243.008603 read 0x000481 thru 0x001e0d (0x198d bytes) 000217 1771399243.015061 MIRROR_RESYNC 2 mirrors 000218 1771399243.871587 trunc from 0x001e0e to 0x03c1cf (0x3a3c2 bytes) 000219 1771399243.964366 MIRROR_RESYNC 2 mirrors 000220 1771399244.693057 trunc from 0x001f5e to 0x010864 (0xe907 bytes) 000221 1771399244.768792 trunc from 0x0061a6 to 0x008b91 (0x29ec bytes) 000222 1771399244.776739 write 0x03ab41 thru 0x03ffff (0x54bf bytes) 000223 1771399244.789857 punch from 0x02dc32 to 0x039d3d (0xc10c bytes) 000225 1771399244.798607 mapread 0x013140 thru 0x0149d7 (0x1898 bytes) 000226 1771399244.837705 trunc from 0x040000 to 0x027e49 (0x181b6 bytes) 000227 1771399244.883707 trunc from 0x022ed4 to 0x027e49 (0x4f76 bytes) 000228 1771399244.900195 fallocate 0x03234a thru 0x03e104 (0xbdbb bytes) 000232 1771399244.919385 read 0x00b1e9 thru 0x012c49 (0x7a61 bytes) 000233 1771399244.931407 mapread 0x00bb0a thru 0x015b4f (0xa046 bytes) 000235 1771399244.933299 mapread 0x01d980 thru 0x025bbb (0x823c bytes) 000236 1771399244.977158 mapwrite 0x020804 thru 0x027f8f (0x778c bytes) 000237 1771399245.058814 trunc from 0x03e105 to 0x00b344 (0x32dc0 bytes) 000238 1771399245.090243 mapread 0x002772 thru 0x00afbf (0x884e bytes) 000239 1771399245.112718 trunc from 0x0091d6 to 0x00b344 (0x216f bytes) 000240 1771399245.121780 fallocate 0x027e8d thru 0x036e67 (0xefdb bytes) 000243 1771399245.153784 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000244 1771399245.437012 fallocate 0x00a703 thru 0x00d430 (0x2d2e bytes) 000245 1771399245.461089 MIRROR_EXTEND to 2 mirrors 000246 1771399246.417131 trunc from 0x00b345 to 0x0138b2 (0x856e bytes) 000247 1771399246.461390 mapread 0x001d70 thru 0x0093fe (0x768f bytes) 000249 1771399246.494046 write 0x031cd4 thru 0x03ffff (0xe32c bytes) 000250 1771399246.502534 write 0x03008b thru 0x032209 (0x217f bytes) 000251 1771399246.517405 fallocate 0x00756d thru 0x00e8e0 (0x7374 bytes) 000252 1771399246.521300 punch from 0x026163 to 0x02bfe4 (0x5e82 bytes) 000253 1771399246.524412 read 0x00fad1 thru 0x016f68 (0x7498 bytes) 000254 1771399246.532160 read 0x026a08 thru 0x031e31 (0xb42a bytes) 000256 1771399246.541101 read 0x02ce15 thru 0x0388a0 (0xba8c bytes) 000257 1771399246.542570 read 0x02dd59 thru 0x033d0d (0x5fb5 bytes) 000258 1771399246.543871 trunc from 0x0163a9 to 0x01841d (0x2075 bytes) 000259 1771399246.549462 punch from 0x038c83 to 0x03e6f4 (0x5a72 bytes) 000260 1771399246.563665 fallocate 0x0227f9 thru 0x030eeb (0xe6f3 bytes) 000261 1771399246.569728 mapwrite 0x014626 thru 0x01950f (0x4eea bytes) 000262 1771399246.596751 fallocate 0x0162fe thru 0x017f84 (0x1c87 bytes) 000263 1771399246.601819 mapwrite 0x0331c0 thru 0x03cc85 (0x9ac6 bytes) 000264 1771399246.619585 mapwrite 0x03d4c6 thru 0x03ffff (0x2b3a bytes) 000265 1771399246.631915 write 0x0045ad thru 0x009937 (0x538b bytes) 000266 1771399246.633152 punch from 0x00fc16 to 0x012a51 (0x2e3c bytes) 000267 1771399246.646748 trunc from 0x040000 to 0x0201b8 (0x1fe47 bytes) 000268 1771399246.690796 punch from 0x01b6cf to 0x01bd02 (0x634 bytes) 000269 1771399246.701761 trunc from 0x0201b9 to 0x0343ca (0x14212 bytes) 000270 1771399246.742061 mapread 0x029f70 thru 0x02f4a7 (0x5538 bytes) 000271 1771399246.807928 fallocate 0x028b20 thru 0x03638a (0xd86b bytes) 000272 1771399246.818044 write 0x02813e thru 0x0286d1 (0x594 bytes) 000273 1771399246.832887 write 0x000c69 thru 0x00c1fb (0xb593 bytes) 000274 1771399246.834497 trunc from 0x01d526 to 0x0273a8 (0x9e83 bytes) 000275 1771399246.847001 MIRROR_RESYNC 2 mirrors 000276 1771399247.528149 punch from 0x00517a to 0x00dcb1 (0x8b38 bytes) 000277 1771399247.572818 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x343cb): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x343cb): Mirror 2: 0x8036f437 (stale) CRC-32 checksum value for chunk [0, 0x343cb): Mirror 1: 0xc7b55313 Mirror 3: 0xc7b55313 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x343cb: skipped 000278 1771399248.755620 read 0x033099 thru 0x0343ca (0x1332 bytes) 000279 1771399248.798348 MIRROR_RESYNC 3 mirrors 000280 1771399249.718519 mapwrite 0x005450 thru 0x00d068 (0x7c19 bytes) 000281 1771399249.878252 punch from 0x0076bd to 0x0118d6 (0xa21a bytes) 000282 1771399249.888944 mapread 0x01d672 thru 0x023d3e (0x66cd bytes) 000283 1771399249.916949 trunc from 0x0343cb to 0x017c22 (0x1c7a8 bytes) 000284 1771399249.955419 read 0x010cd0 thru 0x014a4f (0x3d80 bytes) 000285 1771399249.968506 read 0x000f25 thru 0x000fc9 (0xa5 bytes) 000286 1771399249.980562 write 0x0341e1 thru 0x03c2f3 (0x8113 bytes) 000287 1771399249.984977 mapwrite 0x03cee9 thru 0x03deac (0xfc4 bytes) 000288 1771399250.040901 read 0x0256dd thru 0x02f8a1 (0xa1c5 bytes) 000290 1771399250.045064 mapwrite 0x02f4c9 thru 0x036989 (0x74c1 bytes) 000291 1771399250.079838 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dead): Mirror 2: 0x61274f0b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dead): Mirror 2: 0x61274f0b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dead: skipped 000292 1771399250.458630 punch from 0x025017 to 0x02d5e4 (0x85ce bytes) 000293 1771399250.466128 read 0x0281d9 thru 0x02fc11 (0x7a39 bytes) 000294 1771399250.477126 read 0x035744 thru 0x036b92 (0x144f bytes) 000296 1771399250.502978 punch from 0x01c36b to 0x020e8b (0x4b21 bytes) 000297 1771399250.508839 trunc from 0x0137c0 to 0x013a7b (0x2bc bytes) 000298 1771399250.514698 punch from 0x00f91b to 0x01a0e1 (0xa7c7 bytes) 000299 1771399250.520616 fallocate 0x035bba thru 0x0384fe (0x2945 bytes) 000300 1771399250.526835 read 0x030f33 thru 0x037ee6 (0x6fb4 bytes) 000301 1771399250.528855 write 0x002a53 thru 0x0099c2 (0x6f70 bytes) 000302 1771399250.536868 punch from 0x00a813 to 0x00acc0 (0x4ae bytes) 000304 1771399250.545846 write 0x015964 thru 0x01b365 (0x5a02 bytes) 000305 1771399250.560023 write 0x010a60 thru 0x012fb1 (0x2552 bytes) 000306 1771399250.566723 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000307 1771399250.808415 fallocate 0x02c975 thru 0x039573 (0xcbff bytes) 000308 1771399250.828147 trunc from 0x03dead to 0x00c20a (0x31ca2 bytes) 000309 1771399250.848645 punch from 0x0049e5 to 0x00c20a (0x7826 bytes) 000310 1771399250.859623 write 0x01c8cf thru 0x029638 (0xcd6a bytes) 000311 1771399250.861499 trunc from 0x029639 to 0x022274 (0x73c4 bytes) 000312 1771399250.905788 read 0x00e2a2 thru 0x015910 (0x766f bytes) 000313 1771399250.923807 trunc from 0x016a65 to 0x022274 (0xb810 bytes) 000314 1771399250.943277 fallocate 0x0234bd thru 0x02cd31 (0x9875 bytes) 000315 1771399250.955860 mapread 0x001f34 thru 0x00a6cf (0x879c bytes) 000317 1771399251.009880 trunc from 0x017652 to 0x025b12 (0xe4c1 bytes) 000318 1771399251.028118 mapwrite 0x035fdc thru 0x03ffff (0xa024 bytes) 000319 1771399251.128719 MIRROR_EXTEND to 2 mirrors 000320 1771399251.868239 punch from 0x032a4a to 0x038323 (0x58da bytes) 000321 1771399251.933569 MIRROR_RESYNC 2 mirrors 000322 1771399252.579511 trunc from 0x040000 to 0x03ab98 (0x5467 bytes) 000323 1771399252.667666 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000324 1771399252.958169 trunc from 0x02c2ce to 0x03448e (0x81c1 bytes) 000325 1771399252.964391 trunc from 0x00ac5f to 0x00e870 (0x3c12 bytes) 000326 1771399252.974783 write 0x00e501 thru 0x019d75 (0xb875 bytes) 000327 1771399252.993691 punch from 0x036200 to 0x0374b6 (0x12b7 bytes) 000328 1771399253.001208 read 0x01f975 thru 0x021f29 (0x25b5 bytes) 000329 1771399253.011399 read 0x00d871 thru 0x01832a (0xaaba bytes) 000332 1771399253.033535 write 0x02ff08 thru 0x032fb6 (0x30af bytes) 000334 1771399253.041579 read 0x0207e3 thru 0x026b34 (0x6352 bytes) 000336 1771399253.043701 mapwrite 0x0153e7 thru 0x024dcc (0xf9e6 bytes) 000337 1771399253.070939 write 0x003316 thru 0x0112c2 (0xdfad bytes) 000338 1771399253.073725 mapread 0x008614 thru 0x015861 (0xd24e bytes) 000339 1771399253.081779 MIRROR_EXTEND to 2 mirrors 000340 1771399253.790698 punch from 0x01ed0f to 0x0234a4 (0x4796 bytes) 000342 1771399253.850622 mapread 0x01258c thru 0x012ffc (0xa71 bytes) 000343 1771399253.855814 fallocate 0x002c70 thru 0x00e3b2 (0xb743 bytes) 000344 1771399253.860813 trunc from 0x01d6de to 0x021ebe (0x47e1 bytes) 000345 1771399253.865809 mapread 0x01c69f thru 0x02a3d3 (0xdd35 bytes) 000346 1771399253.915212 trunc from 0x027108 to 0x0315b8 (0xa4b1 bytes) 000348 1771399253.922915 mapread 0x0169b1 thru 0x01c1a6 (0x57f6 bytes) 000349 1771399253.933334 MIRROR_RESYNC 2 mirrors 000350 1771399255.165058 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000351 1771399255.320987 MIRROR_EXTEND to 2 mirrors 000352 1771399255.952390 punch from 0x009f2e to 0x013b3a (0x9c0d bytes) 000353 1771399255.978009 trunc from 0x03ab99 to 0x0389bc (0x21dc bytes) 000354 1771399256.004972 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000355 1771399256.118192 mapread 0x023475 thru 0x0238db (0x467 bytes) 000356 1771399256.124413 read 0x00989d thru 0x00d2ae (0x3a12 bytes) 000357 1771399256.135491 fallocate 0x034546 thru 0x03f8ae (0xb369 bytes) 000358 1771399256.140333 punch from 0x01ec5f to 0x02d71e (0xeac0 bytes) 000359 1771399256.145259 mapwrite 0x030b61 thru 0x03d5df (0xca7f bytes) 000360 1771399256.209362 trunc from 0x01acf4 to 0x020c9e (0x5fab bytes) 000361 1771399256.220029 fallocate 0x00aa36 thru 0x01795f (0xcf2a bytes) 000362 1771399256.225377 mapread 0x00d218 thru 0x00d7b3 (0x59c bytes) 000363 1771399256.227002 read 0x02c0e1 thru 0x02f314 (0x3234 bytes) 000364 1771399256.229250 mapread 0x00b0c1 thru 0x00d693 (0x25d3 bytes) 000365 1771399256.230426 trunc from 0x03d5e0 to 0x0303c4 (0xd21b bytes) 000366 1771399256.253983 MIRROR_EXTEND to 2 mirrors 000367 1771399256.842844 punch from 0x0063cc to 0x01382f (0xd464 bytes) 000368 1771399256.892953 MIRROR_RESYNC 2 mirrors 000370 1771399257.511979 mapread 0x00531c thru 0x007409 (0x20ee bytes) 000371 1771399257.538712 fallocate 0x01758e thru 0x022321 (0xad94 bytes) 000372 1771399257.572998 read 0x028ee6 thru 0x0303c4 (0x74df bytes) 000373 1771399257.585035 write 0x0132e7 thru 0x01bb5c (0x8876 bytes) 000374 1771399257.596152 mapwrite 0x00c449 thru 0x01ae1d (0xe9d5 bytes) 000375 1771399257.618170 fallocate 0x030500 thru 0x033e65 (0x3966 bytes) 000376 1771399257.622662 mapread 0x011294 thru 0x01cec6 (0xbc33 bytes) 000377 1771399257.635336 punch from 0x0278b5 to 0x0303c4 (0x8b10 bytes) 000378 1771399257.640318 read 0x02e9fa thru 0x0303c4 (0x19cb bytes) 000379 1771399257.650208 fallocate 0x013802 thru 0x01d8bc (0xa0bb bytes) 000380 1771399257.656280 read 0x023d01 thru 0x02ea90 (0xad90 bytes) 000381 1771399257.657727 read 0x004f9f thru 0x005d4f (0xdb1 bytes) 000382 1771399257.658964 trunc from 0x01da67 to 0x01f594 (0x1b2e bytes) 000383 1771399257.666923 fallocate 0x006950 thru 0x00a173 (0x3824 bytes) 000384 1771399257.673056 fallocate 0x030a70 thru 0x032f84 (0x2515 bytes) 000385 1771399257.678585 fallocate 0x00a7b8 thru 0x0194f4 (0xed3d bytes) 000386 1771399257.684001 fallocate 0x01c07f thru 0x01d323 (0x12a5 bytes) 000387 1771399257.699285 MIRROR_RESYNC 2 mirrors 000388 1771399258.662787 punch from 0x00c2d4 to 0x012955 (0x6682 bytes) 000389 1771399258.695731 punch from 0x01fb99 to 0x02dd93 (0xe1fb bytes) 000390 1771399258.701476 mapwrite 0x032e43 thru 0x03ffff (0xd1bd bytes) 000391 1771399258.780334 mapread 0x03ad55 thru 0x03ffff (0x52ab bytes) 000392 1771399258.781689 mapwrite 0x023154 thru 0x02e61e (0xb4cb bytes) 000393 1771399258.823230 trunc from 0x007fee to 0x00affc (0x300f bytes) 000394 1771399258.829706 fallocate 0x0250ad thru 0x026948 (0x189c bytes) 000395 1771399258.839946 trunc from 0x026309 to 0x0287ba (0x24b2 bytes) 000396 1771399258.851823 trunc from 0x023734 to 0x031044 (0xd911 bytes) 000397 1771399258.858535 punch from 0x03f621 to 0x03ffff (0x9df bytes) 000399 1771399258.867403 fallocate 0x01e165 thru 0x022fe9 (0x4e85 bytes) 000401 1771399258.878891 trunc from 0x00924a to 0x0191a9 (0xff60 bytes) 000402 1771399258.897791 trunc from 0x010b65 to 0x019105 (0x85a1 bytes) 000403 1771399258.928010 fallocate 0x023350 thru 0x0262e6 (0x2f97 bytes) 000404 1771399258.937335 trunc from 0x040000 to 0x021047 (0x1efb8 bytes) 000405 1771399258.965859 mapwrite 0x00f590 thru 0x01e3f0 (0xee61 bytes) 000406 1771399259.056385 mapwrite 0x0063ea thru 0x013f11 (0xdb28 bytes) 000407 1771399259.110997 read 0x014008 thru 0x021047 (0xd040 bytes) 000408 1771399259.111780 trunc from 0x0180f8 to 0x01d872 (0x577b bytes) 000409 1771399259.122699 read 0x0172b2 thru 0x020d72 (0x9ac1 bytes) 000410 1771399259.144578 read 0x0094c4 thru 0x00a497 (0xfd4 bytes) 000411 1771399259.146331 fallocate 0x035978 thru 0x038d1c (0x33a5 bytes) 000412 1771399259.152814 fallocate 0x00ba81 thru 0x011190 (0x5710 bytes) 000413 1771399259.162331 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000414 1771399259.299182 trunc from 0x02a4f5 to 0x032990 (0x849c bytes) 000415 1771399259.305881 mapread 0x011cba thru 0x01a30a (0x8651 bytes) 000416 1771399259.332550 mapread 0x00695c thru 0x01663f (0xfce4 bytes) 000418 1771399259.349324 mapwrite 0x02fe41 thru 0x036c74 (0x6e34 bytes) 000419 1771399259.377676 mapwrite 0x0260eb thru 0x02d736 (0x764c bytes) 000420 1771399259.403308 write 0x02b89f thru 0x03b03d (0xf79f bytes) 000421 1771399259.409425 mapwrite 0x0052c5 thru 0x010818 (0xb554 bytes) 000422 1771399259.432209 mapwrite 0x02f301 thru 0x035bf4 (0x68f4 bytes) 000423 1771399259.452910 mapwrite 0x01b9f8 thru 0x02290d (0x6f16 bytes) 000424 1771399259.485389 write 0x013326 thru 0x0228d1 (0xf5ac bytes) 000425 1771399259.490321 trunc from 0x030ba0 to 0x03330c (0x276d bytes) 000426 1771399259.497613 fallocate 0x025ba9 thru 0x027bd3 (0x202b bytes) 000427 1771399259.503656 trunc from 0x03b03e to 0x001c8d (0x393b0 bytes) 000428 1771399259.522636 write 0x036b59 thru 0x03f926 (0x8dce bytes) 000429 1771399259.525631 mapwrite 0x01edf5 thru 0x0213b9 (0x25c5 bytes) 000430 1771399259.560765 write 0x00cc51 thru 0x010af9 (0x3ea9 bytes) 000432 1771399259.589354 write 0x02b4b8 thru 0x031d80 (0x68c9 bytes) 000433 1771399259.606961 write 0x032fd0 thru 0x033c60 (0xc91 bytes) 000434 1771399259.608516 mapwrite 0x02adf7 thru 0x02ee7d (0x4087 bytes) 000435 1771399259.638282 mapread 0x000914 thru 0x00bd59 (0xb446 bytes) 000436 1771399259.654938 fallocate 0x0158ca thru 0x01d843 (0x7f7a bytes) 000437 1771399259.663041 trunc from 0x033a9c to 0x03543a (0x199f bytes) 000438 1771399259.672895 punch from 0x03667b to 0x03f926 (0x92ac bytes) 000439 1771399259.683018 trunc from 0x0197bf to 0x0228e9 (0x912b bytes) 000440 1771399259.719005 MIRROR_EXTEND to 2 mirrors 000441 1771399260.495749 mapwrite 0x002b07 thru 0x00a130 (0x762a bytes) 000443 1771399260.594596 fallocate 0x00dcfc thru 0x01067e (0x2983 bytes) 000444 1771399260.601570 mapread 0x015856 thru 0x01dbdb (0x8386 bytes) 000446 1771399260.612560 write 0x014388 thru 0x024315 (0xff8e bytes) 000447 1771399260.626708 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f927): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f927): Mirror 2: 0xd7a46395 (stale) CRC-32 checksum value for chunk [0, 0x3f927): Mirror 1: 0x1818f9fa Mirror 3: 0x1818f9fa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f927: skipped 000448 1771399261.541396 fallocate 0x029e7e thru 0x03492d (0xaab0 bytes) 000449 1771399261.578935 mapwrite 0x02a6b7 thru 0x02bf0d (0x1857 bytes) 000450 1771399261.601972 punch from 0x02ce3e to 0x030a58 (0x3c1b bytes) 000452 1771399261.608746 write 0x00ee01 thru 0x01d77b (0xe97b bytes) 000455 1771399261.632901 trunc from 0x02a870 to 0x039f21 (0xf6b2 bytes) 000456 1771399261.641805 fallocate 0x02041d thru 0x02da82 (0xd666 bytes) 000457 1771399261.647214 read 0x015947 thru 0x01f9bb (0xa075 bytes) 000458 1771399261.653691 trunc from 0x03f927 to 0x006a53 (0x38ed3 bytes) 000459 1771399261.681562 fallocate 0x017781 thru 0x022eed (0xb76d bytes) 000460 1771399261.691218 trunc from 0x0212b0 to 0x022eed (0x1c3e bytes) 000461 1771399261.698254 fallocate 0x02664e thru 0x026858 (0x20b bytes) 000462 1771399261.716773 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26859): Mirror 3: 0xdb672cc6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26859): Mirror 3: 0xdb672cc6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26859: skipped 000463 1771399262.155713 write 0x03b576 thru 0x03ffff (0x4a8a bytes) 000465 1771399262.173341 mapread 0x000da9 thru 0x008325 (0x757d bytes) 000466 1771399262.253704 trunc from 0x040000 to 0x00389d (0x3c762 bytes) 000467 1771399262.274543 mapwrite 0x00328a thru 0x003d6e (0xae5 bytes) 000468 1771399262.316183 punch from 0x003aab to 0x003d6e (0x2c4 bytes) 000469 1771399262.324620 punch from 0x000aa7 to 0x002a9c (0x1ff6 bytes) 000470 1771399262.329684 fallocate 0x024644 thru 0x028b3c (0x44f9 bytes) 000471 1771399262.334608 write 0x038d74 thru 0x03c4a3 (0x3730 bytes) 000472 1771399262.337897 mapread 0x00d36b thru 0x01c6c2 (0xf358 bytes) 000473 1771399262.351788 mapread 0x002213 thru 0x0093d3 (0x71c1 bytes) 000474 1771399262.369866 write 0x02364b thru 0x0281f8 (0x4bae bytes) 000475 1771399262.379481 trunc from 0x03c4a4 to 0x03080c (0xbc97 bytes) 000476 1771399262.399753 punch from 0x002301 to 0x00a3e0 (0x80e0 bytes) 000477 1771399262.405316 read 0x011ec5 thru 0x0211cf (0xf30b bytes) 000478 1771399262.405982 trunc from 0x03080d to 0x03f690 (0xee84 bytes) 000479 1771399262.419814 punch from 0x01db27 to 0x02724b (0x9725 bytes) 000480 1771399262.435741 punch from 0x00cd68 to 0x013023 (0x62bc bytes) 000481 1771399262.440576 trunc from 0x03f691 to 0x01adb9 (0x248d7 bytes) 000482 1771399262.458348 trunc from 0x012106 to 0x01975a (0x7655 bytes) 000483 1771399262.465739 trunc from 0x011b53 to 0x018aa7 (0x6f55 bytes) 000484 1771399262.478716 write 0x033d26 thru 0x03ffff (0xc2da bytes) 000485 1771399262.484014 write 0x03e334 thru 0x03fd9d (0x1a6a bytes) 000486 1771399262.487672 punch from 0x01da30 to 0x022923 (0x4ef4 bytes) 000487 1771399262.493368 mapread 0x00ff91 thru 0x011fe2 (0x2052 bytes) 000489 1771399262.510633 trunc from 0x007275 to 0x0122ee (0xb07a bytes) 000490 1771399262.516965 mapwrite 0x008885 thru 0x009bd8 (0x1354 bytes) 000491 1771399262.559301 trunc from 0x00f0aa to 0x00f422 (0x379 bytes) 000492 1771399262.567372 write 0x03240b thru 0x036097 (0x3c8d bytes) 000493 1771399262.593353 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9282e4a0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe4419e89 Mirror 4: 0xe4419e89 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000494 1771399264.161332 punch from 0x03807b to 0x03c8b1 (0x4837 bytes) 000495 1771399264.262111 read 0x0226b2 thru 0x03107e (0xe9cd bytes) 000496 1771399264.278322 write 0x00c2ac thru 0x01a5da (0xe32f bytes) 000497 1771399264.311659 trunc from 0x040000 to 0x036642 (0x99bd bytes) 000498 1771399264.349686 read 0x028260 thru 0x0344d2 (0xc273 bytes) 000499 1771399264.373791 punch from 0x01aabd to 0x01e653 (0x3b97 bytes) 000500 1771399264.398382 read 0x015f8a thru 0x01ce1e (0x6e95 bytes) 000501 1771399264.442346 trunc from 0x036643 to 0x027e3e (0xe804 bytes) 000502 1771399264.482730 read 0x00db78 thru 0x01cd13 (0xf19c bytes) 000503 1771399264.486209 fallocate 0x01b824 thru 0x023039 (0x7816 bytes) 000504 1771399264.505925 trunc from 0x027e3f to 0x001dc0 (0x2607e bytes) 000505 1771399264.526799 mapwrite 0x00591b thru 0x0123a0 (0xca86 bytes) 000506 1771399264.606114 fallocate 0x029f6e thru 0x032ab5 (0x8b48 bytes) 000507 1771399264.618685 mapwrite 0x0318f6 thru 0x0330ed (0x17f8 bytes) 000508 1771399264.714057 read 0x030369 thru 0x0330ed (0x2d85 bytes) 000509 1771399264.715977 write 0x02e0db thru 0x0355a7 (0x74cd bytes) 000510 1771399264.722062 write 0x011de0 thru 0x01beb6 (0xa0d7 bytes) 000511 1771399264.731003 mapread 0x020301 thru 0x029a54 (0x9754 bytes) 000512 1771399264.739968 MIRROR_RESYNC 3 mirrors 000513 1771399266.011364 punch from 0x014518 to 0x022839 (0xe322 bytes) 000514 1771399266.050356 read 0x01cc80 thru 0x01e3ad (0x172e bytes) 000515 1771399266.055859 punch from 0x011333 to 0x018bf4 (0x78c2 bytes) 000517 1771399266.062745 write 0x00cdbe thru 0x018ce2 (0xbf25 bytes) 000518 1771399266.070267 fallocate 0x036b4f thru 0x03c2e3 (0x5795 bytes) 000520 1771399266.074300 mapread 0x006527 thru 0x006f56 (0xa30 bytes) 000521 1771399266.090819 punch from 0x01248e to 0x01a782 (0x82f5 bytes) 000522 1771399266.104286 trunc from 0x00c36c to 0x0176b9 (0xb34e bytes) 000523 1771399266.109191 mapread 0x01c0a0 thru 0x0238d5 (0x7836 bytes) 000524 1771399266.121261 trunc from 0x0355a8 to 0x007559 (0x2e04e bytes) 000525 1771399266.147041 trunc from 0x0025fe to 0x007559 (0x4f5c bytes) 000527 1771399266.180509 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x755a): Mirror 4: 0xbadf2f95 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x755a): Mirror 4: 0xbadf2f95 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x755a: skipped 000528 1771399266.573914 fallocate 0x029342 thru 0x02e07f (0x4d3e bytes) 000530 1771399266.586367 trunc from 0x00755a to 0x01591f (0xe3c6 bytes) 000531 1771399266.622663 fallocate 0x01ba9b thru 0x0256f2 (0x9c58 bytes) 000532 1771399266.634597 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000533 1771399266.835252 mapread 0x0071e0 thru 0x0137b1 (0xc5d2 bytes) 000534 1771399266.884980 MIRROR_EXTEND to 2 mirrors 000535 1771399267.696000 punch from 0x00f0c5 to 0x01591f (0x685b bytes) 000536 1771399267.741581 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) 000537 1771399267.870630 mapread 0x01275c thru 0x01591f (0x31c4 bytes) 000538 1771399267.891425 fallocate 0x01f6e9 thru 0x02c5c3 (0xcedb bytes) 000539 1771399267.896225 mapread 0x02300e thru 0x02532f (0x2322 bytes) 000540 1771399267.911602 fallocate 0x0136ac thru 0x015cd8 (0x262d bytes) 000541 1771399267.916744 write 0x01f31d thru 0x02224f (0x2f33 bytes) 000542 1771399267.921678 trunc from 0x01a2dc to 0x024b19 (0xa83e bytes) 000543 1771399267.932791 read 0x00fed3 thru 0x0115cf (0x16fd bytes) 000544 1771399267.933683 fallocate 0x037ee0 thru 0x03ffff (0x8120 bytes) 000545 1771399267.938635 write 0x02998f thru 0x033a91 (0xa103 bytes) 000546 1771399267.941267 mapwrite 0x03f54d thru 0x03ffff (0xab3 bytes) 000547 1771399267.976943 mapwrite 0x01eb6f thru 0x024e58 (0x62ea bytes) 000548 1771399268.002640 read 0x03ea7a thru 0x03ffff (0x1586 bytes) 000549 1771399268.003340 trunc from 0x009ac8 to 0x00ceb8 (0x33f1 bytes) 000550 1771399268.007744 mapread 0x00c33b thru 0x01a7b9 (0xe47f bytes) 000552 1771399268.017873 mapread 0x030fba thru 0x03912b (0x8172 bytes) 000553 1771399268.019726 trunc from 0x040000 to 0x0166bc (0x29943 bytes) 000554 1771399268.034628 mapwrite 0x02feb0 thru 0x03c173 (0xc2c4 bytes) 000555 1771399268.126681 mapwrite 0x03add6 thru 0x03b9bc (0xbe7 bytes) 000556 1771399268.143776 MIRROR_EXTEND to 2 mirrors 000557 1771399268.843323 MIRROR_EXTEND to 3 mirrors 000558 1771399269.888781 punch from 0x03b3db to 0x03c173 (0xd99 bytes) 000559 1771399269.995284 write 0x0337d0 thru 0x039a76 (0x62a7 bytes) 000560 1771399270.030965 write 0x0111ce thru 0x01c4d6 (0xb309 bytes) 000561 1771399270.070556 punch from 0x00f48e to 0x01a430 (0xafa3 bytes) 000562 1771399270.099609 read 0x00a57e thru 0x019108 (0xeb8b bytes) 000563 1771399270.107802 read 0x003eb0 thru 0x00ce23 (0x8f74 bytes) 000564 1771399270.122526 read 0x009193 thru 0x013831 (0xa69f bytes) 000565 1771399270.123510 punch from 0x019ebd to 0x01ca25 (0x2b69 bytes) 000566 1771399270.128141 mapwrite 0x023510 thru 0x0333fc (0xfeed bytes) 000567 1771399270.168937 write 0x006379 thru 0x009ed8 (0x3b60 bytes) 000568 1771399270.173921 fallocate 0x0078d2 thru 0x01177c (0x9eab bytes) 000569 1771399270.180176 trunc from 0x012bed to 0x01e574 (0xb988 bytes) 000570 1771399270.190492 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3c174): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c174): Mirror 2: 0x625b5878 (stale) Mirror 3: 0x625b5878 (stale) CRC-32 checksum value for chunk [0, 0x3c174): Mirror 1: 0x331ea9e0 Mirror 4: 0x331ea9e0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c174: skipped 000571 1771399271.293287 mapread 0x0027ea thru 0x006b57 (0x436e bytes) 000572 1771399271.349597 fallocate 0x03cb05 thru 0x03ffff (0x34fb bytes) 000573 1771399271.419600 mapwrite 0x030706 thru 0x03ffff (0xf8fa bytes) 000574 1771399271.516224 punch from 0x00608f to 0x00b138 (0x50aa bytes) 000575 1771399271.522865 trunc from 0x03b37a to 0x03ffff (0x4c86 bytes) 000576 1771399271.530669 mapread 0x02ba63 thru 0x02cd09 (0x12a7 bytes) 000578 1771399271.532899 trunc from 0x040000 to 0x011cec (0x2e313 bytes) 000579 1771399271.608800 read 0x0023fb thru 0x008646 (0x624c bytes) 000580 1771399271.622075 write 0x027a01 thru 0x029e3e (0x243e bytes) 000581 1771399271.624586 punch from 0x00468e to 0x007376 (0x2ce9 bytes) 000582 1771399271.632189 mapwrite 0x012b29 thru 0x01b2a8 (0x8780 bytes) 000583 1771399271.680636 punch from 0x006d40 to 0x01359d (0xc85e bytes) 000584 1771399271.690588 write 0x02f104 thru 0x03eab2 (0xf9af bytes) 000585 1771399271.694883 mapwrite 0x035457 thru 0x03b392 (0x5f3c bytes) 000586 1771399271.720471 write 0x01aa70 thru 0x0228fa (0x7e8b bytes) 000587 1771399271.722936 trunc from 0x03eab3 to 0x02c365 (0x1274d bytes) 000588 1771399271.753194 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2c366): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c366): Mirror 2: 0xad21e751 (stale) Mirror 3: 0xad21e751 (stale) Mirror 4: 0xea634734 (stale) CRC-32 checksum value for chunk [0, 0x2c366): Mirror 1: 0x75055379 Mirror 5: 0x75055379 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c366: skipped 000589 1771399272.952804 punch from 0x011a89 to 0x01f7c9 (0xdd41 bytes) 000590 1771399273.013638 trunc from 0x02c366 to 0x01cf77 (0xf3ee bytes) 000591 1771399273.037092 fallocate 0x01ea33 thru 0x024a1f (0x5fed bytes) 000592 1771399273.043491 punch from 0x0089e2 to 0x00bdf4 (0x3413 bytes) 000593 1771399273.050173 write 0x039770 thru 0x03ffff (0x6890 bytes) 000594 1771399273.052553 mapwrite 0x016a7e thru 0x01db3f (0x70c2 bytes) 000595 1771399273.100662 mapwrite 0x0332df thru 0x03f9dc (0xc6fe bytes) 000596 1771399273.144672 read 0x01e45a thru 0x02a8d4 (0xc47b bytes) 000598 1771399273.156256 mapread 0x000f59 thru 0x00aa94 (0x9b3c bytes) 000599 1771399273.174731 write 0x0274f0 thru 0x03390f (0xc420 bytes) 000600 1771399273.177586 mapwrite 0x03d153 thru 0x03ffff (0x2ead bytes) 000601 1771399273.197153 trunc from 0x040000 to 0x02c900 (0x136ff bytes) 000602 1771399273.222600 fallocate 0x019ee8 thru 0x024a09 (0xab22 bytes) 000603 1771399273.232875 mapwrite 0x02fcb6 thru 0x03c827 (0xcb72 bytes) 000604 1771399273.303107 mapread 0x0236c8 thru 0x030f67 (0xd8a0 bytes) 000605 1771399273.304465 mapread 0x019fa7 thru 0x027b0d (0xdb67 bytes) 000606 1771399273.305720 write 0x008d92 thru 0x00cddb (0x404a bytes) 000607 1771399273.309063 fallocate 0x013730 thru 0x01d522 (0x9df3 bytes) 000608 1771399273.315617 fallocate 0x011ca9 thru 0x0135ba (0x1912 bytes) 000609 1771399273.326527 trunc from 0x03c828 to 0x004db5 (0x37a72 bytes) 000610 1771399273.347918 punch from 0x002649 to 0x004db5 (0x276d bytes) 000611 1771399273.354723 fallocate 0x025267 thru 0x02f524 (0xa2be bytes) 000612 1771399273.360906 punch from 0x02287c to 0x0275a9 (0x4d2e bytes) 000613 1771399273.365801 trunc from 0x02f525 to 0x0235f6 (0xbf2e bytes) 000614 1771399273.390619 read 0x00661d thru 0x00cd22 (0x6706 bytes) 000615 1771399273.404547 trunc from 0x0235f7 to 0x001d27 (0x218cf bytes) 000616 1771399273.424608 fallocate 0x019aa7 thru 0x02385d (0x9db7 bytes) 000617 1771399273.431331 punch from 0x0003da to 0x001d27 (0x194e bytes) 000619 1771399273.439389 trunc from 0x001d28 to 0x01c996 (0x1ac6f bytes) 000620 1771399273.462920 fallocate 0x0110f7 thru 0x0184f2 (0x73fc bytes) 000621 1771399273.475175 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1c997): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c997): Mirror 2: 0x41629be2 (stale) Mirror 3: 0x41629be2 (stale) Mirror 4: 0xddcbc4a3 (stale) Mirror 5: 0x629775d6 (stale) CRC-32 checksum value for chunk [0, 0x1c997): Mirror 1: 0x41629be2 Mirror 6: 0x41629be2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c997: skipped 000622 1771399274.693499 trunc from 0x00fdea to 0x010adc (0xcf3 bytes) 000623 1771399274.755413 write 0x011d1d thru 0x017526 (0x580a bytes) 000624 1771399274.797165 mapread 0x015224 thru 0x01bd1b (0x6af8 bytes) 000625 1771399274.830027 fallocate 0x00bdcb thru 0x00f435 (0x366b bytes) 000626 1771399274.846070 MIRROR_RESYNC 6 mirrors 000627 1771399276.326313 punch from 0x00ddfa to 0x01c996 (0xeb9d bytes) 000628 1771399276.367996 mapwrite 0x004b24 thru 0x013ea8 (0xf385 bytes) 000629 1771399276.476391 fallocate 0x024961 thru 0x029f6f (0x560f bytes) 000631 1771399276.484706 trunc from 0x01c997 to 0x018646 (0x4350 bytes) 000632 1771399276.527860 mapread 0x00b824 thru 0x017074 (0xb851 bytes) 000633 1771399276.530804 trunc from 0x010c25 to 0x0162ce (0x56aa bytes) 000634 1771399276.548290 trunc from 0x005942 to 0x00e09d (0x875c bytes) 000635 1771399276.556275 mapread 0x0064e7 thru 0x014157 (0xdc71 bytes) 000636 1771399276.582721 trunc from 0x018647 to 0x033478 (0x1ae32 bytes) 000637 1771399276.614786 write 0x03fea7 thru 0x03ffff (0x159 bytes) 000638 1771399276.617589 fallocate 0x004544 thru 0x00b04d (0x6b0a bytes) 000639 1771399276.628548 MIRROR_RESYNC 6 mirrors 000640 1771399278.431871 punch from 0x0374ed to 0x03ffff (0x8b13 bytes) 000641 1771399278.470122 trunc from 0x0087ee to 0x015731 (0xcf44 bytes) 000642 1771399278.475303 fallocate 0x03eb06 thru 0x03ffff (0x14fa bytes) 000643 1771399278.480232 mapwrite 0x0124be thru 0x01b610 (0x9153 bytes) 000644 1771399278.518340 trunc from 0x012db1 to 0x01e103 (0xb353 bytes) 000645 1771399278.524646 mapwrite 0x03ac7b thru 0x03ffff (0x5385 bytes) 000646 1771399278.559824 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x269f3a96 (stale) Mirror 4: 0x269f3a96 (stale) Mirror 5: 0x269f3a96 (stale) Mirror 6: 0x269f3a96 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x269f3a96 (stale) Mirror 4: 0x269f3a96 (stale) Mirror 5: 0x269f3a96 (stale) Mirror 6: 0x269f3a96 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000647 1771399278.948844 trunc from 0x00e40f to 0x017fb0 (0x9ba2 bytes) 000648 1771399278.959327 MIRROR_RESYNC 5 mirrors 000649 1771399280.072542 trunc from 0x031a93 to 0x03c083 (0xa5f1 bytes) 000652 1771399280.115333 trunc from 0x0038a4 to 0x00f215 (0xb972 bytes) 000653 1771399280.122358 punch from 0x01c84a to 0x021838 (0x4fef bytes) 000656 1771399280.132784 trunc from 0x039267 to 0x03c9d7 (0x3771 bytes) 000657 1771399280.139212 punch from 0x0040fe to 0x007052 (0x2f55 bytes) 000658 1771399280.144599 trunc from 0x040000 to 0x01b1eb (0x24e14 bytes) 000659 1771399280.165283 fallocate 0x035a4f thru 0x03c1c8 (0x677a bytes) 000660 1771399280.174531 mapwrite 0x00d170 thru 0x011729 (0x45ba bytes) 000661 1771399280.214171 fallocate 0x0077cd thru 0x0167cf (0xf003 bytes) 000662 1771399280.219730 read 0x0367bc thru 0x03955e (0x2da3 bytes) 000663 1771399280.233697 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c1c9): Mirror 4: 0x71528d30 (stale) Mirror 5: 0x71528d30 (stale) Mirror 6: 0x71528d30 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c1c9): Mirror 4: 0x71528d30 (stale) Mirror 5: 0x71528d30 (stale) Mirror 6: 0x71528d30 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c1c9: skipped 000664 1771399280.701314 mapread 0x034b0e thru 0x03c1c8 (0x76bb bytes) 000666 1771399280.724569 mapwrite 0x00f44c thru 0x0107e7 (0x139c bytes) 000667 1771399280.763548 read 0x02efed thru 0x03a6d6 (0xb6ea bytes) 000668 1771399280.769683 mapwrite 0x038bad thru 0x03ffff (0x7453 bytes) 000669 1771399280.821370 write 0x027c2c thru 0x02c089 (0x445e bytes) 000670 1771399280.842574 mapread 0x00e06c thru 0x00f505 (0x149a bytes) 000671 1771399280.850341 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x119149df (stale) Mirror 5: 0x119149df (stale) Mirror 6: 0x119149df (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa192aa05 Mirror 7: 0xa192aa05 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000672 1771399281.951047 MIRROR_RESYNC 5 mirrors 000674 1771399283.364582 write 0x026a52 thru 0x02ef7e (0x852d bytes) 000675 1771399283.415464 trunc from 0x030348 to 0x03a0bd (0x9d76 bytes) 000676 1771399283.421420 read 0x017c24 thru 0x027bab (0xff88 bytes) 000678 1771399283.435928 write 0x03c532 thru 0x03ffff (0x3ace bytes) 000679 1771399283.458118 punch from 0x0227a8 to 0x02e95f (0xc1b8 bytes) 000680 1771399283.472277 mapread 0x0309b6 thru 0x03ff17 (0xf562 bytes) 000681 1771399283.505930 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xa192aa05 (stale) Mirror 6: 0xa192aa05 (stale) Mirror 7: 0xa192aa05 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xa192aa05 (stale) Mirror 6: 0xa192aa05 (stale) Mirror 7: 0xa192aa05 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000682 1771399283.896069 punch from 0x001454 to 0x008cd5 (0x7882 bytes) 000683 1771399283.901120 punch from 0x00cb67 to 0x0191af (0xc649 bytes) 000684 1771399283.905866 punch from 0x001738 to 0x0040d5 (0x299e bytes) 000685 1771399283.911167 fallocate 0x039e24 thru 0x03ebc7 (0x4da4 bytes) 000687 1771399283.919216 read 0x008711 thru 0x017e20 (0xf710 bytes) 000688 1771399283.928711 mapwrite 0x0341be thru 0x036b1a (0x295d bytes) 000689 1771399283.957710 fallocate 0x0304e9 thru 0x0309b1 (0x4c9 bytes) 000690 1771399283.971254 MIRROR_RESYNC 4 mirrors 000691 1771399285.020241 trunc from 0x040000 to 0x00e310 (0x31cef bytes) 000692 1771399285.072341 read 0x00459d thru 0x009bd9 (0x563d bytes) 000693 1771399285.081346 punch from 0x0043b5 to 0x00bc32 (0x787e bytes) 000694 1771399285.085588 write 0x0388cf thru 0x03ffff (0x7731 bytes) 000695 1771399285.089603 fallocate 0x00e08e thru 0x00f235 (0x11a8 bytes) 000696 1771399285.094213 mapwrite 0x009434 thru 0x0147fb (0xb3c8 bytes) 000697 1771399285.131529 fallocate 0x03ced0 thru 0x03ffff (0x3130 bytes) 000698 1771399285.135805 fallocate 0x03a901 thru 0x03ecf9 (0x43f9 bytes) 000699 1771399285.140253 write 0x0337f3 thru 0x039f7a (0x6788 bytes) 000700 1771399285.152267 read 0x004e70 thru 0x00b1c3 (0x6354 bytes) 000701 1771399285.153026 write 0x03603d thru 0x03ffff (0x9fc3 bytes) 000703 1771399285.155733 trunc from 0x035827 to 0x03ffff (0xa7d9 bytes) 000704 1771399285.167313 mapread 0x002140 thru 0x00dfbb (0xbe7c bytes) 000705 1771399285.191987 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x75d6fa13 (stale) Mirror 6: 0x75d6fa13 (stale) Mirror 7: 0x75d6fa13 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xff043359 Mirror 8: 0xff043359 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000706 1771399285.952712 MIRROR_RESYNC 5 mirrors 000707 1771399286.901200 fallocate 0x010ccb thru 0x013b21 (0x2e57 bytes) 000708 1771399286.931775 trunc from 0x019e23 to 0x02290e (0x8aec bytes) 000709 1771399286.939248 mapwrite 0x02dcfd thru 0x0364bb (0x87bf bytes) 000710 1771399286.978188 mapread 0x01f91e thru 0x02a401 (0xaae4 bytes) 000711 1771399286.985509 trunc from 0x040000 to 0x0293ea (0x16c15 bytes) 000712 1771399287.022513 trunc from 0x0293eb to 0x00d3be (0x1c02c bytes) 000713 1771399287.035522 write 0x01d02d thru 0x01ec34 (0x1c08 bytes) 000714 1771399287.040585 trunc from 0x004bbf to 0x009ed4 (0x5316 bytes) 000715 1771399287.043855 punch from 0x01c085 to 0x01ec34 (0x2bb0 bytes) 000716 1771399287.060729 write 0x021867 thru 0x0295fc (0x7d96 bytes) 000717 1771399287.065831 MIRROR_RESYNC 5 mirrors 000718 1771399288.703460 fallocate 0x00d027 thru 0x01c385 (0xf35f bytes) 000719 1771399288.749853 mapwrite 0x039df9 thru 0x03ffff (0x6207 bytes) 000720 1771399288.817853 trunc from 0x01ac38 to 0x022d4d (0x8116 bytes) 000721 1771399288.825536 mapread 0x020b5a thru 0x02149d (0x944 bytes) 000722 1771399288.846068 read 0x016f0c thru 0x01fd0a (0x8dff bytes) 000723 1771399288.853568 trunc from 0x00d345 to 0x016352 (0x900e bytes) 000724 1771399288.860138 read 0x0197eb thru 0x020db4 (0x75ca bytes) 000725 1771399288.862072 write 0x008ffc thru 0x01170a (0x870f bytes) 000726 1771399288.874815 mapread 0x01a51c thru 0x028254 (0xdd39 bytes) 000727 1771399288.877196 punch from 0x031baa to 0x03e97f (0xcdd6 bytes) 000728 1771399288.884367 mapwrite 0x01b974 thru 0x029c05 (0xe292 bytes) 000729 1771399288.911679 mapwrite 0x00155d thru 0x010a5f (0xf503 bytes) 000730 1771399288.993820 trunc from 0x008723 to 0x0093e5 (0xcc3 bytes) 000731 1771399289.010791 mapwrite 0x02594f thru 0x034c6b (0xf31d bytes) 000732 1771399289.098107 punch from 0x02b18a to 0x03aa75 (0xf8ec bytes) 000733 1771399289.111271 mapwrite 0x036ab3 thru 0x03e15c (0x76aa bytes) 000734 1771399289.281151 mapread 0x00d155 thru 0x015abb (0x8967 bytes) 000735 1771399289.284049 trunc from 0x040000 to 0x018828 (0x277d7 bytes) 000736 1771399289.318283 read 0x00663a thru 0x0116d5 (0xb09c bytes) 000737 1771399289.385424 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x18829): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18829): Mirror 4: 0xc9866dc0 (stale) Mirror 6: 0xc9866dc0 (stale) Mirror 7: 0xc9866dc0 (stale) Mirror 8: 0xc9866dc0 (stale) CRC-32 checksum value for chunk [0, 0x18829): Mirror 1: 0xf83aec63 Mirror 9: 0xf83aec63 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18829: skipped 000738 1771399290.558233 MIRROR_RESYNC 6 mirrors 000742 1771399291.396881 write 0x022416 thru 0x02ac56 (0x8841 bytes) 000743 1771399291.448099 write 0x02018d thru 0x021cd5 (0x1b49 bytes) 000744 1771399291.462166 fallocate 0x015636 thru 0x01fd80 (0xa74b bytes) 000745 1771399291.467755 trunc from 0x0074c5 to 0x007aee (0x62a bytes) 000746 1771399291.474149 mapwrite 0x0376f7 thru 0x03f858 (0x8162 bytes) 000747 1771399291.537006 punch from 0x03740b to 0x03f53b (0x8131 bytes) 000748 1771399291.545363 write 0x010ddd thru 0x0140ee (0x3312 bytes) 000749 1771399291.558660 read 0x01eeb5 thru 0x01fbf1 (0xd3d bytes) 000750 1771399291.566285 mapread 0x015d4a thru 0x025d16 (0xffcd bytes) 000751 1771399291.599888 read 0x009875 thru 0x01076c (0x6ef8 bytes) 000752 1771399291.600646 fallocate 0x00b828 thru 0x00c7e2 (0xfbb bytes) 000753 1771399291.617983 read 0x03816f thru 0x03aa2a (0x28bc bytes) 000754 1771399291.638770 punch from 0x005fbf to 0x015a33 (0xfa75 bytes) 000755 1771399291.679343 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f859): Mirror 6: 0x1300c795 (stale) Mirror 7: 0x1300c795 (stale) Mirror 8: 0x1300c795 (stale) Mirror 9: 0x1300c795 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f859): Mirror 6: 0x1300c795 (stale) Mirror 7: 0x1300c795 (stale) Mirror 8: 0x1300c795 (stale) Mirror 9: 0x1300c795 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f859: skipped 000756 1771399292.332567 fallocate 0x0043b6 thru 0x00d119 (0x8d64 bytes) 000757 1771399292.350180 trunc from 0x03f859 to 0x03a22b (0x562d bytes) 000758 1771399292.405567 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a22c): Mirror 7: 0xa70e18c1 (stale) Mirror 8: 0xa70e18c1 (stale) Mirror 9: 0xa70e18c1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a22c): Mirror 7: 0xa70e18c1 (stale) Mirror 8: 0xa70e18c1 (stale) Mirror 9: 0xa70e18c1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a22c: skipped 000759 1771399292.946115 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3a22c): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a22c): Mirror 7: 0xa70e18c1 (stale) Mirror 8: 0xa70e18c1 (stale) Mirror 9: 0xa70e18c1 (stale) CRC-32 checksum value for chunk [0, 0x3a22c): Mirror 1: 0xf1bd2925 Mirror 10: 0xf1bd2925 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a22c: skipped 000760 1771399293.945584 trunc from 0x03a22c to 0x00ae51 (0x2f3da bytes) 000761 1771399294.009155 write 0x02cc34 thru 0x037c37 (0xb004 bytes) 000762 1771399294.011152 trunc from 0x037c38 to 0x025264 (0x129d3 bytes) 000763 1771399294.037590 mapwrite 0x02020c thru 0x02f568 (0xf35d bytes) 000764 1771399294.103332 mapwrite 0x0370b8 thru 0x03ffff (0x8f48 bytes) 000765 1771399294.177656 mapread 0x0213a9 thru 0x02b212 (0x9e6a bytes) 000766 1771399294.179804 trunc from 0x027ff2 to 0x036f68 (0xef77 bytes) 000767 1771399294.189314 mapwrite 0x001d96 thru 0x00cf07 (0xb172 bytes) 000768 1771399294.241417 write 0x020b1c thru 0x02c156 (0xb63b bytes) 000769 1771399294.257366 mapread 0x01b621 thru 0x02af5c (0xf93c bytes) 000770 1771399294.258837 write 0x0186fc thru 0x01a18f (0x1a94 bytes) 000771 1771399294.284752 write 0x02a860 thru 0x031472 (0x6c13 bytes) 000774 1771399294.302687 write 0x01d0fc thru 0x028580 (0xb485 bytes) 000775 1771399294.307505 read 0x003c58 thru 0x012805 (0xebae bytes) 000776 1771399294.315913 trunc from 0x02e27c to 0x034e95 (0x6c1a bytes) 000777 1771399294.344590 fallocate 0x02d420 thru 0x039dd9 (0xc9ba bytes) 000778 1771399294.351905 trunc from 0x040000 to 0x0330f0 (0xcf0f bytes) 000779 1771399294.398260 MIRROR_RESYNC 5 mirrors 000780 1771399296.446175 MIRROR_EXTEND to 6 mirrors 000781 1771399297.603627 mapwrite 0x0046be thru 0x00cf87 (0x88ca bytes) 000782 1771399297.716897 mapwrite 0x0123ca thru 0x018c3a (0x6871 bytes) 000783 1771399297.743192 write 0x00c62e thru 0x013239 (0x6c0c bytes) 000784 1771399297.751414 write 0x035138 thru 0x0382f1 (0x31ba bytes) 000785 1771399297.753690 punch from 0x035382 to 0x0360b6 (0xd35 bytes) 000786 1771399297.773159 mapwrite 0x0085a7 thru 0x010ac7 (0x8521 bytes) 000787 1771399297.797307 mapwrite 0x014590 thru 0x015184 (0xbf5 bytes) 000788 1771399297.815752 write 0x035d77 thru 0x03ffff (0xa289 bytes) 000789 1771399297.830804 punch from 0x021307 to 0x029906 (0x8600 bytes) 000790 1771399297.837678 trunc from 0x040000 to 0x002ff5 (0x3d00a bytes) 000791 1771399297.860952 trunc from 0x002ff6 to 0x0002e7 (0x2d0e bytes) 000792 1771399297.877629 read 0x00026e thru 0x0002e7 (0x7a bytes) 000793 1771399297.884547 fallocate 0x001a39 thru 0x011595 (0xfb5d bytes) 000794 1771399297.891373 mapwrite 0x02bb5b thru 0x031354 (0x57fa bytes) 000795 1771399298.050378 mapwrite 0x00c7c1 thru 0x0193b8 (0xcbf8 bytes) 000796 1771399298.163902 read 0x0107ed thru 0x012844 (0x2058 bytes) 000797 1771399298.165828 trunc from 0x031355 to 0x013d81 (0x1d5d3 bytes) 000798 1771399298.205392 write 0x023863 thru 0x02e6bd (0xae5b bytes) 000799 1771399298.207127 punch from 0x01726b to 0x01a3f2 (0x3188 bytes) 000800 1771399298.223154 trunc from 0x027e30 to 0x02e6bd (0x688e bytes) 000801 1771399298.261106 punch from 0x027a69 to 0x027ff9 (0x591 bytes) 000802 1771399298.275768 mapwrite 0x0110e7 thru 0x01c5e2 (0xb4fc bytes) 000803 1771399298.324048 fallocate 0x01ea5e thru 0x02ddcb (0xf36e bytes) 000804 1771399298.340732 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2e6be): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e6be): Mirror 7: 0xbba46cb5 (stale) Mirror 8: 0xbba46cb5 (stale) Mirror 9: 0xbba46cb5 (stale) Mirror 10: 0xbba46cb5 (stale) Mirror 11: 0xbba46cb5 (stale) CRC-32 checksum value for chunk [0, 0x2e6be): Mirror 1: 0x603ced89 Mirror 12: 0x603ced89 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e6be: skipped 000805 1771399299.468659 trunc from 0x001402 to 0x002573 (0x1172 bytes) 000806 1771399299.510345 trunc from 0x02e6be to 0x00a475 (0x24248 bytes) 000807 1771399299.586528 trunc from 0x00a476 to 0x01d4f9 (0x13084 bytes) 000808 1771399299.607340 read 0x014027 thru 0x01af1d (0x6ef7 bytes) 000809 1771399299.628535 trunc from 0x01d4fa to 0x00950a (0x13fef bytes) 000810 1771399299.660017 trunc from 0x0002e0 to 0x004932 (0x4653 bytes) 000812 1771399299.679398 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x950b): Mirror 7: 0x9a24c57a (stale) Mirror 8: 0x9a24c57a (stale) Mirror 10: 0x9a24c57a (stale) Mirror 11: 0x9a24c57a (stale) Mirror 12: 0x60e628c3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x950b): Mirror 7: 0x9a24c57a (stale) Mirror 8: 0x9a24c57a (stale) Mirror 10: 0x9a24c57a (stale) Mirror 11: 0x9a24c57a (stale) Mirror 12: 0x60e628c3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x950b: skipped 000813 1771399300.038106 trunc from 0x006809 to 0x00950a (0x2d02 bytes) 000814 1771399300.043363 write 0x004548 thru 0x008c6a (0x4723 bytes) 000815 1771399300.059427 fallocate 0x03de5a thru 0x03ffff (0x21a6 bytes) 000816 1771399300.065320 trunc from 0x0032d4 to 0x00950a (0x6237 bytes) 000817 1771399300.079302 trunc from 0x00950b to 0x03efe8 (0x35ade bytes) 000818 1771399300.092405 fallocate 0x0340ba thru 0x03ffff (0xbf46 bytes) 000820 1771399300.098992 punch from 0x02eb84 to 0x033b74 (0x4ff1 bytes) 000821 1771399300.102615 trunc from 0x03cb60 to 0x03efe8 (0x2489 bytes) 000822 1771399300.107357 trunc from 0x02e7fd to 0x037e1d (0x9621 bytes) 000823 1771399300.111198 fallocate 0x014dc4 thru 0x020280 (0xb4bd bytes) 000824 1771399300.115606 mapwrite 0x022554 thru 0x02ae32 (0x88df bytes) 000825 1771399300.144869 trunc from 0x03efe9 to 0x036a28 (0x85c0 bytes) 000826 1771399300.159295 fallocate 0x03e53c thru 0x03ffff (0x1ac4 bytes) 000827 1771399300.164586 write 0x0158b1 thru 0x02369d (0xdded bytes) 000828 1771399300.174213 write 0x0157cf thru 0x023043 (0xd875 bytes) 000829 1771399300.175752 trunc from 0x036a29 to 0x013fc8 (0x22a60 bytes) 000830 1771399300.193789 trunc from 0x008402 to 0x013fc8 (0xbbc7 bytes) 000831 1771399300.199130 trunc from 0x013fc9 to 0x034cea (0x20d22 bytes) 000832 1771399300.215620 write 0x02baa1 thru 0x0389ff (0xcf5f bytes) 000833 1771399300.225481 mapread 0x0202fe thru 0x02eb15 (0xe818 bytes) 000834 1771399300.238073 mapwrite 0x037742 thru 0x03ffff (0x88be bytes) 000835 1771399300.284360 read 0x039c85 thru 0x03ced0 (0x324c bytes) 000837 1771399300.292412 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe622cadb (stale) Mirror 10: 0xe622cadb (stale) Mirror 11: 0xe622cadb (stale) Mirror 12: 0xcebad526 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe622cadb (stale) Mirror 10: 0xe622cadb (stale) Mirror 11: 0xe622cadb (stale) Mirror 12: 0xcebad526 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000838 1771399300.871688 trunc from 0x00345b to 0x007455 (0x3ffb bytes) 000839 1771399300.878085 mapread 0x0139f1 thru 0x019009 (0x5619 bytes) 000841 1771399300.900721 trunc from 0x031e97 to 0x03ec04 (0xcd6e bytes) 000842 1771399300.907884 read 0x01a0aa thru 0x01e488 (0x43df bytes) 000843 1771399300.913477 write 0x02234b thru 0x029617 (0x72cd bytes) 000844 1771399300.930617 mapread 0x006201 thru 0x009111 (0x2f11 bytes) 000845 1771399300.944293 trunc from 0x032899 to 0x03ecd0 (0xc438 bytes) 000846 1771399300.951038 punch from 0x01dc56 to 0x02d680 (0xfa2b bytes) 000847 1771399300.971600 mapread 0x025f3a thru 0x02de6e (0x7f35 bytes) 000848 1771399300.988028 trunc from 0x0300dd to 0x03038e (0x2b2 bytes) 000849 1771399300.995333 fallocate 0x017528 thru 0x01c744 (0x521d bytes) 000850 1771399301.003391 punch from 0x037b86 to 0x03ffff (0x847a bytes) 000851 1771399301.010315 mapread 0x01e3ce thru 0x02709a (0x8ccd bytes) 000852 1771399301.014028 read 0x023999 thru 0x02b032 (0x769a bytes) 000853 1771399301.021070 MIRROR_RESYNC 5 mirrors 000854 1771399302.469338 read 0x008c06 thru 0x00b52b (0x2926 bytes) 000855 1771399302.479010 fallocate 0x000c34 thru 0x009af3 (0x8ec0 bytes) 000856 1771399302.547648 read 0x0170a9 thru 0x0172be (0x216 bytes) 000857 1771399302.578630 mapread 0x011571 thru 0x0135c4 (0x2054 bytes) 000858 1771399302.583382 mapwrite 0x007d40 thru 0x015878 (0xdb39 bytes) 000860 1771399302.651581 mapread 0x003512 thru 0x011e15 (0xe904 bytes) 000861 1771399302.653719 read 0x03ea9e thru 0x03ffff (0x1562 bytes) 000862 1771399302.664180 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xd8c56083 (stale) Mirror 10: 0xd8c56083 (stale) Mirror 11: 0xd8c56083 (stale) Mirror 12: 0xd8c56083 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf18211ca Mirror 13: 0xf18211ca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000863 1771399304.023117 mapwrite 0x03e48e thru 0x03ffff (0x1b72 bytes) 000864 1771399304.171764 trunc from 0x0015ea to 0x00ef77 (0xd98e bytes) 000865 1771399304.191797 read 0x025f92 thru 0x02d664 (0x76d3 bytes) 000866 1771399304.214896 read 0x026fc7 thru 0x033a88 (0xcac2 bytes) 000867 1771399304.230711 mapwrite 0x00c86a thru 0x0165f5 (0x9d8c bytes) 000868 1771399304.260993 trunc from 0x040000 to 0x02d158 (0x12ea7 bytes) 000869 1771399304.332789 trunc from 0x027ead to 0x02c075 (0x41c9 bytes) 000870 1771399304.352076 trunc from 0x016b78 to 0x01c5fd (0x5a86 bytes) 000871 1771399304.372337 mapread 0x0203ec thru 0x02d158 (0xcd6d bytes) 000872 1771399304.438365 fallocate 0x01fc40 thru 0x02093e (0xcff bytes) 000873 1771399304.487988 fallocate 0x014dc8 thru 0x014ede (0x117 bytes) 000874 1771399304.499368 mapwrite 0x0375b4 thru 0x03c79c (0x51e9 bytes) 000875 1771399304.633119 punch from 0x02e728 to 0x030f44 (0x281d bytes) 000876 1771399304.643366 fallocate 0x0323dc thru 0x0387c0 (0x63e5 bytes) 000877 1771399304.661695 trunc from 0x022462 to 0x028e16 (0x69b5 bytes) 000878 1771399304.671548 trunc from 0x036bff to 0x03c79c (0x5b9e bytes) 000880 1771399304.693957 mapwrite 0x01004e thru 0x0115e2 (0x1595 bytes) 000881 1771399304.714883 write 0x02db10 thru 0x0379a5 (0x9e96 bytes) 000882 1771399304.722298 fallocate 0x000a18 thru 0x00b505 (0xaaee bytes) 000883 1771399304.730908 write 0x013c0f thru 0x01f01b (0xb40d bytes) 000884 1771399304.759357 read 0x0298f5 thru 0x02acbf (0x13cb bytes) 000885 1771399304.768235 write 0x03e68b thru 0x03ffff (0x1975 bytes) 000886 1771399304.787184 mapread 0x007bf5 thru 0x01311f (0xb52b bytes) 000887 1771399304.800048 read 0x01a5ca thru 0x022f94 (0x89cb bytes) 000888 1771399304.853281 trunc from 0x040000 to 0x01a55f (0x25aa0 bytes) 000889 1771399304.902327 trunc from 0x01a560 to 0x0208b6 (0x6357 bytes) 000890 1771399304.953009 trunc from 0x0208b7 to 0x039734 (0x18e7e bytes) 000891 1771399305.026097 trunc from 0x01314b to 0x016092 (0x2f48 bytes) 000892 1771399305.066675 trunc from 0x012afd to 0x01b4da (0x89de bytes) 000893 1771399305.091709 mapwrite 0x03549e thru 0x03e499 (0x8ffc bytes) 000895 1771399305.268876 read 0x012d79 thru 0x021393 (0xe61b bytes) 000896 1771399305.327652 fallocate 0x02c2d6 thru 0x036063 (0x9d8e bytes) 000897 1771399305.340313 trunc from 0x0155e2 to 0x021295 (0xbcb4 bytes) 000898 1771399305.367487 punch from 0x038042 to 0x03e499 (0x6458 bytes) 000899 1771399305.380347 read 0x00c070 thru 0x00d8fe (0x188f bytes) 000901 1771399305.381746 punch from 0x00b0aa to 0x013c10 (0x8b67 bytes) 000902 1771399305.427832 MIRROR_RESYNC 6 mirrors 000903 1771399306.945986 read 0x0257d9 thru 0x034a8f (0xf2b7 bytes) 000904 1771399306.957055 mapread 0x025836 thru 0x026fcd (0x1798 bytes) 000905 1771399306.958327 write 0x02c2da thru 0x031549 (0x5270 bytes) 000906 1771399307.008414 punch from 0x03c610 to 0x03e499 (0x1e8a bytes) 000907 1771399307.033114 MIRROR_RESYNC 6 mirrors 000908 1771399309.370877 punch from 0x01c237 to 0x02030d (0x40d7 bytes) 000909 1771399309.401642 write 0x01a90b thru 0x02a0f7 (0xf7ed bytes) 000910 1771399309.411300 read 0x005e3a thru 0x014c49 (0xee10 bytes) 000911 1771399309.425741 read 0x00f011 thru 0x017444 (0x8434 bytes) 000912 1771399309.437049 MIRROR_RESYNC 6 mirrors 000913 1771399310.917351 mapread 0x025b18 thru 0x02f931 (0x9e1a bytes) 000914 1771399310.939069 mapwrite 0x000c91 thru 0x00b438 (0xa7a8 bytes) 000915 1771399311.048130 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3e49a): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e49a): Mirror 8: 0xa3040a12 (stale) Mirror 10: 0xa3040a12 (stale) Mirror 11: 0xa3040a12 (stale) Mirror 12: 0xa3040a12 (stale) Mirror 13: 0xa3040a12 (stale) CRC-32 checksum value for chunk [0, 0x3e49a): Mirror 1: 0x276eb772 Mirror 14: 0x276eb772 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e49a: skipped 000916 1771399312.413994 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e49a): Mirror 10: 0xa3040a12 (stale) Mirror 11: 0xa3040a12 (stale) Mirror 12: 0xa3040a12 (stale) Mirror 13: 0xa3040a12 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e49a): Mirror 10: 0xa3040a12 (stale) Mirror 11: 0xa3040a12 (stale) Mirror 12: 0xa3040a12 (stale) Mirror 13: 0xa3040a12 (stale) CRC-32 checksum value for chunk [0, 0x3e49a): Mirror 1: 0x276eb772 Mirror 14: 0x276eb772 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e49a: skipped 000917 1771399313.088778 fallocate 0x02412f thru 0x027915 (0x37e7 bytes) 000918 1771399313.155314 punch from 0x035bde to 0x03e499 (0x88bc bytes) 000919 1771399313.163305 write 0x0026ab thru 0x008734 (0x608a bytes) 000920 1771399313.182164 trunc from 0x026f43 to 0x02eea8 (0x7f66 bytes) 000921 1771399313.188235 trunc from 0x0079cc to 0x010a3d (0x9072 bytes) 000922 1771399313.203209 trunc from 0x029dc3 to 0x031113 (0x7351 bytes) 000923 1771399313.212360 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e49a): Mirror 11: 0xa3040a12 (stale) Mirror 12: 0xa3040a12 (stale) Mirror 13: 0xa3040a12 (stale) Mirror 14: 0x276eb772 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e49a): Mirror 11: 0xa3040a12 (stale) Mirror 12: 0xa3040a12 (stale) Mirror 13: 0xa3040a12 (stale) Mirror 14: 0x276eb772 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e49a: skipped 000925 1771399313.579960 fallocate 0x025101 thru 0x0266c0 (0x15c0 bytes) 000926 1771399313.585250 trunc from 0x00b734 to 0x01967e (0xdf4b bytes) 000927 1771399313.590309 mapwrite 0x00d5d1 thru 0x00fef7 (0x2927 bytes) 000928 1771399313.622614 write 0x0079e0 thru 0x009136 (0x1757 bytes) 000929 1771399313.633645 mapwrite 0x007982 thru 0x01535d (0xd9dc bytes) 000930 1771399313.654345 trunc from 0x03e49a to 0x00e250 (0x30249 bytes) 000931 1771399313.675630 trunc from 0x00cc96 to 0x00e250 (0x15bb bytes) 000932 1771399313.687049 punch from 0x009729 to 0x00e250 (0x4b28 bytes) 000934 1771399313.693963 mapwrite 0x036029 thru 0x036186 (0x15e bytes) 000935 1771399313.743265 trunc from 0x023b2b to 0x0279d2 (0x3ea8 bytes) 000936 1771399313.749583 fallocate 0x03e349 thru 0x03f39b (0x1053 bytes) 000937 1771399313.755046 read 0x0241e2 thru 0x0285ef (0x440e bytes) 000938 1771399313.764576 write 0x01dbbb thru 0x02917e (0xb5c4 bytes) 000939 1771399313.805713 trunc from 0x0217e8 to 0x027f6e (0x6787 bytes) 000940 1771399313.832125 read 0x029eee thru 0x036186 (0xc299 bytes) 000941 1771399313.851297 fallocate 0x02fcf5 thru 0x03da74 (0xdd80 bytes) 000942 1771399313.866418 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3da75): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3da75): Mirror 11: 0xc6e80f8b (stale) Mirror 12: 0xc6e80f8b (stale) Mirror 13: 0xc6e80f8b (stale) Mirror 14: 0x3739f4ed (stale) CRC-32 checksum value for chunk [0, 0x3da75): Mirror 1: 0xbc4fdf3c Mirror 15: 0xbc4fdf3c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3da75: skipped 000943 1771399314.782150 mapread 0x0105a8 thru 0x011ce2 (0x173b bytes) 000944 1771399314.806224 read 0x016367 thru 0x022ea8 (0xcb42 bytes) 000945 1771399314.828586 write 0x01add2 thru 0x02abe5 (0xfe14 bytes) 000946 1771399314.889011 fallocate 0x026495 thru 0x028e7c (0x29e8 bytes) 000947 1771399314.893322 fallocate 0x0294b5 thru 0x029dbc (0x908 bytes) 000948 1771399314.901119 MIRROR_RESYNC 6 mirrors 000949 1771399316.254149 trunc from 0x03cc78 to 0x03da74 (0xdfd bytes) 000950 1771399316.294412 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3da75): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3da75): Mirror 11: 0x90375af7 (stale) Mirror 12: 0x90375af7 (stale) Mirror 13: 0x90375af7 (stale) Mirror 14: 0x90375af7 (stale) Mirror 15: 0x90375af7 (stale) CRC-32 checksum value for chunk [0, 0x3da75): Mirror 1: 0x90375af7 Mirror 16: 0x90375af7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3da75: skipped 000951 1771399317.231218 MIRROR_RESYNC 7 mirrors 000952 1771399319.332036 mapwrite 0x03e715 thru 0x03ffff (0x18eb bytes) 000953 1771399319.395179 fallocate 0x02c6f8 thru 0x02ff35 (0x383e bytes) 000954 1771399319.400198 punch from 0x02975a to 0x032708 (0x8faf bytes) 000955 1771399319.407146 punch from 0x02d573 to 0x02f9e9 (0x2477 bytes) 000956 1771399319.413014 read 0x00296c thru 0x00b261 (0x88f6 bytes) 000957 1771399319.420003 write 0x004340 thru 0x007580 (0x3241 bytes) 000958 1771399319.421421 mapwrite 0x02343b thru 0x0330b2 (0xfc78 bytes) 000959 1771399319.464207 trunc from 0x040000 to 0x0163b3 (0x29c4c bytes) 000960 1771399319.497881 trunc from 0x010ef7 to 0x0163b3 (0x54bd bytes) 000961 1771399319.502900 mapwrite 0x02c74d thru 0x02d459 (0xd0d bytes) 000962 1771399319.546198 trunc from 0x02d45a to 0x002867 (0x2abf2 bytes) 000963 1771399319.565501 read 0x002254 thru 0x002867 (0x614 bytes) 000964 1771399319.566808 trunc from 0x002868 to 0x01e796 (0x1bf2f bytes) 000965 1771399319.579091 mapread 0x000d24 thru 0x005942 (0x4c1f bytes) 000966 1771399319.601264 MIRROR_RESYNC 7 mirrors 000967 1771399320.734343 read 0x015fa6 thru 0x017686 (0x16e1 bytes) 000968 1771399320.741142 trunc from 0x004cee to 0x010c12 (0xbf25 bytes) 000969 1771399320.783228 write 0x01cdc0 thru 0x02790c (0xab4d bytes) 000970 1771399320.790303 read 0x015510 thru 0x01ec28 (0x9719 bytes) 000971 1771399320.801237 mapwrite 0x009ca5 thru 0x0174f8 (0xd854 bytes) 000972 1771399320.827851 trunc from 0x02790d to 0x01e417 (0x94f5 bytes) 000973 1771399320.853466 read 0x0188f6 thru 0x01ca21 (0x412c bytes) 000974 1771399320.854557 punch from 0x00fda7 to 0x01284b (0x2aa5 bytes) 000975 1771399320.863573 trunc from 0x01c8ad to 0x01e417 (0x1b6b bytes) 000976 1771399320.873729 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1e418): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e418): Mirror 11: 0xab572eb4 (stale) Mirror 12: 0xab572eb4 (stale) Mirror 13: 0xab572eb4 (stale) Mirror 14: 0xab572eb4 (stale) Mirror 15: 0xab572eb4 (stale) Mirror 16: 0xab572eb4 (stale) CRC-32 checksum value for chunk [0, 0x1e418): Mirror 1: 0x72c2d4d4 Mirror 17: 0x72c2d4d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e418: skipped 000977 1771399322.237894 mapwrite 0x02b40e thru 0x03afc6 (0xfbb9 bytes) 000978 1771399322.494197 punch from 0x0191f3 to 0x025cab (0xcab9 bytes) 000979 1771399322.512875 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3afc7): Mirror 11: 0x1e6e3d09 (stale) Mirror 13: 0x1e6e3d09 (stale) Mirror 14: 0x1e6e3d09 (stale) Mirror 15: 0x1e6e3d09 (stale) Mirror 16: 0x1e6e3d09 (stale) Mirror 17: 0x244cc36b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3afc7): Mirror 11: 0x1e6e3d09 (stale) Mirror 13: 0x1e6e3d09 (stale) Mirror 14: 0x1e6e3d09 (stale) Mirror 15: 0x1e6e3d09 (stale) Mirror 16: 0x1e6e3d09 (stale) Mirror 17: 0x244cc36b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3afc7: skipped 000980 1771399323.199561 punch from 0x039bd9 to 0x03afc6 (0x13ee bytes) 000981 1771399323.208125 mapread 0x03a257 thru 0x03afc6 (0xd70 bytes) 000982 1771399323.214834 write 0x0222ec thru 0x02a9e8 (0x86fd bytes) 000983 1771399323.239249 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3afc7): Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3afc7): Mirror 11: 0x1e6e3d09 (stale) Mirror 13: 0x1e6e3d09 (stale) Mirror 14: 0x1e6e3d09 (stale) Mirror 15: 0x1e6e3d09 (stale) Mirror 16: 0x1e6e3d09 (stale) Mirror 17: 0x244cc36b (stale) CRC-32 checksum value for chunk [0, 0x3afc7): Mirror 1: 0xf262a1e8 Mirror 18: 0xf262a1e8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3afc7: skipped 000984 1771399324.156984 write 0x021e26 thru 0x02e7dc (0xc9b7 bytes) 000985 1771399324.215269 trunc from 0x021515 to 0x0223cb (0xeb7 bytes) 000986 1771399324.230196 write 0x013c62 thru 0x022cb1 (0xf050 bytes) 000987 1771399324.242951 punch from 0x014774 to 0x01cd16 (0x85a3 bytes) 000989 1771399324.272529 mapwrite 0x03124e thru 0x036df2 (0x5ba5 bytes) 000990 1771399324.335301 trunc from 0x013170 to 0x013e59 (0xcea bytes) 000991 1771399324.351252 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x29:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3afc7): Mirror 11: 0x1e6e3d09 (stale) Mirror 14: 0x1e6e3d09 (stale) Mirror 15: 0x1e6e3d09 (stale) Mirror 16: 0x1e6e3d09 (stale) Mirror 17: 0x244cc36b (stale) Mirror 18: 0xf262a1e8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3afc7): Mirror 11: 0x1e6e3d09 (stale) Mirror 14: 0x1e6e3d09 (stale) Mirror 15: 0x1e6e3d09 (stale) Mirror 16: 0x1e6e3d09 (stale) Mirror 17: 0x244cc36b (stale) Mirror 18: 0xf262a1e8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3afc7: skipped 000992 1771399325.001540 trunc from 0x0120c0 to 0x017a7f (0x59c0 bytes) 000993 1771399325.012810 trunc from 0x0385db to 0x03afc6 (0x29ec bytes) 000994 1771399325.016341 punch from 0x0196f9 to 0x0257ee (0xc0f6 bytes) 000995 1771399325.023339 MIRROR_RESYNC 7 mirrors 000996 1771399327.082373 punch from 0x01b3d1 to 0x01fa76 (0x46a6 bytes) 000997 1771399327.103209 write 0x01a8d2 thru 0x020880 (0x5faf bytes) 000998 1771399327.110775 fallocate 0x0345ac thru 0x03bbc3 (0x7618 bytes) 000999 1771399327.114183 mapwrite 0x000a76 thru 0x0097cf (0x8d5a bytes) 001000 1771399327.137036 punch from 0x005762 to 0x0124a9 (0xcd48 bytes) All operations completed A-OK! PASS 70a (107s) == sanity-flr test 71: check mirror extend parent fid ==== 02:22:14 (1771399334) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1d3:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 02:22:25 (1771399345) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.202559 s, 10.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192563 s, 5.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 02:22:34 (1771399354) Starting client: oleg340-client.virtnet: -o user_xattr,flock 192.168.203.140@tcp:/lustre /mnt/lustre2 Starting client: oleg340-client.virtnet: -o user_xattr,flock 192.168.203.140@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2470496 .. Extending file size to 5670976 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 5833824 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 5761152 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8784000 .. Extending file size to 7688384 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8960448 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8489376 .. failed Extending file size to 9091584 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9242176 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 59837 59839 59840 59841 59842 done 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) 192.168.203.140@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg340-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (81s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 02:23:55 (1771399435) Starting client: oleg340-client.virtnet: -o user_xattr,flock 192.168.203.140@tcp:/lustre /mnt/lustre2 Starting client: oleg340-client.virtnet: -o user_xattr,flock 192.168.203.140@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2399712 .. Extending file size to 8777440 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3935520 .. Extending file size to 8994560 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9029184 .. Extending file size to 9026912 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 61713 61714 61715 61717 61721 failed 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) 192.168.203.140@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg340-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 02:25:16 (1771399516) Starting client: oleg340-client.virtnet: -o user_xattr,flock 192.168.203.140@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2408 1268896 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2068 1269236 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17612 3589408 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25380 3581640 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42992 7171048 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2408 1268896 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2068 1269236 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17612 3589408 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25380 3581640 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42992 7171048 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.172634 s, 12.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.43774 s, 4.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x2801bc85 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) PASS 200c (22s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 02:25:40 (1771399540) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.108302 s, 19.4 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 02:25:48 (1771399548) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x77:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc4:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc4:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc5:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 02:25:56 (1771399556) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.103084 s, 20.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 02:26:05 (1771399565) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0796183 s, 26.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23d:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 02:26:13 (1771399573) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.10757 s, 19.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23f:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 02:26:45 (1771399605) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg340-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765978 s, 13.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x241:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x240:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x241:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x240:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23f:0x0] } PASS 204d (53s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 02:27:38 (1771399658) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg340-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0943392 s, 22.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (52s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 02:28:30 (1771399710) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg340-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg340-server: Pool lustre.test_204f created Adding targets to pool oleg340-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg340-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.137183 s, 15.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg340-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg340-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg340-server: Pool lustre.test_204f destroyed PASS 204f (76s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 02:29:46 (1771399786) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 02:29:54 (1771399794) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 02:30:02 (1771399802) oleg340-server: Pool lustre.test_206 created Adding targets to pool oleg340-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg340-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg340-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg340-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg340-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 02:30:32 (1771399832) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145448 s, 7.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816119 s, 12.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x2bc91702 Mirror 3: 0x2bc91702 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 02:30:44 (1771399844) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 02:30:47 (1771399847) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 02:30:51 (1771399851) Starting client: oleg340-client.virtnet: -o user_xattr,flock 192.168.203.140@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0630194 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2390000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1ca0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1ca0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1ca0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xcf0000 Read this (4 bytes): BBBB 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) PASS 209a (45s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 02:31:36 (1771399896) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0169126 s, 484 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 02:31:44 (1771399904) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573994 s, 18.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x94 lmm_fid: [0x200000405:0x94:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x94 lmm_fid: [0x200000405:0x94:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611284 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-02-18 02:31:46.000000000 -0500 Modify: 2026-02-18 02:31:46.000000000 -0500 Change: 2026-02-18 02:31:48.000000000 -0500 Birth: 2026-02-18 02:31:47.000000000 -0500 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 02:31:53 (1771399913) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270389 s, 38.8 MB/s oleg340-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg340-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg340-server: O/280000401/d18/d2: File not found by ext2_lookup PASS 210b (22s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 02:32:15 (1771399935) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.20031 s, 204 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0169567 s, 242 kB/s size after second write -rw-r--r-- 1 root root 4096 Feb 18 02:32 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x99:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Feb 18 02:32 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 02:32:23 (1771399943) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 02:32:33 (1771399953) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1839 sec =========== 02:32:42 (1771399962) === sanity-flr: start cleanup 02:32:44 (1771399964) === === sanity-flr: finish cleanup 02:32:51 (1771399971) ===