-----============= acceptance-small: sanity-flr ============----- Sat Aug 8 04:28:19 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg250-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 04:28:37 (1786177717) === oleg250-client.virtnet: executing check_config_client /mnt/lustre oleg250-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg250-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9e2007d22800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e2007d22800.idle_timeout=debug disable quota as required oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 04:29:10 (1786177750) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8244] == sanity-flr test 0a: lfs mirror create with -N option == 04:29:13 (1786177753) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 04:29:32 (1786177772) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 04:29:36 (1786177776) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 04:29:40 (1786177780) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 04:30:05 (1786177805) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 04:30:08 (1786177808) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 04:30:11 (1786177811) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9e2007d22800.stats=clear osc.lustre-OST0001-osc-ffff9e2007d22800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 04:30:19 (1786177819) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 04:30:28 (1786177828) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 04:30:37 (1786177837) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 04:30:41 (1786177841) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 04:30:50 (1786177850) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 04:30:58 (1786177858) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 04:31:07 (1786177867) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 04:31:16 (1786177876) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 04:31:23 (1786177883) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 04:31:35 (1786177895) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 04:31:41 (1786177901) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.769281 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503857 s, 20.8 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 04:31:51 (1786177911) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.676331 s, 24.8 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 04:31:59 (1786177919) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 04:32:10 (1786177930) Stopping /mnt/lustre-ost1 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e2007d22800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e2007d22800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg250-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9e2007d22800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e2007d22800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg250-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (79s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 04:33:29 (1786178009) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.239394 s, 13.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e2007d22800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e2007d22800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg250-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 04:34:14 (1786178054) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e2007d22800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e2007d22800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg250-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9e2007d22800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e2007d22800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg250-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (72s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 04:35:26 (1786178126) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.19444 s, 23.9 MB/s -rw-r--r-- 1 root root 52428800 Aug 8 04:35 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.843s user 0m0.000s sys 0m0.148s Stopping /mnt/lustre-ost1 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e2007d22800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e2007d22800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.050s user 0m0.000s sys 0m0.195s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg250-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9e2007d22800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e2007d22800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.105s user 0m0.000s sys 0m0.164s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg250-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg250-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e2007d22800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e2007d22800.ost_server_uuid in FULL state after 0 sec PASS 33b (89s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 04:36:55 (1786178215) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 04:36:59 (1786178219) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 04:37:02 (1786178222) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 04:37:05 (1786178225) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 04:37:12 (1786178232) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.479573 s, 26.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 04:37:23 (1786178243) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046634 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415269 s, 25.3 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 04:37:31 (1786178251) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 04:37:39 (1786178259) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000830278 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0022386 s, 457 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0116261 s, 88.1 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000689606 s, 1.5 MB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 04:37:49 (1786178269) create test files with size 15734748 .. d253303a3b72b21a663980873b925131 - 4a7bfd1df4c43c419f832bcfe67362ec - 8a9449cb6554d7d7d37b5e0b49e4cf00 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg250-client.virtnet /mnt/lustre (opts:) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 04:38:12 (1786178292) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (11s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 04:38:24 (1786178304) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 04:38:31 (1786178311) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472355 s, 22.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185872 s, 56.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (6s) == sanity-flr test 41: lfs mirror resync check =========== 04:38:38 (1786178318) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0851638 s, 24.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.231831 s, 18.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104354 s, 20.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 04:38:53 (1786178333) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 04:38:55 (1786178335) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 04:38:59 (1786178339) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475276 s, 22.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 44a: lfs mirror split check =========== 04:39:05 (1786178345) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.199254 s, 15.8 MB/s PASS 44a (7s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 04:39:13 (1786178353) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 04:39:24 (1786178364) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.335376 s, 31.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg250-client.virtnet /mnt/lustre (opts:) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 04:39:35 (1786178375) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9e201823a800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.318897 s, 32.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg250-client.virtnet /mnt/lustre (opts:) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9e20309dd800.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 04:39:50 (1786178390) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (39s) == sanity-flr test 46: Verify setstripe --copy option ==== 04:40:29 (1786178429) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (43s) == sanity-flr test 47: Verify mirror obj alloc =========== 04:41:12 (1786178472) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 04:41:15 (1786178475) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.280449 s, 11.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.118507 s, 26.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x608bb027 (nosync,stale) Mirror 2: 0x608bb027 (nosync,stale) ** verify mirror 2 doesn't change original checksum: eeae54c0c239482a16a3a4188bb03d80 - mirror 1 checksum: 8ab078c76715522eea842d1d22eccfa5 - mirror 2 checksum: eeae54c0c239482a16a3a4188bb03d80 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-08T04:41:19-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-08T04:41:19-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: eeae54c0c239482a16a3a4188bb03d80 - mirror 1 checksum: 8ab078c76715522eea842d1d22eccfa5 - mirror 2 checksum: 8ab078c76715522eea842d1d22eccfa5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-08T04:41:19-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-08T04:41:19-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 04:41:26 (1786178486) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218891 s, 47.9 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 04:41:34 (1786178494) Failing mds1 on oleg250-server Stopping /mnt/lustre-mds1 (opts:) on oleg250-server 04:41:46 (1786178506) shut down facet: mds1 facet_host: oleg250-server facet_failover_host: oleg250-server Failover mds1 to oleg250-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-MDT0000 04:42:11 (1786178531) targets are mounted 04:42:11 (1786178531) facet_failover done oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049655 s, 21.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } PASS 50A (56s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 04:42:29 (1786178549) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0393341 s, 521 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0112327 s, 1.8 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 04:42:38 (1786178558) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0487426 s, 420 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00928863 s, 2.2 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 04:42:48 (1786178568) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00685621 s, 2.4 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 04:42:55 (1786178575) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0775505 s, 264 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (7s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 04:43:02 (1786178582) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0181458 s, 226 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0244724 s, 167 kB/s PASS 60a (6s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 04:43:09 (1786178589) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 04:43:28 (1786178608) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (26s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 04:43:54 (1786178634) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 04:44:18 (1786178658) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 04:44:41 (1786178681) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 04:44:50 (1786178690) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57565 mirror split pid 57566 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57565 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57566 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 04:45:59 (1786178759) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 8761 000001 1786178761.459871 write 0x02f0d1 thru 0x03947c (0xa3ac bytes) truncating to largest ever: 0x5f6b 000003 1786178761.477578 trunc from 0x03947d to 0x005f6a (0x33512 bytes) 000004 1786178761.495194 punch from 0x005f16 to 0x005f6a (0x55 bytes) 000005 1786178761.503103 read 0x00362d thru 0x005f6a (0x293e bytes) 000006 1786178761.513055 punch from 0x000804 to 0x000f4b (0x748 bytes) 000007 1786178761.522515 write 0x035e7a thru 0x03ffff (0xa186 bytes) 000008 1786178761.524942 read 0x01e19e thru 0x01f617 (0x147a bytes) 000010 1786178761.543832 punch from 0x0108e3 to 0x01d54f (0xcc6d bytes) truncating to largest ever: 0x149b5 000011 1786178761.556451 trunc from 0x040000 to 0x0149b4 (0x2b64b bytes) 000012 1786178761.592740 mapwrite 0x03ef5c thru 0x03ffff (0x10a4 bytes) 000013 1786178761.671349 MIRROR_EXTEND to 2 mirrors 000015 1786178762.338547 punch from 0x03a506 to 0x03ffff (0x5afa bytes) 000016 1786178762.369853 mapwrite 0x00ac22 thru 0x01a0c6 (0xf4a5 bytes) 000017 1786178762.414674 read 0x028b00 thru 0x0372a5 (0xe7a6 bytes) 000018 1786178762.423000 punch from 0x028d48 to 0x0373d0 (0xe689 bytes) truncating to largest ever: 0x22bbf 000019 1786178762.427612 trunc from 0x040000 to 0x022bbe (0x1d441 bytes) 000021 1786178762.451591 write 0x00230e thru 0x0056ec (0x33df bytes) 000022 1786178762.457904 punch from 0x01a6b7 to 0x022bbe (0x8508 bytes) 000025 1786178762.465126 mapwrite 0x03d45c thru 0x03ffff (0x2ba4 bytes) 000026 1786178762.501063 mapwrite 0x00b755 thru 0x014925 (0x91d1 bytes) truncating to largest ever: 0x37f97 000027 1786178762.516574 trunc from 0x040000 to 0x037f96 (0x8069 bytes) fallocating to largest ever: 0x40000 000028 1786178762.531345 fallocate 0x039ca2 thru 0x03ffff (0x635e bytes) 000029 1786178762.536638 mapwrite 0x0123dc thru 0x01dcad (0xb8d2 bytes) 000030 1786178762.567274 read 0x00a3c4 thru 0x018026 (0xdc63 bytes) 000031 1786178762.568406 mapread 0x03c0f5 thru 0x03ffff (0x3f0b bytes) 000032 1786178762.578361 punch from 0x00870c to 0x015b74 (0xd469 bytes) 000033 1786178762.582665 mapread 0x01b6df thru 0x01c7be (0x10e0 bytes) 000034 1786178762.583509 mapwrite 0x036f25 thru 0x03b4ff (0x45db bytes) 000035 1786178762.595868 write 0x003811 thru 0x00d686 (0x9e76 bytes) 000036 1786178762.602579 mapread 0x025e20 thru 0x02b3c2 (0x55a3 bytes) 000038 1786178762.604422 mapread 0x01f586 thru 0x020e3d (0x18b8 bytes) 000039 1786178762.607282 punch from 0x005720 to 0x00abd4 (0x54b5 bytes) 000040 1786178762.624113 trunc from 0x02f957 to 0x02fe11 (0x4bb bytes) 000041 1786178762.628238 read 0x0051b4 thru 0x00b6b3 (0x6500 bytes) 000042 1786178762.644521 trunc from 0x01974c to 0x0206ff (0x6fb4 bytes) 000043 1786178762.653073 mapwrite 0x0205c1 thru 0x024cfa (0x473a bytes) 000044 1786178762.679612 trunc from 0x02c0f8 to 0x03bbfe (0xfb07 bytes) 000045 1786178762.685448 trunc from 0x040000 to 0x0194cc (0x26b33 bytes) 000046 1786178762.698154 trunc from 0x00c9db to 0x0125f6 (0x5c1c bytes) 000047 1786178762.704441 mapwrite 0x02eb71 thru 0x030fc4 (0x2454 bytes) 000048 1786178762.742196 fallocate 0x02510a thru 0x0252b4 (0x1ab bytes) 000049 1786178762.747940 trunc from 0x027020 to 0x02c686 (0x5667 bytes) 000050 1786178762.761217 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) 000051 1786178762.894057 fallocate 0x027ea6 thru 0x03585a (0xd9b5 bytes) 000053 1786178762.899921 trunc from 0x003837 to 0x0052a3 (0x1a6d bytes) 000054 1786178762.912138 mapread 0x01e007 thru 0x02a309 (0xc303 bytes) 000055 1786178762.955090 punch from 0x004855 to 0x00d022 (0x87ce bytes) 000056 1786178762.960881 write 0x02ddd1 thru 0x03d0f0 (0xf320 bytes) 000057 1786178762.970916 read 0x02d812 thru 0x03bfc2 (0xe7b1 bytes) 000060 1786178762.973367 fallocate 0x030554 thru 0x03a999 (0xa446 bytes) 000061 1786178762.978974 read 0x011838 thru 0x016227 (0x49f0 bytes) 000062 1786178762.986788 mapread 0x035117 thru 0x035b11 (0x9fb bytes) 000063 1786178762.988455 punch from 0x02bc5f to 0x03aca3 (0xf045 bytes) 000064 1786178763.009369 mapwrite 0x01f74a thru 0x023f7d (0x4834 bytes) 000065 1786178763.028694 trunc from 0x00ee3b to 0x011d93 (0x2f59 bytes) 000066 1786178763.033677 mapwrite 0x03a7ac thru 0x03ffff (0x5854 bytes) 000067 1786178763.113186 mapread 0x019311 thru 0x023e79 (0xab69 bytes) 000068 1786178763.114333 fallocate 0x004d57 thru 0x00c61b (0x78c5 bytes) 000070 1786178763.126570 read 0x0054df thru 0x00e41f (0x8f41 bytes) 000071 1786178763.162198 trunc from 0x040000 to 0x009a2f (0x365d0 bytes) 000072 1786178763.187302 fallocate 0x029838 thru 0x031d3e (0x8507 bytes) 000073 1786178763.192931 fallocate 0x03e991 thru 0x03ffff (0x166f bytes) 000075 1786178763.207068 trunc from 0x009a30 to 0x020b09 (0x170da bytes) 000076 1786178763.229471 trunc from 0x00c422 to 0x0119e2 (0x55c1 bytes) 000077 1786178763.236737 fallocate 0x0154ce thru 0x02066c (0xb19f bytes) 000078 1786178763.249537 MIRROR_EXTEND to 2 mirrors 000079 1786178763.902172 mapread 0x006a15 thru 0x00ecf2 (0x82de bytes) 000080 1786178763.949964 punch from 0x003a7b to 0x00e3e9 (0xa96f bytes) 000081 1786178763.993622 mapwrite 0x03280d thru 0x03ffff (0xd7f3 bytes) 000082 1786178764.088825 read 0x0079b2 thru 0x011ff2 (0xa641 bytes) 000083 1786178764.100492 mapread 0x00fb71 thru 0x019c4e (0xa0de bytes) 000084 1786178764.110569 write 0x01d8ef thru 0x029c3c (0xc34e bytes) 000085 1786178764.119728 trunc from 0x040000 to 0x02c35c (0x13ca3 bytes) 000087 1786178764.131560 mapread 0x011956 thru 0x017ea7 (0x6552 bytes) 000088 1786178764.132779 mapwrite 0x0398a4 thru 0x03ffff (0x675c bytes) 000089 1786178764.199508 trunc from 0x02d66f to 0x0301c1 (0x2b53 bytes) 000090 1786178764.208250 trunc from 0x024e9c to 0x02dd7b (0x8ee0 bytes) 000091 1786178764.219978 mapwrite 0x02f5c0 thru 0x038c66 (0x96a7 bytes) 000092 1786178764.304892 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x5837af6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xec8a96d Mirror 3: 0xec8a96d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000093 1786178765.233339 read 0x022506 thru 0x02eeb4 (0xc9af bytes) 000094 1786178765.266924 mapwrite 0x02508b thru 0x02a554 (0x54ca bytes) 000095 1786178765.377762 write 0x039cb5 thru 0x03ffff (0x634b bytes) 000096 1786178765.392908 mapread 0x0280cf thru 0x02cce1 (0x4c13 bytes) 000098 1786178765.396584 write 0x02049b thru 0x025c9c (0x5802 bytes) 000099 1786178765.398823 fallocate 0x00deb8 thru 0x0177d5 (0x991e bytes) 000100 1786178765.405334 punch from 0x017347 to 0x0200d2 (0x8d8c bytes) 000101 1786178765.424402 punch from 0x02ccd6 to 0x03bc8b (0xefb6 bytes) 000103 1786178765.430846 write 0x0288c3 thru 0x028d54 (0x492 bytes) 000104 1786178765.433491 trunc from 0x040000 to 0x00a433 (0x35bcc bytes) 000105 1786178765.476077 punch from 0x005f5a to 0x008e93 (0x2f3a bytes) 000106 1786178765.479878 mapwrite 0x03f03f thru 0x03fda3 (0xd65 bytes) 000107 1786178765.515603 fallocate 0x031fe7 thru 0x03d483 (0xb49d bytes) 000108 1786178765.522762 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fda4): Mirror 2: 0x855622ae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fda4): Mirror 2: 0x855622ae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fda4: skipped 000109 1786178765.788957 mapread 0x010d09 thru 0x01ba05 (0xacfd bytes) 000110 1786178765.836232 punch from 0x00ffc4 to 0x01dffc (0xe039 bytes) 000112 1786178765.844203 trunc from 0x01944b to 0x01d1c9 (0x3d7f bytes) 000113 1786178765.849677 trunc from 0x03fda4 to 0x01b6c2 (0x246e1 bytes) 000114 1786178765.872758 mapwrite 0x0017ec thru 0x001de2 (0x5f7 bytes) 000115 1786178765.897285 mapwrite 0x001ee6 thru 0x00c4c1 (0xa5dc bytes) 000116 1786178765.915292 fallocate 0x032180 thru 0x036b98 (0x4a19 bytes) 000117 1786178765.921562 punch from 0x02499a to 0x02fe6f (0xb4d6 bytes) 000118 1786178765.925477 mapwrite 0x012d31 thru 0x01eb6d (0xbe3d bytes) 000119 1786178765.960579 trunc from 0x01c745 to 0x0240ec (0x79a8 bytes) 000120 1786178765.965597 fallocate 0x00bebf thru 0x00c206 (0x348 bytes) 000121 1786178765.969734 trunc from 0x036b99 to 0x035631 (0x1567 bytes) 000122 1786178765.986693 trunc from 0x01e3c7 to 0x0223d0 (0x400a bytes) 000123 1786178765.990088 read 0x0056fd thru 0x00fe70 (0xa774 bytes) 000124 1786178765.991376 trunc from 0x035632 to 0x03fb4d (0xa51c bytes) 000125 1786178766.010861 mapwrite 0x0159e8 thru 0x019a26 (0x403f bytes) 000126 1786178766.022460 trunc from 0x03fb4e to 0x006c2b (0x38f22 bytes) 000127 1786178766.032263 read 0x0030d9 thru 0x006c2b (0x3b53 bytes) 000128 1786178766.032770 punch from 0x004132 to 0x006c2b (0x2afa bytes) 000129 1786178766.035995 mapread 0x0040d3 thru 0x006c2b (0x2b59 bytes) 000130 1786178766.041383 trunc from 0x006c2c to 0x018ce1 (0x120b6 bytes) 000131 1786178766.055744 mapread 0x00a25c thru 0x01870f (0xe4b4 bytes) 000132 1786178766.071083 trunc from 0x018ce2 to 0x037a86 (0x1eda5 bytes) 000133 1786178766.083397 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) 000134 1786178766.199535 punch from 0x000f2e to 0x0012e1 (0x3b4 bytes) 000135 1786178766.205873 mapwrite 0x024b4d thru 0x02edd6 (0xa28a bytes) 000136 1786178766.260354 MIRROR_EXTEND to 2 mirrors 000137 1786178766.900936 fallocate 0x00ffce thru 0x014318 (0x434b bytes) 000138 1786178766.945773 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37a87): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a87): Mirror 2: 0x356fc3cc (stale) CRC-32 checksum value for chunk [0, 0x37a87): Mirror 1: 0x356fc3cc Mirror 3: 0x356fc3cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a87: skipped 000139 1786178767.985656 trunc from 0x037a87 to 0x038261 (0x7db bytes) 000140 1786178768.024112 mapread 0x010867 thru 0x019945 (0x90df bytes) 000141 1786178768.042740 MIRROR_RESYNC 3 mirrors 000142 1786178768.839346 write 0x01af93 thru 0x01f7a7 (0x4815 bytes) 000143 1786178768.882816 MIRROR_RESYNC 3 mirrors 000144 1786178769.903417 punch from 0x00c06a to 0x00eb0c (0x2aa3 bytes) 000145 1786178769.924283 mapread 0x013433 thru 0x01d509 (0xa0d7 bytes) 000146 1786178769.937858 mapread 0x015ba4 thru 0x0165b0 (0xa0d bytes) 000148 1786178769.939690 read 0x0271da thru 0x031d39 (0xab60 bytes) 000149 1786178769.949661 mapwrite 0x03a03d thru 0x03ffff (0x5fc3 bytes) 000150 1786178769.989354 mapwrite 0x019e94 thru 0x0206bb (0x6828 bytes) 000151 1786178770.017327 trunc from 0x028efa to 0x036be7 (0xdcee bytes) 000152 1786178770.022171 fallocate 0x0316a1 thru 0x032460 (0xdc0 bytes) 000153 1786178770.025940 punch from 0x0248dd to 0x02cc3a (0x835e bytes) 000154 1786178770.036996 mapread 0x0380e7 thru 0x038cdc (0xbf6 bytes) 000156 1786178770.038939 trunc from 0x03ca25 to 0x03ffff (0x35db bytes) 000157 1786178770.054173 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x766ba260 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x766ba260 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000160 1786178770.327445 trunc from 0x040000 to 0x00fe39 (0x301c6 bytes) 000161 1786178770.352221 fallocate 0x03113a thru 0x03ba0c (0xa8d3 bytes) 000162 1786178770.357379 mapwrite 0x02b9db thru 0x0379f9 (0xc01f bytes) 000163 1786178770.409942 punch from 0x01a19f to 0x01faba (0x591c bytes) 000164 1786178770.426772 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ba0d): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ba0d): Mirror 3: 0x707cc61b (stale) CRC-32 checksum value for chunk [0, 0x3ba0d): Mirror 1: 0x2e593e7d Mirror 4: 0x2e593e7d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ba0d: skipped 000165 1786178771.677706 MIRROR_RESYNC 3 mirrors 000166 1786178772.307751 write 0x03d9d7 thru 0x03ffff (0x2629 bytes) 000167 1786178772.340001 punch from 0x02ace9 to 0x030424 (0x573c bytes) 000168 1786178772.344845 punch from 0x028013 to 0x02ab62 (0x2b50 bytes) 000169 1786178772.349875 write 0x0033f3 thru 0x00f45c (0xc06a bytes) 000170 1786178772.359852 mapwrite 0x008fbc thru 0x00eef5 (0x5f3a bytes) 000171 1786178772.389272 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x16aac433 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x16aac433 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000173 1786178772.630891 fallocate 0x028214 thru 0x02e802 (0x65ef bytes) 000174 1786178772.637608 mapread 0x01be2b thru 0x022f5b (0x7131 bytes) 000175 1786178772.662439 read 0x03a4dc thru 0x03adab (0x8d0 bytes) 000176 1786178772.667316 punch from 0x02d534 to 0x03aa52 (0xd51f bytes) 000177 1786178772.672908 write 0x00475b thru 0x0049e8 (0x28e bytes) 000178 1786178772.687696 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) 000179 1786178772.845392 read 0x03d261 thru 0x03ffff (0x2d9f bytes) 000180 1786178772.849605 fallocate 0x00c5c1 thru 0x01057c (0x3fbc bytes) 000181 1786178772.858939 trunc from 0x040000 to 0x039cb9 (0x6346 bytes) 000182 1786178772.889498 fallocate 0x03b15f thru 0x03fae7 (0x4989 bytes) 000183 1786178772.898192 trunc from 0x02b7a6 to 0x035b23 (0xa37e bytes) 000184 1786178772.905317 mapread 0x008c55 thru 0x00af46 (0x22f2 bytes) 000185 1786178772.925490 fallocate 0x030f45 thru 0x033fe2 (0x309e bytes) 000186 1786178772.931294 write 0x0227b2 thru 0x02dee9 (0xb738 bytes) 000187 1786178772.946985 write 0x0221b6 thru 0x02e4be (0xc309 bytes) 000188 1786178772.949940 trunc from 0x020a0d to 0x023a30 (0x3024 bytes) 000189 1786178772.964705 trunc from 0x0325e9 to 0x03c533 (0x9f4b bytes) 000190 1786178772.970381 mapread 0x00ee3d thru 0x01a1b7 (0xb37b bytes) 000191 1786178772.979378 punch from 0x037119 to 0x03e025 (0x6f0d bytes) 000192 1786178772.983931 trunc from 0x02a6c8 to 0x0316c5 (0x6ffe bytes) 000194 1786178772.991768 mapwrite 0x01f275 thru 0x0277ae (0x853a bytes) 000195 1786178773.019942 punch from 0x00cceb to 0x016aad (0x9dc3 bytes) 000197 1786178773.025918 trunc from 0x03fae8 to 0x01d899 (0x2224e bytes) 000198 1786178773.041321 trunc from 0x00a927 to 0x014d87 (0xa461 bytes) 000199 1786178773.047947 write 0x02da9c thru 0x03adec (0xd351 bytes) 000200 1786178773.052468 trunc from 0x03aded to 0x01d07f (0x1dd6d bytes) 000201 1786178773.087471 write 0x00a950 thru 0x00aa2c (0xdd bytes) 000202 1786178773.098324 MIRROR_EXTEND to 2 mirrors 000203 1786178773.762543 read 0x007397 thru 0x008cd1 (0x193b bytes) 000204 1786178773.789362 read 0x01ba97 thru 0x01d07f (0x15e9 bytes) 000205 1786178773.848500 mapread 0x018064 thru 0x01d07f (0x501c bytes) 000206 1786178773.882154 mapwrite 0x026c94 thru 0x029470 (0x27dd bytes) 000207 1786178773.962114 punch from 0x006dfa to 0x00b193 (0x439a bytes) 000208 1786178773.966953 mapwrite 0x03d5ee thru 0x03ffff (0x2a12 bytes) 000209 1786178774.006126 trunc from 0x030d29 to 0x03d908 (0xcbe0 bytes) 000210 1786178774.011528 mapread 0x00040d thru 0x005bda (0x57ce bytes) 000212 1786178774.034870 trunc from 0x013726 to 0x01bfc8 (0x88a3 bytes) 000213 1786178774.039291 mapread 0x02cf99 thru 0x02f619 (0x2681 bytes) 000214 1786178774.041214 trunc from 0x040000 to 0x032247 (0xddb8 bytes) 000215 1786178774.057488 fallocate 0x01f540 thru 0x02951c (0x9fdd bytes) 000216 1786178774.062239 write 0x03e37d thru 0x03fd05 (0x1989 bytes) 000217 1786178774.063886 fallocate 0x02b999 thru 0x038834 (0xce9c bytes) 000218 1786178774.068380 punch from 0x01c872 to 0x01ddf0 (0x157f bytes) 000219 1786178774.072324 mapwrite 0x00b687 thru 0x00d118 (0x1a92 bytes) 000220 1786178774.098495 trunc from 0x030267 to 0x03d00d (0xcda7 bytes) 000221 1786178774.102715 write 0x013ac8 thru 0x01ef67 (0xb4a0 bytes) 000222 1786178774.113204 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3fd06): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd06): Mirror 2: 0xa4754b87 (stale) CRC-32 checksum value for chunk [0, 0x3fd06): Mirror 1: 0xbbeaa8c8 Mirror 3: 0xbbeaa8c8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd06: skipped 000223 1786178774.760361 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) 000225 1786178775.104745 read 0x0095e2 thru 0x00f6b2 (0x60d1 bytes) 000226 1786178775.125745 fallocate 0x03ceac thru 0x03f47c (0x25d1 bytes) 000227 1786178775.171558 write 0x02e8d8 thru 0x035fee (0x7717 bytes) 000228 1786178775.183510 write 0x02b110 thru 0x03a9cc (0xf8bd bytes) 000230 1786178775.199206 fallocate 0x024171 thru 0x02e97f (0xa80f bytes) 000231 1786178775.204105 fallocate 0x02b9d8 thru 0x037542 (0xbb6b bytes) 000233 1786178775.209206 trunc from 0x016a59 to 0x017045 (0x5ed bytes) 000234 1786178775.214818 punch from 0x03066d to 0x034262 (0x3bf6 bytes) 000235 1786178775.232867 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3fd06): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd06): Mirror 3: 0xbbeaa8c8 (stale) CRC-32 checksum value for chunk [0, 0x3fd06): Mirror 1: 0xe1821c9b Mirror 4: 0xe1821c9b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd06: skipped 000237 1786178776.195351 mapwrite 0x03e90b thru 0x03ffff (0x16f5 bytes) 000238 1786178776.335441 read 0x01d8e1 thru 0x02975f (0xbe7f bytes) 000239 1786178776.347422 read 0x0097d1 thru 0x016824 (0xd054 bytes) 000240 1786178776.359805 trunc from 0x03d800 to 0x03ffff (0x2800 bytes) 000241 1786178776.367374 write 0x01a7c6 thru 0x023973 (0x91ae bytes) 000242 1786178776.374424 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8c404f90 (stale) Mirror 4: 0xe38f2e6e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3d6521f4 Mirror 5: 0x3d6521f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000243 1786178777.176208 mapread 0x0217a6 thru 0x03051f (0xed7a bytes) 000244 1786178777.221311 read 0x00163d thru 0x00d5e7 (0xbfab bytes) 000245 1786178777.243719 mapread 0x0122d9 thru 0x01d29c (0xafc4 bytes) 000246 1786178777.285592 MIRROR_RESYNC 4 mirrors 000247 1786178778.306294 fallocate 0x0373a7 thru 0x03ffff (0x8c59 bytes) 000248 1786178778.341447 mapread 0x033c5e thru 0x036a9b (0x2e3e bytes) 000249 1786178778.357801 fallocate 0x02a6d2 thru 0x030724 (0x6053 bytes) 000250 1786178778.369164 trunc from 0x040000 to 0x01a86c (0x25793 bytes) 000251 1786178778.435087 mapwrite 0x001dd2 thru 0x0044d4 (0x2703 bytes) 000252 1786178778.486133 mapread 0x011ae2 thru 0x01a86c (0x8d8b bytes) 000253 1786178778.514443 punch from 0x016acc to 0x01a86c (0x3da1 bytes) 000254 1786178778.519580 read 0x015f7c thru 0x01a86c (0x48f1 bytes) 000255 1786178778.534044 trunc from 0x01a86d to 0x034a5f (0x1a1f3 bytes) 000256 1786178778.557803 mapwrite 0x01a37e thru 0x01e094 (0x3d17 bytes) 000258 1786178778.588401 mapwrite 0x0158a4 thru 0x016261 (0x9be bytes) 000259 1786178778.602708 read 0x00addd thru 0x0113b3 (0x65d7 bytes) 000260 1786178778.603569 mapwrite 0x03c221 thru 0x03ffff (0x3ddf bytes) 000261 1786178778.643789 trunc from 0x040000 to 0x031e48 (0xe1b7 bytes) 000262 1786178778.666650 MIRROR_RESYNC 4 mirrors 000263 1786178779.953730 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) 000264 1786178780.283556 MIRROR_RESYNC 3 mirrors 000265 1786178780.808834 mapwrite 0x037074 thru 0x03b344 (0x42d1 bytes) 000266 1786178780.939326 read 0x0332e9 thru 0x03b344 (0x805c bytes) 000267 1786178780.940547 trunc from 0x03b345 to 0x0044a4 (0x36ea0 bytes) 000268 1786178780.959715 mapwrite 0x031098 thru 0x03c92c (0xb895 bytes) 000269 1786178781.007126 punch from 0x0118ea to 0x015934 (0x404b bytes) 000270 1786178781.012139 trunc from 0x03c92d to 0x003308 (0x39624 bytes) 000271 1786178781.034420 read 0x002bfa thru 0x003308 (0x70f bytes) 000273 1786178781.041808 trunc from 0x002d2e to 0x003308 (0x5db bytes) 000274 1786178781.048946 fallocate 0x017861 thru 0x0242c9 (0xca69 bytes) 000275 1786178781.054476 mapread 0x01ad62 thru 0x0242c9 (0x9568 bytes) 000276 1786178781.065173 trunc from 0x012805 to 0x0129cb (0x1c7 bytes) 000277 1786178781.074704 read 0x00fe13 thru 0x0109ce (0xbbc bytes) 000278 1786178781.075853 trunc from 0x0242ca to 0x0133a8 (0x10f21 bytes) 000279 1786178781.096592 punch from 0x00c2f1 to 0x0133a8 (0x70b8 bytes) 000281 1786178781.102990 read 0x00ecee thru 0x0133a8 (0x46bb bytes) 000282 1786178781.110764 fallocate 0x038a02 thru 0x03d2cb (0x48ca bytes) 000283 1786178781.119006 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d2cc): Mirror 4: 0x3d43a2b5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d2cc): Mirror 4: 0x3d43a2b5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d2cc: skipped 000284 1786178781.380150 punch from 0x037704 to 0x03d2cb (0x5bc8 bytes) 000285 1786178781.385118 fallocate 0x036902 thru 0x03ffff (0x96fe bytes) 000286 1786178781.389987 trunc from 0x020163 to 0x02ec34 (0xead2 bytes) 000287 1786178781.394481 mapwrite 0x03e329 thru 0x03e91b (0x5f3 bytes) 000288 1786178781.408438 punch from 0x017f23 to 0x01ca19 (0x4af7 bytes) 000289 1786178781.412344 write 0x01ca5d thru 0x024432 (0x79d6 bytes) 000291 1786178781.426971 read 0x02b816 thru 0x032fd6 (0x77c1 bytes) 000292 1786178781.445814 fallocate 0x02da58 thru 0x0321cc (0x4775 bytes) 000293 1786178781.451911 write 0x03d92b thru 0x03ffff (0x26d5 bytes) 000294 1786178781.459435 write 0x00e39e thru 0x017e4f (0x9ab2 bytes) 000295 1786178781.473665 mapread 0x033689 thru 0x03e87f (0xb1f7 bytes) 000296 1786178781.496658 mapwrite 0x00371d thru 0x00785f (0x4143 bytes) 000297 1786178781.537896 punch from 0x037dc2 to 0x03e0cc (0x630b bytes) 000298 1786178781.548360 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) 000299 1786178781.656709 read 0x0097f2 thru 0x0187dc (0xefeb bytes) 000300 1786178781.667845 trunc from 0x040000 to 0x01f064 (0x20f9b bytes) 000301 1786178781.691595 mapwrite 0x036045 thru 0x036695 (0x651 bytes) 000302 1786178781.721848 write 0x00be27 thru 0x011647 (0x5821 bytes) 000303 1786178781.723678 mapwrite 0x03a19a thru 0x03bad6 (0x193d bytes) 000304 1786178781.772308 read 0x034889 thru 0x0393d6 (0x4b4e bytes) 000307 1786178781.779776 punch from 0x001f9b to 0x003a94 (0x1afa bytes) 000308 1786178781.785353 mapwrite 0x00fddc thru 0x011a5d (0x1c82 bytes) 000309 1786178781.797321 mapread 0x02726a thru 0x028e40 (0x1bd7 bytes) 000310 1786178781.808678 punch from 0x007ee8 to 0x011699 (0x97b2 bytes) 000311 1786178781.813877 fallocate 0x001176 thru 0x005c20 (0x4aab bytes) 000312 1786178781.825341 MIRROR_EXTEND to 2 mirrors 000313 1786178782.422151 mapread 0x00d916 thru 0x0197ab (0xbe96 bytes) 000314 1786178782.465736 punch from 0x01984e to 0x01d29d (0x3a50 bytes) 000315 1786178782.497778 read 0x001a06 thru 0x010a2e (0xf029 bytes) 000316 1786178782.506937 punch from 0x0018c9 to 0x00f8f2 (0xe02a bytes) 000317 1786178782.511815 punch from 0x01ddc6 to 0x020541 (0x277c bytes) 000319 1786178782.517713 write 0x03c1c4 thru 0x03ffff (0x3e3c bytes) 000320 1786178782.522329 trunc from 0x031feb to 0x034e2d (0x2e43 bytes) 000321 1786178782.527629 trunc from 0x040000 to 0x00bb43 (0x344bc bytes) 000322 1786178782.548293 MIRROR_RESYNC 2 mirrors 000323 1786178783.037708 trunc from 0x0024a6 to 0x00bb43 (0x969e bytes) 000324 1786178783.077606 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xbb44): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbb44): Mirror 2: 0x43077e4c (stale) CRC-32 checksum value for chunk [0, 0xbb44): Mirror 1: 0x43077e4c Mirror 3: 0x43077e4c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbb44: skipped 000325 1786178783.845823 write 0x007023 thru 0x008f47 (0x1f25 bytes) 000326 1786178783.934794 MIRROR_RESYNC 3 mirrors 000327 1786178784.508994 mapwrite 0x02ef3b thru 0x032421 (0x34e7 bytes) 000329 1786178784.572152 trunc from 0x0098c9 to 0x017112 (0xd84a bytes) 000330 1786178784.577841 write 0x022924 thru 0x02e337 (0xba14 bytes) 000331 1786178784.584394 mapread 0x01317b thru 0x020d13 (0xdb99 bytes) 000332 1786178784.600247 fallocate 0x00e424 thru 0x015f5f (0x7b3c bytes) 000333 1786178784.608544 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x32422): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32422): Mirror 2: 0xfcd305bf (stale) Mirror 3: 0xfcd305bf (stale) CRC-32 checksum value for chunk [0, 0x32422): Mirror 1: 0x1f367e4c Mirror 4: 0x1f367e4c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32422: skipped 000335 1786178785.393596 mapread 0x029d83 thru 0x032421 (0x869f bytes) 000336 1786178785.437108 fallocate 0x0002b2 thru 0x002478 (0x21c7 bytes) 000337 1786178785.482923 trunc from 0x032422 to 0x03c957 (0xa536 bytes) 000338 1786178785.568827 mapwrite 0x021bfa thru 0x025777 (0x3b7e bytes) 000339 1786178785.637422 trunc from 0x011572 to 0x01ac59 (0x96e8 bytes) 000340 1786178785.649723 trunc from 0x03a50f to 0x03b097 (0xb89 bytes) 000341 1786178785.665328 trunc from 0x0044d4 to 0x00729e (0x2dcb bytes) 000342 1786178785.673560 trunc from 0x034ee0 to 0x03c957 (0x7a78 bytes) 000343 1786178785.682264 punch from 0x0249e6 to 0x025d4d (0x1368 bytes) 000344 1786178785.686860 write 0x015cec thru 0x01c8bb (0x6bd0 bytes) 000345 1786178785.692745 mapread 0x023b07 thru 0x02758b (0x3a85 bytes) 000346 1786178785.706870 write 0x00c721 thru 0x0152f2 (0x8bd2 bytes) 000347 1786178785.713782 trunc from 0x03c958 to 0x008eff (0x33a58 bytes) 000348 1786178785.733676 write 0x0110b8 thru 0x0183be (0x7307 bytes) 000349 1786178785.736415 trunc from 0x01176f to 0x0129a3 (0x1235 bytes) 000350 1786178785.749075 fallocate 0x032a3b thru 0x03ddcc (0xb392 bytes) 000353 1786178785.756153 read 0x00da33 thru 0x00f608 (0x1bd6 bytes) 000354 1786178785.761310 read 0x029065 thru 0x03486f (0xb80b bytes) 000355 1786178785.798172 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ddcd): Mirror 2: 0x143405af (stale) Mirror 4: 0x16b2d0d3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ddcd): Mirror 2: 0x143405af (stale) Mirror 4: 0x16b2d0d3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ddcd: skipped 000356 1786178786.137660 mapwrite 0x03f063 thru 0x03ffff (0xf9d bytes) 000357 1786178786.196744 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa76cd1f5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa76cd1f5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000358 1786178786.502532 trunc from 0x030ecb to 0x03ffff (0xf135 bytes) 000359 1786178786.517663 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) 000360 1786178786.723378 mapread 0x0284fd thru 0x02f643 (0x7147 bytes) 000361 1786178786.756157 trunc from 0x040000 to 0x0394b1 (0x6b4e bytes) 000362 1786178786.783710 mapwrite 0x035ddb thru 0x0389e0 (0x2c06 bytes) 000363 1786178786.834729 write 0x02d842 thru 0x03059d (0x2d5c bytes) 000364 1786178786.837192 punch from 0x01edd2 to 0x02e39b (0xf5ca bytes) 000366 1786178786.857354 mapread 0x022739 thru 0x030acc (0xe394 bytes) 000367 1786178786.888415 mapread 0x00fef5 thru 0x01d0dc (0xd1e8 bytes) 000368 1786178786.916194 read 0x02fb66 thru 0x0394b1 (0x994c bytes) 000369 1786178786.916865 trunc from 0x0394b2 to 0x03c9e4 (0x3533 bytes) 000370 1786178786.933139 read 0x01e188 thru 0x0204bf (0x2338 bytes) 000371 1786178786.933870 punch from 0x03b0db to 0x03c9e4 (0x190a bytes) 000372 1786178786.943288 write 0x00771a thru 0x007b92 (0x479 bytes) 000373 1786178786.965506 mapwrite 0x011a59 thru 0x01ee9c (0xd444 bytes) 000374 1786178786.998475 write 0x00492e thru 0x01349f (0xeb72 bytes) 000376 1786178787.003341 trunc from 0x03a413 to 0x03c9e4 (0x25d2 bytes) 000377 1786178787.007607 punch from 0x01d7d2 to 0x02a54c (0xcd7b bytes) 000378 1786178787.017393 mapread 0x0285d1 thru 0x0302cb (0x7cfb bytes) 000381 1786178787.032516 fallocate 0x036967 thru 0x039ccc (0x3366 bytes) 000382 1786178787.041172 MIRROR_EXTEND to 2 mirrors 000383 1786178787.750130 punch from 0x0225d6 to 0x028d41 (0x676c bytes) 000384 1786178787.784774 trunc from 0x03c9e5 to 0x01c09a (0x2094a bytes) 000385 1786178787.816753 mapwrite 0x02d3f5 thru 0x037e62 (0xaa6e bytes) 000386 1786178787.876925 write 0x005946 thru 0x015113 (0xf7ce bytes) 000387 1786178787.891167 read 0x000b5c thru 0x009054 (0x84f9 bytes) 000388 1786178787.908077 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37e63): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37e63): Mirror 2: 0x22d59768 (stale) CRC-32 checksum value for chunk [0, 0x37e63): Mirror 1: 0x8d7e928e Mirror 3: 0x8d7e928e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37e63: skipped 000389 1786178788.714862 mapread 0x02e90e thru 0x037e62 (0x9555 bytes) 000390 1786178788.750826 read 0x032929 thru 0x037e62 (0x553a bytes) 000391 1786178788.764766 MIRROR_RESYNC 3 mirrors 000392 1786178789.341201 write 0x001aee thru 0x008aae (0x6fc1 bytes) 000393 1786178789.393581 mapwrite 0x031273 thru 0x038005 (0x6d93 bytes) 000394 1786178789.462488 write 0x01052c thru 0x01d22b (0xcd00 bytes) 000395 1786178789.472508 mapwrite 0x00a1ea thru 0x00a9bf (0x7d6 bytes) 000396 1786178789.502293 punch from 0x0143fd to 0x017a24 (0x3628 bytes) 000398 1786178789.507506 write 0x014853 thru 0x023f44 (0xf6f2 bytes) 000399 1786178789.526375 mapwrite 0x02fb17 thru 0x03836e (0x8858 bytes) 000400 1786178789.561479 read 0x032faa thru 0x036745 (0x379c bytes) 000401 1786178789.567067 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3836f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3836f): Mirror 2: 0x543944c1 (stale) Mirror 3: 0x543944c1 (stale) CRC-32 checksum value for chunk [0, 0x3836f): Mirror 1: 0x589b72ae Mirror 4: 0x589b72ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3836f: skipped 000402 1786178790.327138 write 0x02b43d thru 0x031998 (0x655c bytes) 000403 1786178790.407289 fallocate 0x025734 thru 0x0293f1 (0x3cbe bytes) 000404 1786178790.412007 trunc from 0x03836f to 0x02488d (0x13ae1 bytes) 000405 1786178790.457364 trunc from 0x01f66c to 0x02488d (0x5222 bytes) 000407 1786178790.475276 trunc from 0x02488e to 0x009e5a (0x1aa33 bytes) 000408 1786178790.497001 fallocate 0x033d2e thru 0x0340e4 (0x3b7 bytes) 000409 1786178790.505233 trunc from 0x009e5b to 0x0160d5 (0xc27b bytes) 000410 1786178790.526429 mapwrite 0x00a0fd thru 0x015acc (0xb9d0 bytes) 000411 1786178790.575047 fallocate 0x0383aa thru 0x03ffff (0x7c56 bytes) 000412 1786178790.580951 MIRROR_RESYNC 4 mirrors 000413 1786178791.223424 mapread 0x00d6b6 thru 0x014299 (0x6be4 bytes) 000414 1786178791.239572 fallocate 0x03a668 thru 0x03ffff (0x5998 bytes) 000415 1786178791.275488 mapread 0x0154d0 thru 0x0160d5 (0xc06 bytes) 000417 1786178791.285956 write 0x015d44 thru 0x01836c (0x2629 bytes) 000418 1786178791.286965 trunc from 0x00f0f7 to 0x01836c (0x9276 bytes) 000419 1786178791.297401 trunc from 0x001b36 to 0x002966 (0xe31 bytes) 000420 1786178791.302194 trunc from 0x01802f to 0x01836c (0x33e bytes) 000421 1786178791.307096 trunc from 0x017741 to 0x01836c (0xc2c bytes) 000422 1786178791.311285 trunc from 0x01836d to 0x0086a0 (0xfccc bytes) 000423 1786178791.347423 write 0x036e2a thru 0x039e1e (0x2ff5 bytes) 000424 1786178791.350310 mapwrite 0x006db3 thru 0x006df8 (0x46 bytes) 000425 1786178791.366448 trunc from 0x039e1f to 0x03239a (0x7a84 bytes) 000426 1786178791.383077 mapread 0x02283b thru 0x02bbdf (0x93a5 bytes) 000427 1786178791.403433 fallocate 0x01d63c thru 0x02cc84 (0xf649 bytes) 000428 1786178791.409976 trunc from 0x01436b to 0x016641 (0x22d7 bytes) 000429 1786178791.415775 mapwrite 0x027790 thru 0x02c059 (0x48ca bytes) 000431 1786178791.435346 mapread 0x0133bb thru 0x01c05d (0x8ca3 bytes) 000432 1786178791.442774 mapread 0x02dcba thru 0x03239a (0x46e1 bytes) 000433 1786178791.443745 punch from 0x02f41f to 0x03239a (0x2f7c bytes) 000435 1786178791.449859 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3239b): Mirror 2: 0xfb6f9c45 (stale) Mirror 4: 0xfb6f9c45 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3239b): Mirror 2: 0xfb6f9c45 (stale) Mirror 4: 0xfb6f9c45 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3239b: skipped 000436 1786178791.758060 mapwrite 0x009557 thru 0x00d8d1 (0x437b bytes) 000437 1786178791.793947 write 0x034fca thru 0x03dc89 (0x8cc0 bytes) 000438 1786178791.799427 mapwrite 0x00cdde thru 0x0108e5 (0x3b08 bytes) 000439 1786178791.822783 mapread 0x026707 thru 0x02ae7a (0x4774 bytes) 000440 1786178791.831917 fallocate 0x00bb45 thru 0x015386 (0x9842 bytes) 000441 1786178791.836449 mapread 0x02fbca thru 0x0300b5 (0x4ec bytes) 000442 1786178791.845746 trunc from 0x03dc8a to 0x03b0ab (0x2bde bytes) 000444 1786178791.862738 punch from 0x039449 to 0x03b0ab (0x1c63 bytes) 000445 1786178791.872088 trunc from 0x03b0ac to 0x02c2bf (0xedec bytes) 000446 1786178791.889319 trunc from 0x02c2c0 to 0x01e5c7 (0xdcf8 bytes) 000447 1786178791.902811 read 0x013016 thru 0x01b34f (0x833a bytes) 000448 1786178791.911293 read 0x007354 thru 0x01063d (0x92ea bytes) 000449 1786178791.912970 punch from 0x0041d2 to 0x0102fc (0xc12b bytes) 000450 1786178791.917871 fallocate 0x01aa33 thru 0x01d56e (0x2b3c bytes) 000452 1786178791.922946 write 0x013b10 thru 0x018699 (0x4b8a bytes) 000453 1786178791.925317 punch from 0x005422 to 0x015234 (0xfe13 bytes) 000455 1786178791.943442 write 0x02d36a thru 0x02f1fb (0x1e92 bytes) 000456 1786178791.945229 trunc from 0x001b6c to 0x01018c (0xe621 bytes) 000458 1786178791.956464 MIRROR_RESYNC 3 mirrors 000459 1786178792.771793 mapread 0x009b4c thru 0x01245b (0x8910 bytes) 000460 1786178792.793158 trunc from 0x022029 to 0x0230f1 (0x10c9 bytes) 000461 1786178792.829188 MIRROR_RESYNC 3 mirrors 000462 1786178793.661493 fallocate 0x0048e7 thru 0x00a47b (0x5b95 bytes) 000463 1786178793.689568 punch from 0x02e02f to 0x02f1fb (0x11cd bytes) 000464 1786178793.696538 mapread 0x00568e thru 0x00ed4b (0x96be bytes) 000465 1786178793.715775 mapread 0x01e22c thru 0x02bde8 (0xdbbd bytes) 000467 1786178793.735521 trunc from 0x02838f to 0x02f1fb (0x6e6d bytes) 000468 1786178793.740866 read 0x00f0e8 thru 0x01405a (0x4f73 bytes) 000469 1786178793.746926 mapwrite 0x009a55 thru 0x00e68f (0x4c3b bytes) 000470 1786178793.771085 read 0x02298a thru 0x023c3c (0x12b3 bytes) 000471 1786178793.772101 trunc from 0x01254f to 0x014808 (0x22ba bytes) 000472 1786178793.780778 MIRROR_RESYNC 3 mirrors 000473 1786178794.638666 trunc from 0x01d2ea to 0x026b0d (0x9824 bytes) 000475 1786178794.694858 mapread 0x024edc thru 0x02d4f3 (0x8618 bytes) 000478 1786178794.725940 mapread 0x018233 thru 0x02577f (0xd54d bytes) 000479 1786178794.743575 mapwrite 0x01446f thru 0x022efa (0xea8c bytes) 000480 1786178794.803826 trunc from 0x02f1fc to 0x008300 (0x26efb bytes) 000481 1786178794.838463 read 0x002396 thru 0x008300 (0x5f6b bytes) 000482 1786178794.868235 trunc from 0x008301 to 0x018d92 (0x10a92 bytes) 000483 1786178794.904824 write 0x01c428 thru 0x027cbf (0xb898 bytes) 000484 1786178794.930869 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27cc0): Mirror 2: 0x46fb010f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27cc0): Mirror 2: 0x46fb010f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27cc0: skipped 000486 1786178795.252546 read 0x018c0a thru 0x0219cb (0x8dc2 bytes) 000487 1786178795.258116 mapread 0x0237aa thru 0x027cbf (0x4516 bytes) 000488 1786178795.274675 trunc from 0x027cc0 to 0x03c602 (0x14943 bytes) 000489 1786178795.291094 mapwrite 0x004888 thru 0x008f88 (0x4701 bytes) 000490 1786178795.334823 trunc from 0x03c603 to 0x03b32b (0x12d7 bytes) 000491 1786178795.352701 mapwrite 0x016178 thru 0x016af9 (0x982 bytes) 000492 1786178795.392261 punch from 0x000459 to 0x00a258 (0x9e00 bytes) 000493 1786178795.398542 fallocate 0x03a5c8 thru 0x03ffff (0x5a38 bytes) 000494 1786178795.410617 fallocate 0x03b03e thru 0x03f467 (0x442a bytes) 000495 1786178795.416936 read 0x02bfc8 thru 0x0317b8 (0x57f1 bytes) 000496 1786178795.435211 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b32c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b32c): Mirror 2: 0x81180646 (stale) CRC-32 checksum value for chunk [0, 0x3b32c): Mirror 1: 0xe568b3d1 Mirror 3: 0xe568b3d1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b32c: skipped 000497 1786178796.259431 MIRROR_RESYNC 3 mirrors 000498 1786178796.825970 write 0x00690e thru 0x00da30 (0x7123 bytes) 000499 1786178796.887769 mapread 0x038058 thru 0x039146 (0x10ef bytes) 000500 1786178796.903309 mapwrite 0x00a2d0 thru 0x014391 (0xa0c2 bytes) 000501 1786178796.945958 read 0x029d28 thru 0x03203a (0x8313 bytes) 000502 1786178796.957667 fallocate 0x0329f4 thru 0x032df6 (0x403 bytes) 000503 1786178796.962669 trunc from 0x03b32c to 0x02a866 (0x10ac5 bytes) 000504 1786178797.009217 mapwrite 0x016407 thru 0x01b67c (0x5276 bytes) 000505 1786178797.039305 trunc from 0x01ebd4 to 0x025a2f (0x6e5c bytes) 000506 1786178797.044754 fallocate 0x03a57c thru 0x03ffff (0x5a84 bytes) 000507 1786178797.050186 fallocate 0x02b5dd thru 0x031103 (0x5b27 bytes) 000508 1786178797.055616 fallocate 0x007bfa thru 0x016040 (0xe447 bytes) 000509 1786178797.061104 read 0x02b136 thru 0x034270 (0x913b bytes) 000510 1786178797.074569 read 0x02721e thru 0x033f63 (0xcd46 bytes) 000511 1786178797.076131 fallocate 0x03bafd thru 0x03ffff (0x4503 bytes) 000512 1786178797.081198 trunc from 0x040000 to 0x03ecc2 (0x133d bytes) 000513 1786178797.100474 mapread 0x006f9a thru 0x011cec (0xad53 bytes) 000514 1786178797.105725 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3ecc3): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ecc3): Mirror 2: 0xb9225921 (stale) Mirror 3: 0xb9225921 (stale) CRC-32 checksum value for chunk [0, 0x3ecc3): Mirror 1: 0xa8d2a0d8 Mirror 4: 0xa8d2a0d8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ecc3: skipped 000516 1786178797.884663 write 0x03b9fd thru 0x03ffff (0x4603 bytes) 000517 1786178797.935327 write 0x02a871 thru 0x03191e (0x70ae bytes) 000518 1786178797.948625 punch from 0x027c30 to 0x02ba3e (0x3e0f bytes) 000520 1786178797.965705 trunc from 0x01ea0d to 0x022372 (0x3966 bytes) 000521 1786178797.976539 read 0x0374f2 thru 0x0392e0 (0x1def bytes) 000522 1786178797.983511 mapwrite 0x01751a thru 0x01dc4d (0x6734 bytes) 000523 1786178798.028599 MIRROR_RESYNC 4 mirrors 000524 1786178799.429898 punch from 0x01ebb9 to 0x0243bc (0x5804 bytes) 000525 1786178799.520642 trunc from 0x040000 to 0x01b433 (0x24bcc bytes) 000527 1786178799.555225 read 0x0069ab thru 0x010899 (0x9eef bytes) 000528 1786178799.580392 fallocate 0x025006 thru 0x02fad4 (0xaacf bytes) 000529 1786178799.588676 fallocate 0x013f85 thru 0x014af9 (0xb75 bytes) 000530 1786178799.614179 trunc from 0x01b434 to 0x03dad1 (0x2269e bytes) 000531 1786178799.664293 fallocate 0x0268d1 thru 0x02f6fa (0x8e2a bytes) 000532 1786178799.670532 trunc from 0x03dad2 to 0x0241e8 (0x198e9 bytes) 000533 1786178799.718643 trunc from 0x0241e9 to 0x003c5a (0x2058e bytes) 000534 1786178799.744215 mapread 0x00126d thru 0x003c5a (0x29ee bytes) 000535 1786178799.762107 trunc from 0x003c5b to 0x03721a (0x335c0 bytes) 000536 1786178799.787406 trunc from 0x005457 to 0x009129 (0x3cd3 bytes) 000537 1786178799.796603 MIRROR_RESYNC 4 mirrors 000538 1786178800.419494 punch from 0x034db7 to 0x03721a (0x2464 bytes) 000539 1786178800.449536 punch from 0x005612 to 0x0093ef (0x3dde bytes) 000540 1786178800.455241 mapread 0x013c7d thru 0x01b516 (0x789a bytes) 000541 1786178800.473996 write 0x03dd8c thru 0x03ffff (0x2274 bytes) 000542 1786178800.475484 read 0x029329 thru 0x032195 (0x8e6d bytes) 000543 1786178800.484872 trunc from 0x040000 to 0x003096 (0x3cf69 bytes) 000544 1786178800.511054 write 0x001787 thru 0x007f3f (0x67b9 bytes) 000545 1786178800.518097 write 0x00d8b6 thru 0x011a1a (0x4165 bytes) 000546 1786178800.519608 trunc from 0x00aee9 to 0x00d801 (0x2919 bytes) 000547 1786178800.539894 trunc from 0x011a1b to 0x0281db (0x167c1 bytes) 000548 1786178800.554602 read 0x01133a thru 0x01ad0a (0x99d1 bytes) 000549 1786178800.569651 fallocate 0x008296 thru 0x00cbcc (0x4937 bytes) 000550 1786178800.574994 trunc from 0x026cec to 0x0281db (0x14f0 bytes) 000551 1786178800.580642 mapwrite 0x0388d9 thru 0x038934 (0x5c bytes) 000552 1786178800.623593 mapwrite 0x03cbf5 thru 0x03ffff (0x340b bytes) 000553 1786178800.667059 read 0x032078 thru 0x03ab61 (0x8aea bytes) 000554 1786178800.668776 write 0x02f3a6 thru 0x039f2b (0xab86 bytes) 000555 1786178800.676387 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 3: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc9a85457 Mirror 5: 0xc9a85457 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000557 1786178801.840142 MIRROR_RESYNC 5 mirrors 000558 1786178802.998952 punch from 0x00a81a to 0x0151fc (0xa9e3 bytes) 000559 1786178803.029836 write 0x02aec0 thru 0x0301fe (0x533f bytes) 000560 1786178803.039948 trunc from 0x040000 to 0x013b4d (0x2c4b2 bytes) 000561 1786178803.057584 punch from 0x00eb34 to 0x01351d (0x49ea bytes) 000562 1786178803.062677 trunc from 0x00e772 to 0x013b4d (0x53dc bytes) 000563 1786178803.067950 fallocate 0x025ac0 thru 0x0283de (0x291f bytes) 000564 1786178803.076970 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13b4e): Mirror 2: 0x5f063df2 (stale) Mirror 4: 0x5f063df2 (stale) Mirror 5: 0x5f063df2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13b4e): Mirror 2: 0x5f063df2 (stale) Mirror 4: 0x5f063df2 (stale) Mirror 5: 0x5f063df2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13b4e: skipped 000565 1786178803.373224 mapread 0x002674 thru 0x00e90c (0xc299 bytes) 000569 1786178803.399544 MIRROR_RESYNC 4 mirrors 000570 1786178804.188226 trunc from 0x013b4e to 0x036901 (0x22db4 bytes) 000571 1786178804.253454 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x36902): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36902): Mirror 2: 0xdb041dad (stale) Mirror 4: 0xdb041dad (stale) Mirror 5: 0xdb041dad (stale) CRC-32 checksum value for chunk [0, 0x36902): Mirror 1: 0xdb041dad Mirror 6: 0xdb041dad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36902: skipped 000572 1786178805.166321 punch from 0x00d548 to 0x00d8ea (0x3a3 bytes) 000573 1786178805.203644 mapwrite 0x0369e8 thru 0x03e973 (0x7f8c bytes) 000574 1786178805.259135 punch from 0x01cef6 to 0x02163d (0x4748 bytes) 000575 1786178805.264204 mapwrite 0x03d55f thru 0x03ffff (0x2aa1 bytes) 000576 1786178805.296154 trunc from 0x00e565 to 0x0123c6 (0x3e62 bytes) 000577 1786178805.303904 fallocate 0x0024d9 thru 0x00ea2c (0xc554 bytes) 000578 1786178805.311146 write 0x01b58d thru 0x025121 (0x9b95 bytes) 000579 1786178805.326895 trunc from 0x013c46 to 0x01a22c (0x65e7 bytes) 000580 1786178805.331577 write 0x03fef7 thru 0x03ffff (0x109 bytes) 000581 1786178805.332954 fallocate 0x03b2c1 thru 0x03ffff (0x4d3f bytes) 000582 1786178805.341972 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x93f935ed (stale) Mirror 4: 0x93f935ed (stale) Mirror 6: 0x93f935ed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x93f935ed (stale) Mirror 4: 0x93f935ed (stale) Mirror 6: 0x93f935ed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000583 1786178805.790913 read 0x032fcd thru 0x03ffff (0xd033 bytes) 000585 1786178805.809851 mapwrite 0x00a665 thru 0x019973 (0xf30f bytes) 000586 1786178805.869867 trunc from 0x040000 to 0x00465a (0x3b9a5 bytes) 000587 1786178805.893971 mapwrite 0x0130b0 thru 0x01865a (0x55ab bytes) 000588 1786178805.951125 fallocate 0x024e78 thru 0x02821a (0x33a3 bytes) 000589 1786178805.964545 MIRROR_RESYNC 4 mirrors 000590 1786178806.970565 trunc from 0x02821b to 0x02d42a (0x5210 bytes) 000592 1786178807.007141 write 0x02e39f thru 0x038dcb (0xaa2d bytes) 000593 1786178807.009282 mapread 0x019c07 thru 0x023704 (0x9afe bytes) 000594 1786178807.027938 read 0x01ce97 thru 0x02c9cb (0xfb35 bytes) 000595 1786178807.035364 read 0x028505 thru 0x02a880 (0x237c bytes) 000597 1786178807.036780 read 0x02a0b7 thru 0x03000e (0x5f58 bytes) 000598 1786178807.046617 mapread 0x02e943 thru 0x031db7 (0x3475 bytes) 000599 1786178807.048172 trunc from 0x038dcc to 0x0055a1 (0x3382a bytes) 000600 1786178807.068665 mapwrite 0x01cd0b thru 0x0281cd (0xb4c3 bytes) 000601 1786178807.150320 trunc from 0x0281ce to 0x036e27 (0xec5a bytes) 000602 1786178807.166985 trunc from 0x036e28 to 0x00c34c (0x2aadb bytes) 000603 1786178807.185813 fallocate 0x00c734 thru 0x019c1e (0xd4eb bytes) 000605 1786178807.191836 write 0x00543c thru 0x00be38 (0x69fd bytes) 000606 1786178807.205730 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc34d): Mirror 4: 0x53cf16d9 (stale) Mirror 6: 0x53cf16d9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc34d): Mirror 4: 0x53cf16d9 (stale) Mirror 6: 0x53cf16d9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc34d: skipped 000607 1786178807.544733 trunc from 0x007347 to 0x00b154 (0x3e0e bytes) 000608 1786178807.557373 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc34d): Mirror 6: 0x53cf16d9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc34d): Mirror 6: 0x53cf16d9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc34d: skipped 000609 1786178807.902960 MIRROR_RESYNC 2 mirrors 000610 1786178808.521158 fallocate 0x00023e thru 0x0078bf (0x7682 bytes) 000613 1786178808.553551 trunc from 0x006042 to 0x00a29f (0x425e bytes) 000614 1786178808.559336 punch from 0x00171e to 0x00aadb (0x93be bytes) 000615 1786178808.567773 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) 000616 1786178808.679455 mapwrite 0x03ebf3 thru 0x03ffff (0x140d bytes) 000617 1786178808.724953 fallocate 0x03c59c thru 0x03ffff (0x3a64 bytes) 000618 1786178808.730007 trunc from 0x040000 to 0x017593 (0x28a6c bytes) 000619 1786178808.752954 write 0x01b84c thru 0x024f80 (0x9735 bytes) 000620 1786178808.756122 punch from 0x013fdf to 0x016920 (0x2942 bytes) 000621 1786178808.762483 trunc from 0x022f1c to 0x024f80 (0x2065 bytes) 000622 1786178808.773283 mapread 0x011ad5 thru 0x01e16e (0xc69a bytes) 000623 1786178808.788407 trunc from 0x015ee2 to 0x017a3e (0x1b5d bytes) 000624 1786178808.792875 fallocate 0x0158fb thru 0x02164b (0xbd51 bytes) 000625 1786178808.797750 mapwrite 0x0382b1 thru 0x03ce42 (0x4b92 bytes) 000626 1786178808.845762 trunc from 0x00d6e2 to 0x010965 (0x3284 bytes) 000627 1786178808.851117 trunc from 0x03224b to 0x03c754 (0xa50a bytes) 000629 1786178808.857629 punch from 0x001d4d to 0x008c33 (0x6ee7 bytes) 000630 1786178808.862401 fallocate 0x007fde thru 0x014008 (0xc02b bytes) 000631 1786178808.869057 write 0x0395ac thru 0x03c98e (0x33e3 bytes) 000632 1786178808.880825 write 0x006379 thru 0x010376 (0x9ffe bytes) 000633 1786178808.890986 read 0x011c7d thru 0x01c94c (0xacd0 bytes) 000634 1786178808.896145 fallocate 0x0032e4 thru 0x01190d (0xe62a bytes) 000635 1786178808.900321 mapread 0x007171 thru 0x01175c (0xa5ec bytes) 000636 1786178808.901833 fallocate 0x0225fe thru 0x02e5d8 (0xbfdb bytes) 000637 1786178808.906300 trunc from 0x0180fc to 0x020d82 (0x8c87 bytes) 000638 1786178808.910646 read 0x0184e0 thru 0x01e91c (0x643d bytes) 000639 1786178808.918391 read 0x02c045 thru 0x02fb5b (0x3b17 bytes) 000641 1786178808.922979 MIRROR_EXTEND to 2 mirrors 000642 1786178809.909151 write 0x0087e6 thru 0x00edb3 (0x65ce bytes) 000643 1786178809.961588 read 0x02b912 thru 0x037650 (0xbd3f bytes) 000644 1786178809.971882 mapread 0x01b86e thru 0x02affc (0xf78f bytes) 000645 1786178810.001163 mapwrite 0x01b527 thru 0x029772 (0xe24c bytes) 000646 1786178810.040838 trunc from 0x00a34f to 0x019ee3 (0xfb95 bytes) 000647 1786178810.046724 fallocate 0x00d34a thru 0x013239 (0x5ef0 bytes) 000648 1786178810.053985 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) 000649 1786178810.216253 mapwrite 0x0311b3 thru 0x0395cb (0x8419 bytes) 000650 1786178810.269148 mapwrite 0x010f64 thru 0x012f9b (0x2038 bytes) 000651 1786178810.302993 fallocate 0x027ddd thru 0x027fd9 (0x1fd bytes) 000652 1786178810.307606 mapread 0x02cdbd thru 0x03482e (0x7a72 bytes) 000653 1786178810.310087 read 0x0077da thru 0x011ec1 (0xa6e8 bytes) 000654 1786178810.319225 mapwrite 0x01784d thru 0x019b20 (0x22d4 bytes) 000655 1786178810.343967 mapread 0x03ccde thru 0x03ce42 (0x165 bytes) 000656 1786178810.350493 read 0x009e45 thru 0x016fd1 (0xd18d bytes) 000657 1786178810.351080 fallocate 0x00f540 thru 0x013abc (0x457d bytes) 000658 1786178810.354677 trunc from 0x03ce43 to 0x036fc6 (0x5e7c bytes) 000659 1786178810.380919 read 0x015f2d thru 0x0238f5 (0xd9c9 bytes) 000660 1786178810.388940 punch from 0x02d991 to 0x03545a (0x7aca bytes) 000661 1786178810.395611 trunc from 0x00f337 to 0x018d9a (0x9a64 bytes) 000662 1786178810.399766 fallocate 0x0348c1 thru 0x036608 (0x1d48 bytes) 000663 1786178810.405377 trunc from 0x0120e2 to 0x018e8a (0x6da9 bytes) 000664 1786178810.410346 read 0x00b4d6 thru 0x0117a5 (0x62d0 bytes) 000665 1786178810.421103 mapwrite 0x03a9b6 thru 0x03becb (0x1516 bytes) 000666 1786178810.493977 write 0x0138c4 thru 0x018323 (0x4a60 bytes) 000669 1786178810.497454 trunc from 0x035c09 to 0x03becb (0x62c3 bytes) 000670 1786178810.503174 punch from 0x0084cb to 0x012578 (0xa0ae bytes) 000671 1786178810.511906 mapwrite 0x025cf2 thru 0x02fe2b (0xa13a bytes) 000674 1786178810.541384 mapread 0x01e40c thru 0x026d58 (0x894d bytes) 000675 1786178810.545780 trunc from 0x03becc to 0x009c00 (0x322cb bytes) 000676 1786178810.568172 write 0x003538 thru 0x003ed3 (0x99c bytes) 000677 1786178810.569224 mapread 0x002b29 thru 0x009c00 (0x70d8 bytes) 000678 1786178810.579830 fallocate 0x002a87 thru 0x006fac (0x4526 bytes) 000679 1786178810.583793 mapread 0x001577 thru 0x009c00 (0x868a bytes) 000680 1786178810.585207 fallocate 0x029af6 thru 0x038ae5 (0xeff0 bytes) 000681 1786178810.594596 mapwrite 0x03b006 thru 0x03ffff (0x4ffa bytes) 000682 1786178810.640473 punch from 0x00ec4d to 0x010581 (0x1935 bytes) 000683 1786178810.646477 punch from 0x027db3 to 0x02a8a9 (0x2af7 bytes) 000684 1786178810.651511 mapwrite 0x03030b thru 0x03b673 (0xb369 bytes) 000685 1786178810.721339 mapread 0x02e738 thru 0x032392 (0x3c5b bytes) 000687 1786178810.734991 mapwrite 0x029b20 thru 0x035dc2 (0xc2a3 bytes) 000688 1786178810.779786 mapwrite 0x01bfbd thru 0x0281cc (0xc210 bytes) 000689 1786178810.919921 trunc from 0x00ef8e to 0x01a490 (0xb503 bytes) 000690 1786178810.929858 read 0x024bd1 thru 0x0315c3 (0xc9f3 bytes) 000691 1786178810.936583 MIRROR_EXTEND to 2 mirrors 000692 1786178811.554616 trunc from 0x023782 to 0x0300d2 (0xc951 bytes) 000693 1786178811.582605 mapread 0x022d4a thru 0x02ff57 (0xd20e bytes) 000694 1786178811.607611 fallocate 0x01f89d thru 0x025f33 (0x6697 bytes) 000695 1786178811.611967 write 0x00dc2c thru 0x01766f (0x9a44 bytes) 000697 1786178811.628985 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9323288e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x60c1a3f3 Mirror 3: 0x60c1a3f3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000698 1786178812.475838 punch from 0x0384a1 to 0x03df4d (0x5aad bytes) 000699 1786178812.507949 trunc from 0x040000 to 0x008331 (0x37cce bytes) 000700 1786178812.536097 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x8332): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8332): Mirror 2: 0x7d0e93e2 (stale) Mirror 3: 0x7d0e93e2 (stale) CRC-32 checksum value for chunk [0, 0x8332): Mirror 1: 0x7d0e93e2 Mirror 4: 0x7d0e93e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8332: skipped 000701 1786178813.251301 fallocate 0x02ee0d thru 0x037528 (0x871c bytes) 000702 1786178813.297582 punch from 0x019276 to 0x0249ea (0xb775 bytes) 000704 1786178813.303880 mapread 0x007d11 thru 0x00e8c0 (0x6bb0 bytes) 000705 1786178813.323770 fallocate 0x035a6c thru 0x03ffff (0xa594 bytes) 000706 1786178813.328349 trunc from 0x025714 to 0x034b41 (0xf42e bytes) 000707 1786178813.332528 read 0x008373 thru 0x01682b (0xe4b9 bytes) 000708 1786178813.341855 fallocate 0x010422 thru 0x020105 (0xfce4 bytes) 000710 1786178813.347153 read 0x00adec thru 0x01700b (0xc220 bytes) 000711 1786178813.355913 write 0x0316a4 thru 0x03fb71 (0xe4ce bytes) 000712 1786178813.366913 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9323288e (stale) Mirror 3: 0x60c1a3f3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9323288e (stale) Mirror 3: 0x60c1a3f3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000713 1786178813.689026 mapwrite 0x00bbfb thru 0x015fcc (0xa3d2 bytes) 000714 1786178813.745976 fallocate 0x01760b thru 0x0238c6 (0xc2bc bytes) 000715 1786178813.751468 write 0x0110a5 thru 0x01d12e (0xc08a bytes) 000716 1786178813.771892 MIRROR_RESYNC 3 mirrors 000718 1786178814.607540 MIRROR_RESYNC 3 mirrors 000719 1786178814.905924 trunc from 0x01a8ec to 0x01f38a (0x4a9f bytes) 000720 1786178814.931746 trunc from 0x040000 to 0x00d80b (0x327f4 bytes) 000721 1786178814.951800 write 0x02553a thru 0x0336b5 (0xe17c bytes) 000722 1786178814.953627 mapwrite 0x0164d2 thru 0x022738 (0xc267 bytes) 000723 1786178815.011833 punch from 0x02b0c8 to 0x0336b5 (0x85ee bytes) 000724 1786178815.016363 punch from 0x0251ab to 0x0255ff (0x455 bytes) 000725 1786178815.020929 write 0x007426 thru 0x00f4ca (0x80a5 bytes) 000726 1786178815.033729 mapread 0x02ae9d thru 0x0336b5 (0x8819 bytes) 000727 1786178815.051121 trunc from 0x0302a5 to 0x0336b5 (0x3411 bytes) 000728 1786178815.055771 punch from 0x013dba to 0x022dde (0xf025 bytes) 000729 1786178815.074350 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x336b6): Mirror 3: 0x7b442436 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x336b6): Mirror 3: 0x7b442436 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x336b6: skipped 000730 1786178815.295944 punch from 0x01ab18 to 0x026363 (0xb84c bytes) 000731 1786178815.301892 trunc from 0x00b242 to 0x01875a (0xd519 bytes) 000732 1786178815.306236 fallocate 0x02c298 thru 0x035904 (0x966d bytes) 000733 1786178815.313738 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x35905): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35905): Mirror 3: 0xc3a67eaa (stale) CRC-32 checksum value for chunk [0, 0x35905): Mirror 1: 0xe6e1ea85 Mirror 4: 0xe6e1ea85 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35905: skipped 000734 1786178815.961521 trunc from 0x0331f5 to 0x035904 (0x2710 bytes) 000735 1786178815.990150 punch from 0x030773 to 0x034dc0 (0x464e bytes) 000736 1786178815.995639 write 0x008455 thru 0x00a2b5 (0x1e61 bytes) 000737 1786178816.004747 mapread 0x01e3d6 thru 0x02d39d (0xefc8 bytes) 000738 1786178816.024292 fallocate 0x03e324 thru 0x03ffff (0x1cdc bytes) 000739 1786178816.029666 fallocate 0x03105a thru 0x031a94 (0xa3b bytes) 000740 1786178816.039296 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x35905): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35905): Mirror 3: 0xc3a67eaa (stale) Mirror 4: 0xe6e1ea85 (stale) CRC-32 checksum value for chunk [0, 0x35905): Mirror 1: 0xc1a80b54 Mirror 5: 0xc1a80b54 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35905: skipped 000741 1786178816.850642 mapread 0x01738a thru 0x023ba7 (0xc81e bytes) 000742 1786178816.897748 trunc from 0x035905 to 0x0331ae (0x2756 bytes) 000743 1786178816.966471 fallocate 0x015233 thru 0x016c7b (0x1a49 bytes) 000744 1786178816.970703 trunc from 0x0331af to 0x015b92 (0x1d61c bytes) 000745 1786178816.996768 fallocate 0x01f8fd thru 0x02cdd0 (0xd4d4 bytes) 000746 1786178817.002277 trunc from 0x020050 to 0x024206 (0x41b7 bytes) 000747 1786178817.008844 write 0x016967 thru 0x01adab (0x4445 bytes) 000748 1786178817.030348 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cdd1): Mirror 3: 0x2e497728 (stale) Mirror 4: 0x69eb55a8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cdd1): Mirror 3: 0x2e497728 (stale) Mirror 4: 0x69eb55a8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cdd1: skipped 000749 1786178817.291940 write 0x03a3eb thru 0x03ffff (0x5c15 bytes) 000750 1786178817.301001 trunc from 0x040000 to 0x000d47 (0x3f2b8 bytes) 000751 1786178817.314877 mapwrite 0x02da74 thru 0x038615 (0xaba2 bytes) 000752 1786178817.386779 write 0x012091 thru 0x015a45 (0x39b5 bytes) 000753 1786178817.395736 mapwrite 0x01bda9 thru 0x01bee8 (0x140 bytes) 000754 1786178817.410765 mapwrite 0x01fa7f thru 0x02a0ec (0xa66e bytes) 000755 1786178817.440736 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x38616): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38616): Mirror 3: 0x9485b7ed (stale) Mirror 4: 0xfa50ad49 (stale) CRC-32 checksum value for chunk [0, 0x38616): Mirror 1: 0xcca55165 Mirror 5: 0xcca55165 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38616: skipped 000756 1786178818.252430 fallocate 0x02a8c4 thru 0x03165c (0x6d99 bytes) 000757 1786178818.288514 write 0x01b346 thru 0x01c776 (0x1431 bytes) 000758 1786178818.300229 trunc from 0x038616 to 0x007ce3 (0x30932 bytes) 000759 1786178818.350057 trunc from 0x007ce4 to 0x024815 (0x1cb32 bytes) 000760 1786178818.379097 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24816): Mirror 3: 0x6268a82 (stale) Mirror 4: 0xefef26ad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24816): Mirror 3: 0x6268a82 (stale) Mirror 4: 0xefef26ad (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24816: skipped 000761 1786178818.670935 fallocate 0x03e0eb thru 0x03ffff (0x1f15 bytes) 000762 1786178818.680153 mapwrite 0x03b7b8 thru 0x03ffff (0x4848 bytes) 000763 1786178818.719956 mapwrite 0x02781d thru 0x02ca30 (0x5214 bytes) 000764 1786178818.746774 fallocate 0x005ab0 thru 0x01188c (0xbddd bytes) 000765 1786178818.759530 trunc from 0x040000 to 0x02a1b3 (0x15e4c bytes) 000766 1786178818.786744 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2a1b4): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a1b4): Mirror 3: 0x28b05ad (stale) Mirror 4: 0xf4e1c6fb (stale) CRC-32 checksum value for chunk [0, 0x2a1b4): Mirror 1: 0x50a8a755 Mirror 5: 0x50a8a755 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a1b4: skipped 000767 1786178819.481981 punch from 0x0102cc to 0x01e351 (0xe086 bytes) 000768 1786178819.526661 punch from 0x009fd6 to 0x00b932 (0x195d bytes) 000769 1786178819.538411 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a1b4): Mirror 4: 0xf4e1c6fb (stale) Mirror 5: 0x50a8a755 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a1b4): Mirror 4: 0xf4e1c6fb (stale) Mirror 5: 0x50a8a755 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a1b4: skipped 000770 1786178819.824676 trunc from 0x02a1b4 to 0x02ec5f (0x4aac bytes) 000771 1786178819.847909 write 0x03c3fc thru 0x03ffff (0x3c04 bytes) 000772 1786178819.850679 trunc from 0x040000 to 0x039626 (0x69d9 bytes) 000773 1786178819.874891 trunc from 0x039627 to 0x00b0df (0x2e547 bytes) 000774 1786178819.898982 mapread 0x00af43 thru 0x00b0df (0x19d bytes) 000776 1786178819.912526 trunc from 0x00b0e0 to 0x037690 (0x2c5b1 bytes) 000777 1786178819.933511 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37691): Mirror 5: 0x9edd51f0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37691): Mirror 5: 0x9edd51f0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37691: skipped 000778 1786178820.240344 mapwrite 0x015ac4 thru 0x018c4f (0x318c bytes) 000779 1786178820.282533 write 0x005d3c thru 0x0062b7 (0x57c bytes) 000780 1786178820.299951 mapwrite 0x00389d thru 0x009110 (0x5874 bytes) 000781 1786178820.337004 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) 000782 1786178820.460935 mapread 0x02fa6a thru 0x037690 (0x7c27 bytes) 000784 1786178820.492432 mapread 0x00bea7 thru 0x012070 (0x61ca bytes) 000785 1786178820.504473 punch from 0x01d586 to 0x01eb60 (0x15db bytes) 000786 1786178820.509866 trunc from 0x0001c5 to 0x00bdaa (0xbbe6 bytes) 000787 1786178820.515137 trunc from 0x01b6d0 to 0x0279c1 (0xc2f2 bytes) 000788 1786178820.520766 fallocate 0x0109e5 thru 0x01a410 (0x9a2c bytes) 000789 1786178820.526439 mapread 0x01b9fd thru 0x029b19 (0xe11d bytes) 000790 1786178820.545955 read 0x0221d2 thru 0x02e92e (0xc75d bytes) 000791 1786178820.546778 mapread 0x01e20f thru 0x0292c2 (0xb0b4 bytes) 000792 1786178820.548663 punch from 0x0166be to 0x01e50d (0x7e50 bytes) 000794 1786178820.558445 MIRROR_EXTEND to 2 mirrors 000795 1786178821.111411 trunc from 0x037691 to 0x0367cc (0xec4 bytes) 000796 1786178821.172697 write 0x00d478 thru 0x00fe55 (0x29de bytes) 000798 1786178821.184613 read 0x028273 thru 0x02e389 (0x6117 bytes) 000799 1786178821.192843 trunc from 0x015073 to 0x0172d4 (0x2262 bytes) 000801 1786178821.199475 punch from 0x02cdfd to 0x034ad8 (0x7cdc bytes) 000802 1786178821.203849 mapwrite 0x034e4c thru 0x03b981 (0x6b36 bytes) 000803 1786178821.266299 write 0x024ba3 thru 0x029f89 (0x53e7 bytes) 000804 1786178821.271756 read 0x035592 thru 0x03b981 (0x63f0 bytes) 000806 1786178821.272680 mapread 0x0390db thru 0x03b981 (0x28a7 bytes) 000808 1786178821.273800 read 0x02c71f thru 0x03422b (0x7b0d bytes) 000809 1786178821.290306 trunc from 0x03b982 to 0x00ea31 (0x2cf50 bytes) 000810 1786178821.322788 punch from 0x00cd42 to 0x00ea31 (0x1cf0 bytes) 000811 1786178821.327555 mapread 0x0083a2 thru 0x00b98c (0x35eb bytes) 000812 1786178821.332117 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) 000814 1786178821.456527 punch from 0x00bf61 to 0x00ea31 (0x2ad1 bytes) 000815 1786178821.462384 trunc from 0x00ea32 to 0x01e389 (0xf958 bytes) 000816 1786178821.481803 write 0x0390dd thru 0x03bf2b (0x2e4f bytes) 000817 1786178821.482962 punch from 0x038739 to 0x038a44 (0x30c bytes) 000818 1786178821.488099 mapread 0x0086fd thru 0x0088b8 (0x1bc bytes) 000819 1786178821.493190 mapwrite 0x024d90 thru 0x02749b (0x270c bytes) 000820 1786178821.525927 trunc from 0x03bf2c to 0x010b65 (0x2b3c6 bytes) 000822 1786178821.543441 mapread 0x001e1b thru 0x005ada (0x3cc0 bytes) 000824 1786178821.553754 mapread 0x006de0 thru 0x00deff (0x7120 bytes) 000825 1786178821.554988 write 0x00d295 thru 0x017adb (0xa847 bytes) 000826 1786178821.557053 read 0x010183 thru 0x017adb (0x7959 bytes) 000827 1786178821.558095 punch from 0x00f5cd to 0x017569 (0x7f9d bytes) 000828 1786178821.571235 trunc from 0x0086bf to 0x00c6f1 (0x4033 bytes) 000829 1786178821.576469 punch from 0x0083c4 to 0x008649 (0x286 bytes) 000830 1786178821.581169 punch from 0x015332 to 0x017adb (0x27aa bytes) 000831 1786178821.585970 fallocate 0x01b08a thru 0x01d020 (0x1f97 bytes) 000832 1786178821.591479 trunc from 0x017adc to 0x02f8a0 (0x17dc5 bytes) 000833 1786178821.608609 write 0x022ef4 thru 0x02d528 (0xa635 bytes) 000835 1786178821.619943 mapwrite 0x015c7e thru 0x01f031 (0x93b4 bytes) 000836 1786178821.657784 write 0x01e74c thru 0x02a162 (0xba17 bytes) 000837 1786178821.665178 MIRROR_EXTEND to 2 mirrors 000838 1786178822.216401 read 0x025bfe thru 0x02baaa (0x5ead bytes) 000839 1786178822.232804 trunc from 0x02e4f7 to 0x02f8a0 (0x13aa bytes) 000840 1786178822.267704 mapwrite 0x03885f thru 0x03ffff (0x77a1 bytes) 000842 1786178822.355296 mapread 0x00f533 thru 0x01c294 (0xcd62 bytes) 000843 1786178822.375360 trunc from 0x040000 to 0x0371bf (0x8e40 bytes) 000844 1786178822.395609 write 0x00130f thru 0x008e40 (0x7b32 bytes) 000845 1786178822.404714 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x371c0): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x371c0): Mirror 2: 0xdd372742 (stale) CRC-32 checksum value for chunk [0, 0x371c0): Mirror 1: 0x21d09989 Mirror 3: 0x21d09989 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x371c0: skipped 000846 1786178823.215808 fallocate 0x017cc3 thru 0x026e3c (0xf17a bytes) 000847 1786178823.261428 read 0x02c740 thru 0x0371bf (0xaa80 bytes) 000848 1786178823.273791 punch from 0x018d00 to 0x028bd7 (0xfed8 bytes) 000849 1786178823.281816 trunc from 0x0371c0 to 0x02f428 (0x7d97 bytes) 000850 1786178823.311337 trunc from 0x02f429 to 0x031519 (0x20f1 bytes) 000851 1786178823.324814 mapwrite 0x03d450 thru 0x03ffff (0x2bb0 bytes) 000852 1786178823.374776 read 0x01da32 thru 0x020fc4 (0x3593 bytes) 000853 1786178823.385596 trunc from 0x040000 to 0x0156b6 (0x2a949 bytes) 000856 1786178823.417651 punch from 0x009c10 to 0x00ad68 (0x1159 bytes) 000857 1786178823.422584 punch from 0x00793b to 0x0124c2 (0xab88 bytes) 000858 1786178823.435334 MIRROR_RESYNC 3 mirrors 000860 1786178824.017466 MIRROR_EXTEND to 4 mirrors 000861 1786178824.658910 punch from 0x00ab51 to 0x00fd06 (0x51b6 bytes) 000862 1786178824.695946 fallocate 0x01bba8 thru 0x023c27 (0x8080 bytes) 000863 1786178824.707220 mapwrite 0x03a968 thru 0x03ffff (0x5698 bytes) 000864 1786178824.797933 write 0x035450 thru 0x038ba5 (0x3756 bytes) 000866 1786178824.800632 trunc from 0x040000 to 0x032c05 (0xd3fa bytes) 000868 1786178824.821980 mapwrite 0x005408 thru 0x005761 (0x35a bytes) 000869 1786178824.842332 punch from 0x02de2f to 0x032c05 (0x4dd7 bytes) 000870 1786178824.846993 write 0x006a4a thru 0x0080a3 (0x165a bytes) 000871 1786178824.848455 fallocate 0x030a48 thru 0x03a555 (0x9b0e bytes) 000872 1786178824.857453 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32c06): Mirror 2: 0x99541c94 (stale) Mirror 3: 0x99541c94 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32c06): Mirror 2: 0x99541c94 (stale) Mirror 3: 0x99541c94 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32c06: skipped 000873 1786178825.208833 punch from 0x01f2bd to 0x025cc8 (0x6a0c bytes) 000874 1786178825.214380 trunc from 0x032c06 to 0x01cc63 (0x15fa2 bytes) 000876 1786178825.245411 trunc from 0x01cc64 to 0x016f12 (0x5d51 bytes) 000877 1786178825.273760 punch from 0x00e354 to 0x014bc0 (0x686d bytes) 000878 1786178825.284419 fallocate 0x03589c thru 0x03868a (0x2def bytes) 000879 1786178825.297364 trunc from 0x001aec to 0x0036bd (0x1bd2 bytes) 000880 1786178825.311791 mapread 0x002912 thru 0x0124e9 (0xfbd8 bytes) 000881 1786178825.350487 fallocate 0x02149c thru 0x027b76 (0x66db bytes) 000882 1786178825.359811 punch from 0x026b31 to 0x02da20 (0x6ef0 bytes) 000883 1786178825.370989 read 0x007894 thru 0x009f6f (0x26dc bytes) 000884 1786178825.372514 punch from 0x0092e6 to 0x017746 (0xe461 bytes) 000885 1786178825.376978 read 0x01f6ce thru 0x02b73c (0xc06f bytes) 000886 1786178825.414632 trunc from 0x016a97 to 0x025ba5 (0xf10f bytes) 000888 1786178825.424768 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3868b): Mirror 2: 0xd7fb692e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3868b): Mirror 2: 0xd7fb692e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3868b: skipped 000889 1786178825.755955 fallocate 0x02f236 thru 0x038dba (0x9b85 bytes) 000890 1786178825.761179 write 0x02b3c8 thru 0x03226e (0x6ea7 bytes) 000891 1786178825.799779 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3868b): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3868b): Mirror 2: 0xd7fb692e (stale) CRC-32 checksum value for chunk [0, 0x3868b): Mirror 1: 0x823dc5d7 Mirror 3: 0x823dc5d7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3868b: skipped 000892 1786178826.703998 trunc from 0x03868b to 0x003483 (0x35207 bytes) 000894 1786178826.762940 trunc from 0x003484 to 0x0336b5 (0x30232 bytes) 000895 1786178826.788685 fallocate 0x02bb78 thru 0x032c74 (0x70fd bytes) 000897 1786178826.794736 punch from 0x000fcd to 0x009065 (0x8099 bytes) 000898 1786178826.802460 trunc from 0x0336b6 to 0x023a77 (0xfc3e bytes) 000900 1786178826.839154 write 0x01c239 thru 0x01c64b (0x413 bytes) 000902 1786178826.850465 trunc from 0x00bd51 to 0x015930 (0x9be0 bytes) 000903 1786178826.856742 mapread 0x0118d9 thru 0x01b698 (0x9dc0 bytes) 000905 1786178826.880325 mapwrite 0x00703c thru 0x010a6f (0x9a34 bytes) 000906 1786178826.914262 MIRROR_RESYNC 3 mirrors 000907 1786178827.636166 write 0x00eaa2 thru 0x01814a (0x96a9 bytes) 000908 1786178827.676660 mapread 0x009fb3 thru 0x00aa35 (0xa83 bytes) 000909 1786178827.686880 trunc from 0x023a78 to 0x01388b (0x101ec bytes) 000910 1786178827.721341 mapwrite 0x016942 thru 0x02662b (0xfcea bytes) 000911 1786178827.794393 punch from 0x01d2e2 to 0x02662b (0x934a bytes) 000912 1786178827.798324 read 0x00783a thru 0x00d7ba (0x5f81 bytes) 000913 1786178827.810156 write 0x00f682 thru 0x01504f (0x59ce bytes) 000914 1786178827.812376 mapwrite 0x012a3e thru 0x01cf01 (0xa4c4 bytes) 000915 1786178827.836577 trunc from 0x02662c to 0x00ba92 (0x1ab99 bytes) 000916 1786178827.865646 read 0x004a24 thru 0x00ba92 (0x706f bytes) 000917 1786178827.876108 trunc from 0x008ca3 to 0x00ba92 (0x2df0 bytes) 000919 1786178827.893845 mapwrite 0x035ace thru 0x03ffff (0xa532 bytes) 000920 1786178827.995413 mapread 0x011394 thru 0x01af70 (0x9bdd bytes) 000921 1786178827.998657 mapread 0x01cb13 thru 0x01e2f0 (0x17de bytes) 000922 1786178828.004513 read 0x01f73a thru 0x0295aa (0x9e71 bytes) 000923 1786178828.008370 trunc from 0x03c60e to 0x03d959 (0x134c bytes) 000924 1786178828.023143 read 0x019d99 thru 0x01a92f (0xb97 bytes) 000925 1786178828.023983 fallocate 0x01f084 thru 0x0209bb (0x1938 bytes) 000927 1786178828.033293 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x917f88e6 (stale) Mirror 3: 0x917f88e6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x96729f9a Mirror 4: 0x96729f9a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000928 1786178828.863696 trunc from 0x02c517 to 0x02dae6 (0x15d0 bytes) 000929 1786178828.895942 fallocate 0x0360c6 thru 0x03bd59 (0x5c94 bytes) 000931 1786178828.905701 MIRROR_RESYNC 4 mirrors 000932 1786178829.951853 mapread 0x00da90 thru 0x017a34 (0x9fa5 bytes) 000933 1786178829.977317 MIRROR_EXTEND to 5 mirrors 000934 1786178830.834712 trunc from 0x040000 to 0x0028af (0x3d750 bytes) 000935 1786178830.910579 trunc from 0x001911 to 0x0028af (0xf9f bytes) 000936 1786178830.922451 read 0x001304 thru 0x0028af (0x15ac bytes) 000938 1786178830.939539 mapread 0x0020b7 thru 0x0028af (0x7f9 bytes) 000939 1786178830.951298 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x28b0): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28b0): Mirror 2: 0xd345a971 (stale) Mirror 3: 0xd345a971 (stale) Mirror 4: 0xd345a971 (stale) Mirror 5: 0xd345a971 (stale) CRC-32 checksum value for chunk [0, 0x28b0): Mirror 1: 0xd345a971 Mirror 6: 0xd345a971 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28b0: skipped 000941 1786178831.771513 mapwrite 0x020a58 thru 0x02389e (0x2e47 bytes) 000942 1786178831.845841 trunc from 0x02389f to 0x010d7f (0x12b1f bytes) 000943 1786178831.864937 trunc from 0x00f0d5 to 0x010d7f (0x1cab bytes) 000945 1786178831.873175 read 0x007cdb thru 0x010d7f (0x90a5 bytes) 000946 1786178831.880835 write 0x0232c2 thru 0x026066 (0x2da5 bytes) 000947 1786178831.882729 punch from 0x01f534 to 0x026066 (0x6b33 bytes) 000949 1786178831.893734 trunc from 0x00bef9 to 0x011f2a (0x6032 bytes) 000950 1786178831.899506 trunc from 0x026067 to 0x014d61 (0x11305 bytes) 000951 1786178831.916955 punch from 0x0007be to 0x00aa66 (0xa2a9 bytes) 000952 1786178831.921408 trunc from 0x014d62 to 0x0368b1 (0x21b50 bytes) 000953 1786178831.942787 trunc from 0x02eb9b to 0x036011 (0x7477 bytes) 000954 1786178831.947795 read 0x017d9d thru 0x022368 (0xa5cc bytes) 000955 1786178831.969542 mapread 0x009ff5 thru 0x00dcdd (0x3ce9 bytes) 000956 1786178831.971165 trunc from 0x0368b2 to 0x029cdf (0xcbd2 bytes) 000957 1786178831.987795 write 0x00e3fa thru 0x01b571 (0xd178 bytes) 000958 1786178831.989391 trunc from 0x029ce0 to 0x025cdb (0x4004 bytes) 000959 1786178832.007839 write 0x025575 thru 0x028355 (0x2de1 bytes) 000960 1786178832.009648 read 0x002ba6 thru 0x00ba84 (0x8edf bytes) 000961 1786178832.011343 mapwrite 0x034c55 thru 0x03ffff (0xb3ab bytes) 000962 1786178832.066529 trunc from 0x040000 to 0x01033c (0x2fcc3 bytes) 000963 1786178832.085747 punch from 0x0014a0 to 0x0071a4 (0x5d05 bytes) 000964 1786178832.090315 trunc from 0x009bbf to 0x00d0a4 (0x34e6 bytes) 000965 1786178832.094839 fallocate 0x0395d7 thru 0x03ffff (0x6a29 bytes) 000966 1786178832.104133 mapread 0x028ac4 thru 0x03043f (0x797c bytes) 000967 1786178832.139157 fallocate 0x039f73 thru 0x03ffff (0x608d bytes) 000968 1786178832.148568 write 0x02fbbe thru 0x03eecf (0xf312 bytes) 000969 1786178832.155946 mapread 0x000e23 thru 0x003242 (0x2420 bytes) 000970 1786178832.176559 write 0x009ac5 thru 0x0170cb (0xd607 bytes) 000971 1786178832.201567 mapread 0x012520 thru 0x016d4e (0x482f bytes) 000972 1786178832.205707 read 0x0210cb thru 0x0279d3 (0x6909 bytes) 000973 1786178832.220646 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x96729f9a (stale) Mirror 4: 0x96729f9a (stale) Mirror 5: 0x96729f9a (stale) Mirror 6: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x96729f9a (stale) Mirror 4: 0x96729f9a (stale) Mirror 5: 0x96729f9a (stale) Mirror 6: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000974 1786178832.569785 trunc from 0x040000 to 0x03f0bc (0xf43 bytes) 000975 1786178832.596536 mapread 0x012c80 thru 0x01c871 (0x9bf2 bytes) 000976 1786178832.611579 write 0x003603 thru 0x00959d (0x5f9b bytes) 000977 1786178832.624498 MIRROR_RESYNC 5 mirrors 000978 1786178833.629623 write 0x0233c3 thru 0x02c21d (0x8e5b bytes) 000979 1786178833.663662 punch from 0x01353b to 0x022de4 (0xf8aa bytes) 000980 1786178833.672799 MIRROR_RESYNC 5 mirrors 000981 1786178834.824539 mapwrite 0x013e1b thru 0x01c370 (0x8556 bytes) 000982 1786178834.901104 fallocate 0x0381ac thru 0x03e8ea (0x673f bytes) 000983 1786178834.906905 read 0x016a56 thru 0x020e7f (0xa42a bytes) 000984 1786178834.917095 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f0bd): Mirror 4: 0x9e767a4b (stale) Mirror 5: 0x9e767a4b (stale) Mirror 6: 0x9e767a4b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f0bd): Mirror 4: 0x9e767a4b (stale) Mirror 5: 0x9e767a4b (stale) Mirror 6: 0x9e767a4b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f0bd: skipped 000985 1786178835.201497 mapread 0x036b08 thru 0x03f0bc (0x85b5 bytes) 000986 1786178835.225646 trunc from 0x03f0bd to 0x014e4f (0x2a26d bytes) 000987 1786178835.249834 trunc from 0x014e50 to 0x008fba (0xbe95 bytes) 000988 1786178835.268441 trunc from 0x008fbb to 0x034347 (0x2b38d bytes) 000989 1786178835.284789 trunc from 0x034348 to 0x00eaa3 (0x258a4 bytes) 000990 1786178835.295351 trunc from 0x00eaa4 to 0x002ab7 (0xbfec bytes) 000991 1786178835.310187 mapwrite 0x001dc8 thru 0x007384 (0x55bd bytes) 000992 1786178835.338354 trunc from 0x0024f0 to 0x003f92 (0x1aa3 bytes) 000994 1786178835.342840 write 0x0177a0 thru 0x01f288 (0x7ae9 bytes) 000995 1786178835.344713 trunc from 0x00e3bd to 0x010ef2 (0x2b36 bytes) 000996 1786178835.351697 fallocate 0x0202cb thru 0x02897f (0x86b5 bytes) 000997 1786178835.354920 punch from 0x01ccf8 to 0x01f288 (0x2591 bytes) 000998 1786178835.363700 trunc from 0x000d03 to 0x00a5b2 (0x98b0 bytes) 000999 1786178835.367258 mapread 0x01dcff thru 0x01f288 (0x158a bytes) 001000 1786178835.373890 write 0x02dc43 thru 0x03355d (0x591b bytes) All operations completed A-OK! PASS 70a (80s) == sanity-flr test 71: check mirror extend parent fid ==== 04:47:19 (1786178839) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x2d:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (9s) == sanity-flr test 72: check append on FLR file ========== 04:47:28 (1786178848) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.297939 s, 7.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126185 s, 8.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10d:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10d:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 04:47:37 (1786178857) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre2 Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5781472 .. Extending file size to 5070624 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7106112 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8119744 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7179776 .. Extending file size to 8850144 .. Extending file size to 9192128 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8347936 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8463328 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61472 61473 61475 61476 61480 done 192.168.202.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre2 (opts:) 192.168.202.150@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 04:48:56 (1786178936) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre2 Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6858464 .. Extending file size to 2107584 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8190720 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4130560 .. failed Extending file size to 8357760 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9199648 .. Extending file size to 8957824 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 63221 63222 63223 63226 63227 failed 192.168.202.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre2 (opts:) 192.168.202.150@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 04:50:13 (1786179013) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2208 1269096 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17856 3589164 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25700 3581320 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43556 7170484 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2208 1269096 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17856 3589164 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25700 3581320 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43556 7170484 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0783313 s, 26.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.174848 s, 12.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xee29be5f (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre2 (opts:) PASS 200c (15s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 04:50:30 (1786179030) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.147276 s, 14.2 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 04:50:36 (1786179036) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3c:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x123:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x123:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x124:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x124:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 04:50:44 (1786179044) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0706552 s, 29.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x276:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x274:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x275:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 04:50:52 (1786179052) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.120139 s, 17.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x125:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x126:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x125:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x125:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x126:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x125:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x126:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 04:51:01 (1786179061) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.132621 s, 15.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x277:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x276:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x277:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x276:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x277:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 04:51:35 (1786179095) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg250-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699193 s, 15.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x278:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x279:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x278:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x278:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x279:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x278:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x279:0x0] } PASS 204d (46s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 04:52:21 (1786179141) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg250-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x27a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x27b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.160493 s, 13.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x27a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x27b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x27a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x27a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x27b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x27a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (45s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 04:53:06 (1786179186) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg250-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg250-server: Pool lustre.test_204f created Adding targets to pool oleg250-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg250-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.157217 s, 13.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x127:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x127:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x127:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x127:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg250-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg250-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg250-server: Pool lustre.test_204f destroyed PASS 204f (70s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 04:54:16 (1786179256) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 04:54:23 (1786179263) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 04:54:29 (1786179269) oleg250-server: Pool lustre.test_206 created Adding targets to pool oleg250-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg250-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg250-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg250-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg250-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 04:54:55 (1786179295) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088958 s, 11.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555668 s, 18.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x737a5eda Mirror 3: 0x737a5eda PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 04:55:05 (1786179305) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 04:55:08 (1786179308) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 04:55:12 (1786179312) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0485415 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xdd0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1840000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1840000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1840000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2060000 Read this (4 bytes): BBBB 192.168.202.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre2 (opts:) PASS 209a (35s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 04:55:46 (1786179346) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0350125 s, 234 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 04:55:53 (1786179353) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263346 s, 39.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x59 lmm_fid: [0x200000405:0x59:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x59 lmm_fid: [0x200000405:0x59:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611225 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-08 04:55:55.000000000 -0400 Modify: 2026-08-08 04:55:55.000000000 -0400 Change: 2026-08-08 04:55:56.000000000 -0400 Birth: 2026-08-08 04:55:54.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 04:56:01 (1786179361) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024845 s, 42.2 MB/s oleg250-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg250-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg250-server: O/280000401/d17/131: File not found by ext2_lookup PASS 210b (22s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 04:56:23 (1786179383) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.238237 s, 172 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0278813 s, 147 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 8 04:56 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5e:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 8 04:56 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 04:56:30 (1786179390) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 04:56:38 (1786179398) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1707 sec =========== 04:56:48 (1786179408) === sanity-flr: start cleanup 04:56:50 (1786179410) === === sanity-flr: finish cleanup 04:56:58 (1786179418) ===