-----============= acceptance-small: sanity-flr ============----- Sat Aug 22 02:54:11 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg640-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 02:54:30 (1787381670) === oleg640-client.virtnet: executing check_config_client /mnt/lustre oleg640-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg640-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9c7102b68800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c7102b68800.idle_timeout=debug disable quota as required oleg640-server: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 02:55:06 (1787381706) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8224] == sanity-flr test 0a: lfs mirror create with -N option == 02:55:08 (1787381708) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 02:55:23 (1787381723) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 02:55:26 (1787381726) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 02:55:30 (1787381730) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 02:55:54 (1787381754) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 02:55:57 (1787381757) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 02:55:59 (1787381759) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9c7102b68800.stats=clear osc.lustre-OST0001-osc-ffff9c7102b68800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 02:56:06 (1787381766) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 02:56:16 (1787381776) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 02:56:24 (1787381784) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 02:56:28 (1787381788) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 02:56:35 (1787381795) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 02:56:44 (1787381804) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 02:56:51 (1787381811) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 02:56:59 (1787381819) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 02:57:07 (1787381827) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 02:57:18 (1787381838) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 02:57:24 (1787381844) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 1.3159 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094438 s, 11.1 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 02:57:35 (1787381855) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.26225 s, 36.0 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 02:57:45 (1787381865) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 02:57:56 (1787381876) Stopping /mnt/lustre-ost1 (opts:) on oleg640-server oleg640-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c7102b68800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c7102b68800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg640-server: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg640-client: oleg640-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg640-server: oleg640-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg640-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg640-server oleg640-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c7102b68800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c7102b68800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg640-server: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg640-client: oleg640-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg640-server: oleg640-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg640-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (70s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 02:59:06 (1787381946) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.386386 s, 19.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg640-server oleg640-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c7102b68800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c7102b68800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg640-server: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg640-client: oleg640-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg640-server: oleg640-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg640-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (43s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 02:59:49 (1787381989) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg640-server oleg640-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c7102b68800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c7102b68800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg640-server: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg640-client: oleg640-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg640-server: oleg640-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg640-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg640-server oleg640-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c7102b68800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c7102b68800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg640-server: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg640-client: oleg640-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg640-server: oleg640-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg640-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (76s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 03:01:05 (1787382065) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.65669 s, 14.3 MB/s -rw-r--r-- 1 root root 52428800 Aug 22 03:01 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.955s user 0m0.000s sys 0m0.167s Stopping /mnt/lustre-ost1 (opts:) on oleg640-server oleg640-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c7102b68800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c7102b68800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.301s user 0m0.000s sys 0m0.196s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg640-server: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg640-client: oleg640-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg640-server: oleg640-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg640-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg640-server oleg640-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c7102b68800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c7102b68800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.244s user 0m0.001s sys 0m0.175s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg640-server: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg640-client: oleg640-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg640-server: oleg640-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg640-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg640-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c7102b68800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c7102b68800.ost_server_uuid in FULL state after 0 sec PASS 33b (90s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 03:02:35 (1787382155) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 03:02:39 (1787382159) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 03:02:42 (1787382162) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 03:02:46 (1787382166) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 03:02:54 (1787382174) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.473668 s, 26.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 03:03:06 (1787382186) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473384 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576108 s, 18.2 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 03:03:14 (1787382194) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 03:03:23 (1787382203) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000987344 s, 1.0 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00166419 s, 615 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00165977 s, 617 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00208482 s, 491 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 03:03:33 (1787382213) create test files with size 15741327 .. 7491af50e994a4550e5d0a318a9c7e6e - 33795d44234fe7087193d0acb5d972cc - 1a7158e3cdcf708f6405c7b2c709d88c - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg640-client.virtnet /mnt/lustre (opts:) Starting client: oleg640-client.virtnet: -o user_xattr,flock 192.168.206.140@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 03:03:58 (1787382238) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 03:04:10 (1787382250) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 03:04:17 (1787382257) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662968 s, 15.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370159 s, 28.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 03:04:26 (1787382266) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0933172 s, 22.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.324749 s, 12.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.244346 s, 8.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 03:04:42 (1787382282) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (5s) == sanity-flr test 43a: mirror pick on write ============= 03:04:46 (1787382286) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 03:04:50 (1787382290) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680849 s, 15.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 44a: lfs mirror split check =========== 03:05:00 (1787382300) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.328221 s, 9.6 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 03:05:11 (1787382311) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc2:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc2:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 03:05:21 (1787382321) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.489249 s, 21.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg640-client.virtnet /mnt/lustre (opts:) Starting client: oleg640-client.virtnet: -o user_xattr,flock 192.168.206.140@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 03:05:32 (1787382332) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9c7120417000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.382661 s, 27.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg640-client.virtnet /mnt/lustre (opts:) Starting client: oleg640-client.virtnet: -o user_xattr,flock 192.168.206.140@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9c7120410800.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 03:05:48 (1787382348) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (37s) == sanity-flr test 46: Verify setstripe --copy option ==== 03:06:24 (1787382384) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (34s) == sanity-flr test 47: Verify mirror obj alloc =========== 03:06:58 (1787382418) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 03:07:02 (1787382422) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.258226 s, 12.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.118671 s, 26.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xe8625290 (nosync,stale) Mirror 2: 0xe8625290 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 7d8104df7f0a37a58f53a3a6697a836c - mirror 1 checksum: c56a4df5b00895747c3cf45207d971ca - mirror 2 checksum: 7d8104df7f0a37a58f53a3a6697a836c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-22T03:07:05-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-22T03:07:05-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 7d8104df7f0a37a58f53a3a6697a836c - mirror 1 checksum: c56a4df5b00895747c3cf45207d971ca - mirror 2 checksum: c56a4df5b00895747c3cf45207d971ca - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-22T03:07:05-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-22T03:07:05-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 03:07:13 (1787382433) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334789 s, 31.3 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 03:07:21 (1787382441) Failing mds1 on oleg640-server Stopping /mnt/lustre-mds1 (opts:) on oleg640-server 03:07:34 (1787382454) shut down facet: mds1 facet_host: oleg640-server facet_failover_host: oleg640-server Failover mds1 to oleg640-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg640-server: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg640-client: oleg640-server: ssh exited with exit code 1 Started lustre-MDT0000 03:07:57 (1787382477) targets are mounted 03:07:57 (1787382477) facet_failover done oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380229 s, 27.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } PASS 50A (55s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 03:08:16 (1787382496) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0611203 s, 335 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0290652 s, 705 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (12s) == sanity-flr test 50b: mirror rsync handles sparseness == 03:08:28 (1787382508) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0442173 s, 463 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0293609 s, 698 kB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 03:08:37 (1787382517) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00617902 s, 2.7 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 03:08:46 (1787382526) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0462615 s, 443 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (7s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 03:08:53 (1787382533) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00768329 s, 533 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0219725 s, 186 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 03:09:02 (1787382542) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 03:09:27 (1787382567) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 03:09:54 (1787382594) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 03:10:18 (1787382618) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 03:10:45 (1787382645) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 03:10:54 (1787382654) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57388 mirror split pid 57389 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57388 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57389 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 03:12:03 (1787382723) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 2725 skipping zero size read skipping zero length zero range 000005 1787382725.864951 mapwrite 0x03e5aa thru 0x03ffff (0x1a56 bytes) truncating to largest ever: 0xcd23 000006 1787382725.924383 trunc from 0x040000 to 0x00cd22 (0x332dd bytes) truncating to largest ever: 0x34fe8 000007 1787382725.973778 trunc from 0x00cd23 to 0x034fe7 (0x282c5 bytes) 000008 1787382726.002185 read 0x026d96 thru 0x02b9f4 (0x4c5f bytes) 000009 1787382726.032978 write 0x0362d7 thru 0x03def3 (0x7c1d bytes) 000010 1787382726.040579 trunc from 0x03def4 to 0x026867 (0x1768c bytes) 000011 1787382726.062959 read 0x012b7c thru 0x012e8e (0x313 bytes) 000012 1787382726.090623 mapread 0x01ab84 thru 0x026867 (0xbce4 bytes) 000014 1787382726.095109 mapread 0x016042 thru 0x01797b (0x193a bytes) 000016 1787382726.106949 mapwrite 0x016d2f thru 0x025795 (0xea67 bytes) fallocating to largest ever: 0x3bce7 000017 1787382726.170661 fallocate 0x03075d thru 0x03bce6 (0xb58a bytes) 000018 1787382726.181417 trunc from 0x03bce7 to 0x004589 (0x3775d bytes) 000019 1787382726.218174 write 0x02c578 thru 0x02db2c (0x15b5 bytes) 000020 1787382726.219301 punch from 0x022c80 to 0x02db2c (0xaead bytes) 000021 1787382726.252796 trunc from 0x02db2d to 0x01927f (0x148ad bytes) 000022 1787382726.274740 read 0x010aa3 thru 0x01927f (0x87dd bytes) 000023 1787382726.296420 read 0x018c72 thru 0x01927f (0x60e bytes) 000025 1787382726.298173 read 0x015a5f thru 0x0190ad (0x364f bytes) 000026 1787382726.305429 punch from 0x017036 to 0x01809a (0x1065 bytes) 000027 1787382726.315387 mapread 0x00e528 thru 0x01927f (0xad58 bytes) 000028 1787382726.331429 read 0x011135 thru 0x01927f (0x814b bytes) 000029 1787382726.332136 punch from 0x000b6a to 0x0081af (0x7646 bytes) 000030 1787382726.349669 fallocate 0x01e6ed thru 0x0203b7 (0x1ccb bytes) 000031 1787382726.358896 mapwrite 0x0186d9 thru 0x01cf2e (0x4856 bytes) 000032 1787382726.418804 punch from 0x014ca1 to 0x017154 (0x24b4 bytes) 000033 1787382726.434070 mapread 0x0045f9 thru 0x01419a (0xfba2 bytes) 000034 1787382726.453333 mapread 0x0028c8 thru 0x0107f9 (0xdf32 bytes) 000035 1787382726.454885 trunc from 0x0203b8 to 0x012740 (0xdc77 bytes) 000036 1787382726.481802 mapwrite 0x038656 thru 0x03d81d (0x51c8 bytes) 000037 1787382726.533855 mapwrite 0x0084a3 thru 0x00a7aa (0x2308 bytes) fallocating to largest ever: 0x3ec22 000038 1787382726.569831 fallocate 0x030adb thru 0x03ec21 (0xe147 bytes) 000039 1787382726.576192 trunc from 0x03ec22 to 0x0118f8 (0x2d329 bytes) 000040 1787382726.612836 trunc from 0x00a466 to 0x00c73b (0x22d6 bytes) 000041 1787382726.618084 fallocate 0x02cce2 thru 0x033fab (0x72ca bytes) 000042 1787382726.624179 write 0x01bcd3 thru 0x024da9 (0x90d7 bytes) 000043 1787382726.647801 punch from 0x00c880 to 0x011c5a (0x53db bytes) 000044 1787382726.658104 mapwrite 0x016038 thru 0x01ff12 (0x9edb bytes) fallocating to largest ever: 0x40000 000045 1787382726.758995 fallocate 0x03f2a1 thru 0x03ffff (0xd5f bytes) 000046 1787382726.766574 fallocate 0x02f433 thru 0x03e38a (0xef58 bytes) 000047 1787382726.771456 punch from 0x013ae2 to 0x020572 (0xca91 bytes) 000048 1787382726.777160 punch from 0x00ab29 to 0x00be7f (0x1357 bytes) 000049 1787382726.787052 read 0x03d520 thru 0x03e38a (0xe6b bytes) 000052 1787382726.816713 mapread 0x033055 thru 0x03e38a (0xb336 bytes) 000053 1787382726.818091 mapread 0x037bc5 thru 0x03b368 (0x37a4 bytes) 000054 1787382726.822432 trunc from 0x03e38b to 0x0376b1 (0x6cd9 bytes) 000055 1787382726.843654 mapwrite 0x0062cf thru 0x00eb75 (0x88a7 bytes) 000056 1787382726.878116 mapwrite 0x019a1e thru 0x01fa5c (0x603f bytes) 000057 1787382726.902772 trunc from 0x0376b2 to 0x01f407 (0x182aa bytes) 000059 1787382726.927787 trunc from 0x01f408 to 0x015b81 (0x9886 bytes) 000062 1787382726.976220 mapwrite 0x006740 thru 0x014606 (0xdec7 bytes) 000063 1787382727.021236 trunc from 0x015b82 to 0x00ac34 (0xaf4d bytes) 000064 1787382727.042992 mapread 0x001f41 thru 0x00567c (0x373c bytes) 000065 1787382727.044111 punch from 0x0023ac to 0x0060fc (0x3d51 bytes) 000066 1787382727.062046 mapwrite 0x024f81 thru 0x02b2a7 (0x6327 bytes) 000067 1787382727.189516 mapwrite 0x01b679 thru 0x02066e (0x4ff6 bytes) 000068 1787382727.247874 trunc from 0x00aee6 to 0x00cf16 (0x2031 bytes) 000070 1787382727.255242 trunc from 0x02b2a8 to 0x0035fb (0x27cac bytes) 000073 1787382727.279641 mapread 0x0026e5 thru 0x0035fb (0xf17 bytes) 000074 1787382727.291309 write 0x01d7f6 thru 0x0211cc (0x39d7 bytes) 000075 1787382727.294033 write 0x034076 thru 0x03ba11 (0x799c bytes) 000076 1787382727.296300 trunc from 0x03ba12 to 0x01b06c (0x209a5 bytes) 000078 1787382727.340147 trunc from 0x01b06d to 0x01b0df (0x73 bytes) 000079 1787382727.359460 mapwrite 0x00f5c2 thru 0x01bacb (0xc50a bytes) 000080 1787382727.503979 punch from 0x012377 to 0x013872 (0x14fc bytes) 000081 1787382727.520970 read 0x0154c1 thru 0x01bacb (0x660b bytes) 000082 1787382727.521962 read 0x0150f7 thru 0x01796f (0x2879 bytes) 000083 1787382727.529432 trunc from 0x00a721 to 0x00d3aa (0x2c8a bytes) 000084 1787382727.548641 mapwrite 0x033716 thru 0x039f79 (0x6864 bytes) 000085 1787382727.667758 trunc from 0x004ee5 to 0x00d9a4 (0x8ac0 bytes) 000086 1787382727.677683 punch from 0x0233e7 to 0x02d184 (0x9d9e bytes) 000087 1787382727.691670 fallocate 0x027201 thru 0x033b92 (0xc992 bytes) 000089 1787382727.717922 punch from 0x011980 to 0x0127aa (0xe2b bytes) 000090 1787382727.730116 fallocate 0x013506 thru 0x01a4b6 (0x6fb1 bytes) 000091 1787382727.748292 trunc from 0x039f7a to 0x033da1 (0x61d8 bytes) 000092 1787382727.845293 mapread 0x02250c thru 0x02264b (0x140 bytes) 000093 1787382727.846303 fallocate 0x0086cc thru 0x015e20 (0xd755 bytes) 000094 1787382727.854449 mapwrite 0x0311df thru 0x032226 (0x1048 bytes) 000096 1787382727.888318 trunc from 0x033da2 to 0x0234e9 (0x108b8 bytes) 000097 1787382727.928749 read 0x0194ff thru 0x0234e9 (0x9feb bytes) 000098 1787382727.945450 read 0x010d89 thru 0x01e261 (0xd4d9 bytes) 000099 1787382727.949218 trunc from 0x0234ea to 0x034f45 (0x11a5c bytes) 000100 1787382727.991257 trunc from 0x034f46 to 0x031e90 (0x30b5 bytes) 000101 1787382728.037691 punch from 0x02f2e7 to 0x031e90 (0x2baa bytes) 000102 1787382728.042363 mapread 0x020b9a thru 0x026a67 (0x5ece bytes) 000104 1787382728.075925 trunc from 0x00709a to 0x0166bd (0xf624 bytes) 000105 1787382728.091061 mapread 0x0292d1 thru 0x031e90 (0x8bc0 bytes) 000106 1787382728.092372 mapread 0x00c41a thru 0x0108da (0x44c1 bytes) 000107 1787382728.152072 mapwrite 0x0017d9 thru 0x002181 (0x9a9 bytes) 000108 1787382728.186692 trunc from 0x031e91 to 0x024cd4 (0xd1bc bytes) 000109 1787382728.212335 read 0x003f16 thru 0x00d2c3 (0x93ae bytes) 000110 1787382728.254449 fallocate 0x026be9 thru 0x02a0ec (0x3504 bytes) 000113 1787382728.262722 trunc from 0x012ed4 to 0x0199ee (0x6b1b bytes) 000114 1787382728.269193 trunc from 0x024cd5 to 0x028046 (0x3372 bytes) 000115 1787382728.294208 mapwrite 0x009afc thru 0x0156fc (0xbc01 bytes) 000116 1787382728.354873 write 0x02ff5d thru 0x030d3f (0xde3 bytes) 000117 1787382728.357270 trunc from 0x030d40 to 0x007733 (0x2960c bytes) 000118 1787382728.382934 fallocate 0x02d9d5 thru 0x037a8f (0xa0bb bytes) 000119 1787382728.392197 trunc from 0x00a66e to 0x01862f (0xdfc2 bytes) 000120 1787382728.399090 write 0x00a4f9 thru 0x00e28a (0x3d92 bytes) 000123 1787382728.422720 mapread 0x029967 thru 0x036060 (0xc6fa bytes) 000125 1787382728.424939 trunc from 0x037a90 to 0x02600d (0x11a82 bytes) 000127 1787382728.454084 mapread 0x023ad8 thru 0x02600d (0x2536 bytes) 000128 1787382728.485701 read 0x01e6b1 thru 0x02600d (0x795d bytes) 000129 1787382728.514302 trunc from 0x02600e to 0x0076ab (0x1e962 bytes) 000130 1787382728.547168 mapread 0x0041fa thru 0x004f64 (0xd6b bytes) 000132 1787382728.549212 write 0x01e946 thru 0x028070 (0x972b bytes) 000133 1787382728.557202 punch from 0x00ea25 to 0x016f73 (0x854f bytes) 000134 1787382728.567374 trunc from 0x001f6a to 0x009161 (0x71f8 bytes) 000135 1787382728.598446 write 0x000afc thru 0x001b06 (0x100b bytes) 000137 1787382728.612113 trunc from 0x028071 to 0x02f4a2 (0x7432 bytes) 000138 1787382728.638003 trunc from 0x00945b to 0x0165b4 (0xd15a bytes) 000139 1787382728.653132 MIRROR_EXTEND to 2 mirrors 000140 1787382729.593269 punch from 0x02edac to 0x02f4a2 (0x6f7 bytes) 000141 1787382729.655973 trunc from 0x02f4a3 to 0x03e0db (0xec39 bytes) 000142 1787382729.698785 fallocate 0x031d1b thru 0x0341fa (0x24e0 bytes) 000143 1787382729.707097 write 0x010bc3 thru 0x01a2d9 (0x9717 bytes) 000144 1787382729.725745 write 0x01bd92 thru 0x01fe13 (0x4082 bytes) 000145 1787382729.737053 mapwrite 0x0292d9 thru 0x02a1bd (0xee5 bytes) 000147 1787382729.780068 write 0x01efd1 thru 0x025e72 (0x6ea2 bytes) 000148 1787382729.791666 write 0x02affb thru 0x0361d4 (0xb1da bytes) 000149 1787382729.800706 punch from 0x034324 to 0x034485 (0x162 bytes) 000151 1787382729.822059 trunc from 0x0080d7 to 0x0087b3 (0x6dd bytes) 000152 1787382729.829858 mapwrite 0x015499 thru 0x025397 (0xfeff bytes) 000153 1787382729.871761 write 0x039543 thru 0x03ce6b (0x3929 bytes) 000154 1787382729.889783 fallocate 0x039b37 thru 0x03ee55 (0x531f bytes) 000155 1787382729.897702 read 0x0049b3 thru 0x00a774 (0x5dc2 bytes) 000156 1787382729.912188 punch from 0x00e27c to 0x015f51 (0x7cd6 bytes) 000157 1787382729.925301 MIRROR_RESYNC 2 mirrors 000158 1787382730.865184 read 0x00e207 thru 0x0199fb (0xb7f5 bytes) 000159 1787382730.882266 mapwrite 0x019433 thru 0x01cd98 (0x3966 bytes) 000161 1787382731.030845 mapwrite 0x01621f thru 0x0175be (0x13a0 bytes) 000162 1787382731.050443 punch from 0x016f12 to 0x0206c7 (0x97b6 bytes) 000163 1787382731.055795 mapread 0x03c900 thru 0x03ee55 (0x2556 bytes) 000164 1787382731.074160 punch from 0x030d56 to 0x034e8f (0x413a bytes) 000165 1787382731.081144 write 0x02e69a thru 0x03148e (0x2df5 bytes) 000167 1787382731.103321 trunc from 0x03ee56 to 0x02466c (0x1a7e9 bytes) 000168 1787382731.125167 write 0x02b8a0 thru 0x02de93 (0x25f4 bytes) 000169 1787382731.130089 mapread 0x02d54b thru 0x02de93 (0x949 bytes) 000170 1787382731.133827 punch from 0x025dcf to 0x02bb8c (0x5dbe bytes) 000171 1787382731.147313 fallocate 0x03736e thru 0x03e8c3 (0x7556 bytes) 000172 1787382731.160109 punch from 0x017595 to 0x01d4fd (0x5f69 bytes) 000173 1787382731.166141 read 0x0180f9 thru 0x0199bf (0x18c7 bytes) 000174 1787382731.184943 fallocate 0x03c87c thru 0x03e5e8 (0x1d6d bytes) 000175 1787382731.202374 write 0x005ea1 thru 0x0156a9 (0xf809 bytes) 000176 1787382731.210008 fallocate 0x015044 thru 0x022764 (0xd721 bytes) 000177 1787382731.217319 write 0x00854a thru 0x0147cb (0xc282 bytes) 000178 1787382731.226046 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2de94): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2de94): Mirror 2: 0xd9082ead (stale) CRC-32 checksum value for chunk [0, 0x2de94): Mirror 1: 0x3ed01867 Mirror 3: 0x3ed01867 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2de94: skipped 000179 1787382732.289334 fallocate 0x036589 thru 0x0376a1 (0x1119 bytes) 000180 1787382732.354485 mapread 0x019268 thru 0x01a9ad (0x1746 bytes) 000182 1787382732.369099 read 0x019434 thru 0x024077 (0xac44 bytes) 000183 1787382732.383148 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2de94): Mirror 3: 0x3ed01867 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2de94): Mirror 3: 0x3ed01867 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2de94: skipped 000184 1787382732.648935 mapwrite 0x02c40f thru 0x039e6d (0xda5f bytes) 000185 1787382732.766834 mapwrite 0x009e3b thru 0x018463 (0xe629 bytes) 000186 1787382732.873587 trunc from 0x00b61b to 0x012c40 (0x7626 bytes) 000187 1787382732.882045 fallocate 0x028d3b thru 0x033ddb (0xb0a1 bytes) 000188 1787382732.896971 punch from 0x021f83 to 0x02960a (0x7688 bytes) 000189 1787382732.908744 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) 000190 1787382733.165033 trunc from 0x02afe3 to 0x0304dc (0x54fa bytes) 000192 1787382733.176350 write 0x018b61 thru 0x02652c (0xd9cc bytes) 000193 1787382733.206102 MIRROR_EXTEND to 2 mirrors 000194 1787382733.939173 write 0x00a122 thru 0x015e12 (0xbcf1 bytes) 000195 1787382734.007441 MIRROR_RESYNC 2 mirrors 000197 1787382734.949201 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) 000198 1787382735.142349 mapwrite 0x0007da thru 0x001959 (0x1180 bytes) 000199 1787382735.179222 mapread 0x027b6a thru 0x02a78f (0x2c26 bytes) 000200 1787382735.193622 mapwrite 0x00d104 thru 0x01b5f3 (0xe4f0 bytes) 000201 1787382735.227113 punch from 0x033de8 to 0x0374e4 (0x36fd bytes) 000202 1787382735.232126 mapread 0x02edc8 thru 0x03700a (0x8243 bytes) 000203 1787382735.241869 mapread 0x0046d5 thru 0x00a363 (0x5c8f bytes) 000204 1787382735.248063 trunc from 0x039e6e to 0x002f76 (0x36ef7 bytes) 000205 1787382735.267235 trunc from 0x002f77 to 0x00b6b9 (0x8743 bytes) 000207 1787382735.284952 mapwrite 0x00f883 thru 0x01b893 (0xc011 bytes) 000208 1787382735.360952 punch from 0x006520 to 0x014e17 (0xe8f8 bytes) 000209 1787382735.372316 write 0x03d25e thru 0x03f36c (0x210f bytes) 000210 1787382735.375166 fallocate 0x0029d1 thru 0x002c07 (0x237 bytes) 000211 1787382735.384826 trunc from 0x03f36d to 0x0199c2 (0x259aa bytes) 000212 1787382735.414653 punch from 0x00ad47 to 0x00c2cd (0x1587 bytes) 000213 1787382735.423002 write 0x03fb08 thru 0x03ffff (0x4f8 bytes) 000214 1787382735.427871 write 0x01052a thru 0x01bedf (0xb9b6 bytes) 000215 1787382735.464298 trunc from 0x040000 to 0x017f96 (0x28069 bytes) 000217 1787382735.514921 write 0x037c42 thru 0x03e7e6 (0x6ba5 bytes) 000218 1787382735.525002 fallocate 0x000a3d thru 0x00e847 (0xde0b bytes) 000219 1787382735.546963 punch from 0x030a8e to 0x033064 (0x25d7 bytes) 000220 1787382735.553919 mapwrite 0x02cd01 thru 0x03b8f9 (0xebf9 bytes) 000221 1787382735.627227 fallocate 0x007d17 thru 0x00ccc8 (0x4fb2 bytes) 000222 1787382735.650974 MIRROR_EXTEND to 2 mirrors 000227 1787382736.567771 write 0x03de7b thru 0x03ffff (0x2185 bytes) 000229 1787382736.617898 trunc from 0x037656 to 0x03fbdb (0x8586 bytes) 000230 1787382736.636302 read 0x000ebd thru 0x00f930 (0xea74 bytes) 000231 1787382736.649139 trunc from 0x040000 to 0x006a2c (0x395d3 bytes) 000232 1787382736.669948 punch from 0x0030e8 to 0x006a2c (0x3945 bytes) 000233 1787382736.681397 read 0x001405 thru 0x006a2c (0x5628 bytes) 000234 1787382736.687921 write 0x026992 thru 0x02780a (0xe79 bytes) 000235 1787382736.689052 trunc from 0x02780b to 0x029b24 (0x231a bytes) 000236 1787382736.714182 trunc from 0x029b25 to 0x00ee63 (0x1acc1 bytes) 000237 1787382736.744655 fallocate 0x0249b0 thru 0x030391 (0xb9e2 bytes) 000238 1787382736.750309 read 0x000da3 thru 0x00637a (0x55d8 bytes) 000239 1787382736.751672 write 0x00b07e thru 0x00ccd2 (0x1c55 bytes) 000240 1787382736.760454 fallocate 0x0231ff thru 0x02ccdd (0x9adf bytes) 000241 1787382736.766677 write 0x001a8b thru 0x004b48 (0x30be bytes) 000243 1787382736.773909 MIRROR_RESYNC 2 mirrors 000244 1787382737.359529 write 0x020a02 thru 0x022965 (0x1f64 bytes) 000246 1787382737.406169 trunc from 0x0154ae to 0x01beca (0x6a1d bytes) 000247 1787382737.412234 mapread 0x0045f5 thru 0x0048a5 (0x2b1 bytes) 000248 1787382737.419178 mapread 0x008618 thru 0x009abe (0x14a7 bytes) 000249 1787382737.437190 trunc from 0x022966 to 0x00bbe4 (0x16d81 bytes) 000250 1787382737.460312 fallocate 0x013bb1 thru 0x019e9d (0x62ed bytes) 000251 1787382737.467165 trunc from 0x002848 to 0x008626 (0x5ddf bytes) 000252 1787382737.474133 trunc from 0x019e9e to 0x015607 (0x4896 bytes) 000253 1787382737.499792 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x15608): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15608): Mirror 2: 0xd70d8036 (stale) CRC-32 checksum value for chunk [0, 0x15608): Mirror 1: 0x14b3f82e Mirror 3: 0x14b3f82e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15608: skipped 000254 1787382738.855154 mapwrite 0x037bc3 thru 0x03a0b5 (0x24f3 bytes) 000255 1787382738.973248 trunc from 0x019737 to 0x01f766 (0x6030 bytes) 000257 1787382738.997129 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a0b6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a0b6): Mirror 2: 0x47e2d532 (stale) Mirror 3: 0x761ae28 (stale) CRC-32 checksum value for chunk [0, 0x3a0b6): Mirror 1: 0xceabd774 Mirror 4: 0xceabd774 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a0b6: skipped 000258 1787382740.027166 mapread 0x00707e thru 0x00f24f (0x81d2 bytes) 000259 1787382740.051700 punch from 0x029701 to 0x02fedd (0x67dd bytes) 000260 1787382740.100746 trunc from 0x01efe0 to 0x0239ae (0x49cf bytes) 000261 1787382740.109250 fallocate 0x0154d0 thru 0x01bcb3 (0x67e4 bytes) 000262 1787382740.124460 trunc from 0x03a0b6 to 0x003834 (0x36881 bytes) 000263 1787382740.163972 mapwrite 0x002e2c thru 0x00fc7d (0xce52 bytes) 000264 1787382740.240967 mapwrite 0x01b2da thru 0x02729b (0xbfc2 bytes) 000265 1787382740.312038 punch from 0x01fd9e to 0x02729b (0x74fe bytes) 000266 1787382740.318471 trunc from 0x010402 to 0x010466 (0x65 bytes) 000267 1787382740.324139 write 0x01b02e thru 0x0219b1 (0x6984 bytes) 000268 1787382740.332001 fallocate 0x0258f2 thru 0x029c70 (0x437f bytes) 000269 1787382740.339229 punch from 0x00cbf1 to 0x00ec0c (0x201c bytes) 000270 1787382740.347151 trunc from 0x029c71 to 0x0145c8 (0x156a8 bytes) 000271 1787382740.368814 read 0x00f17e thru 0x01113a (0x1fbd bytes) 000272 1787382740.379991 fallocate 0x00288a thru 0x006e77 (0x45ee bytes) 000274 1787382740.391703 fallocate 0x0110c9 thru 0x01acb7 (0x9bef bytes) 000275 1787382740.405536 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1acb8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1acb8): Mirror 2: 0xfd1f4471 (stale) Mirror 3: 0x784d8a4 (stale) Mirror 4: 0x784d8a4 (stale) CRC-32 checksum value for chunk [0, 0x1acb8): Mirror 1: 0xc51994a8 Mirror 5: 0xc51994a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1acb8: skipped 000276 1787382741.435618 mapread 0x00be4c thru 0x011102 (0x52b7 bytes) 000277 1787382741.471162 fallocate 0x00e4aa thru 0x00ff3b (0x1a92 bytes) 000279 1787382741.517758 trunc from 0x0041e7 to 0x00c78b (0x85a5 bytes) 000280 1787382741.532166 MIRROR_RESYNC 5 mirrors 000281 1787382743.036479 mapread 0x019939 thru 0x019edb (0x5a3 bytes) 000282 1787382743.044981 MIRROR_EXTEND to 6 mirrors 000283 1787382744.041098 mapread 0x00a9f1 thru 0x00cf25 (0x2535 bytes) 000284 1787382744.059237 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) 000285 1787382744.382847 trunc from 0x01acb8 to 0x007a6e (0x13249 bytes) 000286 1787382744.429028 trunc from 0x003187 to 0x007a6e (0x48e8 bytes) 000287 1787382744.442030 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x7a6f): Mirror 2: 0xb7537811 (stale) Mirror 3: 0xb7537811 (stale) Mirror 4: 0xb7537811 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7a6f): Mirror 2: 0xb7537811 (stale) Mirror 3: 0xb7537811 (stale) Mirror 4: 0xb7537811 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7a6f: skipped 000288 1787382744.796143 write 0x02f47b thru 0x0361a4 (0x6d2a bytes) 000289 1787382744.806833 write 0x02955c thru 0x0330ee (0x9b93 bytes) 000290 1787382744.817613 fallocate 0x0132c7 thru 0x021f9a (0xecd4 bytes) 000291 1787382744.824984 punch from 0x020101 to 0x02fc19 (0xfb19 bytes) 000292 1787382744.873609 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x361a5): Mirror 3: 0xff1be96d (stale) Mirror 4: 0xff1be96d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x361a5): Mirror 3: 0xff1be96d (stale) Mirror 4: 0xff1be96d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x361a5: skipped 000293 1787382745.366950 trunc from 0x0361a5 to 0x01d0ff (0x190a5 bytes) 000294 1787382745.391043 write 0x00a17b thru 0x019423 (0xf2a9 bytes) 000296 1787382745.404237 mapwrite 0x026d07 thru 0x02aab6 (0x3db0 bytes) 000297 1787382745.459062 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2aab7): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aab7): Mirror 3: 0x412e47e3 (stale) Mirror 4: 0x412e47e3 (stale) CRC-32 checksum value for chunk [0, 0x2aab7): Mirror 1: 0xe8fceae9 Mirror 5: 0xe8fceae9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aab7: skipped 000298 1787382746.995287 read 0x015ad3 thru 0x019d9c (0x42ca bytes) 000299 1787382747.009891 trunc from 0x02aab7 to 0x02a018 (0xa9e bytes) 000300 1787382747.140801 mapwrite 0x0123ee thru 0x01aa87 (0x869a bytes) 000301 1787382747.264829 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a019): Mirror 4: 0x16df5026 (stale) Mirror 5: 0xf6231449 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a019): Mirror 4: 0x16df5026 (stale) Mirror 5: 0xf6231449 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a019: skipped 000303 1787382747.670113 mapwrite 0x02253c thru 0x0233ff (0xec4 bytes) 000304 1787382747.699195 mapwrite 0x03b710 thru 0x03ffff (0x48f0 bytes) 000305 1787382747.759728 mapread 0x028c30 thru 0x02a3d6 (0x17a7 bytes) 000306 1787382747.766415 trunc from 0x00285a to 0x011404 (0xebab bytes) 000307 1787382747.774933 fallocate 0x036bc5 thru 0x03fba2 (0x8fde bytes) 000308 1787382747.788962 trunc from 0x014917 to 0x019926 (0x5010 bytes) 000309 1787382747.796333 trunc from 0x040000 to 0x01cb2a (0x234d5 bytes) 000310 1787382747.823810 trunc from 0x00dca1 to 0x0127ab (0x4b0b bytes) 000311 1787382747.834665 mapread 0x0016f9 thru 0x00763a (0x5f42 bytes) 000312 1787382747.849864 write 0x008268 thru 0x010bb0 (0x8949 bytes) 000313 1787382747.870329 mapread 0x00787c thru 0x009013 (0x1798 bytes) 000314 1787382747.875663 mapread 0x012750 thru 0x01a5f5 (0x7ea6 bytes) 000315 1787382747.895192 write 0x0329b0 thru 0x03e4da (0xbb2b bytes) 000316 1787382747.900173 punch from 0x0267c4 to 0x032f8b (0xc7c8 bytes) 000317 1787382747.928047 read 0x027bab thru 0x031939 (0x9d8f bytes) 000320 1787382747.941085 mapwrite 0x030951 thru 0x030b9d (0x24d bytes) 000321 1787382747.959516 punch from 0x027053 to 0x028654 (0x1602 bytes) 000322 1787382747.971038 mapwrite 0x01e0a2 thru 0x01e107 (0x66 bytes) 000323 1787382748.005276 write 0x018ee2 thru 0x028998 (0xfab7 bytes) 000324 1787382748.016259 read 0x01fead thru 0x029655 (0x97a9 bytes) 000325 1787382748.017392 mapwrite 0x026cdc thru 0x028a03 (0x1d28 bytes) 000327 1787382748.048654 read 0x02670b thru 0x031d5d (0xb653 bytes) 000328 1787382748.051585 trunc from 0x031607 to 0x032cc8 (0x16c2 bytes) 000329 1787382748.061187 trunc from 0x03e4db to 0x002355 (0x3c185 bytes) 000330 1787382748.092588 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2356): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2356): Mirror 4: 0xa07caa45 (stale) Mirror 5: 0xa07caa45 (stale) CRC-32 checksum value for chunk [0, 0x2356): Mirror 1: 0xa07caa45 Mirror 6: 0xa07caa45 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2356: skipped 000331 1787382749.019244 trunc from 0x002356 to 0x03637e (0x34029 bytes) 000332 1787382749.101163 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3637f): Mirror 5: 0x506f2f26 (stale) Mirror 6: 0xdc79a6ea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3637f): Mirror 5: 0x506f2f26 (stale) Mirror 6: 0xdc79a6ea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3637f: skipped 000333 1787382749.544470 write 0x013e89 thru 0x020586 (0xc6fe bytes) 000334 1787382749.564873 trunc from 0x03637f to 0x038a18 (0x269a bytes) 000335 1787382749.605292 fallocate 0x039ab8 thru 0x03ffff (0x6548 bytes) 000336 1787382749.617917 punch from 0x003aa3 to 0x00fc8c (0xc1ea bytes) 000337 1787382749.626394 punch from 0x01b00c to 0x0267ab (0xb7a0 bytes) 000338 1787382749.680955 mapread 0x013901 thru 0x0169c8 (0x30c8 bytes) 000339 1787382749.682999 fallocate 0x002d42 thru 0x005d25 (0x2fe4 bytes) 000340 1787382749.689059 mapwrite 0x002fef thru 0x012cbc (0xfcce bytes) 000342 1787382749.736117 read 0x00071f thru 0x0027c2 (0x20a4 bytes) 000343 1787382749.742401 mapread 0x02df18 thru 0x038a18 (0xab01 bytes) 000344 1787382749.774332 MIRROR_RESYNC 3 mirrors 000345 1787382750.706229 trunc from 0x038a19 to 0x02dc15 (0xae03 bytes) 000346 1787382750.762654 read 0x005819 thru 0x0077c8 (0x1fb0 bytes) 000347 1787382750.775789 mapwrite 0x014aec thru 0x018238 (0x374d bytes) 000348 1787382750.880687 read 0x01e313 thru 0x021de1 (0x3acf bytes) 000349 1787382750.895551 MIRROR_RESYNC 3 mirrors 000350 1787382751.815830 trunc from 0x01d06f to 0x0210ad (0x403f bytes) 000351 1787382751.879963 mapwrite 0x0047d4 thru 0x00d693 (0x8ec0 bytes) 000352 1787382751.950649 trunc from 0x017f5b to 0x01f0e2 (0x7188 bytes) 000353 1787382751.956422 read 0x00a299 thru 0x017699 (0xd401 bytes) 000354 1787382751.965921 trunc from 0x00d39c to 0x0114ce (0x4133 bytes) 000355 1787382751.973364 trunc from 0x02bfe5 to 0x02dc15 (0x1c31 bytes) 000356 1787382751.987126 MIRROR_RESYNC 3 mirrors 000357 1787382752.873600 punch from 0x01d816 to 0x01e49d (0xc88 bytes) 000358 1787382752.907159 write 0x0086e0 thru 0x0131ea (0xab0b bytes) 000359 1787382752.919381 punch from 0x02d963 to 0x02dc15 (0x2b3 bytes) 000360 1787382752.924075 mapread 0x01ed02 thru 0x0257d2 (0x6ad1 bytes) 000362 1787382752.945668 mapread 0x02105d thru 0x021def (0xd93 bytes) 000364 1787382752.947745 mapread 0x010f7c thru 0x01647e (0x5503 bytes) 000365 1787382752.958671 mapread 0x00ce83 thru 0x018533 (0xb6b1 bytes) 000366 1787382752.960869 read 0x02d05d thru 0x02dc15 (0xbb9 bytes) 000367 1787382752.966891 mapwrite 0x03c594 thru 0x03ffff (0x3a6c bytes) 000368 1787382753.026287 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x634a4663 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x634a4663 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000369 1787382753.451905 write 0x01570e thru 0x01c8a6 (0x7199 bytes) 000370 1787382753.483090 trunc from 0x0167b0 to 0x0198d1 (0x3122 bytes) 000372 1787382753.503583 mapread 0x00acb4 thru 0x014e8f (0xa1dc bytes) 000374 1787382753.547813 punch from 0x022b1c to 0x0295ca (0x6aaf bytes) 000375 1787382753.555662 trunc from 0x021fca to 0x023a93 (0x1aca bytes) 000376 1787382753.562589 punch from 0x027155 to 0x031892 (0xa73e bytes) 000377 1787382753.583653 MIRROR_RESYNC 2 mirrors 000378 1787382754.496995 mapread 0x013230 thru 0x01ca5a (0x982b bytes) 000379 1787382754.529010 MIRROR_RESYNC 2 mirrors 000380 1787382754.955262 trunc from 0x040000 to 0x024b3b (0x1b4c4 bytes) 000381 1787382755.007389 fallocate 0x00680d thru 0x00f125 (0x8919 bytes) 000382 1787382755.013226 read 0x013bf0 thru 0x013c36 (0x47 bytes) 000383 1787382755.020135 mapread 0x013ba8 thru 0x022279 (0xe6d2 bytes) 000384 1787382755.056848 trunc from 0x01bd2d to 0x024b3b (0x8e0f bytes) 000385 1787382755.064862 trunc from 0x011996 to 0x01d92b (0xbf96 bytes) 000386 1787382755.071263 trunc from 0x021a19 to 0x024b3b (0x3123 bytes) 000387 1787382755.076866 trunc from 0x024b3c to 0x0269fd (0x1ec2 bytes) 000388 1787382755.104808 mapread 0x00448c thru 0x004737 (0x2ac bytes) 000389 1787382755.118834 trunc from 0x0269fe to 0x0073fa (0x1f603 bytes) 000390 1787382755.141901 mapwrite 0x011da2 thru 0x01b9be (0x9c1d bytes) 000391 1787382755.246704 trunc from 0x00d2b1 to 0x0162b1 (0x9001 bytes) 000392 1787382755.255849 trunc from 0x01b9bf to 0x00a309 (0x116b5 bytes) 000393 1787382755.293031 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) 000394 1787382755.431739 punch from 0x009db9 to 0x00a309 (0x551 bytes) 000395 1787382755.440370 mapread 0x004f1d thru 0x005e78 (0xf5c bytes) 000397 1787382755.461987 fallocate 0x00bfaa thru 0x01221a (0x6271 bytes) 000398 1787382755.486162 mapwrite 0x0272c0 thru 0x02fb36 (0x8877 bytes) 000399 1787382755.598810 mapread 0x021495 thru 0x029fa3 (0x8b0f bytes) 000400 1787382755.604962 trunc from 0x02fb37 to 0x02bffe (0x3b38 bytes) 000401 1787382755.630219 trunc from 0x0134fd to 0x01a23b (0x6d3f bytes) 000402 1787382755.636851 mapread 0x024017 thru 0x026e25 (0x2e0f bytes) 000404 1787382755.647992 mapread 0x0249f1 thru 0x028262 (0x3872 bytes) 000405 1787382755.650078 read 0x01cb45 thru 0x0235c1 (0x6a7d bytes) 000406 1787382755.677472 fallocate 0x00e307 thru 0x01b839 (0xd533 bytes) 000408 1787382755.684570 mapread 0x00188c thru 0x00401d (0x2792 bytes) 000409 1787382755.698646 fallocate 0x022f72 thru 0x02cfc1 (0xa050 bytes) 000410 1787382755.703258 mapwrite 0x00f6ab thru 0x019664 (0x9fba bytes) 000411 1787382755.785625 read 0x0273fa thru 0x02cfc1 (0x5bc8 bytes) 000412 1787382755.806084 read 0x001949 thru 0x010c3b (0xf2f3 bytes) 000413 1787382755.807714 mapwrite 0x00767f thru 0x0161e7 (0xeb69 bytes) 000414 1787382755.852206 write 0x037b98 thru 0x03ffff (0x8468 bytes) 000416 1787382755.858011 punch from 0x009a6a to 0x017d2c (0xe2c3 bytes) 000417 1787382755.865226 mapread 0x03a9d7 thru 0x03bdff (0x1429 bytes) 000420 1787382755.874821 trunc from 0x040000 to 0x01cc65 (0x2339a bytes) 000421 1787382755.902778 trunc from 0x01cc66 to 0x02ff1a (0x132b5 bytes) 000422 1787382755.930208 fallocate 0x01e976 thru 0x022c1b (0x42a6 bytes) 000423 1787382755.936350 trunc from 0x02056a to 0x02d4b1 (0xcf48 bytes) 000424 1787382755.942383 mapread 0x01458e thru 0x014a0d (0x480 bytes) 000425 1787382755.970122 trunc from 0x0245dd to 0x02f845 (0xb269 bytes) 000426 1787382755.976652 read 0x00f55f thru 0x01c753 (0xd1f5 bytes) 000427 1787382755.977474 trunc from 0x02ff1b to 0x024dd4 (0xb146 bytes) 000428 1787382756.006030 trunc from 0x00f0b8 to 0x0160b1 (0x6ffa bytes) 000429 1787382756.033218 trunc from 0x01ca1c to 0x024dd4 (0x83b9 bytes) 000430 1787382756.055184 fallocate 0x03350e thru 0x03ee7d (0xb970 bytes) 000432 1787382756.110909 MIRROR_EXTEND to 2 mirrors 000433 1787382757.250732 read 0x0129d6 thru 0x0193c0 (0x69eb bytes) 000434 1787382757.276886 fallocate 0x02d0b4 thru 0x03a356 (0xd2a3 bytes) 000435 1787382757.332208 mapwrite 0x024e93 thru 0x0271d3 (0x2341 bytes) 000436 1787382757.414811 write 0x02109a thru 0x02c1d3 (0xb13a bytes) 000437 1787382757.427840 fallocate 0x00ad7e thru 0x013825 (0x8aa8 bytes) 000438 1787382757.447787 trunc from 0x016e04 to 0x023016 (0xc213 bytes) 000439 1787382757.487341 fallocate 0x015d4f thru 0x0206fe (0xa9b0 bytes) 000440 1787382757.496720 fallocate 0x039368 thru 0x03ffff (0x6c98 bytes) 000441 1787382757.504095 mapwrite 0x0307dc thru 0x032a89 (0x22ae bytes) 000442 1787382757.538950 read 0x023742 thru 0x03179b (0xe05a bytes) 000443 1787382757.579906 read 0x02ffd8 thru 0x03926c (0x9295 bytes) 000444 1787382757.590069 MIRROR_RESYNC 2 mirrors 000445 1787382758.799745 MIRROR_RESYNC 2 mirrors 000446 1787382759.372600 mapwrite 0x020808 thru 0x02ff24 (0xf71d bytes) 000447 1787382759.539365 trunc from 0x00dfe7 to 0x0193a2 (0xb3bc bytes) 000448 1787382759.549064 trunc from 0x040000 to 0x022d9a (0x1d265 bytes) 000450 1787382759.585893 read 0x015f04 thru 0x0190b3 (0x31b0 bytes) 000451 1787382759.603149 fallocate 0x01f732 thru 0x02a0cc (0xa99b bytes) 000452 1787382759.615634 write 0x03b741 thru 0x03ffff (0x48bf bytes) 000453 1787382759.618363 read 0x0054d7 thru 0x00984a (0x4374 bytes) 000454 1787382759.630198 mapread 0x011194 thru 0x017ab4 (0x6921 bytes) 000457 1787382759.635025 punch from 0x00bdee to 0x018d1d (0xcf30 bytes) 000458 1787382759.656176 trunc from 0x040000 to 0x0101da (0x2fe25 bytes) 000459 1787382759.688114 write 0x0067da thru 0x010655 (0x9e7c bytes) 000462 1787382759.701660 fallocate 0x0303d3 thru 0x03a4b7 (0xa0e5 bytes) 000463 1787382759.722647 MIRROR_RESYNC 2 mirrors 000464 1787382760.307863 trunc from 0x03a4b8 to 0x01d519 (0x1cf9e bytes) 000466 1787382760.383573 mapwrite 0x0095a0 thru 0x00ebe1 (0x5642 bytes) 000467 1787382760.447350 trunc from 0x0066aa to 0x006e73 (0x7ca bytes) 000468 1787382760.456126 trunc from 0x0037ed to 0x00b282 (0x7a96 bytes) 000469 1787382760.467106 MIRROR_RESYNC 2 mirrors 000470 1787382761.183174 trunc from 0x009bf4 to 0x0199d6 (0xfde3 bytes) 000471 1787382761.248019 trunc from 0x00826b to 0x010685 (0x841b bytes) 000472 1787382761.263594 trunc from 0x001646 to 0x003316 (0x1cd1 bytes) 000473 1787382761.271030 write 0x00215e thru 0x0084a4 (0x6347 bytes) 000474 1787382761.286982 write 0x03d158 thru 0x03ffff (0x2ea8 bytes) 000476 1787382761.289207 trunc from 0x038f43 to 0x03d6b9 (0x4777 bytes) 000477 1787382761.308425 mapwrite 0x036516 thru 0x03ffff (0x9aea bytes) 000478 1787382761.379735 mapread 0x01519c thru 0x01f5fc (0xa461 bytes) 000479 1787382761.416719 read 0x033d14 thru 0x0370c3 (0x33b0 bytes) 000480 1787382761.419103 mapwrite 0x0138aa thru 0x0220fc (0xe853 bytes) 000481 1787382761.449410 punch from 0x00e7fe to 0x010a8e (0x2291 bytes) 000482 1787382761.460635 fallocate 0x021c23 thru 0x025c9f (0x407d bytes) 000483 1787382761.474757 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8fb193e5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc33802b0 Mirror 3: 0xc33802b0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000485 1787382762.476844 mapwrite 0x03143f thru 0x0339f8 (0x25ba bytes) 000486 1787382762.572417 mapread 0x0010bd thru 0x003fbe (0x2f02 bytes) 000487 1787382762.621785 mapwrite 0x03e714 thru 0x03ffff (0x18ec bytes) 000489 1787382762.657987 trunc from 0x028a98 to 0x02ad4b (0x22b4 bytes) 000490 1787382762.662129 mapread 0x033068 thru 0x03ffff (0xcf98 bytes) 000491 1787382762.663579 read 0x008a23 thru 0x0160fa (0xd6d8 bytes) 000492 1787382762.669626 trunc from 0x01aa5f to 0x02a156 (0xf6f8 bytes) 000493 1787382762.674253 fallocate 0x029cca thru 0x02a01a (0x351 bytes) 000494 1787382762.679588 mapread 0x000851 thru 0x00b4cb (0xac7b bytes) 000495 1787382762.685570 mapread 0x02418a thru 0x02f8b4 (0xb72b bytes) 000496 1787382762.698816 fallocate 0x018748 thru 0x01f49d (0x6d56 bytes) 000497 1787382762.703072 trunc from 0x02acd1 to 0x02e2ab (0x35db bytes) 000498 1787382762.712660 trunc from 0x01607f to 0x01af59 (0x4edb bytes) 000499 1787382762.718332 write 0x03694c thru 0x0399ad (0x3062 bytes) 000500 1787382762.723211 fallocate 0x03a314 thru 0x03ffff (0x5cec bytes) 000501 1787382762.728751 punch from 0x017a2f to 0x017b96 (0x168 bytes) 000502 1787382762.733853 write 0x00d8ce thru 0x01d46e (0xfba1 bytes) 000503 1787382762.735743 punch from 0x03e2f6 to 0x03ffff (0x1d0a bytes) 000504 1787382762.744183 fallocate 0x0290f3 thru 0x033c38 (0xab46 bytes) 000505 1787382762.750094 write 0x039264 thru 0x03a0f4 (0xe91 bytes) 000506 1787382762.752073 mapread 0x013b0c thru 0x01af9f (0x7494 bytes) 000507 1787382762.754444 trunc from 0x040000 to 0x031788 (0xe877 bytes) 000509 1787382762.815133 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31789): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31789): Mirror 2: 0xc2a87c78 (stale) Mirror 3: 0xa98a7033 (stale) CRC-32 checksum value for chunk [0, 0x31789): Mirror 1: 0xfe1823d5 Mirror 4: 0xfe1823d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31789: skipped 000510 1787382763.968746 trunc from 0x00a8c0 to 0x012a03 (0x8144 bytes) 000511 1787382764.084283 mapread 0x01ced2 thru 0x023303 (0x6432 bytes) 000512 1787382764.118794 trunc from 0x004cc5 to 0x0141da (0xf516 bytes) 000513 1787382764.126411 trunc from 0x031789 to 0x005c4c (0x2bb3c bytes) 000514 1787382764.184268 MIRROR_RESYNC 4 mirrors 000515 1787382765.513184 mapread 0x003008 thru 0x005c4c (0x2c45 bytes) 000516 1787382765.559126 mapread 0x0052dd thru 0x005c4c (0x970 bytes) 000517 1787382765.560722 punch from 0x0035a4 to 0x005c4c (0x26a9 bytes) 000518 1787382765.634110 mapread 0x002652 thru 0x005c4c (0x35fb bytes) 000519 1787382765.649103 punch from 0x00290c to 0x005c4c (0x3341 bytes) 000520 1787382765.656333 punch from 0x002c9a to 0x005c4c (0x2fb3 bytes) 000521 1787382765.664939 read 0x00046a thru 0x005c4c (0x57e3 bytes) 000522 1787382765.675938 fallocate 0x019083 thru 0x02552b (0xc4a9 bytes) 000523 1787382765.682788 trunc from 0x003766 to 0x00dd74 (0xa60f bytes) 000524 1787382765.691960 write 0x014d07 thru 0x0199fb (0x4cf5 bytes) 000525 1787382765.705052 mapwrite 0x03ecdb thru 0x03ffff (0x1325 bytes) 000526 1787382765.769127 trunc from 0x040000 to 0x002de0 (0x3d21f bytes) 000528 1787382765.792281 read 0x00010e thru 0x002de0 (0x2cd3 bytes) 000529 1787382765.793847 mapread 0x00060d thru 0x002de0 (0x27d4 bytes) 000531 1787382765.796595 write 0x03a0ac thru 0x03ffff (0x5f54 bytes) 000533 1787382765.803818 mapwrite 0x014e26 thru 0x015af9 (0xcd4 bytes) 000534 1787382765.877206 trunc from 0x00acb2 to 0x00f4be (0x480d bytes) 000535 1787382765.885013 fallocate 0x009298 thru 0x010f1e (0x7c87 bytes) 000536 1787382765.903053 mapread 0x0178a8 thru 0x01b91d (0x4076 bytes) 000537 1787382765.904329 trunc from 0x03551b to 0x03ffff (0xaae5 bytes) 000538 1787382765.919824 read 0x0099b1 thru 0x00eeca (0x551a bytes) 000539 1787382765.936191 trunc from 0x01b3c2 to 0x021b9d (0x67dc bytes) 000541 1787382765.945683 trunc from 0x040000 to 0x03c3bb (0x3c44 bytes) 000542 1787382765.973112 write 0x035257 thru 0x03ffff (0xada9 bytes) 000543 1787382765.977760 read 0x033a5f thru 0x03ed67 (0xb309 bytes) 000544 1787382765.979786 mapwrite 0x00ee5c thru 0x01a6f4 (0xb899 bytes) 000545 1787382766.021393 mapwrite 0x029b04 thru 0x02fb54 (0x6051 bytes) 000546 1787382766.048876 MIRROR_RESYNC 4 mirrors 000547 1787382767.481833 read 0x018688 thru 0x01d100 (0x4a79 bytes) 000548 1787382767.492670 fallocate 0x028048 thru 0x02ddbc (0x5d75 bytes) 000549 1787382767.577649 mapread 0x022406 thru 0x0267da (0x43d5 bytes) 000550 1787382767.624156 trunc from 0x010e5c to 0x019f23 (0x90c8 bytes) 000551 1787382767.634661 mapwrite 0x013bbc thru 0x017557 (0x399c bytes) 000552 1787382767.683789 write 0x03bd4d thru 0x03dea3 (0x2157 bytes) 000553 1787382767.693874 trunc from 0x040000 to 0x01a089 (0x25f76 bytes) 000554 1787382767.717994 write 0x029b0c thru 0x036bf6 (0xd0eb bytes) 000556 1787382767.723243 trunc from 0x017cae to 0x026573 (0xe8c6 bytes) 000558 1787382767.730662 fallocate 0x017a28 thru 0x0242fa (0xc8d3 bytes) 000559 1787382767.735724 fallocate 0x02a99f thru 0x0395de (0xec40 bytes) 000560 1787382767.741907 fallocate 0x031c20 thru 0x03ffff (0xe3e0 bytes) 000561 1787382767.748005 mapwrite 0x0292e2 thru 0x03068d (0x73ac bytes) 000562 1787382767.765840 mapread 0x009135 thru 0x00b3e5 (0x22b1 bytes) 000563 1787382767.778131 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc9b8f566 (stale) Mirror 3: 0xc9b8f566 (stale) Mirror 4: 0xc9b8f566 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8b686242 Mirror 5: 0x8b686242 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000564 1787382769.055405 trunc from 0x040000 to 0x03e83c (0x17c3 bytes) 000565 1787382769.119253 fallocate 0x012225 thru 0x0166c2 (0x449e bytes) 000566 1787382769.127037 mapwrite 0x005fc7 thru 0x00f29a (0x92d4 bytes) 000567 1787382769.176189 write 0x00e5fa thru 0x01a930 (0xc337 bytes) 000568 1787382769.185815 fallocate 0x024c11 thru 0x02de07 (0x91f7 bytes) 000569 1787382769.195922 punch from 0x02f73c to 0x036b8b (0x7450 bytes) 000570 1787382769.202810 fallocate 0x004d14 thru 0x00d342 (0x862f bytes) 000571 1787382769.208129 punch from 0x03e4ee to 0x03e83c (0x34f bytes) 000572 1787382769.213307 read 0x01d05d thru 0x026231 (0x91d5 bytes) 000573 1787382769.222176 trunc from 0x03e83d to 0x033ae1 (0xad5b bytes) 000574 1787382769.249261 read 0x02b8c4 thru 0x03022a (0x4967 bytes) 000575 1787382769.267288 write 0x03490f thru 0x03ffff (0xb6f1 bytes) 000576 1787382769.271010 write 0x02a8c4 thru 0x0307bb (0x5ef8 bytes) 000577 1787382769.273978 trunc from 0x040000 to 0x01c061 (0x23f9e bytes) 000578 1787382769.292535 read 0x01aea8 thru 0x01c061 (0x11ba bytes) 000579 1787382769.299779 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1c062): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c062): Mirror 2: 0x1a2948e1 (stale) Mirror 3: 0x1a2948e1 (stale) Mirror 4: 0x1a2948e1 (stale) Mirror 5: 0xf5a42021 (stale) CRC-32 checksum value for chunk [0, 0x1c062): Mirror 1: 0x88201eb3 Mirror 6: 0x88201eb3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c062: skipped 000581 1787382770.287120 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1c062): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c062): Mirror 2: 0x1a2948e1 (stale) Mirror 3: 0x1a2948e1 (stale) Mirror 4: 0x1a2948e1 (stale) Mirror 5: 0xf5a42021 (stale) CRC-32 checksum value for chunk [0, 0x1c062): Mirror 1: 0x88201eb3 Mirror 6: 0x88201eb3 Mirror 7: 0x88201eb3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c062: skipped 000582 1787382771.764109 mapread 0x00a7d2 thru 0x010561 (0x5d90 bytes) 000583 1787382771.795949 trunc from 0x001554 to 0x00f8ff (0xe3ac bytes) 000584 1787382771.906755 trunc from 0x013900 to 0x01c061 (0x8762 bytes) 000585 1787382771.922351 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1c062): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c062): Mirror 2: 0x1a2948e1 (stale) Mirror 3: 0x1a2948e1 (stale) Mirror 4: 0x1a2948e1 (stale) Mirror 5: 0xf5a42021 (stale) Mirror 6: 0x88201eb3 (stale) Mirror 7: 0x88201eb3 (stale) CRC-32 checksum value for chunk [0, 0x1c062): Mirror 1: 0x27aa2efc Mirror 8: 0x27aa2efc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c062: skipped 000586 1787382773.460086 fallocate 0x03a26c thru 0x03ffff (0x5d94 bytes) 000587 1787382773.524207 mapread 0x00c946 thru 0x0126fe (0x5db9 bytes) 000589 1787382773.550080 read 0x015c32 thru 0x01ac10 (0x4fdf bytes) 000590 1787382773.569863 read 0x0325b4 thru 0x03ffff (0xda4c bytes) 000591 1787382773.581844 fallocate 0x00a836 thru 0x00da9e (0x3269 bytes) 000593 1787382773.591983 punch from 0x0137ac to 0x01a44a (0x6c9f bytes) 000594 1787382773.610014 trunc from 0x03114a to 0x03a3b6 (0x926d bytes) 000596 1787382773.621277 read 0x0137a0 thru 0x013a20 (0x281 bytes) 000598 1787382773.649895 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc9b8f566 (stale) Mirror 3: 0xc9b8f566 (stale) Mirror 4: 0xc9b8f566 (stale) Mirror 5: 0x8b686242 (stale) Mirror 6: 0x9d2509e7 (stale) Mirror 7: 0x9d2509e7 (stale) Mirror 8: 0x8966d571 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x61c3a31f Mirror 9: 0x61c3a31f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000599 1787382775.063460 write 0x024334 thru 0x02c790 (0x845d bytes) 000600 1787382775.154033 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc9b8f566 (stale) Mirror 3: 0xc9b8f566 (stale) Mirror 4: 0xc9b8f566 (stale) Mirror 5: 0x8b686242 (stale) Mirror 6: 0x9d2509e7 (stale) Mirror 7: 0x9d2509e7 (stale) Mirror 8: 0x8966d571 (stale) Mirror 9: 0x61c3a31f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9e948f8c Mirror 10: 0x9e948f8c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000601 1787382777.001084 write 0x02030d thru 0x02258d (0x2281 bytes) 000602 1787382777.108453 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc9b8f566 (stale) Mirror 3: 0xc9b8f566 (stale) Mirror 4: 0xc9b8f566 (stale) Mirror 5: 0x8b686242 (stale) Mirror 6: 0x9d2509e7 (stale) Mirror 7: 0x9d2509e7 (stale) Mirror 8: 0x8966d571 (stale) Mirror 9: 0x61c3a31f (stale) Mirror 10: 0x9e948f8c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcc933a4f Mirror 11: 0xcc933a4f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000603 1787382778.715573 mapread 0x01bbbc thru 0x0228bc (0x6d01 bytes) 000604 1787382778.782641 write 0x03b342 thru 0x03ffff (0x4cbe bytes) 000605 1787382778.832048 write 0x00c4af thru 0x00dfdb (0x1b2d bytes) 000606 1787382778.848470 MIRROR_RESYNC 11 mirrors 000607 1787382783.419788 read 0x026e25 thru 0x02a2ba (0x3496 bytes) 000608 1787382783.427459 trunc from 0x000113 to 0x006ef4 (0x6de2 bytes) 000609 1787382783.515752 MIRROR_RESYNC 11 mirrors 000610 1787382787.425343 fallocate 0x00b44a thru 0x012759 (0x7310 bytes) 000611 1787382787.502025 punch from 0x03c966 to 0x03ffff (0x369a bytes) 000612 1787382787.516806 trunc from 0x03647c to 0x03c984 (0x6509 bytes) 000613 1787382787.523385 mapread 0x011ce6 thru 0x014087 (0x23a2 bytes) 000615 1787382787.566594 trunc from 0x00b3b9 to 0x019f73 (0xebbb bytes) 000616 1787382787.574685 read 0x00184c thru 0x0081cb (0x6980 bytes) 000617 1787382787.593770 trunc from 0x03073c to 0x031f46 (0x180b bytes) 000618 1787382787.602814 trunc from 0x033969 to 0x034467 (0xaff bytes) 000620 1787382787.617313 mapread 0x023f26 thru 0x02adfb (0x6ed6 bytes) 000621 1787382787.637334 trunc from 0x040000 to 0x002c26 (0x3d3d9 bytes) 000622 1787382787.664908 mapread 0x000b81 thru 0x002c26 (0x20a6 bytes) 000623 1787382787.678601 mapwrite 0x006e60 thru 0x00fb99 (0x8d3a bytes) 000624 1787382787.735915 read 0x003393 thru 0x007803 (0x4471 bytes) 000626 1787382787.738686 punch from 0x00d485 to 0x00fb99 (0x2715 bytes) 000627 1787382787.752345 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xfb9a): Mirror 3: 0x89eb58b2 (stale) Mirror 4: 0x89eb58b2 (stale) Mirror 5: 0x89eb58b2 (stale) Mirror 6: 0x89eb58b2 (stale) Mirror 7: 0x89eb58b2 (stale) Mirror 8: 0x89eb58b2 (stale) Mirror 9: 0x89eb58b2 (stale) Mirror 10: 0x89eb58b2 (stale) Mirror 11: 0x89eb58b2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfb9a): Mirror 3: 0x89eb58b2 (stale) Mirror 4: 0x89eb58b2 (stale) Mirror 5: 0x89eb58b2 (stale) Mirror 6: 0x89eb58b2 (stale) Mirror 7: 0x89eb58b2 (stale) Mirror 8: 0x89eb58b2 (stale) Mirror 9: 0x89eb58b2 (stale) Mirror 10: 0x89eb58b2 (stale) Mirror 11: 0x89eb58b2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfb9a: skipped 000628 1787382788.291793 punch from 0x00fb7d to 0x00fb99 (0x1d bytes) 000629 1787382788.297362 read 0x002e57 thru 0x00fb99 (0xcd43 bytes) 000630 1787382788.307756 fallocate 0x02d068 thru 0x02e58c (0x1525 bytes) 000631 1787382788.317286 write 0x025c4e thru 0x031f8b (0xc33e bytes) 000632 1787382788.332333 fallocate 0x0064d9 thru 0x00c5cc (0x60f4 bytes) 000633 1787382788.337705 mapwrite 0x01479e thru 0x01dda0 (0x9603 bytes) 000634 1787382788.405989 MIRROR_RESYNC 10 mirrors 000635 1787382791.368984 read 0x0258e5 thru 0x02f17b (0x9897 bytes) 000636 1787382791.380466 trunc from 0x031f8c to 0x003081 (0x2ef0a bytes) 000637 1787382791.439528 mapread 0x000b97 thru 0x003081 (0x24eb bytes) 000638 1787382791.463092 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3082): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3082): Mirror 3: 0x8363344f (stale) Mirror 4: 0x8363344f (stale) Mirror 5: 0x8363344f (stale) Mirror 6: 0x8363344f (stale) Mirror 7: 0x8363344f (stale) Mirror 8: 0x8363344f (stale) Mirror 9: 0x8363344f (stale) Mirror 10: 0x8363344f (stale) Mirror 11: 0x8363344f (stale) CRC-32 checksum value for chunk [0, 0x3082): Mirror 1: 0x8363344f Mirror 12: 0x8363344f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3082: skipped 000639 1787382792.536156 trunc from 0x002fe3 to 0x003081 (0x9f bytes) 000640 1787382792.616268 punch from 0x000210 to 0x003081 (0x2e72 bytes) 000641 1787382792.647998 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3082): Mirror 3: 0x8363344f (stale) Mirror 4: 0x8363344f (stale) Mirror 5: 0x8363344f (stale) Mirror 6: 0x8363344f (stale) Mirror 8: 0x8363344f (stale) Mirror 9: 0x8363344f (stale) Mirror 10: 0x8363344f (stale) Mirror 11: 0x8363344f (stale) Mirror 12: 0x8363344f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3082): Mirror 3: 0x8363344f (stale) Mirror 4: 0x8363344f (stale) Mirror 5: 0x8363344f (stale) Mirror 6: 0x8363344f (stale) Mirror 8: 0x8363344f (stale) Mirror 9: 0x8363344f (stale) Mirror 10: 0x8363344f (stale) Mirror 11: 0x8363344f (stale) Mirror 12: 0x8363344f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3082: skipped 000642 1787382793.339983 punch from 0x0019c5 to 0x003081 (0x16bd bytes) 000643 1787382793.350949 read 0x000d6d thru 0x003081 (0x2315 bytes) 000644 1787382793.366332 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3082): Mirror 3: 0x8363344f (stale) Mirror 4: 0x8363344f (stale) Mirror 6: 0x8363344f (stale) Mirror 8: 0x8363344f (stale) Mirror 9: 0x8363344f (stale) Mirror 10: 0x8363344f (stale) Mirror 11: 0x8363344f (stale) Mirror 12: 0x8363344f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3082): Mirror 3: 0x8363344f (stale) Mirror 4: 0x8363344f (stale) Mirror 6: 0x8363344f (stale) Mirror 8: 0x8363344f (stale) Mirror 9: 0x8363344f (stale) Mirror 10: 0x8363344f (stale) Mirror 11: 0x8363344f (stale) Mirror 12: 0x8363344f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3082: skipped 000645 1787382793.796041 mapread 0x001d3c thru 0x003081 (0x1346 bytes) 000646 1787382793.818614 read 0x001455 thru 0x003081 (0x1c2d bytes) 000647 1787382793.819733 mapread 0x0014a1 thru 0x003081 (0x1be1 bytes) 000648 1787382793.822932 mapwrite 0x01e97a thru 0x026ff4 (0x867b bytes) 000649 1787382793.922367 trunc from 0x026169 to 0x026873 (0x70b bytes) 000650 1787382793.934004 punch from 0x00ab95 to 0x010e3b (0x62a7 bytes) 000651 1787382793.946895 trunc from 0x026ff5 to 0x014dea (0x1220a bytes) 000654 1787382793.979163 fallocate 0x039007 thru 0x03b899 (0x2893 bytes) 000655 1787382793.987862 mapwrite 0x0247d8 thru 0x024981 (0x1aa bytes) 000656 1787382794.073772 mapwrite 0x0264a9 thru 0x02b9ce (0x5526 bytes) 000657 1787382794.134088 fallocate 0x0380d4 thru 0x03bfc9 (0x3ef6 bytes) 000658 1787382794.143593 write 0x008a07 thru 0x00bd9d (0x3397 bytes) 000659 1787382794.163990 trunc from 0x03bfca to 0x00746f (0x34b5a bytes) 000660 1787382794.184996 fallocate 0x038341 thru 0x03ffff (0x7cbf bytes) 000661 1787382794.195385 trunc from 0x007470 to 0x005557 (0x1f18 bytes) 000662 1787382794.214089 write 0x00425c thru 0x00f900 (0xb6a5 bytes) 000663 1787382794.225480 trunc from 0x00f901 to 0x03b7d0 (0x2bed0 bytes) 000664 1787382794.249580 mapread 0x01130e thru 0x01c26f (0xaf62 bytes) 000665 1787382794.282667 mapwrite 0x01614f thru 0x0253ec (0xf29e bytes) 000666 1787382794.359602 punch from 0x0249c2 to 0x0297cc (0x4e0b bytes) 000668 1787382794.367694 read 0x01904d thru 0x0201db (0x718f bytes) 000669 1787382794.368996 trunc from 0x007295 to 0x0150dc (0xde48 bytes) 000670 1787382794.379781 write 0x02dbfe thru 0x034da9 (0x71ac bytes) 000671 1787382794.395835 write 0x0051dd thru 0x0126df (0xd503 bytes) 000673 1787382794.403566 trunc from 0x03b7d1 to 0x013e1f (0x279b1 bytes) 000674 1787382794.439929 mapwrite 0x01beed thru 0x0244ff (0x8613 bytes) 000675 1787382794.641001 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x24500): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24500): Mirror 3: 0x659339c3 (stale) Mirror 4: 0x659339c3 (stale) Mirror 6: 0x659339c3 (stale) Mirror 8: 0x659339c3 (stale) Mirror 9: 0x659339c3 (stale) Mirror 10: 0x659339c3 (stale) Mirror 11: 0x659339c3 (stale) Mirror 12: 0xc4c37415 (stale) CRC-32 checksum value for chunk [0, 0x24500): Mirror 1: 0x23670e1a Mirror 13: 0x23670e1a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24500: skipped 000676 1787382796.094855 trunc from 0x024500 to 0x005f69 (0x1e596 bytes) 000677 1787382796.195687 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x5f6a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5f6a): Mirror 3: 0xbfa56091 (stale) Mirror 4: 0xbfa56091 (stale) Mirror 6: 0xbfa56091 (stale) Mirror 8: 0xbfa56091 (stale) Mirror 9: 0xbfa56091 (stale) Mirror 10: 0xbfa56091 (stale) Mirror 11: 0xbfa56091 (stale) Mirror 12: 0xbfa56091 (stale) Mirror 13: 0x621c07a5 (stale) CRC-32 checksum value for chunk [0, 0x5f6a): Mirror 1: 0x621c07a5 Mirror 14: 0x621c07a5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5f6a: skipped 000678 1787382797.581296 trunc from 0x000fd8 to 0x005f69 (0x4f92 bytes) 000679 1787382797.667136 read 0x005e6f thru 0x005f69 (0xfb bytes) 000680 1787382797.677567 trunc from 0x001b69 to 0x005f69 (0x4401 bytes) 000681 1787382797.702432 trunc from 0x0052db to 0x005f69 (0xc8f bytes) 000683 1787382797.740924 read 0x004915 thru 0x005f69 (0x1655 bytes) 000684 1787382797.758056 trunc from 0x0012bf to 0x005f69 (0x4cab bytes) 000686 1787382797.774652 fallocate 0x02f2b2 thru 0x0374ff (0x824e bytes) 000688 1787382797.791318 fallocate 0x00e9e2 thru 0x012712 (0x3d31 bytes) 000689 1787382797.811823 trunc from 0x005183 to 0x00b9c0 (0x683e bytes) 000690 1787382797.825123 trunc from 0x037500 to 0x01627f (0x21280 bytes) 000691 1787382797.893839 write 0x02269a thru 0x02ef7d (0xc8e4 bytes) 000692 1787382797.911027 trunc from 0x02d3b8 to 0x02ef7d (0x1bc6 bytes) 000693 1787382797.940995 punch from 0x004582 to 0x006314 (0x1d93 bytes) 000694 1787382797.951631 fallocate 0x038283 thru 0x03a27e (0x1ffc bytes) 000696 1787382797.992106 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a27f): Mirror 3: 0xb285d62e (stale) Mirror 4: 0xb285d62e (stale) Mirror 6: 0xb285d62e (stale) Mirror 8: 0xb285d62e (stale) Mirror 9: 0xb285d62e (stale) Mirror 11: 0xb285d62e (stale) Mirror 12: 0x558872d9 (stale) Mirror 13: 0xb5d5c537 (stale) Mirror 14: 0xaa25c449 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a27f): Mirror 3: 0xb285d62e (stale) Mirror 4: 0xb285d62e (stale) Mirror 6: 0xb285d62e (stale) Mirror 8: 0xb285d62e (stale) Mirror 9: 0xb285d62e (stale) Mirror 11: 0xb285d62e (stale) Mirror 12: 0x558872d9 (stale) Mirror 13: 0xb5d5c537 (stale) Mirror 14: 0xaa25c449 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a27f: skipped 000697 1787382798.765358 read 0x003b43 thru 0x00f809 (0xbcc7 bytes) 000698 1787382798.791474 punch from 0x02e44c to 0x0319d5 (0x358a bytes) 000699 1787382798.797643 fallocate 0x0240fb thru 0x02ec45 (0xab4b bytes) 000700 1787382798.803585 write 0x00796d thru 0x00cb3b (0x51cf bytes) 000701 1787382798.812227 mapread 0x011341 thru 0x01314b (0x1e0b bytes) 000702 1787382798.833817 read 0x02e729 thru 0x03986b (0xb143 bytes) 000703 1787382798.845031 trunc from 0x03a27f to 0x01c5b1 (0x1dccd bytes) 000704 1787382798.861249 read 0x010c13 thru 0x01a86b (0x9c59 bytes) 000705 1787382798.863101 mapwrite 0x000ea3 thru 0x0082bd (0x741b bytes) 000706 1787382798.886327 trunc from 0x019eaf to 0x01c5b1 (0x2703 bytes) 000707 1787382798.896008 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x1c5b2): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c5b2): Mirror 3: 0x68142080 (stale) Mirror 4: 0x68142080 (stale) Mirror 6: 0x68142080 (stale) Mirror 8: 0x68142080 (stale) Mirror 9: 0x68142080 (stale) Mirror 11: 0x68142080 (stale) Mirror 12: 0xf02b9183 (stale) Mirror 13: 0xcb43875b (stale) Mirror 14: 0x1e86ddd2 (stale) CRC-32 checksum value for chunk [0, 0x1c5b2): Mirror 1: 0x2aa57f0f Mirror 15: 0x2aa57f0f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c5b2: skipped 000708 1787382800.343094 trunc from 0x00eebf to 0x01368d (0x47cf bytes) 000709 1787382800.403902 mapread 0x00f036 thru 0x010269 (0x1234 bytes) 000710 1787382800.432155 mapread 0x006dac thru 0x00f8ef (0x8b44 bytes) 000711 1787382800.452431 trunc from 0x00606b to 0x010c52 (0xabe8 bytes) 000712 1787382800.466866 trunc from 0x01c5b2 to 0x028da5 (0xc7f4 bytes) 000714 1787382800.502809 read 0x00baec thru 0x00e368 (0x287d bytes) 000715 1787382800.513718 mapread 0x0115a5 thru 0x0144be (0x2f1a bytes) 000716 1787382800.526726 read 0x00fb35 thru 0x01b2c1 (0xb78d bytes) 000717 1787382800.544444 punch from 0x006b88 to 0x00d014 (0x648d bytes) 000718 1787382800.552224 read 0x012746 thru 0x01961e (0x6ed9 bytes) 000719 1787382800.554460 trunc from 0x028da6 to 0x01090a (0x1849b bytes) 000720 1787382800.578473 mapread 0x00ccc1 thru 0x00e0e3 (0x1423 bytes) 000721 1787382800.594700 trunc from 0x00db13 to 0x01090a (0x2df8 bytes) 000722 1787382800.600573 mapwrite 0x03204f thru 0x03d453 (0xb405 bytes) 000723 1787382800.679813 fallocate 0x0075fe thru 0x00ae3d (0x3840 bytes) 000724 1787382800.688752 mapwrite 0x00a452 thru 0x016985 (0xc534 bytes) 000725 1787382800.749647 trunc from 0x01f57e to 0x02cacb (0xd54e bytes) 000726 1787382800.758737 read 0x005b53 thru 0x012eb6 (0xd364 bytes) 000727 1787382800.760135 trunc from 0x00d49b to 0x01d1e1 (0xfd47 bytes) 000728 1787382800.776746 punch from 0x030661 to 0x03d453 (0xcdf3 bytes) 000729 1787382800.783866 trunc from 0x01018f to 0x0109e1 (0x853 bytes) 000730 1787382800.790713 punch from 0x001805 to 0x004fdc (0x37d8 bytes) 000731 1787382800.798164 mapread 0x01af62 thru 0x0278ac (0xc94b bytes) 000732 1787382800.811090 trunc from 0x03d454 to 0x027c2b (0x15828 bytes) 000733 1787382800.830829 punch from 0x007fef to 0x0158bc (0xd8ce bytes) 000734 1787382800.851678 punch from 0x0253e5 to 0x0254c9 (0xe5 bytes) 000735 1787382800.864917 trunc from 0x01f43d to 0x027c2b (0x87ef bytes) 000736 1787382800.878958 mapwrite 0x03e5cc thru 0x03ffff (0x1a34 bytes) 000737 1787382800.948733 mapwrite 0x00414d thru 0x0041f0 (0xa4 bytes) 000738 1787382800.996331 trunc from 0x040000 to 0x023294 (0x1cd6b bytes) 000739 1787382801.033254 trunc from 0x023295 to 0x03fb3b (0x1c8a7 bytes) 000740 1787382801.050333 trunc from 0x03fb3c to 0x0075ec (0x3854f bytes) 000742 1787382801.088763 MIRROR_RESYNC 11 mirrors 000743 1787382803.878929 mapread 0x006187 thru 0x0075ec (0x1466 bytes) 000744 1787382803.896575 trunc from 0x0075ed to 0x006974 (0xc78 bytes) 000746 1787382803.952242 read 0x003bae thru 0x006974 (0x2dc7 bytes) 000747 1787382803.963816 punch from 0x00376c to 0x006974 (0x3209 bytes) 000748 1787382803.972205 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6975): Mirror 4: 0xea72bb64 (stale) Mirror 6: 0xea72bb64 (stale) Mirror 8: 0xea72bb64 (stale) Mirror 9: 0xea72bb64 (stale) Mirror 11: 0xea72bb64 (stale) Mirror 12: 0xea72bb64 (stale) Mirror 13: 0xea72bb64 (stale) Mirror 14: 0xea72bb64 (stale) Mirror 15: 0xea72bb64 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6975): Mirror 4: 0xea72bb64 (stale) Mirror 6: 0xea72bb64 (stale) Mirror 8: 0xea72bb64 (stale) Mirror 9: 0xea72bb64 (stale) Mirror 11: 0xea72bb64 (stale) Mirror 12: 0xea72bb64 (stale) Mirror 13: 0xea72bb64 (stale) Mirror 14: 0xea72bb64 (stale) Mirror 15: 0xea72bb64 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6975: skipped 000749 1787382804.753347 trunc from 0x0066aa to 0x006974 (0x2cb bytes) 000750 1787382804.763656 punch from 0x002de7 to 0x005eec (0x3106 bytes) 000751 1787382804.779577 MIRROR_RESYNC 10 mirrors 000752 1787382807.137072 trunc from 0x006975 to 0x023001 (0x1c68d bytes) 000753 1787382807.227394 trunc from 0x023002 to 0x023c4c (0xc4b bytes) 000754 1787382807.262139 punch from 0x00e232 to 0x0175e6 (0x93b5 bytes) 000755 1787382807.268522 mapread 0x00df4c thru 0x00e66e (0x723 bytes) 000756 1787382807.290870 mapwrite 0x02d190 thru 0x031d92 (0x4c03 bytes) 000757 1787382807.359042 punch from 0x00fd34 to 0x01905b (0x9328 bytes) 000758 1787382807.367379 mapread 0x00d352 thru 0x018e13 (0xbac2 bytes) 000759 1787382807.403387 trunc from 0x003962 to 0x0058b3 (0x1f52 bytes) 000760 1787382807.429410 mapwrite 0x009bcc thru 0x00d548 (0x397d bytes) 000761 1787382807.461259 read 0x00baad thru 0x00f6c0 (0x3c14 bytes) 000762 1787382807.463069 fallocate 0x03aaf5 thru 0x03ffff (0x550b bytes) 000763 1787382807.481835 fallocate 0x0068fb thru 0x00cd16 (0x641c bytes) 000764 1787382807.488544 mapwrite 0x03aaa2 thru 0x03cce7 (0x2246 bytes) 000766 1787382807.547572 trunc from 0x040000 to 0x004ccb (0x3b334 bytes) 000767 1787382807.584844 mapread 0x0021ad thru 0x004ccb (0x2b1f bytes) 000768 1787382807.600314 trunc from 0x003ae3 to 0x004ccb (0x11e9 bytes) 000769 1787382807.604465 mapread 0x001967 thru 0x004ccb (0x3365 bytes) 000770 1787382807.615484 mapread 0x00493a thru 0x004ccb (0x392 bytes) 000771 1787382807.616667 trunc from 0x004ccc to 0x03e219 (0x3954e bytes) 000772 1787382807.660409 read 0x0232bc thru 0x0303e5 (0xd12a bytes) 000773 1787382807.690980 mapwrite 0x03c6bb thru 0x03ffff (0x3945 bytes) 000774 1787382807.747757 read 0x02ddaa thru 0x039c40 (0xbe97 bytes) 000775 1787382807.748864 write 0x02629d thru 0x031ee3 (0xbc47 bytes) 000776 1787382807.753646 mapread 0x03819d thru 0x03ffff (0x7e63 bytes) 000777 1787382807.760439 MIRROR_RESYNC 10 mirrors 000778 1787382810.609681 write 0x00fa7b thru 0x010b35 (0x10bb bytes) 000779 1787382810.719123 fallocate 0x0128af thru 0x01886a (0x5fbc bytes) 000780 1787382810.729433 punch from 0x0146eb to 0x017d7c (0x3692 bytes) 000781 1787382810.763686 MIRROR_RESYNC 10 mirrors 000782 1787382814.312197 trunc from 0x040000 to 0x000531 (0x3face bytes) 000783 1787382814.356255 trunc from 0x000532 to 0x035e72 (0x35941 bytes) 000784 1787382814.373822 punch from 0x018496 to 0x01d4cf (0x503a bytes) 000785 1787382814.378353 write 0x03115b thru 0x03ffff (0xeea5 bytes) 000786 1787382814.384640 fallocate 0x02b0a1 thru 0x034d2e (0x9c8e bytes) 000787 1787382814.389578 mapwrite 0x0245db thru 0x033c29 (0xf64f bytes) 000788 1787382814.445753 write 0x0086c1 thru 0x010809 (0x8149 bytes) 000789 1787382814.455354 fallocate 0x0354a2 thru 0x037ecf (0x2a2e bytes) 000790 1787382814.459901 mapwrite 0x012001 thru 0x015c78 (0x3c78 bytes) 000791 1787382814.482397 read 0x000d6f thru 0x00b61b (0xa8ad bytes) 000792 1787382814.491140 read 0x03a2ee thru 0x03ffff (0x5d12 bytes) 000793 1787382814.491972 punch from 0x00953f to 0x017313 (0xddd5 bytes) 000795 1787382814.498477 mapwrite 0x02fcc2 thru 0x03c94f (0xcc8e bytes) 000796 1787382814.517045 trunc from 0x040000 to 0x02e75e (0x118a1 bytes) 000798 1787382814.536189 trunc from 0x02e75f to 0x024693 (0xa0cb bytes) 000799 1787382814.559646 MIRROR_RESYNC 10 mirrors 000800 1787382816.753685 read 0x011fd7 thru 0x01a353 (0x837d bytes) 000802 1787382816.771010 trunc from 0x00dd4f to 0x019cef (0xbfa1 bytes) 000803 1787382816.831276 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24694): Mirror 4: 0x654f5f06 (stale) Mirror 8: 0x654f5f06 (stale) Mirror 9: 0x654f5f06 (stale) Mirror 11: 0x654f5f06 (stale) Mirror 12: 0x654f5f06 (stale) Mirror 13: 0x654f5f06 (stale) Mirror 14: 0x654f5f06 (stale) Mirror 15: 0x654f5f06 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24694): Mirror 4: 0x654f5f06 (stale) Mirror 8: 0x654f5f06 (stale) Mirror 9: 0x654f5f06 (stale) Mirror 11: 0x654f5f06 (stale) Mirror 12: 0x654f5f06 (stale) Mirror 13: 0x654f5f06 (stale) Mirror 14: 0x654f5f06 (stale) Mirror 15: 0x654f5f06 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24694: skipped 000804 1787382817.680576 fallocate 0x037ea2 thru 0x03ffff (0x815e bytes) 000805 1787382817.705329 mapwrite 0x0285f0 thru 0x0326c2 (0xa0d3 bytes) 000806 1787382817.809789 MIRROR_RESYNC 9 mirrors 000807 1787382820.224440 fallocate 0x03cf2d thru 0x03f1d8 (0x22ac bytes) 000808 1787382820.302677 write 0x03336b thru 0x036d9e (0x3a34 bytes) 000809 1787382820.330882 mapwrite 0x03db7c thru 0x03ffff (0x2484 bytes) 000810 1787382820.463540 MIRROR_RESYNC 9 mirrors 000811 1787382823.753463 trunc from 0x02b235 to 0x03a58f (0xf35b bytes) 000812 1787382823.786925 mapwrite 0x01ea3b thru 0x0276c3 (0x8c89 bytes) 000813 1787382823.834943 write 0x00d353 thru 0x014b9e (0x784c bytes) 000814 1787382823.849976 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xddd1a9ca (stale) Mirror 8: 0xddd1a9ca (stale) Mirror 9: 0xddd1a9ca (stale) Mirror 11: 0xddd1a9ca (stale) Mirror 12: 0xddd1a9ca (stale) Mirror 13: 0xddd1a9ca (stale) Mirror 14: 0xddd1a9ca (stale) Mirror 15: 0xddd1a9ca (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x81925fb5 Mirror 16: 0x81925fb5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000816 1787382824.911009 read 0x00ac96 thru 0x01a644 (0xf9af bytes) 000817 1787382824.932939 MIRROR_RESYNC 10 mirrors 000819 1787382828.085458 MIRROR_EXTEND to 11 mirrors 000820 1787382829.337511 trunc from 0x029256 to 0x029f0f (0xcba bytes) 000821 1787382829.415206 trunc from 0x040000 to 0x02ff2d (0x100d2 bytes) 000822 1787382829.467163 write 0x00ad15 thru 0x010bd8 (0x5ec4 bytes) 000823 1787382829.506313 trunc from 0x01ea2b to 0x02db59 (0xf12f bytes) 000824 1787382829.519276 write 0x03a0d8 thru 0x03a707 (0x630 bytes) 000825 1787382829.527847 read 0x023af3 thru 0x02e243 (0xa751 bytes) 000826 1787382829.548009 mapread 0x002121 thru 0x00e2fe (0xc1de bytes) 000827 1787382829.581119 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3a708): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a708): Mirror 4: 0xa4ae9ab3 (stale) Mirror 8: 0xa4ae9ab3 (stale) Mirror 9: 0xa4ae9ab3 (stale) Mirror 11: 0xa4ae9ab3 (stale) Mirror 12: 0xa4ae9ab3 (stale) Mirror 13: 0xa4ae9ab3 (stale) Mirror 14: 0xa4ae9ab3 (stale) Mirror 15: 0xa4ae9ab3 (stale) Mirror 16: 0xa4ae9ab3 (stale) Mirror 17: 0xa4ae9ab3 (stale) CRC-32 checksum value for chunk [0, 0x3a708): Mirror 1: 0xe476a874 Mirror 18: 0xe476a874 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a708: skipped 000829 1787382831.197478 punch from 0x001a53 to 0x00cb38 (0xb0e6 bytes) 000830 1787382831.287476 trunc from 0x03a708 to 0x0318ad (0x8e5a bytes) 000832 1787382831.315330 write 0x01c386 thru 0x02761a (0xb295 bytes) 000833 1787382831.325890 write 0x031b77 thru 0x03ac15 (0x909f bytes) 000835 1787382831.334649 punch from 0x028954 to 0x0333f1 (0xaa9e bytes) 000837 1787382831.353599 mapread 0x029de2 thru 0x032c11 (0x8e30 bytes) 000838 1787382831.373271 fallocate 0x032501 thru 0x03b4c2 (0x8fc2 bytes) 000839 1787382831.385326 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3b4c3): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b4c3): Mirror 4: 0xe239157a (stale) Mirror 8: 0xe239157a (stale) Mirror 9: 0xe239157a (stale) Mirror 11: 0xe239157a (stale) Mirror 12: 0xe239157a (stale) Mirror 13: 0xe239157a (stale) Mirror 14: 0xe239157a (stale) Mirror 15: 0xe239157a (stale) Mirror 16: 0xe239157a (stale) Mirror 17: 0xe239157a (stale) Mirror 18: 0x14625484 (stale) CRC-32 checksum value for chunk [0, 0x3b4c3): Mirror 1: 0x2016d076 Mirror 19: 0x2016d076 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b4c3: skipped 000841 1787382832.910057 trunc from 0x03b4c3 to 0x031d4d (0x9775 bytes) 000842 1787382832.966387 trunc from 0x01568c to 0x0245a4 (0xef19 bytes) 000843 1787382832.973392 read 0x024cae thru 0x02f59c (0xa8ef bytes) 000844 1787382832.985437 trunc from 0x0240e2 to 0x031d4d (0xdc6c bytes) 000845 1787382832.991805 mapwrite 0x03347c thru 0x037209 (0x3d8e bytes) 000846 1787382833.065908 trunc from 0x03720a to 0x006d0c (0x304fd bytes) 000847 1787382833.084849 trunc from 0x0052c9 to 0x006d0c (0x1a44 bytes) 000848 1787382833.093929 fallocate 0x03feb3 thru 0x03ffff (0x14d bytes) 000849 1787382833.099569 write 0x0363bf thru 0x03ffff (0x9c41 bytes) 000850 1787382833.102567 write 0x0247f2 thru 0x02e102 (0x9911 bytes) 000852 1787382833.116476 write 0x003062 thru 0x0121d7 (0xf176 bytes) 000853 1787382833.132163 read 0x03168e thru 0x036a6b (0x53de bytes) 000854 1787382833.144682 write 0x0264d9 thru 0x02b9fb (0x5523 bytes) 000855 1787382833.146458 mapwrite 0x039415 thru 0x03ffff (0x6beb bytes) 000856 1787382833.175291 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x81925fb5 (stale) Mirror 8: 0x81925fb5 (stale) Mirror 9: 0x81925fb5 (stale) Mirror 11: 0x81925fb5 (stale) Mirror 12: 0x81925fb5 (stale) Mirror 13: 0x81925fb5 (stale) Mirror 14: 0x81925fb5 (stale) Mirror 15: 0x81925fb5 (stale) Mirror 16: 0x81925fb5 (stale) Mirror 17: 0x81925fb5 (stale) Mirror 18: 0x57f8de2d (stale) Mirror 19: 0x6cefbfb4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7f711ffe Mirror 20: 0x7f711ffe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000857 1787382834.519973 write 0x01e1c9 thru 0x028fc9 (0xae01 bytes) 000858 1787382834.580221 write 0x00dcb9 thru 0x017747 (0x9a8f bytes) 000859 1787382834.591292 read 0x0109aa thru 0x01fc68 (0xf2bf bytes) 000861 1787382834.601391 trunc from 0x040000 to 0x03053d (0xfac2 bytes) 000862 1787382834.640568 punch from 0x011b85 to 0x01b6fd (0x9b79 bytes) 000863 1787382834.665242 fallocate 0x00549b thru 0x00bfec (0x6b52 bytes) 000864 1787382834.674853 MIRROR_SPLIT mirror 13 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3053e): Mirror 4: 0x73dc2443 (stale) Mirror 8: 0x73dc2443 (stale) Mirror 9: 0x73dc2443 (stale) Mirror 11: 0x73dc2443 (stale) Mirror 12: 0x73dc2443 (stale) Mirror 13: 0x73dc2443 (stale) Mirror 14: 0x73dc2443 (stale) Mirror 15: 0x73dc2443 (stale) Mirror 16: 0x73dc2443 (stale) Mirror 17: 0x73dc2443 (stale) Mirror 18: 0x4fb844ee (stale) Mirror 19: 0x5e1ff890 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3053e): Mirror 4: 0x73dc2443 (stale) Mirror 8: 0x73dc2443 (stale) Mirror 9: 0x73dc2443 (stale) Mirror 11: 0x73dc2443 (stale) Mirror 12: 0x73dc2443 (stale) Mirror 13: 0x73dc2443 (stale) Mirror 14: 0x73dc2443 (stale) Mirror 15: 0x73dc2443 (stale) Mirror 16: 0x73dc2443 (stale) Mirror 17: 0x73dc2443 (stale) Mirror 18: 0x4fb844ee (stale) Mirror 19: 0x5e1ff890 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3053e: skipped 000865 1787382835.393083 fallocate 0x017918 thru 0x01acd0 (0x33b9 bytes) 000867 1787382835.404257 read 0x021fbe thru 0x02d5d5 (0xb618 bytes) 000868 1787382835.425914 mapread 0x01f43e thru 0x02c137 (0xccfa bytes) 000869 1787382835.440271 fallocate 0x02f743 thru 0x039bc7 (0xa485 bytes) 000870 1787382835.447976 punch from 0x0318d9 to 0x039bc7 (0x82ef bytes) 000871 1787382835.455453 trunc from 0x02cb67 to 0x03283c (0x5cd6 bytes) 000872 1787382835.462195 trunc from 0x039bc8 to 0x0163ff (0x237c8 bytes) 000873 1787382835.485091 trunc from 0x0038a4 to 0x011466 (0xdbc3 bytes) 000874 1787382835.498620 trunc from 0x016400 to 0x001024 (0x153db bytes) 000875 1787382835.533763 mapread 0x00024f thru 0x001024 (0xdd6 bytes) 000876 1787382835.545354 mapwrite 0x00ef9e thru 0x01a13a (0xb19d bytes) 000877 1787382835.653555 punch from 0x017010 to 0x01a13a (0x312b bytes) 000878 1787382835.668438 fallocate 0x03d9ea thru 0x03ffff (0x2616 bytes) 000879 1787382835.688763 trunc from 0x040000 to 0x02b7c7 (0x14838 bytes) 000880 1787382835.731435 read 0x014fb9 thru 0x01e81a (0x9862 bytes) 000882 1787382835.763556 trunc from 0x026694 to 0x028615 (0x1f82 bytes) 000883 1787382835.772174 mapwrite 0x005ad3 thru 0x0105e3 (0xab11 bytes) 000884 1787382835.795532 punch from 0x00b6a4 to 0x0143c7 (0x8d24 bytes) 000885 1787382835.801202 write 0x01f8a6 thru 0x0244d0 (0x4c2b bytes) 000886 1787382835.804106 fallocate 0x02e11b thru 0x03548e (0x7374 bytes) 000887 1787382835.809272 write 0x022485 thru 0x032034 (0xfbb0 bytes) 000888 1787382835.817126 read 0x032240 thru 0x03503e (0x2dff bytes) 000889 1787382835.843150 trunc from 0x024285 to 0x02f2cc (0xb048 bytes) 000890 1787382835.873455 fallocate 0x01aa5e thru 0x01b02a (0x5cd bytes) 000891 1787382835.890806 punch from 0x008ba4 to 0x00d954 (0x4db1 bytes) 000892 1787382835.910407 punch from 0x01d2ce to 0x023922 (0x6655 bytes) 000893 1787382835.921870 trunc from 0x006015 to 0x013a08 (0xd9f4 bytes) 000894 1787382835.928302 read 0x023605 thru 0x02c5ef (0x8feb bytes) 000895 1787382835.943012 mapwrite 0x02652b thru 0x03468e (0xe164 bytes) 000896 1787382835.973928 write 0x031ef0 thru 0x0389b5 (0x6ac6 bytes) 000897 1787382835.977604 mapread 0x00b677 thru 0x0131d4 (0x7b5e bytes) 000898 1787382835.979722 mapwrite 0x03ca37 thru 0x03f779 (0x2d43 bytes) 000901 1787382836.034464 read 0x011e61 thru 0x01e81b (0xc9bb bytes) 000902 1787382836.037099 punch from 0x014e96 to 0x01cb5d (0x7cc8 bytes) 000903 1787382836.056404 punch from 0x00da89 to 0x012af2 (0x506a bytes) 000906 1787382836.073096 fallocate 0x014686 thru 0x01d824 (0x919f bytes) 000907 1787382836.090728 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x3f77a): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f77a): Mirror 4: 0x980f557c (stale) Mirror 8: 0x980f557c (stale) Mirror 9: 0x980f557c (stale) Mirror 11: 0x980f557c (stale) Mirror 12: 0x980f557c (stale) Mirror 13: 0x980f557c (stale) Mirror 14: 0x980f557c (stale) Mirror 15: 0x980f557c (stale) Mirror 16: 0x980f557c (stale) Mirror 17: 0x980f557c (stale) Mirror 18: 0x6772d1a5 (stale) Mirror 19: 0xc50e78ba (stale) CRC-32 checksum value for chunk [0, 0x3f77a): Mirror 1: 0xb7b77b4e Mirror 20: 0xb7b77b4e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f77a: skipped 000910 1787382837.639340 trunc from 0x03f77a to 0x03f2f2 (0x487 bytes) 000912 1787382837.752467 trunc from 0x02a162 to 0x03348a (0x9329 bytes) 000913 1787382837.757556 mapread 0x008262 thru 0x012313 (0xa0b2 bytes) 000914 1787382837.785327 mapread 0x036a2d thru 0x03d52d (0x6b01 bytes) 000915 1787382837.803690 trunc from 0x0354d7 to 0x0356bd (0x1e7 bytes) 000916 1787382837.814624 trunc from 0x03f2f3 to 0x0066e9 (0x38c09 bytes) 000917 1787382837.845002 trunc from 0x005a52 to 0x0066e9 (0xc98 bytes) 000918 1787382837.852162 trunc from 0x0066ea to 0x02a236 (0x23b4d bytes) 000919 1787382837.876320 mapwrite 0x01879b thru 0x023909 (0xb16f bytes) 000920 1787382837.934159 read 0x00d7a3 thru 0x00daf0 (0x34e bytes) 000921 1787382837.939666 mapwrite 0x03e07f thru 0x03ffff (0x1f81 bytes) 000922 1787382837.975294 trunc from 0x006eb5 to 0x009c8e (0x2dda bytes) 000923 1787382837.983555 write 0x0332a2 thru 0x03ffff (0xcd5e bytes) 000924 1787382837.991561 punch from 0x00f363 to 0x01a3ab (0xb049 bytes) 000925 1787382838.002608 MIRROR_RESYNC 14 mirrors 000926 1787382841.915082 trunc from 0x040000 to 0x00c13f (0x33ec0 bytes) 000927 1787382841.974405 trunc from 0x00c140 to 0x03c3fa (0x302bb bytes) 000928 1787382841.997900 punch from 0x01d760 to 0x02b375 (0xdc16 bytes) 000929 1787382842.003380 trunc from 0x03c3fb to 0x024bc9 (0x17831 bytes) 000930 1787382842.023635 punch from 0x0147de to 0x0158cb (0x10ee bytes) 000932 1787382842.033904 MIRROR_RESYNC 14 mirrors 000933 1787382844.461645 write 0x006865 thru 0x012156 (0xb8f2 bytes) 000934 1787382844.525293 mapread 0x00282a thru 0x002f86 (0x75d bytes) 000935 1787382844.536754 mapwrite 0x0376ac thru 0x03ffff (0x8954 bytes) 000936 1787382844.631197 write 0x000405 thru 0x00f705 (0xf301 bytes) 000938 1787382844.649345 mapread 0x02491f thru 0x033b47 (0xf229 bytes) 000939 1787382844.669322 punch from 0x01fdbe to 0x02d2c4 (0xd507 bytes) 000940 1787382844.679603 fallocate 0x01f738 thru 0x0253e8 (0x5cb1 bytes) 000942 1787382844.728382 MIRROR_SPLIT mirror 6 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xe20eea22 (stale) Mirror 11: 0xe20eea22 (stale) Mirror 12: 0xe20eea22 (stale) Mirror 14: 0xe20eea22 (stale) Mirror 15: 0xe20eea22 (stale) Mirror 16: 0xe20eea22 (stale) Mirror 17: 0xe20eea22 (stale) Mirror 18: 0xe20eea22 (stale) Mirror 19: 0xe20eea22 (stale) Mirror 20: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xe20eea22 (stale) Mirror 11: 0xe20eea22 (stale) Mirror 12: 0xe20eea22 (stale) Mirror 14: 0xe20eea22 (stale) Mirror 15: 0xe20eea22 (stale) Mirror 16: 0xe20eea22 (stale) Mirror 17: 0xe20eea22 (stale) Mirror 18: 0xe20eea22 (stale) Mirror 19: 0xe20eea22 (stale) Mirror 20: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000943 1787382845.721412 write 0x036891 thru 0x03ffff (0x976f bytes) 000944 1787382845.741250 MIRROR_RESYNC 13 mirrors 000945 1787382848.096439 trunc from 0x004e43 to 0x0073af (0x256d bytes) 000946 1787382848.152349 mapread 0x002dcf thru 0x00ac77 (0x7ea9 bytes) 000947 1787382848.183440 trunc from 0x00ec65 to 0x018357 (0x96f3 bytes) 000948 1787382848.202262 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x9cee3b81 (stale) Mirror 8: 0x9cee3b81 (stale) Mirror 9: 0x9cee3b81 (stale) Mirror 11: 0x9cee3b81 (stale) Mirror 12: 0x9cee3b81 (stale) Mirror 14: 0x9cee3b81 (stale) Mirror 15: 0x9cee3b81 (stale) Mirror 16: 0x9cee3b81 (stale) Mirror 17: 0x9cee3b81 (stale) Mirror 18: 0x9cee3b81 (stale) Mirror 19: 0x9cee3b81 (stale) Mirror 20: 0x9cee3b81 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xeea726a4 Mirror 21: 0xeea726a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000949 1787382849.595124 trunc from 0x040000 to 0x00f0b1 (0x30f4e bytes) 000950 1787382849.675909 MIRROR_RESYNC 14 mirrors 000951 1787382852.738385 fallocate 0x003d79 thru 0x010263 (0xc4eb bytes) 000952 1787382852.816863 mapwrite 0x02e2b8 thru 0x034727 (0x6470 bytes) 000953 1787382852.908097 MIRROR_SPLIT mirror 7 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34728): Mirror 4: 0x20d54bc8 (stale) Mirror 8: 0x20d54bc8 (stale) Mirror 9: 0x20d54bc8 (stale) Mirror 11: 0x20d54bc8 (stale) Mirror 12: 0x20d54bc8 (stale) Mirror 14: 0x20d54bc8 (stale) Mirror 16: 0x20d54bc8 (stale) Mirror 17: 0x20d54bc8 (stale) Mirror 18: 0x20d54bc8 (stale) Mirror 19: 0x20d54bc8 (stale) Mirror 20: 0x20d54bc8 (stale) Mirror 21: 0x20d54bc8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34728): Mirror 4: 0x20d54bc8 (stale) Mirror 8: 0x20d54bc8 (stale) Mirror 9: 0x20d54bc8 (stale) Mirror 11: 0x20d54bc8 (stale) Mirror 12: 0x20d54bc8 (stale) Mirror 14: 0x20d54bc8 (stale) Mirror 16: 0x20d54bc8 (stale) Mirror 17: 0x20d54bc8 (stale) Mirror 18: 0x20d54bc8 (stale) Mirror 19: 0x20d54bc8 (stale) Mirror 20: 0x20d54bc8 (stale) Mirror 21: 0x20d54bc8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34728: skipped 000954 1787382853.717474 fallocate 0x03faf7 thru 0x03ffff (0x509 bytes) 000955 1787382853.725082 trunc from 0x0235cf to 0x02f738 (0xc16a bytes) 000956 1787382853.729760 fallocate 0x002ece thru 0x00355d (0x690 bytes) 000957 1787382853.739405 mapread 0x019342 thru 0x026a55 (0xd714 bytes) 000958 1787382853.802847 trunc from 0x034728 to 0x013e04 (0x20923 bytes) 000959 1787382853.841327 mapread 0x00c2d7 thru 0x013e04 (0x7b2e bytes) 000960 1787382853.862913 punch from 0x005beb to 0x00e887 (0x8c9d bytes) 000961 1787382853.869759 trunc from 0x011592 to 0x013e04 (0x2873 bytes) 000962 1787382853.878280 trunc from 0x006377 to 0x010871 (0xa4fb bytes) 000963 1787382853.891682 write 0x035c68 thru 0x03ffff (0xa398 bytes) 000964 1787382853.895404 write 0x01d27c thru 0x02650f (0x9294 bytes) 000965 1787382853.907663 mapread 0x03dea4 thru 0x03ffff (0x215c bytes) 000966 1787382853.909172 punch from 0x013d61 to 0x023693 (0xf933 bytes) 000967 1787382853.926648 trunc from 0x00ef1b to 0x016971 (0x7a57 bytes) 000968 1787382853.931581 read 0x02ab6e thru 0x0312ed (0x6780 bytes) 000969 1787382853.946925 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xf1f85a05 (stale) Mirror 9: 0xf1f85a05 (stale) Mirror 11: 0xf1f85a05 (stale) Mirror 12: 0xf1f85a05 (stale) Mirror 14: 0xf1f85a05 (stale) Mirror 16: 0xf1f85a05 (stale) Mirror 17: 0xf1f85a05 (stale) Mirror 18: 0xf1f85a05 (stale) Mirror 19: 0xf1f85a05 (stale) Mirror 20: 0xf1f85a05 (stale) Mirror 21: 0xf1f85a05 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xf1f85a05 (stale) Mirror 9: 0xf1f85a05 (stale) Mirror 11: 0xf1f85a05 (stale) Mirror 12: 0xf1f85a05 (stale) Mirror 14: 0xf1f85a05 (stale) Mirror 16: 0xf1f85a05 (stale) Mirror 17: 0xf1f85a05 (stale) Mirror 18: 0xf1f85a05 (stale) Mirror 19: 0xf1f85a05 (stale) Mirror 20: 0xf1f85a05 (stale) Mirror 21: 0xf1f85a05 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000970 1787382854.857717 mapwrite 0x03007a thru 0x039be7 (0x9b6e bytes) 000971 1787382854.906749 read 0x01d07c thru 0x02ae39 (0xddbe bytes) 000972 1787382854.928449 MIRROR_RESYNC 12 mirrors 000973 1787382857.930004 read 0x0247ed thru 0x032e41 (0xe655 bytes) 000974 1787382857.949942 write 0x022416 thru 0x0242f3 (0x1ede bytes) 000975 1787382857.996036 punch from 0x035ba2 to 0x03ffff (0xa45e bytes) 000976 1787382858.005338 mapwrite 0x011940 thru 0x0143c0 (0x2a81 bytes) 000977 1787382858.067143 trunc from 0x040000 to 0x0029eb (0x3d614 bytes) 000979 1787382858.095151 read 0x001cf5 thru 0x0029eb (0xcf7 bytes) 000980 1787382858.106167 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x29ec): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29ec): Mirror 8: 0x1d280983 (stale) Mirror 9: 0x1d280983 (stale) Mirror 11: 0x1d280983 (stale) Mirror 12: 0x1d280983 (stale) Mirror 14: 0x1d280983 (stale) Mirror 16: 0x1d280983 (stale) Mirror 17: 0x1d280983 (stale) Mirror 18: 0x1d280983 (stale) Mirror 19: 0x1d280983 (stale) Mirror 20: 0x1d280983 (stale) Mirror 21: 0x1d280983 (stale) CRC-32 checksum value for chunk [0, 0x29ec): Mirror 1: 0x1d280983 Mirror 22: 0x1d280983 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29ec: skipped 000981 1787382858.957419 MIRROR_RESYNC 13 mirrors 000982 1787382860.243405 punch from 0x0028fa to 0x0029eb (0xf2 bytes) 000983 1787382860.322426 trunc from 0x000fd8 to 0x0029eb (0x1a14 bytes) 000984 1787382860.328709 mapread 0x001ed6 thru 0x0029eb (0xb16 bytes) 000985 1787382860.346545 write 0x009768 thru 0x00cf05 (0x379e bytes) 000986 1787382860.354307 fallocate 0x01b268 thru 0x02a9dd (0xf776 bytes) 000989 1787382860.369040 fallocate 0x02165f thru 0x023bf9 (0x259b bytes) 000990 1787382860.374232 read 0x01eaed thru 0x0295b2 (0xaac6 bytes) 000991 1787382860.390784 MIRROR_SPLIT mirror 7 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a9de): Mirror 8: 0x6b21f431 (stale) Mirror 9: 0x6b21f431 (stale) Mirror 11: 0x6b21f431 (stale) Mirror 12: 0x6b21f431 (stale) Mirror 14: 0x6b21f431 (stale) Mirror 16: 0x6b21f431 (stale) Mirror 18: 0x6b21f431 (stale) Mirror 19: 0x6b21f431 (stale) Mirror 20: 0x6b21f431 (stale) Mirror 21: 0x6b21f431 (stale) Mirror 22: 0x6b21f431 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a9de): Mirror 8: 0x6b21f431 (stale) Mirror 9: 0x6b21f431 (stale) Mirror 11: 0x6b21f431 (stale) Mirror 12: 0x6b21f431 (stale) Mirror 14: 0x6b21f431 (stale) Mirror 16: 0x6b21f431 (stale) Mirror 18: 0x6b21f431 (stale) Mirror 19: 0x6b21f431 (stale) Mirror 20: 0x6b21f431 (stale) Mirror 21: 0x6b21f431 (stale) Mirror 22: 0x6b21f431 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a9de: skipped 000992 1787382860.984579 punch from 0x017d15 to 0x0242b3 (0xc59f bytes) 000993 1787382860.991130 mapwrite 0x01c988 thru 0x01dc20 (0x1299 bytes) 000994 1787382861.014601 mapwrite 0x005f57 thru 0x009b40 (0x3bea bytes) 000995 1787382861.059563 punch from 0x023e47 to 0x024210 (0x3ca bytes) 000996 1787382861.087764 mapread 0x01682c thru 0x01f4fc (0x8cd1 bytes) 000997 1787382861.135254 MIRROR_RESYNC 12 mirrors 000998 1787382865.472103 MIRROR_RESYNC 12 mirrors 000999 1787382866.612429 fallocate 0x014016 thru 0x016653 (0x263e bytes) 001000 1787382866.675961 read 0x015a47 thru 0x023d88 (0xe342 bytes) All operations completed A-OK! PASS 70a (149s) == sanity-flr test 71: check mirror extend parent fid ==== 03:14:32 (1787382872) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x18c:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 03:14:44 (1787382884) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2cc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.224455 s, 9.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ae:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2cc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145706 s, 7.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ae:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2cd:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2cc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ae:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2cd:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2cc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2af:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 03:14:53 (1787382893) Starting client: oleg640-client.virtnet: -o user_xattr,flock 192.168.206.140@tcp:/lustre /mnt/lustre2 Starting client: oleg640-client.virtnet: -o user_xattr,flock 192.168.206.140@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5400832 .. Extending file size to 1359744 .. Extending file size to 5146912 .. Extending file size to 5463968 .. Extending file size to 6480352 .. Extending file size to 6884160 .. Extending file size to 8598144 .. Extending file size to 5444896 .. Extending file size to 7907360 .. Extending file size to 8882720 .. Extending file size to 8492288 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9180992 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 61146 61147 61148 61152 61154 192.168.206.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg640-client.virtnet /mnt/lustre2 (opts:) 192.168.206.140@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg640-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (78s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 03:16:11 (1787382971) Starting client: oleg640-client.virtnet: -o user_xattr,flock 192.168.206.140@tcp:/lustre /mnt/lustre2 Starting client: oleg640-client.virtnet: -o user_xattr,flock 192.168.206.140@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 780096 .. Extending file size to 7194144 .. Extending file size to 8909760 .. Extending file size to 3095296 .. Extending file size to 4530048 .. Extending file size to 7479776 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9178816 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 62427 62428 62430 62431 62434 done 192.168.206.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg640-client.virtnet /mnt/lustre2 (opts:) 192.168.206.140@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg640-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 03:17:33 (1787383053) Starting client: oleg640-client.virtnet: -o user_xattr,flock 192.168.206.140@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1268860 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2108 1269196 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18920 3588100 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24764 3582256 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43684 7170356 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1268860 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2108 1269196 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18920 3588100 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24764 3582256 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43684 7170356 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104526 s, 20.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.202938 s, 10.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x4bbb44b9 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg640-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 03:17:51 (1787383071) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0930052 s, 22.5 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 03:17:59 (1787383079) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x19b:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e3:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e3:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e4:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c4:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 03:18:06 (1787383086) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.103332 s, 20.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x58:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 03:18:14 (1787383094) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.12232 s, 17.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c6:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 03:18:22 (1787383102) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.134525 s, 15.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c8:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 03:18:54 (1787383134) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg640-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572638 s, 18.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x59:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x59:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5a:0x0] } PASS 204d (46s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 03:19:40 (1787383180) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg640-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0920151 s, 22.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (45s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 03:20:25 (1787383225) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg640-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg640-server: Pool lustre.test_204f created Adding targets to pool oleg640-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg640-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0989516 s, 21.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg640-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg640-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg640-server: Pool lustre.test_204f destroyed PASS 204f (70s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 03:21:36 (1787383296) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 03:21:44 (1787383304) PASS 205b (11s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 03:21:56 (1787383316) oleg640-server: Pool lustre.test_206 created Adding targets to pool oleg640-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg640-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg640-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg640-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg640-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 03:22:24 (1787383344) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685917 s, 15.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556554 s, 18.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x9b119829 Mirror 3: 0x9b119829 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 03:22:35 (1787383355) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 03:22:38 (1787383358) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 03:22:43 (1787383363) Starting client: oleg640-client.virtnet: -o user_xattr,flock 192.168.206.140@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0975811 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2600000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x23f0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x23f0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x23f0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2650000 Read this (4 bytes): BBBB 192.168.206.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg640-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 03:23:22 (1787383402) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0253596 s, 323 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 03:23:31 (1787383411) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463124 s, 22.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1ba lmm_fid: [0x200000405:0x1ba:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1ba lmm_fid: [0x200000405:0x1ba:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f4:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611578 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-22 03:23:33.000000000 -0400 Modify: 2026-08-22 03:23:33.000000000 -0400 Change: 2026-08-22 03:23:35.000000000 -0400 Birth: 2026-08-22 03:23:33.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 03:23:41 (1787383421) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681697 s, 15.4 MB/s oleg640-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg640-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg640-server: O/280000401/d21/2f5: File not found by ext2_lookup PASS 210b (25s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 03:24:06 (1787383446) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.287172 s, 143 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0510349 s, 80.3 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 22 03:24 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1bf:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 22 03:24 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 03:24:15 (1787383455) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 03:24:22 (1787383462) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1818 sec =========== 03:24:31 (1787383471) === sanity-flr: start cleanup 03:24:32 (1787383472) === === sanity-flr: finish cleanup 03:24:42 (1787383482) ===