-----============= acceptance-small: sanity-flr ============----- Sat Apr 11 01:10:51 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg228-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 01:11:04 (1775884264) === oleg228-client.virtnet: executing check_config_client /mnt/lustre oleg228-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg228-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9c9c09d14000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c9c09d14000.idle_timeout=debug disable quota as required oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 01:11:26 (1775884286) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8171] == sanity-flr test 0a: lfs mirror create with -N option == 01:11:27 (1775884287) PASS 0a (9s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:11:36 (1775884296) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:11:38 (1775884298) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:11:40 (1775884300) PASS 0d (13s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:11:54 (1775884314) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:11:56 (1775884316) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 01:11:58 (1775884318) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9c9c09d14000.stats=clear osc.lustre-OST0001-osc-ffff9c9c09d14000.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:12:03 (1775884323) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (8s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:12:10 (1775884330) PASS 0j (5s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:12:15 (1775884335) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 01:12:17 (1775884337) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (5s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:12:22 (1775884342) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 01:12:26 (1775884346) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:12:30 (1775884350) PASS 3 (5s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:12:35 (1775884355) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:12:39 (1775884359) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (4s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:12:45 (1775884365) PASS 7 (4s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:12:49 (1775884369) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.479368 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346134 s, 30.3 MB/s PASS 21 (5s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:12:54 (1775884374) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.134576 s, 62.3 MB/s PASS 22 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:12:59 (1775884379) Stopping /mnt/lustre-ost1 (opts:) on oleg228-server oleg228-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c9c09d14000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c9c09d14000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg228-server: oleg228-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg228-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg228-server oleg228-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c9c09d14000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c9c09d14000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg228-server: oleg228-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg228-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (39s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:13:38 (1775884418) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.25273 s, 33.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg228-server oleg228-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c9c09d14000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c9c09d14000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg228-server: oleg228-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg228-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (22s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:14:00 (1775884440) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg228-server oleg228-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c9c09d14000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c9c09d14000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg228-server: oleg228-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg228-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg228-server oleg228-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c9c09d14000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c9c09d14000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg228-server: oleg228-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg228-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (44s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:14:44 (1775884484) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.62513 s, 32.3 MB/s -rw-r--r-- 1 root root 52428800 Apr 11 01:14 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.614s user 0m0.000s sys 0m0.171s Stopping /mnt/lustre-ost1 (opts:) on oleg228-server oleg228-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c9c09d14000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c9c09d14000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.593s user 0m0.001s sys 0m0.168s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg228-server: oleg228-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg228-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg228-server oleg228-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c9c09d14000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c9c09d14000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.673s user 0m0.002s sys 0m0.167s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg228-server: oleg228-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg228-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg228-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c9c09d14000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c9c09d14000.ost_server_uuid in FULL state after 0 sec PASS 33b (53s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:15:37 (1775884537) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:15:39 (1775884539) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:15:41 (1775884541) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 01:15:43 (1775884543) PASS 35 (4s) == sanity-flr test 36a: write to mirrored files ========== 01:15:47 (1775884547) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.274044 s, 45.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (7s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:15:54 (1775884554) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162773 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203718 s, 51.5 MB/s PASS 36b (4s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:15:58 (1775884558) PASS 36c (5s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:16:03 (1775884563) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00083 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00100004 s, 1.0 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00143987 s, 711 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00167221 s, 612 kB/s flv=10 olv=10 PASS 36d (5s) == sanity-flr test 37: mirror I/O API verification ======= 01:16:09 (1775884569) create test files with size 15761104 .. 7352915c6bdc7250be0423101a337551 - 5aea484f8965c4164ed9d59602bd7db4 - 87c3354fcec9dba4ccf3284a7e949f97 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre PASS 37 (11s) == sanity-flr test 38: resync ============================ 01:16:20 (1775884580) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (7s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:16:28 (1775884588) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:16:31 (1775884591) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307825 s, 34.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3e:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170639 s, 61.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (5s) == sanity-flr test 41: lfs mirror resync check =========== 01:16:36 (1775884596) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0478804 s, 43.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.186468 s, 22.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0956608 s, 21.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (8s) == sanity-flr test 42: lfs mirror verify ================= 01:16:44 (1775884604) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 01:16:46 (1775884606) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:16:48 (1775884608) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359678 s, 29.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (5s) == sanity-flr test 44a: lfs mirror split check =========== 01:16:53 (1775884613) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.180967 s, 17.4 MB/s PASS 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:16:58 (1775884618) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4b:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4a:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4a:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:17:02 (1775884622) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.1718 s, 61.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:17:12 (1775884632) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9c9c09d15000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.171944 s, 61.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9c9c118a3800.xattr_cache=1 PASS 44e (7s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:17:19 (1775884639) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (20s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:17:39 (1775884659) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (21s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:18:00 (1775884680) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 01:18:02 (1775884682) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.135715 s, 23.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0916263 s, 34.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x5c896b3b (stale) Mirror 2: 0x5c896b3b (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 2561e9ac431c69b61ad988e603160a14 - mirror 1 checksum: 6b5fcd35a884d08478c08012e25ebdc2 - mirror 2 checksum: 2561e9ac431c69b61ad988e603160a14 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sat Apr 11 01:18:04 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sat Apr 11 01:18:04 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 2561e9ac431c69b61ad988e603160a14 - mirror 1 checksum: 6b5fcd35a884d08478c08012e25ebdc2 - mirror 2 checksum: 6b5fcd35a884d08478c08012e25ebdc2 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sat Apr 11 01:18:04 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sat Apr 11 01:18:04 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } PASS 48 (6s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 01:18:08 (1775884688) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139987 s, 74.9 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (4s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:18:12 (1775884692) Failing mds1 on oleg228-server Stopping /mnt/lustre-mds1 (opts:) on oleg228-server 01:18:21 (1775884701) shut down facet: mds1 facet_host: oleg228-server facet_failover_host: oleg228-server Failover mds1 to oleg228-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-MDT0000 01:18:40 (1775884720) targets are mounted 01:18:40 (1775884720) facet_failover done oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156744 s, 66.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x56:0x0] } PASS 50A (37s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:18:49 (1775884729) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0318592 s, 643 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0146563 s, 1.4 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (6s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:18:55 (1775884735) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0558185 s, 367 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0056867 s, 3.6 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (5s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:19:00 (1775884740) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00465947 s, 3.5 MB/s PASS 50c (4s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:19:04 (1775884744) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0290973 s, 704 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:19:09 (1775884749) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00622609 s, 658 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0258657 s, 158 kB/s PASS 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:19:13 (1775884753) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (13s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:19:26 (1775884766) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (22s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:19:49 (1775884789) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:20:09 (1775884809) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:20:28 (1775884828) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (4s) == sanity-flr test 70: mirror create and split race ====== 01:20:32 (1775884832) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 56422 mirror split pid 56423 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 56422 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 56423 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (64s) == sanity-flr test 70a: flr mode fsx test ================ 01:21:36 (1775884896) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 4897 000001 1775884897.885343 mapwrite 0x03b7e6 thru 0x03ffff (0x481a bytes) fallocating to largest ever: 0x1bbf8 000002 1775884897.940814 fallocate 0x010c54 thru 0x01bbf7 (0xafa4 bytes) zero_range to largest ever: 0x3c855 000003 1775884897.945797 trunc from 0x02e631 to 0x03c854 (0xe224 bytes) 000005 1775884897.949980 trunc from 0x02bed5 to 0x030b46 (0x4c72 bytes) 000006 1775884897.952843 punch from 0x00a379 to 0x01746d (0xd0f5 bytes) 000007 1775884897.960274 MIRROR_EXTEND to 2 mirrors 000008 1775884898.319131 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) 000009 1775884898.421452 punch from 0x020048 to 0x020544 (0x4fd bytes) 000010 1775884898.433179 trunc from 0x028d9f to 0x029673 (0x8d5 bytes) 000011 1775884898.436986 trunc from 0x013cc7 to 0x021bf8 (0xdf32 bytes) 000012 1775884898.441168 read 0x035c46 thru 0x037d1a (0x20d5 bytes) 000013 1775884898.449480 MIRROR_EXTEND to 2 mirrors 000014 1775884898.853639 punch from 0x035afa to 0x038aec (0x2ff3 bytes) 000015 1775884898.885080 trunc from 0x040000 to 0x02380b (0x1c7f4 bytes) 000016 1775884898.915105 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2380c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2380c): Mirror 2: 0x255e61f6 (stale) CRC-32 checksum value for chunk [0, 0x2380c): Mirror 1: 0x255e61f6 Mirror 3: 0x255e61f6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2380c: skipped 000017 1775884899.512652 MIRROR_RESYNC 3 mirrors 000019 1775884899.848238 trunc from 0x01603f to 0x01f396 (0x9358 bytes) 000021 1775884899.890517 write 0x02a40f thru 0x032989 (0x857b bytes) 000023 1775884899.894392 write 0x019472 thru 0x02385a (0xa3e9 bytes) 000024 1775884899.906155 mapread 0x005089 thru 0x00ecc6 (0x9c3e bytes) 000026 1775884899.922966 trunc from 0x03298a to 0x020ca4 (0x11ce5 bytes) 000028 1775884899.952795 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20ca5): Mirror 3: 0xc2e10bf3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20ca5): Mirror 3: 0xc2e10bf3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20ca5: skipped 000029 1775884900.183565 read 0x000d46 thru 0x00ebf5 (0xdeb0 bytes) 000030 1775884900.193558 trunc from 0x020ca5 to 0x00a66d (0x16637 bytes) 000031 1775884900.211139 trunc from 0x008fa6 to 0x00a66d (0x16c8 bytes) 000032 1775884900.215540 fallocate 0x027ecb thru 0x02f056 (0x718c bytes) 000033 1775884900.219966 write 0x013843 thru 0x019a02 (0x61c0 bytes) 000034 1775884900.229431 mapread 0x01206a thru 0x01cc88 (0xac1f bytes) 000035 1775884900.248956 mapwrite 0x002898 thru 0x011466 (0xebcf bytes) 000036 1775884900.271460 trunc from 0x00969e to 0x00dbd9 (0x453c bytes) 000037 1775884900.275064 trunc from 0x010a28 to 0x01c8c7 (0xbea0 bytes) 000038 1775884900.279426 fallocate 0x026b9a thru 0x02a3a9 (0x3810 bytes) 000039 1775884900.283853 mapread 0x02e15f thru 0x02f056 (0xef8 bytes) 000040 1775884900.292326 read 0x0282fa thru 0x02a0c5 (0x1dcc bytes) 000041 1775884900.293232 write 0x03a1ca thru 0x03ffff (0x5e36 bytes) 000042 1775884900.294262 write 0x02d337 thru 0x031060 (0x3d2a bytes) 000043 1775884900.299992 fallocate 0x017536 thru 0x0238f2 (0xc3bd bytes) 000044 1775884900.304582 trunc from 0x040000 to 0x00dc72 (0x3238d bytes) 000045 1775884900.316893 read 0x00a024 thru 0x00dc72 (0x3c4f bytes) 000046 1775884900.317788 trunc from 0x00dc73 to 0x019abc (0xbe4a bytes) 000047 1775884900.340456 write 0x026994 thru 0x031c59 (0xb2c6 bytes) 000048 1775884900.343579 mapread 0x01a02b thru 0x028209 (0xe1df bytes) 000049 1775884900.353201 mapread 0x01a2c1 thru 0x0278b1 (0xd5f1 bytes) 000051 1775884900.355723 read 0x01a642 thru 0x021429 (0x6de8 bytes) 000052 1775884900.360755 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x31c5a): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31c5a): Mirror 3: 0xd33a81f8 (stale) CRC-32 checksum value for chunk [0, 0x31c5a): Mirror 1: 0x1d7bb476 Mirror 4: 0x1d7bb476 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31c5a: skipped 000053 1775884900.915889 mapread 0x019f78 thru 0x0215bb (0x7644 bytes) 000054 1775884900.935322 trunc from 0x031c5a to 0x01654b (0x1b70e bytes) 000055 1775884900.975665 trunc from 0x01654c to 0x00c694 (0x9eb7 bytes) 000056 1775884900.987810 trunc from 0x0053b0 to 0x008aff (0x3750 bytes) 000057 1775884900.991018 read 0x000e05 thru 0x00c694 (0xb890 bytes) 000059 1775884901.000757 write 0x010bd5 thru 0x020a4f (0xfe7b bytes) 000060 1775884901.003544 write 0x034f92 thru 0x0387f3 (0x3862 bytes) 000062 1775884901.005470 write 0x00b1d5 thru 0x00d4aa (0x22d6 bytes) 000063 1775884901.009047 write 0x03eeb0 thru 0x03ffff (0x1150 bytes) 000064 1775884901.010074 fallocate 0x00423f thru 0x00899d (0x475f bytes) 000065 1775884901.016218 punch from 0x00d0a6 to 0x017831 (0xa78c bytes) 000066 1775884901.034778 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 4: 0xc2b473a0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x30628ef8 Mirror 5: 0x30628ef8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000067 1775884901.681335 mapwrite 0x00e5b1 thru 0x0148ef (0x633f bytes) 000068 1775884901.776819 read 0x01480b thru 0x017168 (0x295e bytes) fallocating to largest ever: 0x3e642 000069 1775884901.779809 fallocate 0x038f67 thru 0x03e641 (0x56db bytes) 000072 1775884901.786445 read 0x022846 thru 0x02639d (0x3b58 bytes) 000073 1775884901.803893 MIRROR_RESYNC 4 mirrors 000074 1775884902.851593 fallocate 0x00c018 thru 0x00e338 (0x2321 bytes) 000077 1775884902.875594 trunc from 0x040000 to 0x0246be (0x1b941 bytes) 000079 1775884902.899682 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x246bf): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x246bf): Mirror 3: 0x5dcdd31e (stale) Mirror 4: 0x5dcdd31e (stale) Mirror 5: 0x5dcdd31e (stale) CRC-32 checksum value for chunk [0, 0x246bf): Mirror 1: 0x5dcdd31e Mirror 6: 0x5dcdd31e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x246bf: skipped 000080 1775884903.640281 mapwrite 0x00f28b thru 0x00f6ac (0x422 bytes) 000081 1775884903.719797 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x246bf): Mirror 3: 0x5dcdd31e (stale) Mirror 4: 0x5dcdd31e (stale) Mirror 6: 0x5dcdd31e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x246bf): Mirror 3: 0x5dcdd31e (stale) Mirror 4: 0x5dcdd31e (stale) Mirror 6: 0x5dcdd31e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x246bf: skipped 000084 1775884904.032848 fallocate 0x010344 thru 0x01c5a6 (0xc263 bytes) 000085 1775884904.042279 write 0x03d0aa thru 0x03ffff (0x2f56 bytes) 000086 1775884904.048013 fallocate 0x02d3af thru 0x03b661 (0xe2b3 bytes) fallocating to largest ever: 0x40000 000087 1775884904.057145 fallocate 0x03dd1e thru 0x03ffff (0x22e2 bytes) 000088 1775884904.060822 mapwrite 0x010450 thru 0x019479 (0x902a bytes) 000089 1775884904.102347 trunc from 0x040000 to 0x037803 (0x87fc bytes) 000090 1775884904.116396 punch from 0x019538 to 0x0234ac (0x9f75 bytes) 000091 1775884904.121358 write 0x034c49 thru 0x03ffff (0xb3b7 bytes) 000092 1775884904.128857 fallocate 0x00bc6a thru 0x015664 (0x99fb bytes) 000093 1775884904.132517 write 0x00ce0b thru 0x00e384 (0x157a bytes) 000095 1775884904.134118 punch from 0x003b1a to 0x01258e (0xea75 bytes) 000096 1775884904.146786 mapwrite 0x01c6b7 thru 0x02bb69 (0xf4b3 bytes) 000097 1775884904.174190 write 0x0172c2 thru 0x01e481 (0x71c0 bytes) 000098 1775884904.177334 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xb907050c (stale) Mirror 4: 0xb907050c (stale) Mirror 6: 0xc444d96b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x19a13e26 Mirror 7: 0x19a13e26 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000099 1775884904.706398 fallocate 0x004897 thru 0x0079ed (0x3157 bytes) 000100 1775884904.745956 mapwrite 0x03919e thru 0x03dd40 (0x4ba3 bytes) 000101 1775884904.765719 trunc from 0x040000 to 0x039475 (0x6b8a bytes) 000102 1775884904.790657 write 0x039c50 thru 0x03e837 (0x4be8 bytes) 000103 1775884904.791864 write 0x00d46d thru 0x01cd9e (0xf932 bytes) 000104 1775884904.799558 trunc from 0x00afac to 0x00c843 (0x1898 bytes) 000105 1775884904.802344 trunc from 0x02fab3 to 0x03d1ad (0xd6fb bytes) 000106 1775884904.810321 fallocate 0x01043a thru 0x01b919 (0xb4e0 bytes) 000107 1775884904.813096 read 0x016ee4 thru 0x01fca4 (0x8dc1 bytes) 000108 1775884904.817479 trunc from 0x038904 to 0x03e837 (0x5f34 bytes) 000109 1775884904.821705 read 0x029d97 thru 0x036277 (0xc4e1 bytes) 000110 1775884904.831573 fallocate 0x001116 thru 0x00d929 (0xc814 bytes) 000111 1775884904.835862 punch from 0x023929 to 0x0297bf (0x5e97 bytes) 000112 1775884904.841575 read 0x02b84c thru 0x03b2cf (0xfa84 bytes) 000113 1775884904.845956 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e838): Mirror 3: 0xafb04c2f (stale) Mirror 6: 0xf60a9c7d (stale) Mirror 7: 0x21504b6c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e838): Mirror 3: 0xafb04c2f (stale) Mirror 6: 0xf60a9c7d (stale) Mirror 7: 0x21504b6c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e838: skipped 000114 1775884905.172561 read 0x02583f thru 0x02a0e3 (0x48a5 bytes) 000117 1775884905.184948 read 0x027bcb thru 0x027eb4 (0x2ea bytes) 000118 1775884905.185628 read 0x0372e1 thru 0x03962a (0x234a bytes) 000119 1775884905.198008 trunc from 0x03e838 to 0x015903 (0x28f34 bytes) 000120 1775884905.215414 fallocate 0x020f00 thru 0x023a5e (0x2b5f bytes) 000121 1775884905.221128 read 0x0179b3 thru 0x022f95 (0xb5e3 bytes) 000122 1775884905.232709 fallocate 0x009a05 thru 0x0115aa (0x7ba6 bytes) 000124 1775884905.241949 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23a5f): Mirror 3: 0x8e995241 (stale) Mirror 6: 0x8e995241 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23a5f): Mirror 3: 0x8e995241 (stale) Mirror 6: 0x8e995241 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23a5f: skipped 000125 1775884905.473151 trunc from 0x01a6a6 to 0x01e460 (0x3dbb bytes) 000126 1775884905.477359 mapread 0x01c7a0 thru 0x023a5e (0x72bf bytes) 000127 1775884905.492326 mapwrite 0x027784 thru 0x02c439 (0x4cb6 bytes) 000128 1775884905.540948 MIRROR_RESYNC 3 mirrors 000129 1775884906.209696 write 0x01ddd4 thru 0x020d0d (0x2f3a bytes) 000130 1775884906.239824 fallocate 0x00edb0 thru 0x010fd9 (0x222a bytes) 000131 1775884906.243709 punch from 0x00c1b5 to 0x00c8d0 (0x71c bytes) 000132 1775884906.253773 trunc from 0x0166b3 to 0x023955 (0xd2a3 bytes) 000133 1775884906.271200 write 0x030163 thru 0x0333d7 (0x3275 bytes) 000134 1775884906.272504 fallocate 0x038187 thru 0x03da42 (0x58bc bytes) 000135 1775884906.276841 read 0x01f0c0 thru 0x027b8e (0x8acf bytes) 000136 1775884906.287740 read 0x039f7b thru 0x03da42 (0x3ac8 bytes) 000137 1775884906.301669 write 0x00d793 thru 0x00ddd8 (0x646 bytes) 000138 1775884906.305332 MIRROR_RESYNC 3 mirrors 000139 1775884907.013896 mapwrite 0x01abad thru 0x01c206 (0x165a bytes) 000140 1775884907.100249 read 0x027653 thru 0x032d94 (0xb742 bytes) 000142 1775884907.109305 mapread 0x023fc4 thru 0x02a559 (0x6596 bytes) 000143 1775884907.114774 trunc from 0x03da43 to 0x011948 (0x2c0fa bytes) 000144 1775884907.167092 trunc from 0x00d583 to 0x011948 (0x43c6 bytes) 000145 1775884907.174911 read 0x00f62e thru 0x011948 (0x231b bytes) 000146 1775884907.183712 trunc from 0x011949 to 0x008361 (0x95e7 bytes) 000149 1775884907.203472 fallocate 0x00ecb8 thru 0x01c645 (0xd98e bytes) 000150 1775884907.216155 trunc from 0x008362 to 0x0135c9 (0xb268 bytes) 000151 1775884907.229368 punch from 0x0113b7 to 0x0124cc (0x1116 bytes) 000152 1775884907.236493 mapwrite 0x022914 thru 0x0232f3 (0x9e0 bytes) 000153 1775884907.280787 mapread 0x0007a2 thru 0x00e641 (0xdea0 bytes) 000154 1775884907.308710 fallocate 0x0239bc thru 0x02c4c8 (0x8b0d bytes) 000155 1775884907.312283 punch from 0x02196c to 0x0232f3 (0x1988 bytes) 000156 1775884907.317381 trunc from 0x0067b9 to 0x00b706 (0x4f4e bytes) 000157 1775884907.321235 mapread 0x0007f1 thru 0x003bbd (0x33cd bytes) 000158 1775884907.322320 read 0x01ecb7 thru 0x01efe3 (0x32d bytes) 000159 1775884907.322910 punch from 0x0143c9 to 0x019497 (0x50cf bytes) 000160 1775884907.329245 mapwrite 0x0017b1 thru 0x00a803 (0x9053 bytes) 000161 1775884907.359674 mapread 0x007ef2 thru 0x00a903 (0x2a12 bytes) 000162 1775884907.361415 trunc from 0x0232f4 to 0x03584c (0x12559 bytes) 000163 1775884907.384691 trunc from 0x03584d to 0x0187fc (0x1d050 bytes) 000164 1775884907.404002 trunc from 0x0187fd to 0x01644c (0x23b0 bytes) 000165 1775884907.418182 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1644d): Mirror 6: 0xaf8fedef (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1644d): Mirror 6: 0xaf8fedef (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1644d: skipped 000166 1775884907.629693 write 0x0285a2 thru 0x02cbc7 (0x4626 bytes) 000168 1775884907.636871 mapread 0x01477e thru 0x01ed7d (0xa600 bytes) 000169 1775884907.665769 punch from 0x00f968 to 0x01aa6c (0xb105 bytes) 000170 1775884907.678906 mapwrite 0x03abeb thru 0x03ffff (0x5415 bytes) 000171 1775884907.730193 trunc from 0x02acab to 0x031bfa (0x6f50 bytes) 000172 1775884907.736288 write 0x02e75e thru 0x03c45b (0xdcfe bytes) 000173 1775884907.747300 MIRROR_RESYNC 2 mirrors 000174 1775884908.391633 read 0x018bd9 thru 0x01fcf6 (0x711e bytes) 000175 1775884908.400937 trunc from 0x040000 to 0x00d904 (0x326fb bytes) 000177 1775884908.441070 read 0x003a1b thru 0x00d904 (0x9eea bytes) 000179 1775884908.454246 mapwrite 0x02d7b1 thru 0x02e3ec (0xc3c bytes) 000182 1775884908.503408 read 0x021bc6 thru 0x027cd8 (0x6113 bytes) 000183 1775884908.504550 trunc from 0x000b70 to 0x009d8a (0x921b bytes) 000185 1775884908.511080 fallocate 0x01bb10 thru 0x02617a (0xa66b bytes) 000186 1775884908.521554 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) 000187 1775884908.627445 trunc from 0x00800b to 0x00c837 (0x482d bytes) 000188 1775884908.633617 trunc from 0x00f53d to 0x013c12 (0x46d6 bytes) 000189 1775884908.643000 trunc from 0x02e3ed to 0x023b20 (0xa8cc bytes) 000190 1775884908.661106 punch from 0x01a387 to 0x021477 (0x70f1 bytes) 000191 1775884908.666345 write 0x010bc1 thru 0x0144e0 (0x3920 bytes) 000192 1775884908.674970 trunc from 0x023b21 to 0x024e4e (0x132e bytes) 000193 1775884908.696796 mapwrite 0x0066e7 thru 0x00ec47 (0x8561 bytes) 000194 1775884908.721170 write 0x03ef39 thru 0x03ffff (0x10c7 bytes) 000195 1775884908.722209 trunc from 0x040000 to 0x026289 (0x19d76 bytes) 000196 1775884908.734196 trunc from 0x00e9eb to 0x018322 (0x9938 bytes) 000197 1775884908.739103 trunc from 0x00a148 to 0x0136ef (0x95a8 bytes) 000198 1775884908.743878 trunc from 0x01a9ef to 0x026289 (0xb89b bytes) 000200 1775884908.753976 mapwrite 0x03d3dd thru 0x03ffff (0x2c23 bytes) 000201 1775884908.787906 MIRROR_EXTEND to 2 mirrors 000202 1775884909.343608 fallocate 0x028d68 thru 0x035760 (0xc9f9 bytes) 000203 1775884909.372322 mapread 0x0168b4 thru 0x01f27d (0x89ca bytes) 000204 1775884909.397946 read 0x018379 thru 0x019f18 (0x1ba0 bytes) 000205 1775884909.399807 mapread 0x020551 thru 0x02a69f (0xa14f bytes) 000206 1775884909.415390 mapread 0x0339dc thru 0x03ffff (0xc624 bytes) 000207 1775884909.431757 read 0x01d3a9 thru 0x0299c0 (0xc618 bytes) 000208 1775884909.435226 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) 000210 1775884909.544018 trunc from 0x03c98a to 0x03d457 (0xace bytes) 000211 1775884909.550398 fallocate 0x01f3e8 thru 0x02724c (0x7e65 bytes) 000214 1775884909.561916 trunc from 0x040000 to 0x021c94 (0x1e36b bytes) 000216 1775884909.575932 trunc from 0x004662 to 0x008780 (0x411f bytes) 000217 1775884909.580494 mapread 0x0064af thru 0x0098f1 (0x3443 bytes) 000218 1775884909.607173 punch from 0x007a81 to 0x00b25b (0x37db bytes) 000219 1775884909.610851 read 0x0181c8 thru 0x021c94 (0x9acd bytes) 000221 1775884909.616289 punch from 0x01a2af to 0x020efb (0x6c4d bytes) 000222 1775884909.619112 read 0x00bfb2 thru 0x016d52 (0xada1 bytes) 000223 1775884909.623628 mapread 0x01a720 thru 0x01c560 (0x1e41 bytes) 000224 1775884909.624320 write 0x0358f5 thru 0x03ffff (0xa70b bytes) 000225 1775884909.625273 trunc from 0x040000 to 0x021804 (0x1e7fb bytes) 000227 1775884909.638447 trunc from 0x021805 to 0x02c61b (0xae17 bytes) 000228 1775884909.650359 mapwrite 0x037e58 thru 0x0382b3 (0x45c bytes) 000229 1775884909.678490 trunc from 0x0382b4 to 0x03dfed (0x5d3a bytes) 000230 1775884909.691981 trunc from 0x02c2f7 to 0x038619 (0xc323 bytes) 000232 1775884909.698903 mapread 0x039c52 thru 0x03dfed (0x439c bytes) 000233 1775884909.705471 mapread 0x024a63 thru 0x032559 (0xdaf7 bytes) 000234 1775884909.727397 mapwrite 0x03e705 thru 0x03ffff (0x18fb bytes) 000235 1775884909.752746 mapread 0x0319f4 thru 0x038d2f (0x733c bytes) 000236 1775884909.773551 fallocate 0x00e10d thru 0x017299 (0x918d bytes) 000237 1775884909.777995 trunc from 0x040000 to 0x007333 (0x38ccc bytes) 000238 1775884909.797324 mapwrite 0x018c08 thru 0x021660 (0x8a59 bytes) 000239 1775884909.882927 write 0x035e7b thru 0x03ffff (0xa185 bytes) 000240 1775884909.884530 fallocate 0x027ac9 thru 0x02fe13 (0x834b bytes) 000241 1775884909.889569 punch from 0x0204be to 0x024fca (0x4b0d bytes) 000242 1775884909.894958 mapwrite 0x02bf0c thru 0x035a40 (0x9b35 bytes) 000243 1775884909.959288 mapread 0x0151fe thru 0x01d1ce (0x7fd1 bytes) 000244 1775884909.980862 read 0x01e242 thru 0x029bbd (0xb97c bytes) 000246 1775884909.993571 trunc from 0x00df55 to 0x01ba8b (0xdb37 bytes) 000247 1775884909.997945 mapread 0x004354 thru 0x00da7f (0x972c bytes) 000248 1775884910.004936 write 0x030ae1 thru 0x032b62 (0x2082 bytes) 000251 1775884910.008538 trunc from 0x0030a1 to 0x011ace (0xea2e bytes) 000252 1775884910.013053 fallocate 0x0003c2 thru 0x00bcd1 (0xb910 bytes) 000255 1775884910.023266 trunc from 0x032236 to 0x0383c0 (0x618b bytes) 000256 1775884910.035835 write 0x0068cc thru 0x01058b (0x9cc0 bytes) 000258 1775884910.048479 read 0x035a42 thru 0x03b1db (0x579a bytes) 000259 1775884910.062564 write 0x027c05 thru 0x028ccf (0x10cb bytes) 000260 1775884910.063638 trunc from 0x040000 to 0x00b0c7 (0x34f38 bytes) 000261 1775884910.085599 punch from 0x00817c to 0x00b0c7 (0x2f4c bytes) 000262 1775884910.110775 fallocate 0x0382f8 thru 0x03a404 (0x210d bytes) 000263 1775884910.132826 trunc from 0x039376 to 0x03a404 (0x108f bytes) 000264 1775884910.137919 trunc from 0x032218 to 0x032f24 (0xd0d bytes) 000265 1775884910.145371 trunc from 0x03a405 to 0x01ecd3 (0x1b731 bytes) 000266 1775884910.190158 trunc from 0x01dab6 to 0x01ecd3 (0x121e bytes) 000267 1775884910.198414 punch from 0x009ced to 0x014629 (0xa93d bytes) 000268 1775884910.202936 read 0x00c961 thru 0x017fd0 (0xb670 bytes) 000269 1775884910.212334 mapwrite 0x036f8d thru 0x037b08 (0xb7c bytes) 000270 1775884910.242077 read 0x014431 thru 0x01acb2 (0x6882 bytes) 000271 1775884910.242554 write 0x017bed thru 0x02166c (0x9a80 bytes) 000272 1775884910.243541 trunc from 0x01f50a to 0x020d62 (0x1859 bytes) 000273 1775884910.254345 fallocate 0x0324b0 thru 0x0367eb (0x433c bytes) 000274 1775884910.258882 fallocate 0x01a4ca thru 0x025f82 (0xbab9 bytes) 000275 1775884910.263428 mapwrite 0x034d6a thru 0x036de9 (0x2080 bytes) 000276 1775884910.272510 write 0x01475b thru 0x01f643 (0xaee9 bytes) 000277 1775884910.277385 read 0x01d4a3 thru 0x020e40 (0x399e bytes) 000278 1775884910.282100 punch from 0x015ddf to 0x017b0d (0x1d2f bytes) 000279 1775884910.294304 mapread 0x024abd thru 0x02d0f9 (0x863d bytes) 000280 1775884910.295697 read 0x01add5 thru 0x01af03 (0x12f bytes) 000281 1775884910.298924 trunc from 0x01a65b to 0x02a18a (0xfb30 bytes) 000282 1775884910.303843 mapread 0x001191 thru 0x00bf00 (0xad70 bytes) 000283 1775884910.305516 trunc from 0x037b09 to 0x0256f8 (0x12410 bytes) 000285 1775884910.321961 trunc from 0x00e52b to 0x015f48 (0x7a1e bytes) 000286 1775884910.326474 punch from 0x004d24 to 0x013bfe (0xeedb bytes) 000287 1775884910.332395 trunc from 0x016a83 to 0x022aeb (0xc069 bytes) 000288 1775884910.335797 mapwrite 0x022ad7 thru 0x024f1e (0x2448 bytes) 000289 1775884910.349766 trunc from 0x025644 to 0x0256f8 (0xb5 bytes) 000290 1775884910.354603 fallocate 0x01d755 thru 0x02b346 (0xdbf2 bytes) 000291 1775884910.360919 MIRROR_EXTEND to 2 mirrors 000292 1775884910.849493 mapwrite 0x01efed thru 0x0288d1 (0x98e5 bytes) 000293 1775884910.967016 write 0x00c11b thru 0x01607c (0x9f62 bytes) 000294 1775884910.980943 mapread 0x012877 thru 0x018b95 (0x631f bytes) 000295 1775884910.991388 mapread 0x015da1 thru 0x01a246 (0x44a6 bytes) 000296 1775884910.992665 mapwrite 0x0354ff thru 0x03f5b4 (0xa0b6 bytes) 000297 1775884911.054150 read 0x03bdd8 thru 0x03f5b4 (0x37dd bytes) 000298 1775884911.054952 punch from 0x02b056 to 0x035fde (0xaf89 bytes) 000299 1775884911.067950 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) 000303 1775884911.171955 punch from 0x015ae3 to 0x01ed4d (0x926b bytes) 000305 1775884911.183412 MIRROR_EXTEND to 2 mirrors 000306 1775884911.753500 punch from 0x02462f to 0x02ac00 (0x65d2 bytes) 000307 1775884911.768896 mapread 0x03abd6 thru 0x03b78e (0xbb9 bytes) 000308 1775884911.774550 mapread 0x03c21b thru 0x03ed0e (0x2af4 bytes) 000309 1775884911.785632 MIRROR_RESYNC 2 mirrors 000310 1775884912.329553 write 0x0350dd thru 0x03ffff (0xaf23 bytes) 000311 1775884912.361728 trunc from 0x0310c9 to 0x03b0ea (0xa022 bytes) 000312 1775884912.379179 read 0x01bf45 thru 0x0276f1 (0xb7ad bytes) 000313 1775884912.387749 write 0x024839 thru 0x02aeb6 (0x667e bytes) 000314 1775884912.392192 write 0x029cd2 thru 0x0375cf (0xd8fe bytes) 000315 1775884912.397410 read 0x011cb5 thru 0x016dee (0x513a bytes) 000316 1775884912.405177 mapwrite 0x03d4f9 thru 0x03ffff (0x2b07 bytes) 000318 1775884912.421162 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x893efad2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf2093ff3 Mirror 3: 0xf2093ff3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000319 1775884912.951558 punch from 0x00dcae to 0x018d0b (0xb05e bytes) 000320 1775884912.986111 read 0x0006aa thru 0x000f28 (0x87f bytes) 000321 1775884912.990495 trunc from 0x040000 to 0x022a63 (0x1d59c bytes) 000322 1775884913.052160 mapwrite 0x0031d1 thru 0x010a25 (0xd855 bytes) 000323 1775884913.158103 MIRROR_RESYNC 3 mirrors 000324 1775884913.686199 mapread 0x022122 thru 0x022a63 (0x942 bytes) 000325 1775884913.691788 fallocate 0x02b01b thru 0x0398bc (0xe8a2 bytes) 000326 1775884913.733304 read 0x033bbb thru 0x036f7b (0x33c1 bytes) 000327 1775884913.738239 trunc from 0x0398bd to 0x00b628 (0x2e294 bytes) 000329 1775884913.776108 trunc from 0x00b629 to 0x029eef (0x1e8c7 bytes) 000330 1775884913.790980 write 0x02b06f thru 0x02f2c1 (0x4253 bytes) 000331 1775884913.792709 punch from 0x00fc3b to 0x01b7e6 (0xbbac bytes) 000332 1775884913.799345 trunc from 0x02f2c2 to 0x02b4c8 (0x3df9 bytes) 000333 1775884913.814583 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2b4c9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b4c9): Mirror 2: 0x237e8c98 (stale) Mirror 3: 0x237e8c98 (stale) CRC-32 checksum value for chunk [0, 0x2b4c9): Mirror 1: 0x41801cfd Mirror 4: 0x41801cfd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b4c9: skipped 000334 1775884914.401486 trunc from 0x02b4c9 to 0x0052f0 (0x261d8 bytes) 000335 1775884914.447747 trunc from 0x002952 to 0x0052f0 (0x299f bytes) 000336 1775884914.452779 read 0x0026ba thru 0x0052f0 (0x2c37 bytes) 000337 1775884914.459126 mapwrite 0x02cf27 thru 0x039d4d (0xce27 bytes) 000339 1775884914.534356 trunc from 0x039d4e to 0x014498 (0x258b5 bytes) 000340 1775884914.547971 read 0x0119d3 thru 0x014498 (0x2ac6 bytes) 000341 1775884914.553589 punch from 0x0038a6 to 0x004dee (0x1549 bytes) 000342 1775884914.556617 trunc from 0x014499 to 0x036d37 (0x2289f bytes) 000343 1775884914.570517 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36d38): Mirror 3: 0x65c8c5a1 (stale) Mirror 4: 0x43a3c9d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36d38): Mirror 3: 0x65c8c5a1 (stale) Mirror 4: 0x43a3c9d6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36d38: skipped 000344 1775884914.770853 mapwrite 0x010235 thru 0x01c4f8 (0xc2c4 bytes) 000345 1775884914.796935 mapwrite 0x006a5a thru 0x0143f0 (0xd997 bytes) 000346 1775884914.820986 mapwrite 0x03180e thru 0x037cb6 (0x64a9 bytes) 000347 1775884914.877705 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x37cb7): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37cb7): Mirror 3: 0x8e046ad2 (stale) Mirror 4: 0x2830de51 (stale) CRC-32 checksum value for chunk [0, 0x37cb7): Mirror 1: 0x6947fbf0 Mirror 5: 0x6947fbf0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37cb7: skipped 000348 1775884915.575827 punch from 0x00a6ce to 0x00fd06 (0x5639 bytes) 000349 1775884915.611513 trunc from 0x037cb7 to 0x00233a (0x3597c bytes) 000351 1775884915.638153 mapread 0x00064b thru 0x00233a (0x1cf0 bytes) 000352 1775884915.653991 mapread 0x0004bf thru 0x00233a (0x1e7c bytes) 000353 1775884915.656048 write 0x03b742 thru 0x03c4a3 (0xd62 bytes) 000354 1775884915.657545 mapread 0x0079a8 thru 0x00f8d9 (0x7f32 bytes) 000355 1775884915.673546 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3c4a4): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c4a4): Mirror 3: 0x4f509828 (stale) Mirror 4: 0x7f8af4ae (stale) Mirror 5: 0xac44dc9a (stale) CRC-32 checksum value for chunk [0, 0x3c4a4): Mirror 1: 0xfca54486 Mirror 6: 0xfca54486 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c4a4: skipped 000357 1775884916.241615 punch from 0x01163e to 0x01c7c5 (0xb188 bytes) 000358 1775884916.277413 fallocate 0x038159 thru 0x03db4a (0x59f2 bytes) 000359 1775884916.281085 fallocate 0x03070c thru 0x0387ed (0x80e2 bytes) 000360 1775884916.284375 mapread 0x030aa4 thru 0x03c4a3 (0xba00 bytes) 000361 1775884916.303793 punch from 0x009ecf to 0x00cc19 (0x2d4b bytes) 000362 1775884916.307985 trunc from 0x0396da to 0x03c4a3 (0x2dca bytes) 000363 1775884916.313988 fallocate 0x03034c thru 0x0333b3 (0x3068 bytes) 000364 1775884916.323002 write 0x01cbb6 thru 0x021d35 (0x5180 bytes) 000366 1775884916.336848 trunc from 0x00594d to 0x006489 (0xb3d bytes) 000367 1775884916.342650 trunc from 0x035133 to 0x039314 (0x41e2 bytes) 000368 1775884916.347051 punch from 0x02aa91 to 0x035428 (0xa998 bytes) 000369 1775884916.353119 trunc from 0x03c4a4 to 0x01697a (0x25b29 bytes) 000370 1775884916.397181 mapwrite 0x00c3a1 thru 0x01078f (0x43ef bytes) 000371 1775884916.416521 read 0x00398f thru 0x0052b7 (0x1929 bytes) 000372 1775884916.420963 trunc from 0x01697b to 0x0133ca (0x35b0 bytes) 000373 1775884916.436372 fallocate 0x019c1a thru 0x01d99a (0x3d81 bytes) 000374 1775884916.440975 trunc from 0x0133cb to 0x03b322 (0x27f58 bytes) 000375 1775884916.454920 mapread 0x029379 thru 0x035243 (0xbecb bytes) 000376 1775884916.474961 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b323): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b323): Mirror 3: 0x90e793e0 (stale) Mirror 4: 0x9bc83e17 (stale) Mirror 5: 0x3501f336 (stale) Mirror 6: 0x84e0d277 (stale) CRC-32 checksum value for chunk [0, 0x3b323): Mirror 1: 0x37860bea Mirror 7: 0x37860bea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b323: skipped 000378 1775884917.083865 trunc from 0x03b323 to 0x01412a (0x271f8 bytes) 000381 1775884917.160146 trunc from 0x011f6a to 0x01412a (0x21c1 bytes) 000382 1775884917.163952 trunc from 0x010241 to 0x011d75 (0x1b35 bytes) 000383 1775884917.173330 mapwrite 0x01c6d7 thru 0x01fd1b (0x3645 bytes) 000384 1775884917.222521 mapread 0x00b9b6 thru 0x00fadd (0x4128 bytes) 000385 1775884917.228127 trunc from 0x01fd1c to 0x00f767 (0x105b4 bytes) 000386 1775884917.238431 mapwrite 0x036510 thru 0x038abf (0x25b0 bytes) 000387 1775884917.278409 write 0x015bb7 thru 0x019bd6 (0x4020 bytes) 000388 1775884917.294324 read 0x0193f0 thru 0x01f7b1 (0x63c2 bytes) 000389 1775884917.299587 trunc from 0x03308f to 0x038abf (0x5a31 bytes) 000390 1775884917.303839 trunc from 0x038ac0 to 0x024c1c (0x13ea3 bytes) 000391 1775884917.316851 mapread 0x006e66 thru 0x0090fb (0x2296 bytes) 000392 1775884917.318410 read 0x01ae9a thru 0x0227f5 (0x795c bytes) 000393 1775884917.324409 trunc from 0x024c1d to 0x032397 (0xd77b bytes) 000394 1775884917.336679 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x32398): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32398): Mirror 3: 0xc1611eb9 (stale) Mirror 4: 0xbafee66 (stale) Mirror 5: 0xfd2a000e (stale) Mirror 6: 0x86d873d8 (stale) Mirror 7: 0xf8035e26 (stale) CRC-32 checksum value for chunk [0, 0x32398): Mirror 1: 0x469cfc5e Mirror 8: 0x469cfc5e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32398: skipped 000395 1775884918.054165 write 0x01efa6 thru 0x022a1b (0x3a76 bytes) 000396 1775884918.093735 trunc from 0x032398 to 0x008d0e (0x29689 bytes) 000399 1775884918.119401 read 0x0059aa thru 0x008d0e (0x3365 bytes) 000400 1775884918.123994 fallocate 0x01aae4 thru 0x01e156 (0x3673 bytes) 000401 1775884918.127533 write 0x0226ad thru 0x03008d (0xd9e1 bytes) 000402 1775884918.129768 trunc from 0x03008e to 0x01d438 (0x12c55 bytes) 000403 1775884918.144951 write 0x00a67d thru 0x00df9c (0x3920 bytes) 000404 1775884918.160306 write 0x0336ba thru 0x03ffff (0xc946 bytes) 000405 1775884918.166874 mapwrite 0x031fb8 thru 0x037190 (0x51d9 bytes) 000406 1775884918.191514 mapread 0x014cf7 thru 0x022a27 (0xdd31 bytes) 000407 1775884918.215072 punch from 0x023bb9 to 0x024ff4 (0x143c bytes) 000409 1775884918.222554 trunc from 0x040000 to 0x021378 (0x1ec87 bytes) 000410 1775884918.238744 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x21379): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21379): Mirror 3: 0xcffa39c4 (stale) Mirror 4: 0xa81d60c9 (stale) Mirror 5: 0xad7b686c (stale) Mirror 6: 0xb1ce649d (stale) Mirror 7: 0xb22ed5ab (stale) Mirror 8: 0x88c4c979 (stale) CRC-32 checksum value for chunk [0, 0x21379): Mirror 1: 0xa81e0339 Mirror 9: 0xa81e0339 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21379: skipped 000411 1775884918.835804 punch from 0x00cbec to 0x00f5c0 (0x29d5 bytes) 000412 1775884918.868346 trunc from 0x021379 to 0x02487e (0x3506 bytes) 000413 1775884918.909757 punch from 0x011dd9 to 0x01d4e0 (0xb708 bytes) 000414 1775884918.921444 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2487f): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2487f): Mirror 3: 0xa59dc706 (stale) Mirror 4: 0x11cdfc82 (stale) Mirror 5: 0xa0dd59c2 (stale) Mirror 6: 0x44320360 (stale) Mirror 7: 0x24395215 (stale) Mirror 8: 0xe4658c86 (stale) Mirror 9: 0xd42cc41d (stale) CRC-32 checksum value for chunk [0, 0x2487f): Mirror 1: 0x62ef6c4c Mirror 10: 0x62ef6c4c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2487f: skipped 000415 1775884919.794297 write 0x026ab6 thru 0x0318ed (0xae38 bytes) 000416 1775884919.849247 read 0x022977 thru 0x029271 (0x68fb bytes) 000417 1775884919.860565 punch from 0x01c329 to 0x021a58 (0x5730 bytes) 000419 1775884919.868200 fallocate 0x038ed3 thru 0x039245 (0x373 bytes) 000420 1775884919.875580 punch from 0x024c01 to 0x025c90 (0x1090 bytes) 000422 1775884919.884238 mapwrite 0x008466 thru 0x015719 (0xd2b4 bytes) 000423 1775884919.961250 write 0x010272 thru 0x010e81 (0xc10 bytes) 000424 1775884919.971544 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x39246): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39246): Mirror 3: 0x3ad64cc6 (stale) Mirror 4: 0x5471f5b8 (stale) Mirror 5: 0xcd17e163 (stale) Mirror 6: 0x8905cfc9 (stale) Mirror 7: 0xdc2c92dc (stale) Mirror 8: 0x87ca8099 (stale) Mirror 9: 0xf4ebb213 (stale) Mirror 10: 0x4cceecba (stale) CRC-32 checksum value for chunk [0, 0x39246): Mirror 1: 0x39f3dc88 Mirror 11: 0x39f3dc88 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39246: skipped 000425 1775884920.931167 read 0x005471 thru 0x0069fa (0x158a bytes) 000426 1775884920.942744 trunc from 0x038b37 to 0x039245 (0x70f bytes) 000427 1775884920.977107 MIRROR_RESYNC 10 mirrors 000428 1775884922.759756 trunc from 0x039246 to 0x00f19a (0x2a0ab bytes) 000429 1775884922.802012 read 0x00d7e5 thru 0x00f19a (0x19b6 bytes) 000430 1775884922.811492 trunc from 0x00f19b to 0x012203 (0x3069 bytes) 000432 1775884922.823933 trunc from 0x012204 to 0x005400 (0xce03 bytes) 000433 1775884922.837477 read 0x001226 thru 0x002463 (0x123e bytes) 000434 1775884922.844728 MIRROR_RESYNC 10 mirrors 000435 1775884923.695600 MIRROR_RESYNC 10 mirrors 000436 1775884924.082921 mapread 0x001033 thru 0x005400 (0x43ce bytes) 000437 1775884924.107128 trunc from 0x0022e6 to 0x005030 (0x2d4b bytes) 000438 1775884924.150266 mapread 0x0013ba thru 0x005400 (0x4047 bytes) 000439 1775884924.158381 mapread 0x003c05 thru 0x005400 (0x17fc bytes) 000440 1775884924.159913 punch from 0x00424d to 0x005400 (0x11b4 bytes) 000444 1775884924.164970 mapwrite 0x016395 thru 0x01fe7f (0x9aeb bytes) 000445 1775884924.216473 mapwrite 0x019dab thru 0x022a85 (0x8cdb bytes) 000446 1775884924.246237 mapread 0x0120d9 thru 0x017f03 (0x5e2b bytes) 000447 1775884924.247330 read 0x00bfe4 thru 0x013fdd (0x7ffa bytes) 000448 1775884924.252601 trunc from 0x022a86 to 0x03fb90 (0x1d10b bytes) 000449 1775884924.264811 mapwrite 0x02d5d5 thru 0x035282 (0x7cae bytes) 000450 1775884924.288878 write 0x015f79 thru 0x023bbe (0xdc46 bytes) 000451 1775884924.297389 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fb91): Mirror 3: 0xa6f6f83d (stale) Mirror 4: 0xa6f6f83d (stale) Mirror 5: 0xa6f6f83d (stale) Mirror 7: 0xa6f6f83d (stale) Mirror 8: 0xa6f6f83d (stale) Mirror 9: 0xa6f6f83d (stale) Mirror 10: 0xa6f6f83d (stale) Mirror 11: 0xa6f6f83d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fb91): Mirror 3: 0xa6f6f83d (stale) Mirror 4: 0xa6f6f83d (stale) Mirror 5: 0xa6f6f83d (stale) Mirror 7: 0xa6f6f83d (stale) Mirror 8: 0xa6f6f83d (stale) Mirror 9: 0xa6f6f83d (stale) Mirror 10: 0xa6f6f83d (stale) Mirror 11: 0xa6f6f83d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fb91: skipped 000452 1775884924.709144 punch from 0x011073 to 0x01d286 (0xc214 bytes) 000453 1775884924.717500 trunc from 0x03fb91 to 0x01fa82 (0x2010e bytes) 000454 1775884924.734213 mapread 0x017716 thru 0x01fa82 (0x836d bytes) 000455 1775884924.748668 read 0x01ada5 thru 0x01f044 (0x42a0 bytes) 000457 1775884924.749474 read 0x01aa75 thru 0x01e54a (0x3ad6 bytes) 000458 1775884924.749936 write 0x028cc9 thru 0x032d81 (0xa0b9 bytes) 000459 1775884924.752607 punch from 0x02056b to 0x02355c (0x2ff2 bytes) 000460 1775884924.759689 fallocate 0x022c80 thru 0x026b1f (0x3ea0 bytes) 000461 1775884924.765394 fallocate 0x00575e thru 0x00ce63 (0x7706 bytes) 000462 1775884924.769411 read 0x0218ff thru 0x02d618 (0xbd1a bytes) 000464 1775884924.777648 write 0x022f3c thru 0x029fb0 (0x7075 bytes) 000465 1775884924.778922 read 0x0154af thru 0x01a88a (0x53dc bytes) 000466 1775884924.779587 trunc from 0x032d82 to 0x01a516 (0x1886b bytes) 000467 1775884924.798762 fallocate 0x031c53 thru 0x032d4f (0x10fd bytes) 000468 1775884924.807118 read 0x005e55 thru 0x0071d9 (0x1385 bytes) 000469 1775884924.807743 mapread 0x01a0f8 thru 0x01a516 (0x41f bytes) 000470 1775884924.808667 trunc from 0x0059e6 to 0x00c839 (0x6e54 bytes) 000471 1775884924.816928 trunc from 0x01a517 to 0x01faa7 (0x5591 bytes) 000472 1775884924.832811 read 0x01f9ea thru 0x01faa7 (0xbe bytes) 000473 1775884924.841476 fallocate 0x03ee46 thru 0x03ffff (0x11ba bytes) 000474 1775884924.846953 punch from 0x03a552 to 0x03ffff (0x5aae bytes) 000476 1775884924.854477 trunc from 0x040000 to 0x007b71 (0x3848e bytes) 000477 1775884924.869932 read 0x001494 thru 0x004518 (0x3085 bytes) 000478 1775884924.870494 mapread 0x0069c3 thru 0x007b71 (0x11af bytes) 000479 1775884924.871249 mapwrite 0x02fad7 thru 0x030155 (0x67f bytes) 000480 1775884924.901956 fallocate 0x0245e0 thru 0x02e445 (0x9e66 bytes) 000481 1775884924.906202 trunc from 0x030156 to 0x014144 (0x1c011 bytes) 000482 1775884924.917802 trunc from 0x014145 to 0x02194a (0xd806 bytes) 000483 1775884924.929218 mapread 0x0035e4 thru 0x0053a1 (0x1dbe bytes) 000484 1775884924.930255 mapread 0x0080ac thru 0x00c9cc (0x4921 bytes) 000485 1775884924.931931 fallocate 0x025781 thru 0x026eb3 (0x1733 bytes) 000486 1775884924.935886 write 0x02000c thru 0x02aebb (0xaeb0 bytes) 000487 1775884924.939836 mapread 0x0123d0 thru 0x012ed4 (0xb05 bytes) 000489 1775884924.945955 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2aebc): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aebc): Mirror 3: 0x9b71374a (stale) Mirror 4: 0x9b71374a (stale) Mirror 5: 0x9b71374a (stale) Mirror 7: 0x9b71374a (stale) Mirror 8: 0x9b71374a (stale) Mirror 9: 0x9b71374a (stale) Mirror 10: 0x9b71374a (stale) Mirror 11: 0x9b71374a (stale) CRC-32 checksum value for chunk [0, 0x2aebc): Mirror 1: 0x445b15f9 Mirror 12: 0x445b15f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aebc: skipped 000490 1775884925.546080 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2aebc): Mirror 4: 0x9b71374a (stale) Mirror 5: 0x9b71374a (stale) Mirror 7: 0x9b71374a (stale) Mirror 8: 0x9b71374a (stale) Mirror 9: 0x9b71374a (stale) Mirror 10: 0x9b71374a (stale) Mirror 11: 0x9b71374a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aebc): Mirror 4: 0x9b71374a (stale) Mirror 5: 0x9b71374a (stale) Mirror 7: 0x9b71374a (stale) Mirror 8: 0x9b71374a (stale) Mirror 9: 0x9b71374a (stale) Mirror 10: 0x9b71374a (stale) Mirror 11: 0x9b71374a (stale) CRC-32 checksum value for chunk [0, 0x2aebc): Mirror 1: 0x445b15f9 Mirror 12: 0x445b15f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aebc: skipped 000491 1775884926.134387 trunc from 0x0081a8 to 0x01657d (0xe3d6 bytes) 000492 1775884926.174785 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2aebc): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aebc): Mirror 4: 0x9b71374a (stale) Mirror 5: 0x9b71374a (stale) Mirror 7: 0x9b71374a (stale) Mirror 8: 0x9b71374a (stale) Mirror 9: 0x9b71374a (stale) Mirror 10: 0x9b71374a (stale) Mirror 11: 0x9b71374a (stale) Mirror 12: 0x445b15f9 (stale) CRC-32 checksum value for chunk [0, 0x2aebc): Mirror 1: 0x445b15f9 Mirror 13: 0x445b15f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aebc: skipped 000493 1775884926.817981 mapwrite 0x02c974 thru 0x0358db (0x8f68 bytes) 000494 1775884926.910328 mapwrite 0x03fc4a thru 0x03ffff (0x3b6 bytes) 000495 1775884926.957681 fallocate 0x02f2c3 thru 0x034dbf (0x5afd bytes) 000496 1775884926.961968 trunc from 0x040000 to 0x001907 (0x3e6f8 bytes) 000497 1775884926.977694 trunc from 0x000614 to 0x001907 (0x12f4 bytes) 000498 1775884926.984241 trunc from 0x001908 to 0x01a5f1 (0x18cea bytes) 000499 1775884926.995746 trunc from 0x014ef6 to 0x015340 (0x44b bytes) 000500 1775884927.000645 punch from 0x010166 to 0x0184f4 (0x838f bytes) 000501 1775884927.005884 trunc from 0x01227f to 0x01a5f1 (0x8373 bytes) 000502 1775884927.009721 mapwrite 0x034a40 thru 0x03f3db (0xa99c bytes) 000504 1775884927.079630 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3f3dc): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f3dc): Mirror 4: 0x98262ed4 (stale) Mirror 5: 0x98262ed4 (stale) Mirror 7: 0x98262ed4 (stale) Mirror 8: 0x98262ed4 (stale) Mirror 9: 0x98262ed4 (stale) Mirror 10: 0x98262ed4 (stale) Mirror 11: 0x98262ed4 (stale) Mirror 12: 0xa2ba0796 (stale) Mirror 13: 0xa2ba0796 (stale) CRC-32 checksum value for chunk [0, 0x3f3dc): Mirror 1: 0xbab102a4 Mirror 14: 0xbab102a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f3dc: skipped 000505 1775884927.690749 trunc from 0x0107cd to 0x019f51 (0x9785 bytes) 000506 1775884927.720668 read 0x0064b5 thru 0x00ecc7 (0x8813 bytes) 000507 1775884927.727886 write 0x03bfec thru 0x03ffff (0x4014 bytes) 000508 1775884927.737606 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xe20eea22 (stale) Mirror 10: 0xe20eea22 (stale) Mirror 11: 0xe20eea22 (stale) Mirror 12: 0xcc1da09 (stale) Mirror 13: 0xcc1da09 (stale) Mirror 14: 0xdf9c0e53 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xe20eea22 (stale) Mirror 10: 0xe20eea22 (stale) Mirror 11: 0xe20eea22 (stale) Mirror 12: 0xcc1da09 (stale) Mirror 13: 0xcc1da09 (stale) Mirror 14: 0xdf9c0e53 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000509 1775884928.087813 write 0x01a2da thru 0x02348f (0x91b6 bytes) 000510 1775884928.128919 mapwrite 0x031c5b thru 0x03bc5f (0xa005 bytes) 000511 1775884928.176894 trunc from 0x040000 to 0x01b07b (0x24f84 bytes) 000512 1775884928.223947 fallocate 0x03e9f2 thru 0x03ffff (0x160e bytes) 000513 1775884928.235460 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x1b07c): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b07c): Mirror 5: 0xd12d705d (stale) Mirror 7: 0xd12d705d (stale) Mirror 8: 0xd12d705d (stale) Mirror 9: 0xd12d705d (stale) Mirror 10: 0xd12d705d (stale) Mirror 11: 0xd12d705d (stale) Mirror 12: 0xd12d705d (stale) Mirror 13: 0xd12d705d (stale) Mirror 14: 0xd12d705d (stale) CRC-32 checksum value for chunk [0, 0x1b07c): Mirror 1: 0x7020005e Mirror 15: 0x7020005e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b07c: skipped 000514 1775884928.966913 write 0x0294e8 thru 0x02d2f8 (0x3e11 bytes) 000515 1775884929.009490 mapwrite 0x00a5a6 thru 0x0189c4 (0xe41f bytes) 000517 1775884929.081431 mapwrite 0x01b13f thru 0x0283bb (0xd27d bytes) 000518 1775884929.104433 trunc from 0x007bf5 to 0x00aa3e (0x2e4a bytes) 000520 1775884929.110886 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2d2f9): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d2f9): Mirror 5: 0xef2efe7a (stale) Mirror 7: 0xef2efe7a (stale) Mirror 8: 0xef2efe7a (stale) Mirror 9: 0xef2efe7a (stale) Mirror 10: 0xef2efe7a (stale) Mirror 11: 0xef2efe7a (stale) Mirror 12: 0x50412bfc (stale) Mirror 13: 0x50412bfc (stale) Mirror 14: 0xef2efe7a (stale) Mirror 15: 0x8492a680 (stale) CRC-32 checksum value for chunk [0, 0x2d2f9): Mirror 1: 0xb25b088a Mirror 16: 0xb25b088a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d2f9: skipped 000521 1775884929.555867 trunc from 0x02d2f9 to 0x038509 (0xb211 bytes) 000522 1775884929.621106 mapwrite 0x01f195 thru 0x02d6e3 (0xe54f bytes) 000523 1775884929.671175 mapwrite 0x00d96a thru 0x019127 (0xb7be bytes) 000524 1775884929.709620 write 0x016e25 thru 0x021325 (0xa501 bytes) 000525 1775884929.712482 mapread 0x01006d thru 0x01657f (0x6513 bytes) 000526 1775884929.713699 trunc from 0x03850a to 0x0372f6 (0x1213 bytes) 000527 1775884929.735359 read 0x0054c9 thru 0x01298c (0xd4c4 bytes) 000528 1775884929.742150 trunc from 0x0372f7 to 0x02b31a (0xbfdc bytes) 000529 1775884929.769885 write 0x01a1ac thru 0x01d4fe (0x3353 bytes) 000530 1775884929.770760 punch from 0x009bfe to 0x015abf (0xbec2 bytes) 000531 1775884929.775530 fallocate 0x019bcd thru 0x0292b4 (0xf6e8 bytes) 000532 1775884929.784239 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x2b31b): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b31b): Mirror 5: 0xa2fb561c (stale) Mirror 7: 0xa2fb561c (stale) Mirror 8: 0xa2fb561c (stale) Mirror 9: 0xa2fb561c (stale) Mirror 10: 0xa2fb561c (stale) Mirror 11: 0xa2fb561c (stale) Mirror 12: 0x9335a7d4 (stale) Mirror 13: 0x9335a7d4 (stale) Mirror 14: 0xa2fb561c (stale) Mirror 15: 0x737395c1 (stale) Mirror 16: 0xaed8534f (stale) CRC-32 checksum value for chunk [0, 0x2b31b): Mirror 1: 0xe9dc72bf Mirror 17: 0xe9dc72bf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b31b: skipped 000533 1775884930.502532 trunc from 0x00295d to 0x004459 (0x1afd bytes) 000534 1775884930.528692 write 0x01381f thru 0x01e495 (0xac77 bytes) 000535 1775884930.538912 fallocate 0x019c86 thru 0x01e5ef (0x496a bytes) 000536 1775884930.548828 MIRROR_RESYNC 13 mirrors 000537 1775884931.672464 mapread 0x02ab65 thru 0x02b31a (0x7b6 bytes) 000538 1775884931.684254 mapwrite 0x030912 thru 0x03bee6 (0xb5d5 bytes) 000539 1775884931.770939 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x3bee7): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bee7): Mirror 5: 0x930a220d (stale) Mirror 7: 0x930a220d (stale) Mirror 8: 0x930a220d (stale) Mirror 9: 0x930a220d (stale) Mirror 10: 0x930a220d (stale) Mirror 11: 0x930a220d (stale) Mirror 12: 0x930a220d (stale) Mirror 13: 0x930a220d (stale) Mirror 14: 0x930a220d (stale) Mirror 15: 0x930a220d (stale) Mirror 16: 0x930a220d (stale) Mirror 17: 0x930a220d (stale) CRC-32 checksum value for chunk [0, 0x3bee7): Mirror 1: 0xc9c211a6 Mirror 18: 0xc9c211a6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bee7: skipped 000540 1775884932.482357 write 0x00b2b4 thru 0x0102fe (0x504b bytes) 000541 1775884932.533807 fallocate 0x010af3 thru 0x019f76 (0x9484 bytes) 000542 1775884932.546362 MIRROR_RESYNC 14 mirrors 000543 1775884934.220748 trunc from 0x03bee7 to 0x025940 (0x165a6 bytes) 000544 1775884934.276943 MIRROR_RESYNC 14 mirrors 000545 1775884935.446812 mapread 0x0049f2 thru 0x007012 (0x2621 bytes) 000547 1775884935.470737 MIRROR_SPLIT mirror 13 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) 000548 1775884935.804367 punch from 0x006b4d to 0x00991e (0x2dd2 bytes) 000549 1775884935.844513 mapwrite 0x01ba76 thru 0x0254bf (0x9a4a bytes) 000550 1775884935.881647 mapread 0x013902 thru 0x01fa88 (0xc187 bytes) 000551 1775884935.895507 read 0x01c633 thru 0x025940 (0x930e bytes) 000552 1775884935.901543 MIRROR_SPLIT mirror 4 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25941): Mirror 5: 0x8cbf876a (stale) Mirror 7: 0x8cbf876a (stale) Mirror 8: 0x8cbf876a (stale) Mirror 10: 0x8cbf876a (stale) Mirror 11: 0x8cbf876a (stale) Mirror 12: 0x8cbf876a (stale) Mirror 13: 0x8cbf876a (stale) Mirror 14: 0x8cbf876a (stale) Mirror 15: 0x8cbf876a (stale) Mirror 16: 0x8cbf876a (stale) Mirror 17: 0x8cbf876a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25941): Mirror 5: 0x8cbf876a (stale) Mirror 7: 0x8cbf876a (stale) Mirror 8: 0x8cbf876a (stale) Mirror 10: 0x8cbf876a (stale) Mirror 11: 0x8cbf876a (stale) Mirror 12: 0x8cbf876a (stale) Mirror 13: 0x8cbf876a (stale) Mirror 14: 0x8cbf876a (stale) Mirror 15: 0x8cbf876a (stale) Mirror 16: 0x8cbf876a (stale) Mirror 17: 0x8cbf876a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25941: skipped 000553 1775884936.421700 trunc from 0x025941 to 0x010238 (0x15708 bytes) 000554 1775884936.434907 trunc from 0x00a3ff to 0x010238 (0x5e3a bytes) 000555 1775884936.441799 fallocate 0x0387c2 thru 0x03ffff (0x783e bytes) 000556 1775884936.445983 fallocate 0x0328a9 thru 0x03ffff (0xd757 bytes) 000557 1775884936.450819 read 0x00e508 thru 0x016982 (0x847b bytes) 000558 1775884936.459512 mapwrite 0x015512 thru 0x0217f3 (0xc2e2 bytes) 000559 1775884936.497536 read 0x0115bd thru 0x016c0f (0x5653 bytes) 000560 1775884936.498144 trunc from 0x040000 to 0x03f150 (0xeaf bytes) 000561 1775884936.507854 mapwrite 0x012eee thru 0x019d8f (0x6ea2 bytes) 000562 1775884936.520897 mapwrite 0x00dcb2 thru 0x012e7a (0x51c9 bytes) 000563 1775884936.537748 write 0x01f2d0 thru 0x0277a6 (0x84d7 bytes) 000564 1775884936.542063 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3f151): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f151): Mirror 5: 0x814cf59a (stale) Mirror 7: 0x814cf59a (stale) Mirror 8: 0x814cf59a (stale) Mirror 10: 0x814cf59a (stale) Mirror 11: 0x814cf59a (stale) Mirror 12: 0x814cf59a (stale) Mirror 13: 0x814cf59a (stale) Mirror 14: 0x814cf59a (stale) Mirror 15: 0x814cf59a (stale) Mirror 16: 0x814cf59a (stale) Mirror 17: 0x814cf59a (stale) CRC-32 checksum value for chunk [0, 0x3f151): Mirror 1: 0xb9347ff3 Mirror 18: 0xb9347ff3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f151: skipped 000565 1775884937.272695 punch from 0x01bd8b to 0x029235 (0xd4ab bytes) 000566 1775884937.309956 mapwrite 0x01cf5d thru 0x01ff32 (0x2fd6 bytes) 000567 1775884937.337631 MIRROR_SPLIT mirror 4 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f151): Mirror 5: 0x814cf59a (stale) Mirror 7: 0x814cf59a (stale) Mirror 8: 0x814cf59a (stale) Mirror 11: 0x814cf59a (stale) Mirror 12: 0x814cf59a (stale) Mirror 13: 0x814cf59a (stale) Mirror 14: 0x814cf59a (stale) Mirror 15: 0x814cf59a (stale) Mirror 16: 0x814cf59a (stale) Mirror 17: 0x814cf59a (stale) Mirror 18: 0xb9347ff3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f151): Mirror 5: 0x814cf59a (stale) Mirror 7: 0x814cf59a (stale) Mirror 8: 0x814cf59a (stale) Mirror 11: 0x814cf59a (stale) Mirror 12: 0x814cf59a (stale) Mirror 13: 0x814cf59a (stale) Mirror 14: 0x814cf59a (stale) Mirror 15: 0x814cf59a (stale) Mirror 16: 0x814cf59a (stale) Mirror 17: 0x814cf59a (stale) Mirror 18: 0xb9347ff3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f151: skipped 000568 1775884937.892141 read 0x010438 thru 0x015b3b (0x5704 bytes) 000569 1775884937.906772 MIRROR_SPLIT mirror 4 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f151): Mirror 5: 0x814cf59a (stale) Mirror 7: 0x814cf59a (stale) Mirror 8: 0x814cf59a (stale) Mirror 12: 0x814cf59a (stale) Mirror 13: 0x814cf59a (stale) Mirror 14: 0x814cf59a (stale) Mirror 15: 0x814cf59a (stale) Mirror 16: 0x814cf59a (stale) Mirror 17: 0x814cf59a (stale) Mirror 18: 0xb9347ff3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f151): Mirror 5: 0x814cf59a (stale) Mirror 7: 0x814cf59a (stale) Mirror 8: 0x814cf59a (stale) Mirror 12: 0x814cf59a (stale) Mirror 13: 0x814cf59a (stale) Mirror 14: 0x814cf59a (stale) Mirror 15: 0x814cf59a (stale) Mirror 16: 0x814cf59a (stale) Mirror 17: 0x814cf59a (stale) Mirror 18: 0xb9347ff3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f151: skipped 000570 1775884938.477280 mapread 0x01b239 thru 0x02aaee (0xf8b6 bytes) 000571 1775884938.540551 punch from 0x008b35 to 0x011034 (0x8500 bytes) 000572 1775884938.547210 punch from 0x007b29 to 0x00b332 (0x380a bytes) 000573 1775884938.553339 trunc from 0x038bd3 to 0x03e8a7 (0x5cd5 bytes) 000574 1775884938.559052 read 0x034cc9 thru 0x035e20 (0x1158 bytes) 000575 1775884938.564131 fallocate 0x02dfda thru 0x03a8e1 (0xc908 bytes) 000576 1775884938.569246 read 0x033302 thru 0x03f150 (0xbe4f bytes) 000577 1775884938.582343 read 0x02b6ce thru 0x02c1e6 (0xb19 bytes) 000578 1775884938.583636 mapread 0x009d2a thru 0x00dd68 (0x403f bytes) 000579 1775884938.589263 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f151): Mirror 5: 0x814cf59a (stale) Mirror 7: 0x814cf59a (stale) Mirror 8: 0x814cf59a (stale) Mirror 12: 0x814cf59a (stale) Mirror 13: 0x814cf59a (stale) Mirror 14: 0x814cf59a (stale) Mirror 16: 0x814cf59a (stale) Mirror 17: 0x814cf59a (stale) Mirror 18: 0xb9347ff3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f151): Mirror 5: 0x814cf59a (stale) Mirror 7: 0x814cf59a (stale) Mirror 8: 0x814cf59a (stale) Mirror 12: 0x814cf59a (stale) Mirror 13: 0x814cf59a (stale) Mirror 14: 0x814cf59a (stale) Mirror 16: 0x814cf59a (stale) Mirror 17: 0x814cf59a (stale) Mirror 18: 0xb9347ff3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f151: skipped 000580 1775884939.084995 trunc from 0x03f151 to 0x00cd1c (0x32434 bytes) 000581 1775884939.116420 mapwrite 0x034bc4 thru 0x03e2d3 (0x9710 bytes) 000582 1775884939.207809 mapread 0x03daea thru 0x03e2d3 (0x7ea bytes) 000583 1775884939.209837 trunc from 0x03e2d4 to 0x02422e (0x1a0a5 bytes) 000584 1775884939.228527 punch from 0x00ba7c to 0x014059 (0x85de bytes) 000585 1775884939.232204 write 0x03470c thru 0x03ffff (0xb8f4 bytes) 000586 1775884939.237433 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xbb131230 (stale) Mirror 7: 0xbb131230 (stale) Mirror 8: 0xbb131230 (stale) Mirror 12: 0xbb131230 (stale) Mirror 14: 0xbb131230 (stale) Mirror 16: 0xbb131230 (stale) Mirror 17: 0xbb131230 (stale) Mirror 18: 0x8c9439a7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xbb131230 (stale) Mirror 7: 0xbb131230 (stale) Mirror 8: 0xbb131230 (stale) Mirror 12: 0xbb131230 (stale) Mirror 14: 0xbb131230 (stale) Mirror 16: 0xbb131230 (stale) Mirror 17: 0xbb131230 (stale) Mirror 18: 0x8c9439a7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000587 1775884939.623092 trunc from 0x040000 to 0x020580 (0x1fa7f bytes) 000588 1775884939.648939 write 0x011965 thru 0x013f22 (0x25be bytes) 000589 1775884939.658714 punch from 0x00149d to 0x00f25a (0xddbe bytes) 000590 1775884939.663837 mapwrite 0x00f89b thru 0x00f8ae (0x14 bytes) 000591 1775884939.683206 mapread 0x00ed4c thru 0x01c654 (0xd909 bytes) 000592 1775884939.694492 punch from 0x01d699 to 0x020580 (0x2ee8 bytes) 000593 1775884939.699866 write 0x0119ac thru 0x01f6ea (0xdd3f bytes) 000595 1775884939.706697 write 0x036dd5 thru 0x03e5e9 (0x7815 bytes) 000596 1775884939.708596 trunc from 0x03e5ea to 0x00c229 (0x323c0 bytes) 000598 1775884939.724443 trunc from 0x00c22a to 0x03ac04 (0x2e9db bytes) 000599 1775884939.738529 trunc from 0x0311b3 to 0x03ac04 (0x9a52 bytes) 000600 1775884939.742617 mapwrite 0x033894 thru 0x03bd88 (0x84f5 bytes) 000601 1775884939.783135 write 0x03498b thru 0x03d1ce (0x8844 bytes) 000602 1775884939.785720 fallocate 0x0382bb thru 0x03fdef (0x7b35 bytes) 000606 1775884939.791600 mapwrite 0x035400 thru 0x03940f (0x4010 bytes) 000607 1775884939.804844 read 0x033ec1 thru 0x03c4eb (0x862b bytes) 000608 1775884939.805628 read 0x009cc1 thru 0x019ba8 (0xfee8 bytes) 000609 1775884939.811947 write 0x0075ce thru 0x012e18 (0xb84b bytes) 000610 1775884939.813670 write 0x033923 thru 0x036168 (0x2846 bytes) 000611 1775884939.814655 trunc from 0x03d1cf to 0x002519 (0x3acb5 bytes) 000612 1775884939.826607 mapread 0x001fc9 thru 0x002519 (0x551 bytes) 000613 1775884939.827698 fallocate 0x002a33 thru 0x0100bd (0xd68b bytes) 000614 1775884939.832225 trunc from 0x0086a9 to 0x00a676 (0x1fce bytes) 000616 1775884939.838468 fallocate 0x03cf28 thru 0x03ffff (0x30d8 bytes) 000617 1775884939.842858 mapread 0x0029d3 thru 0x008a97 (0x60c5 bytes) 000618 1775884939.866670 write 0x03c668 thru 0x03ffff (0x3998 bytes) 000619 1775884939.868330 fallocate 0x02028c thru 0x02c268 (0xbfdd bytes) 000620 1775884939.873925 fallocate 0x012e80 thru 0x022946 (0xfac7 bytes) 000621 1775884939.877671 read 0x03a173 thru 0x03de6f (0x3cfd bytes) 000622 1775884939.885097 punch from 0x02c303 to 0x038055 (0xbd53 bytes) 000623 1775884939.888707 trunc from 0x03d4c5 to 0x03ffff (0x2b3b bytes) 000624 1775884939.901857 trunc from 0x040000 to 0x03f1cd (0xe32 bytes) 000625 1775884939.917076 trunc from 0x00583e to 0x00c8a9 (0x706c bytes) 000626 1775884939.920425 mapread 0x004b9a thru 0x004d3a (0x1a1 bytes) 000627 1775884939.921356 fallocate 0x01b43e thru 0x01fb30 (0x46f3 bytes) 000628 1775884939.924510 trunc from 0x03f1ce to 0x00e318 (0x30eb5 bytes) 000629 1775884939.937759 trunc from 0x00b626 to 0x00e318 (0x2cf3 bytes) 000630 1775884939.946174 MIRROR_RESYNC 9 mirrors 000631 1775884940.838212 mapwrite 0x00d92e thru 0x00daec (0x1bf bytes) 000632 1775884940.908019 trunc from 0x00e319 to 0x01b538 (0xd220 bytes) 000633 1775884940.926369 mapread 0x0027f0 thru 0x0097ce (0x6fdf bytes) 000636 1775884940.939117 punch from 0x003f18 to 0x0130a5 (0xf18e bytes) 000637 1775884940.949389 mapwrite 0x00952c thru 0x00e6f7 (0x51cc bytes) 000638 1775884940.971812 mapwrite 0x01c238 thru 0x022e40 (0x6c09 bytes) 000639 1775884941.006081 trunc from 0x022e41 to 0x002bd5 (0x2026b bytes) 000641 1775884941.026628 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2bd6): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bd6): Mirror 5: 0x59e71525 (stale) Mirror 7: 0x59e71525 (stale) Mirror 8: 0x59e71525 (stale) Mirror 12: 0x59e71525 (stale) Mirror 14: 0x59e71525 (stale) Mirror 16: 0x59e71525 (stale) Mirror 17: 0x59e71525 (stale) Mirror 18: 0x59e71525 (stale) CRC-32 checksum value for chunk [0, 0x2bd6): Mirror 1: 0x59e71525 Mirror 19: 0x59e71525 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bd6: skipped 000642 1775884941.642386 fallocate 0x006b17 thru 0x014307 (0xd7f1 bytes) 000643 1775884941.679600 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2bd6): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bd6): Mirror 5: 0x59e71525 (stale) Mirror 7: 0x59e71525 (stale) Mirror 8: 0x59e71525 (stale) Mirror 12: 0x59e71525 (stale) Mirror 14: 0x59e71525 (stale) Mirror 16: 0x59e71525 (stale) Mirror 17: 0x59e71525 (stale) Mirror 18: 0x59e71525 (stale) Mirror 19: 0x59e71525 (stale) CRC-32 checksum value for chunk [0, 0x2bd6): Mirror 1: 0x59e71525 Mirror 20: 0x59e71525 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bd6: skipped 000644 1775884942.285706 trunc from 0x002bd6 to 0x02c745 (0x29b70 bytes) 000645 1775884942.323269 read 0x003474 thru 0x0093d8 (0x5f65 bytes) 000647 1775884942.333251 punch from 0x00ae1d to 0x01231e (0x7502 bytes) 000648 1775884942.336858 mapwrite 0x03fb75 thru 0x03ffff (0x48b bytes) 000649 1775884942.363471 punch from 0x027f0f to 0x031e46 (0x9f38 bytes) 000651 1775884942.367683 write 0x01bf05 thru 0x02576b (0x9867 bytes) 000652 1775884942.374431 punch from 0x03a0da to 0x03e01e (0x3f45 bytes) 000653 1775884942.377291 read 0x017362 thru 0x021b8c (0xa82b bytes) 000654 1775884942.383835 trunc from 0x040000 to 0x0279c6 (0x18639 bytes) 000655 1775884942.399343 read 0x007e38 thru 0x016abb (0xec84 bytes) 000656 1775884942.400084 mapread 0x0152ed thru 0x016b56 (0x186a bytes) 000658 1775884942.402755 read 0x024101 thru 0x0279c6 (0x38c6 bytes) 000659 1775884942.403908 mapwrite 0x01b0d3 thru 0x02abbb (0xfae9 bytes) 000660 1775884942.437982 trunc from 0x01c27c to 0x02abbb (0xe940 bytes) 000662 1775884942.442401 read 0x00c038 thru 0x015bfa (0x9bc3 bytes) 000663 1775884942.443105 mapread 0x0158ea thru 0x01c85b (0x6f72 bytes) 000664 1775884942.450755 mapwrite 0x0100e8 thru 0x012614 (0x252d bytes) 000665 1775884942.459335 write 0x01cc8a thru 0x029d84 (0xd0fb bytes) 000666 1775884942.460960 punch from 0x0082f0 to 0x01716b (0xee7c bytes) 000667 1775884942.466828 read 0x0246fc thru 0x02abbb (0x64c0 bytes) 000668 1775884942.470746 write 0x0338fe thru 0x03f2c6 (0xb9c9 bytes) 000669 1775884942.474532 trunc from 0x0182f3 to 0x024724 (0xc432 bytes) 000670 1775884942.488073 punch from 0x001f26 to 0x010e1c (0xeef7 bytes) 000671 1775884942.496834 fallocate 0x02eb88 thru 0x039c47 (0xb0c0 bytes) 000672 1775884942.502991 trunc from 0x02b46b to 0x03744a (0xbfe0 bytes) 000673 1775884942.507483 trunc from 0x0109d2 to 0x01b1db (0xa80a bytes) 000674 1775884942.524162 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f2c7): Mirror 7: 0x8c6a3478 (stale) Mirror 8: 0x8c6a3478 (stale) Mirror 12: 0x8c6a3478 (stale) Mirror 14: 0x8c6a3478 (stale) Mirror 16: 0x8c6a3478 (stale) Mirror 17: 0x8c6a3478 (stale) Mirror 18: 0x8c6a3478 (stale) Mirror 19: 0x8c6a3478 (stale) Mirror 20: 0x8c6a3478 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f2c7): Mirror 7: 0x8c6a3478 (stale) Mirror 8: 0x8c6a3478 (stale) Mirror 12: 0x8c6a3478 (stale) Mirror 14: 0x8c6a3478 (stale) Mirror 16: 0x8c6a3478 (stale) Mirror 17: 0x8c6a3478 (stale) Mirror 18: 0x8c6a3478 (stale) Mirror 19: 0x8c6a3478 (stale) Mirror 20: 0x8c6a3478 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f2c7: skipped 000675 1775884942.985562 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3f2c7): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f2c7): Mirror 7: 0x8c6a3478 (stale) Mirror 8: 0x8c6a3478 (stale) Mirror 12: 0x8c6a3478 (stale) Mirror 14: 0x8c6a3478 (stale) Mirror 16: 0x8c6a3478 (stale) Mirror 17: 0x8c6a3478 (stale) Mirror 18: 0x8c6a3478 (stale) Mirror 19: 0x8c6a3478 (stale) Mirror 20: 0x8c6a3478 (stale) CRC-32 checksum value for chunk [0, 0x3f2c7): Mirror 1: 0xe1cbd887 Mirror 21: 0xe1cbd887 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f2c7: skipped 000676 1775884943.739729 fallocate 0x0150d3 thru 0x015bb5 (0xae3 bytes) 000677 1775884943.780571 read 0x019f80 thru 0x02785f (0xd8e0 bytes) 000679 1775884943.800242 MIRROR_RESYNC 11 mirrors 000680 1775884945.472148 write 0x01f997 thru 0x029322 (0x998c bytes) 000681 1775884945.512542 read 0x00d710 thru 0x01043c (0x2d2d bytes) 000682 1775884945.524467 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f2c7): Mirror 7: 0xe1cbd887 (stale) Mirror 8: 0xe1cbd887 (stale) Mirror 12: 0xe1cbd887 (stale) Mirror 14: 0xe1cbd887 (stale) Mirror 16: 0xe1cbd887 (stale) Mirror 17: 0xe1cbd887 (stale) Mirror 19: 0xe1cbd887 (stale) Mirror 20: 0xe1cbd887 (stale) Mirror 21: 0xe1cbd887 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f2c7): Mirror 7: 0xe1cbd887 (stale) Mirror 8: 0xe1cbd887 (stale) Mirror 12: 0xe1cbd887 (stale) Mirror 14: 0xe1cbd887 (stale) Mirror 16: 0xe1cbd887 (stale) Mirror 17: 0xe1cbd887 (stale) Mirror 19: 0xe1cbd887 (stale) Mirror 20: 0xe1cbd887 (stale) Mirror 21: 0xe1cbd887 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f2c7: skipped 000683 1775884946.029847 mapread 0x022344 thru 0x027f38 (0x5bf5 bytes) 000684 1775884946.044134 trunc from 0x03f2c7 to 0x0036b1 (0x3bc15 bytes) 000686 1775884946.068294 MIRROR_RESYNC 10 mirrors 000690 1775884946.986456 fallocate 0x029ee9 thru 0x031a05 (0x7b1d bytes) 000691 1775884947.022702 read 0x002202 thru 0x00bc78 (0x9a77 bytes) 000692 1775884947.038710 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x31a06): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31a06): Mirror 7: 0xbe882c63 (stale) Mirror 8: 0xbe882c63 (stale) Mirror 12: 0xbe882c63 (stale) Mirror 14: 0xbe882c63 (stale) Mirror 16: 0xbe882c63 (stale) Mirror 17: 0xbe882c63 (stale) Mirror 19: 0xbe882c63 (stale) Mirror 20: 0xbe882c63 (stale) Mirror 21: 0xbe882c63 (stale) CRC-32 checksum value for chunk [0, 0x31a06): Mirror 1: 0xbe882c63 Mirror 22: 0xbe882c63 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31a06: skipped 000693 1775884947.674488 write 0x006aed thru 0x012541 (0xba55 bytes) 000695 1775884947.721243 punch from 0x00a2f1 to 0x015995 (0xb6a5 bytes) 000697 1775884947.737679 trunc from 0x00ab29 to 0x0163a7 (0xb87f bytes) 000698 1775884947.742889 punch from 0x00fbd3 to 0x012d5d (0x318b bytes) 000699 1775884947.749411 fallocate 0x0254b9 thru 0x02a6fa (0x5242 bytes) 000700 1775884947.755608 fallocate 0x022188 thru 0x031da1 (0xfc1a bytes) 000701 1775884947.765424 mapread 0x027488 thru 0x0274d1 (0x4a bytes) 000703 1775884947.782493 trunc from 0x031a06 to 0x0331fd (0x17f8 bytes) 000704 1775884947.808485 trunc from 0x0331fe to 0x01ef46 (0x142b7 bytes) 000706 1775884947.825601 write 0x036ead thru 0x03986a (0x29be bytes) 000707 1775884947.828012 trunc from 0x03986b to 0x021298 (0x185d2 bytes) 000708 1775884947.852694 write 0x02d936 thru 0x02fd60 (0x242b bytes) 000709 1775884947.857619 read 0x014ac2 thru 0x020a01 (0xbf40 bytes) 000710 1775884947.878484 mapwrite 0x036533 thru 0x038b96 (0x2664 bytes) 000711 1775884947.951240 fallocate 0x001d25 thru 0x009952 (0x7c2e bytes) 000713 1775884947.961085 MIRROR_RESYNC 11 mirrors 000714 1775884949.687785 MIRROR_EXTEND to 12 mirrors 000715 1775884950.175968 mapread 0x020d28 thru 0x023c5c (0x2f35 bytes) 000716 1775884950.195465 mapread 0x034e7a thru 0x038b96 (0x3d1d bytes) 000717 1775884950.203251 trunc from 0x01c336 to 0x01d53d (0x1208 bytes) 000718 1775884950.237825 mapread 0x01e8be thru 0x0258d8 (0x701b bytes) 000719 1775884950.244477 trunc from 0x038b97 to 0x033744 (0x5452 bytes) 000720 1775884950.260700 fallocate 0x020399 thru 0x02097b (0x5e3 bytes) 000721 1775884950.264892 mapread 0x0088c3 thru 0x00fde8 (0x7526 bytes) 000722 1775884950.272733 mapwrite 0x0239c6 thru 0x02e3a1 (0xa9dc bytes) 000724 1775884950.291316 punch from 0x00fa9b to 0x010edb (0x1441 bytes) 000725 1775884950.297766 punch from 0x00cdcd to 0x00f7a1 (0x29d5 bytes) 000726 1775884950.301098 write 0x037617 thru 0x039f0a (0x28f4 bytes) 000727 1775884950.304612 read 0x0219da thru 0x0264c3 (0x4aea bytes) 000728 1775884950.307209 write 0x005783 thru 0x009c30 (0x44ae bytes) 000729 1775884950.308838 punch from 0x0217fd to 0x027820 (0x6024 bytes) 000730 1775884950.318142 MIRROR_RESYNC 12 mirrors 000731 1775884951.934152 fallocate 0x03c793 thru 0x03ffff (0x386d bytes) 000732 1775884951.962160 write 0x01b226 thru 0x020979 (0x5754 bytes) 000734 1775884951.968823 trunc from 0x0026a3 to 0x00af2c (0x888a bytes) 000735 1775884951.975010 read 0x00a0c7 thru 0x014b78 (0xaab2 bytes) 000736 1775884951.980946 punch from 0x0167e5 to 0x01879d (0x1fb9 bytes) 000738 1775884951.984670 punch from 0x0346b9 to 0x039f0a (0x5852 bytes) 000739 1775884951.987305 mapwrite 0x02cd30 thru 0x02d269 (0x53a bytes) 000740 1775884952.007100 mapwrite 0x009de1 thru 0x00ff32 (0x6152 bytes) 000741 1775884952.030250 mapread 0x027686 thru 0x031986 (0xa301 bytes) 000742 1775884952.031487 mapwrite 0x03e795 thru 0x03ffff (0x186b bytes) 000743 1775884952.058276 trunc from 0x040000 to 0x00e104 (0x31efb bytes) 000744 1775884952.069611 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0xe105): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe105): Mirror 7: 0x80f00f1b (stale) Mirror 8: 0x80f00f1b (stale) Mirror 12: 0x80f00f1b (stale) Mirror 14: 0x80f00f1b (stale) Mirror 16: 0x80f00f1b (stale) Mirror 17: 0x80f00f1b (stale) Mirror 19: 0x80f00f1b (stale) Mirror 20: 0x80f00f1b (stale) Mirror 21: 0x80f00f1b (stale) Mirror 22: 0x80f00f1b (stale) Mirror 23: 0x80f00f1b (stale) CRC-32 checksum value for chunk [0, 0xe105): Mirror 1: 0xe0ddd401 Mirror 24: 0xe0ddd401 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe105: skipped 000745 1775884952.669499 trunc from 0x000a36 to 0x00b1c0 (0xa78b bytes) 000746 1775884952.693744 fallocate 0x011efa thru 0x01ab14 (0x8c1b bytes) 000747 1775884952.701393 mapwrite 0x02b73b thru 0x03299f (0x7265 bytes) 000748 1775884952.750854 mapwrite 0x01006d thru 0x0130fb (0x308f bytes) 000749 1775884952.764652 mapwrite 0x00dfc3 thru 0x0174c9 (0x9507 bytes) 000750 1775884952.781429 mapwrite 0x033799 thru 0x03ffff (0xc867 bytes) 000751 1775884952.819652 read 0x02871d thru 0x02906c (0x950 bytes) 000752 1775884952.820396 fallocate 0x029217 thru 0x031b7b (0x8965 bytes) 000753 1775884952.831674 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x160c22b (stale) Mirror 8: 0x160c22b (stale) Mirror 12: 0x160c22b (stale) Mirror 14: 0x160c22b (stale) Mirror 16: 0x160c22b (stale) Mirror 17: 0x160c22b (stale) Mirror 19: 0x160c22b (stale) Mirror 20: 0x160c22b (stale) Mirror 21: 0x160c22b (stale) Mirror 22: 0x160c22b (stale) Mirror 23: 0x160c22b (stale) Mirror 24: 0x3d168bed (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9571a3df Mirror 25: 0x9571a3df lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000754 1775884953.473714 fallocate 0x011702 thru 0x01c478 (0xad77 bytes) 000755 1775884953.516521 mapwrite 0x016dd1 thru 0x02113a (0xa36a bytes) 000756 1775884953.561641 trunc from 0x040000 to 0x036f91 (0x906e bytes) 000758 1775884953.584316 write 0x031d13 thru 0x035ccf (0x3fbd bytes) 000759 1775884953.595741 read 0x00a4ef thru 0x0195cb (0xf0dd bytes) 000760 1775884953.604001 punch from 0x0097f3 to 0x00eae9 (0x52f7 bytes) 000761 1775884953.608319 mapread 0x01ddf4 thru 0x02511c (0x7329 bytes) 000762 1775884953.613075 MIRROR_RESYNC 14 mirrors 000763 1775884955.340977 trunc from 0x036f92 to 0x027941 (0xf650 bytes) 000764 1775884955.371342 write 0x03e200 thru 0x03ffff (0x1e00 bytes) 000765 1775884955.372486 read 0x00b70e thru 0x0166bb (0xafae bytes) 000766 1775884955.382839 trunc from 0x03bbe4 to 0x03ffff (0x441c bytes) 000767 1775884955.396516 trunc from 0x040000 to 0x0245ef (0x1ba10 bytes) 000768 1775884955.414341 trunc from 0x01f20e to 0x0245ef (0x53e2 bytes) 000769 1775884955.419133 mapwrite 0x009b02 thru 0x010f6b (0x746a bytes) 000770 1775884955.450704 trunc from 0x018e33 to 0x021bc5 (0x8d93 bytes) 000771 1775884955.455774 read 0x003afe thru 0x007d40 (0x4243 bytes) 000772 1775884955.464711 write 0x03f548 thru 0x03ffff (0xab8 bytes) 000774 1775884955.470459 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x932db93d (stale) Mirror 8: 0x932db93d (stale) Mirror 12: 0x932db93d (stale) Mirror 14: 0x932db93d (stale) Mirror 16: 0x932db93d (stale) Mirror 17: 0x932db93d (stale) Mirror 19: 0x932db93d (stale) Mirror 20: 0x932db93d (stale) Mirror 21: 0x932db93d (stale) Mirror 22: 0x932db93d (stale) Mirror 23: 0x932db93d (stale) Mirror 24: 0x932db93d (stale) Mirror 25: 0x932db93d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8c4b6bb3 Mirror 26: 0x8c4b6bb3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000776 1775884956.300084 fallocate 0x00683b thru 0x014c0c (0xe3d2 bytes) 000777 1775884956.329874 MIRROR_SPLIT mirror 5 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x932db93d (stale) Mirror 8: 0x932db93d (stale) Mirror 12: 0x932db93d (stale) Mirror 14: 0x932db93d (stale) Mirror 17: 0x932db93d (stale) Mirror 19: 0x932db93d (stale) Mirror 20: 0x932db93d (stale) Mirror 21: 0x932db93d (stale) Mirror 22: 0x932db93d (stale) Mirror 23: 0x932db93d (stale) Mirror 24: 0x932db93d (stale) Mirror 25: 0x932db93d (stale) Mirror 26: 0x8c4b6bb3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x932db93d (stale) Mirror 8: 0x932db93d (stale) Mirror 12: 0x932db93d (stale) Mirror 14: 0x932db93d (stale) Mirror 17: 0x932db93d (stale) Mirror 19: 0x932db93d (stale) Mirror 20: 0x932db93d (stale) Mirror 21: 0x932db93d (stale) Mirror 22: 0x932db93d (stale) Mirror 23: 0x932db93d (stale) Mirror 24: 0x932db93d (stale) Mirror 25: 0x932db93d (stale) Mirror 26: 0x8c4b6bb3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000778 1775884956.888244 mapread 0x01708a thru 0x022f1a (0xbe91 bytes) 000779 1775884956.916396 punch from 0x01e7c4 to 0x02bc87 (0xd4c4 bytes) 000780 1775884956.923365 write 0x01f0c7 thru 0x024a0d (0x5947 bytes) 000781 1775884956.944620 write 0x037eab thru 0x039693 (0x17e9 bytes) 000782 1775884956.958376 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x932db93d (stale) Mirror 8: 0x932db93d (stale) Mirror 12: 0x932db93d (stale) Mirror 14: 0x932db93d (stale) Mirror 17: 0x932db93d (stale) Mirror 19: 0x932db93d (stale) Mirror 20: 0x932db93d (stale) Mirror 21: 0x932db93d (stale) Mirror 22: 0x932db93d (stale) Mirror 23: 0x932db93d (stale) Mirror 24: 0x932db93d (stale) Mirror 25: 0x932db93d (stale) Mirror 26: 0x8c4b6bb3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x97718ef Mirror 27: 0x97718ef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000783 1775884957.748113 punch from 0x039b0f to 0x03e1cb (0x46bd bytes) 000784 1775884957.780507 trunc from 0x040000 to 0x00228c (0x3dd73 bytes) 000785 1775884957.798368 read 0x000a08 thru 0x00228c (0x1885 bytes) 000786 1775884957.802006 mapwrite 0x019022 thru 0x028501 (0xf4e0 bytes) 000787 1775884957.846122 read 0x01139e thru 0x012279 (0xedc bytes) 000788 1775884957.851685 mapread 0x00779a thru 0x00bb06 (0x436d bytes) 000789 1775884957.852652 write 0x03e4e8 thru 0x03ffff (0x1b18 bytes) 000790 1775884957.853483 mapwrite 0x0245bb thru 0x02da03 (0x9449 bytes) 000791 1775884957.875110 fallocate 0x0314a5 thru 0x03151f (0x7b bytes) 000792 1775884957.881953 read 0x02499e thru 0x028cd5 (0x4338 bytes) 000794 1775884957.884521 punch from 0x006dcf to 0x009f8b (0x31bd bytes) 000795 1775884957.890014 write 0x0261fc thru 0x02950a (0x330f bytes) 000796 1775884957.892968 punch from 0x023ed7 to 0x026c53 (0x2d7d bytes) 000797 1775884957.903575 punch from 0x0277f8 to 0x036053 (0xe85c bytes) 000798 1775884957.908901 mapwrite 0x036135 thru 0x03a7a3 (0x466f bytes) 000799 1775884957.932523 read 0x00c97a thru 0x016396 (0x9a1d bytes) 000800 1775884957.936061 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x932db93d (stale) Mirror 8: 0x932db93d (stale) Mirror 12: 0x932db93d (stale) Mirror 14: 0x932db93d (stale) Mirror 17: 0x932db93d (stale) Mirror 19: 0x932db93d (stale) Mirror 20: 0x932db93d (stale) Mirror 21: 0x932db93d (stale) Mirror 22: 0x932db93d (stale) Mirror 23: 0x932db93d (stale) Mirror 24: 0x932db93d (stale) Mirror 25: 0x932db93d (stale) Mirror 26: 0x8c4b6bb3 (stale) Mirror 27: 0x97718ef (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8d2aaa19 Mirror 28: 0x8d2aaa19 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000801 1775884958.679398 fallocate 0x001aa9 thru 0x010347 (0xe89f bytes) 000802 1775884958.709600 write 0x02b8c7 thru 0x02e675 (0x2daf bytes) 000803 1775884958.721748 fallocate 0x018500 thru 0x02029f (0x7da0 bytes) 000804 1775884958.725783 write 0x018c76 thru 0x01c7e2 (0x3b6d bytes) 000805 1775884958.735792 punch from 0x01e0e2 to 0x02d652 (0xf571 bytes) 000806 1775884958.747604 fallocate 0x02014e thru 0x02cee6 (0xcd99 bytes) 000807 1775884958.751958 read 0x00df24 thru 0x013a32 (0x5b0f bytes) 000808 1775884958.760624 read 0x003299 thru 0x00edda (0xbb42 bytes) 000809 1775884958.767055 punch from 0x00c5a7 to 0x00cb3a (0x594 bytes) 000811 1775884958.778489 MIRROR_SPLIT mirror 12 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x932db93d (stale) Mirror 8: 0x932db93d (stale) Mirror 12: 0x932db93d (stale) Mirror 14: 0x932db93d (stale) Mirror 17: 0x932db93d (stale) Mirror 19: 0x932db93d (stale) Mirror 20: 0x932db93d (stale) Mirror 21: 0x932db93d (stale) Mirror 22: 0x932db93d (stale) Mirror 23: 0x932db93d (stale) Mirror 24: 0x932db93d (stale) Mirror 26: 0x8c4b6bb3 (stale) Mirror 27: 0x97718ef (stale) Mirror 28: 0x8d2aaa19 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x932db93d (stale) Mirror 8: 0x932db93d (stale) Mirror 12: 0x932db93d (stale) Mirror 14: 0x932db93d (stale) Mirror 17: 0x932db93d (stale) Mirror 19: 0x932db93d (stale) Mirror 20: 0x932db93d (stale) Mirror 21: 0x932db93d (stale) Mirror 22: 0x932db93d (stale) Mirror 23: 0x932db93d (stale) Mirror 24: 0x932db93d (stale) Mirror 26: 0x8c4b6bb3 (stale) Mirror 27: 0x97718ef (stale) Mirror 28: 0x8d2aaa19 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000812 1775884959.278376 trunc from 0x040000 to 0x028e1f (0x171e0 bytes) 000813 1775884959.299299 read 0x025b79 thru 0x0268a8 (0xd30 bytes) 000814 1775884959.306566 write 0x02f4c1 thru 0x03b4c1 (0xc001 bytes) 000817 1775884959.316003 MIRROR_RESYNC 15 mirrors 000818 1775884961.413429 trunc from 0x03b4c2 to 0x016597 (0x24f2a bytes) 000819 1775884961.464935 fallocate 0x001cec thru 0x003d4a (0x205f bytes) 000820 1775884961.472885 punch from 0x01527a to 0x016597 (0x131e bytes) 000821 1775884961.482933 fallocate 0x0268b1 thru 0x034f7b (0xe6cb bytes) 000822 1775884961.489981 read 0x00491b thru 0x013ceb (0xf3d1 bytes) 000823 1775884961.501264 fallocate 0x022b96 thru 0x026249 (0x36b4 bytes) 000824 1775884961.509839 trunc from 0x00b213 to 0x01274c (0x753a bytes) 000825 1775884961.514972 write 0x03454c thru 0x039377 (0x4e2c bytes) 000826 1775884961.518478 read 0x03572b thru 0x039377 (0x3c4d bytes) 000827 1775884961.519770 mapwrite 0x000ad8 thru 0x00c840 (0xbd69 bytes) 000828 1775884961.555829 MIRROR_SPLIT mirror 8 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39378): Mirror 7: 0xc892fac4 (stale) Mirror 8: 0xc892fac4 (stale) Mirror 12: 0xc892fac4 (stale) Mirror 14: 0xc892fac4 (stale) Mirror 17: 0xc892fac4 (stale) Mirror 19: 0xc892fac4 (stale) Mirror 20: 0xc892fac4 (stale) Mirror 22: 0xc892fac4 (stale) Mirror 23: 0xc892fac4 (stale) Mirror 24: 0xc892fac4 (stale) Mirror 26: 0xc892fac4 (stale) Mirror 27: 0xc892fac4 (stale) Mirror 28: 0xc892fac4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39378): Mirror 7: 0xc892fac4 (stale) Mirror 8: 0xc892fac4 (stale) Mirror 12: 0xc892fac4 (stale) Mirror 14: 0xc892fac4 (stale) Mirror 17: 0xc892fac4 (stale) Mirror 19: 0xc892fac4 (stale) Mirror 20: 0xc892fac4 (stale) Mirror 22: 0xc892fac4 (stale) Mirror 23: 0xc892fac4 (stale) Mirror 24: 0xc892fac4 (stale) Mirror 26: 0xc892fac4 (stale) Mirror 27: 0xc892fac4 (stale) Mirror 28: 0xc892fac4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39378: skipped 000829 1775884962.087115 read 0x00067e thru 0x003f89 (0x390c bytes) 000830 1775884962.095480 MIRROR_SPLIT mirror 4 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39378): Mirror 7: 0xc892fac4 (stale) Mirror 8: 0xc892fac4 (stale) Mirror 12: 0xc892fac4 (stale) Mirror 17: 0xc892fac4 (stale) Mirror 19: 0xc892fac4 (stale) Mirror 20: 0xc892fac4 (stale) Mirror 22: 0xc892fac4 (stale) Mirror 23: 0xc892fac4 (stale) Mirror 24: 0xc892fac4 (stale) Mirror 26: 0xc892fac4 (stale) Mirror 27: 0xc892fac4 (stale) Mirror 28: 0xc892fac4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39378): Mirror 7: 0xc892fac4 (stale) Mirror 8: 0xc892fac4 (stale) Mirror 12: 0xc892fac4 (stale) Mirror 17: 0xc892fac4 (stale) Mirror 19: 0xc892fac4 (stale) Mirror 20: 0xc892fac4 (stale) Mirror 22: 0xc892fac4 (stale) Mirror 23: 0xc892fac4 (stale) Mirror 24: 0xc892fac4 (stale) Mirror 26: 0xc892fac4 (stale) Mirror 27: 0xc892fac4 (stale) Mirror 28: 0xc892fac4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39378: skipped 000831 1775884962.590636 MIRROR_SPLIT mirror 11 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39378): Mirror 7: 0xc892fac4 (stale) Mirror 8: 0xc892fac4 (stale) Mirror 12: 0xc892fac4 (stale) Mirror 17: 0xc892fac4 (stale) Mirror 19: 0xc892fac4 (stale) Mirror 20: 0xc892fac4 (stale) Mirror 22: 0xc892fac4 (stale) Mirror 23: 0xc892fac4 (stale) Mirror 24: 0xc892fac4 (stale) Mirror 26: 0xc892fac4 (stale) Mirror 28: 0xc892fac4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39378): Mirror 7: 0xc892fac4 (stale) Mirror 8: 0xc892fac4 (stale) Mirror 12: 0xc892fac4 (stale) Mirror 17: 0xc892fac4 (stale) Mirror 19: 0xc892fac4 (stale) Mirror 20: 0xc892fac4 (stale) Mirror 22: 0xc892fac4 (stale) Mirror 23: 0xc892fac4 (stale) Mirror 24: 0xc892fac4 (stale) Mirror 26: 0xc892fac4 (stale) Mirror 28: 0xc892fac4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39378: skipped 000832 1775884963.074126 trunc from 0x039378 to 0x016f4e (0x22429 bytes) 000833 1775884963.085864 mapwrite 0x035d84 thru 0x03de29 (0x80a6 bytes) 000834 1775884963.129134 trunc from 0x03de2a to 0x011dff (0x2c02a bytes) 000835 1775884963.143910 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x11e00): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11e00): Mirror 7: 0xeda896b1 (stale) Mirror 8: 0xeda896b1 (stale) Mirror 12: 0xeda896b1 (stale) Mirror 17: 0xeda896b1 (stale) Mirror 19: 0xeda896b1 (stale) Mirror 20: 0xeda896b1 (stale) Mirror 22: 0xeda896b1 (stale) Mirror 23: 0xeda896b1 (stale) Mirror 24: 0xeda896b1 (stale) Mirror 26: 0xeda896b1 (stale) Mirror 28: 0xeda896b1 (stale) CRC-32 checksum value for chunk [0, 0x11e00): Mirror 1: 0x5ca314ae Mirror 29: 0x5ca314ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11e00: skipped 000836 1775884963.771424 fallocate 0x0045aa thru 0x00b110 (0x6b67 bytes) 000838 1775884963.807976 read 0x00e211 thru 0x011dff (0x3bef bytes) 000839 1775884963.819108 MIRROR_SPLIT mirror 2 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11e00): Mirror 7: 0xeda896b1 (stale) Mirror 12: 0xeda896b1 (stale) Mirror 17: 0xeda896b1 (stale) Mirror 19: 0xeda896b1 (stale) Mirror 20: 0xeda896b1 (stale) Mirror 22: 0xeda896b1 (stale) Mirror 23: 0xeda896b1 (stale) Mirror 24: 0xeda896b1 (stale) Mirror 26: 0xeda896b1 (stale) Mirror 28: 0xeda896b1 (stale) Mirror 29: 0x5ca314ae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11e00): Mirror 7: 0xeda896b1 (stale) Mirror 12: 0xeda896b1 (stale) Mirror 17: 0xeda896b1 (stale) Mirror 19: 0xeda896b1 (stale) Mirror 20: 0xeda896b1 (stale) Mirror 22: 0xeda896b1 (stale) Mirror 23: 0xeda896b1 (stale) Mirror 24: 0xeda896b1 (stale) Mirror 26: 0xeda896b1 (stale) Mirror 28: 0xeda896b1 (stale) Mirror 29: 0x5ca314ae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11e00: skipped 000841 1775884964.145956 mapwrite 0x01a219 thru 0x01e3f6 (0x41de bytes) 000843 1775884964.200133 mapread 0x01881a thru 0x01aaa4 (0x228b bytes) 000844 1775884964.201218 fallocate 0x028fae thru 0x033125 (0xa178 bytes) 000845 1775884964.205559 mapwrite 0x0167ee thru 0x01dd80 (0x7593 bytes) 000846 1775884964.226234 trunc from 0x01e3f7 to 0x03dee8 (0x1faf2 bytes) 000847 1775884964.239538 trunc from 0x03dee9 to 0x025e97 (0x18051 bytes) 000848 1775884964.249866 read 0x006b25 thru 0x013c98 (0xd174 bytes) 000849 1775884964.258237 mapread 0x017939 thru 0x0231bf (0xb887 bytes) 000850 1775884964.263450 mapread 0x01557f thru 0x0186ea (0x316c bytes) 000851 1775884964.264409 trunc from 0x025e98 to 0x03b91d (0x15a86 bytes) 000853 1775884964.280283 trunc from 0x007f55 to 0x016fc7 (0xf073 bytes) 000854 1775884964.284615 trunc from 0x03b91e to 0x018758 (0x231c5 bytes) 000855 1775884964.297208 fallocate 0x0010eb thru 0x009dee (0x8d04 bytes) 000856 1775884964.300501 punch from 0x014b2e to 0x018758 (0x3c2b bytes) 000857 1775884964.304304 punch from 0x00cca2 to 0x012aef (0x5e4e bytes) 000858 1775884964.316639 MIRROR_RESYNC 12 mirrors 000860 1775884965.543869 write 0x030ef5 thru 0x03e4f0 (0xd5fc bytes) 000861 1775884965.570517 read 0x02be74 thru 0x03298a (0x6b17 bytes) 000862 1775884965.575627 mapread 0x01e2ee thru 0x021521 (0x3234 bytes) 000863 1775884965.592112 MIRROR_SPLIT mirror 8 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e4f1): Mirror 7: 0xb86ba3f1 (stale) Mirror 12: 0xb86ba3f1 (stale) Mirror 17: 0xb86ba3f1 (stale) Mirror 19: 0xb86ba3f1 (stale) Mirror 20: 0xb86ba3f1 (stale) Mirror 22: 0xb86ba3f1 (stale) Mirror 23: 0xb86ba3f1 (stale) Mirror 26: 0xb86ba3f1 (stale) Mirror 28: 0xb86ba3f1 (stale) Mirror 29: 0xb86ba3f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e4f1): Mirror 7: 0xb86ba3f1 (stale) Mirror 12: 0xb86ba3f1 (stale) Mirror 17: 0xb86ba3f1 (stale) Mirror 19: 0xb86ba3f1 (stale) Mirror 20: 0xb86ba3f1 (stale) Mirror 22: 0xb86ba3f1 (stale) Mirror 23: 0xb86ba3f1 (stale) Mirror 26: 0xb86ba3f1 (stale) Mirror 28: 0xb86ba3f1 (stale) Mirror 29: 0xb86ba3f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e4f1: skipped 000864 1775884966.034473 punch from 0x01b641 to 0x026898 (0xb258 bytes) 000865 1775884966.043219 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e4f1): Mirror 7: 0xb86ba3f1 (stale) Mirror 12: 0xb86ba3f1 (stale) Mirror 17: 0xb86ba3f1 (stale) Mirror 19: 0xb86ba3f1 (stale) Mirror 20: 0xb86ba3f1 (stale) Mirror 22: 0xb86ba3f1 (stale) Mirror 23: 0xb86ba3f1 (stale) Mirror 28: 0xb86ba3f1 (stale) Mirror 29: 0xb86ba3f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e4f1): Mirror 7: 0xb86ba3f1 (stale) Mirror 12: 0xb86ba3f1 (stale) Mirror 17: 0xb86ba3f1 (stale) Mirror 19: 0xb86ba3f1 (stale) Mirror 20: 0xb86ba3f1 (stale) Mirror 22: 0xb86ba3f1 (stale) Mirror 23: 0xb86ba3f1 (stale) Mirror 28: 0xb86ba3f1 (stale) Mirror 29: 0xb86ba3f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e4f1: skipped 000866 1775884966.388170 mapread 0x013c96 thru 0x01b339 (0x76a4 bytes) 000867 1775884966.410776 trunc from 0x03e4f1 to 0x0056ea (0x38e06 bytes) 000868 1775884966.434765 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x56eb): Mirror 7: 0x7697d3a7 (stale) Mirror 12: 0x7697d3a7 (stale) Mirror 17: 0x7697d3a7 (stale) Mirror 19: 0x7697d3a7 (stale) Mirror 20: 0x7697d3a7 (stale) Mirror 22: 0x7697d3a7 (stale) Mirror 28: 0x7697d3a7 (stale) Mirror 29: 0x7697d3a7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x56eb): Mirror 7: 0x7697d3a7 (stale) Mirror 12: 0x7697d3a7 (stale) Mirror 17: 0x7697d3a7 (stale) Mirror 19: 0x7697d3a7 (stale) Mirror 20: 0x7697d3a7 (stale) Mirror 22: 0x7697d3a7 (stale) Mirror 28: 0x7697d3a7 (stale) Mirror 29: 0x7697d3a7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x56eb: skipped 000869 1775884966.768798 write 0x0084b6 thru 0x011e23 (0x996e bytes) 000870 1775884966.775013 mapread 0x00f085 thru 0x011e23 (0x2d9f bytes) 000871 1775884966.784302 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11e24): Mirror 7: 0x2566bfb3 (stale) Mirror 12: 0x2566bfb3 (stale) Mirror 19: 0x2566bfb3 (stale) Mirror 20: 0x2566bfb3 (stale) Mirror 22: 0x2566bfb3 (stale) Mirror 28: 0x2566bfb3 (stale) Mirror 29: 0x2566bfb3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11e24): Mirror 7: 0x2566bfb3 (stale) Mirror 12: 0x2566bfb3 (stale) Mirror 19: 0x2566bfb3 (stale) Mirror 20: 0x2566bfb3 (stale) Mirror 22: 0x2566bfb3 (stale) Mirror 28: 0x2566bfb3 (stale) Mirror 29: 0x2566bfb3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11e24: skipped 000872 1775884967.168808 trunc from 0x00a770 to 0x00e9f2 (0x4283 bytes) 000874 1775884967.174646 MIRROR_RESYNC 8 mirrors 000875 1775884968.248760 trunc from 0x000b7f to 0x00afb4 (0xa436 bytes) 000876 1775884968.294158 punch from 0x003360 to 0x010acc (0xd76d bytes) 000878 1775884968.298752 trunc from 0x010256 to 0x011e23 (0x1bce bytes) 000879 1775884968.301648 trunc from 0x00799d to 0x011a21 (0xa085 bytes) 000880 1775884968.304613 trunc from 0x0091e1 to 0x011e23 (0x8c43 bytes) 000881 1775884968.307984 trunc from 0x0047d5 to 0x0080e2 (0x390e bytes) 000882 1775884968.310858 write 0x00601d thru 0x014764 (0xe748 bytes) 000883 1775884968.316813 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x14765): Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14765): Mirror 7: 0xf74e9adb (stale) Mirror 12: 0xf74e9adb (stale) Mirror 19: 0xf74e9adb (stale) Mirror 20: 0xf74e9adb (stale) Mirror 22: 0xf74e9adb (stale) Mirror 28: 0xf74e9adb (stale) Mirror 29: 0xf74e9adb (stale) CRC-32 checksum value for chunk [0, 0x14765): Mirror 1: 0xb0b6e336 Mirror 30: 0xb0b6e336 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14765: skipped 000884 1775884968.959690 fallocate 0x0138be thru 0x01bcfe (0x8441 bytes) 000885 1775884968.994287 punch from 0x013317 to 0x014764 (0x144e bytes) 000887 1775884968.998870 read 0x000b84 thru 0x003a12 (0x2e8f bytes) 000888 1775884969.002611 read 0x006a17 thru 0x010a81 (0xa06b bytes) 000889 1775884969.007969 read 0x00d185 thru 0x00fb1c (0x2998 bytes) 000890 1775884969.008464 write 0x0284a1 thru 0x036147 (0xdca7 bytes) 000891 1775884969.016954 MIRROR_RESYNC 9 mirrors 000892 1775884970.193760 MIRROR_EXTEND to 10 mirrors 000893 1775884970.679958 write 0x01645e thru 0x018944 (0x24e7 bytes) 000894 1775884970.714325 trunc from 0x01c105 to 0x021219 (0x5115 bytes) 000895 1775884970.717799 fallocate 0x02eacb thru 0x02fafe (0x1034 bytes) 000896 1775884970.721280 punch from 0x032f92 to 0x036147 (0x31b6 bytes) 000898 1775884970.725403 trunc from 0x01b604 to 0x01d290 (0x1c8d bytes) 000899 1775884970.731822 write 0x012e94 thru 0x01a12b (0x7298 bytes) 000900 1775884970.740194 fallocate 0x013c87 thru 0x020cdd (0xd057 bytes) 000901 1775884970.745770 write 0x009c2f thru 0x01037d (0x674f bytes) 000902 1775884970.752589 mapread 0x025925 thru 0x03043a (0xab16 bytes) 000904 1775884970.767107 mapread 0x011ba8 thru 0x01ad67 (0x91c0 bytes) 000905 1775884970.767873 punch from 0x0336c6 to 0x036147 (0x2a82 bytes) 000906 1775884970.771369 trunc from 0x036148 to 0x012ba2 (0x235a5 bytes) 000907 1775884970.791047 trunc from 0x012ba3 to 0x02108f (0xe4ed bytes) 000908 1775884970.797064 write 0x01d172 thru 0x0226d5 (0x5564 bytes) 000909 1775884970.805930 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x226d6): Mirror 7: 0x992ec750 (stale) Mirror 12: 0x992ec750 (stale) Mirror 20: 0x992ec750 (stale) Mirror 22: 0x992ec750 (stale) Mirror 28: 0x992ec750 (stale) Mirror 29: 0x992ec750 (stale) Mirror 30: 0x992ec750 (stale) Mirror 31: 0x992ec750 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x226d6): Mirror 7: 0x992ec750 (stale) Mirror 12: 0x992ec750 (stale) Mirror 20: 0x992ec750 (stale) Mirror 22: 0x992ec750 (stale) Mirror 28: 0x992ec750 (stale) Mirror 29: 0x992ec750 (stale) Mirror 30: 0x992ec750 (stale) Mirror 31: 0x992ec750 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x226d6: skipped 000910 1775884971.110348 trunc from 0x0226d6 to 0x0168b6 (0xbe1f bytes) 000911 1775884971.123384 mapwrite 0x00568d thru 0x01065c (0xafd0 bytes) 000912 1775884971.148291 mapwrite 0x03bcb6 thru 0x03dd45 (0x2090 bytes) 000913 1775884971.179859 trunc from 0x03dd46 to 0x034c42 (0x9103 bytes) 000914 1775884971.196365 MIRROR_RESYNC 9 mirrors 000915 1775884972.194211 MIRROR_EXTEND to 10 mirrors 000916 1775884972.705465 punch from 0x028fdc to 0x02c54e (0x3573 bytes) 000918 1775884972.733704 write 0x011f6c thru 0x0182ed (0x6382 bytes) 000919 1775884972.740958 trunc from 0x02d3f0 to 0x031ad9 (0x46ea bytes) 000920 1775884972.744324 write 0x008344 thru 0x00aee8 (0x2ba5 bytes) 000921 1775884972.750449 trunc from 0x034c43 to 0x004615 (0x3062d bytes) 000922 1775884972.767425 punch from 0x0034a5 to 0x004615 (0x1171 bytes) 000923 1775884972.771433 write 0x018bd1 thru 0x01a5ed (0x1a1d bytes) 000924 1775884972.773219 write 0x01da80 thru 0x02c084 (0xe605 bytes) 000925 1775884972.775672 mapwrite 0x01d8d6 thru 0x02c83c (0xef67 bytes) 000926 1775884972.825777 write 0x03183d thru 0x03dc47 (0xc40b bytes) 000927 1775884972.828607 write 0x00850c thru 0x00aeb9 (0x29ae bytes) 000928 1775884972.842629 MIRROR_RESYNC 10 mirrors 000929 1775884974.820952 trunc from 0x03dc48 to 0x01986a (0x243dd bytes) 000930 1775884974.857021 fallocate 0x03c606 thru 0x03ffff (0x39fa bytes) 000931 1775884974.862868 mapread 0x037a8a thru 0x03b03c (0x35b3 bytes) 000932 1775884974.878550 read 0x01ca2b thru 0x02a54c (0xdb22 bytes) 000933 1775884974.884587 mapwrite 0x027383 thru 0x030c69 (0x98e7 bytes) 000934 1775884974.908927 mapwrite 0x028d2c thru 0x035022 (0xc2f7 bytes) 000935 1775884974.934396 fallocate 0x01f2f3 thru 0x01f55b (0x269 bytes) 000936 1775884974.942922 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xe6e5d19b (stale) Mirror 12: 0xe6e5d19b (stale) Mirror 20: 0xe6e5d19b (stale) Mirror 22: 0xe6e5d19b (stale) Mirror 28: 0xe6e5d19b (stale) Mirror 29: 0xe6e5d19b (stale) Mirror 30: 0xe6e5d19b (stale) Mirror 31: 0xe6e5d19b (stale) Mirror 32: 0xe6e5d19b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6c53a74d Mirror 33: 0x6c53a74d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000937 1775884975.692246 punch from 0x00c325 to 0x018bda (0xc8b6 bytes) 000938 1775884975.729933 trunc from 0x040000 to 0x036d75 (0x928a bytes) 000939 1775884975.751697 trunc from 0x036d76 to 0x02f61e (0x7757 bytes) 000940 1775884975.765931 mapread 0x02923c thru 0x02f61e (0x63e3 bytes) 000941 1775884975.785922 punch from 0x01b7d6 to 0x01bad4 (0x2ff bytes) 000942 1775884975.792623 read 0x01a6d3 thru 0x01de77 (0x37a5 bytes) 000943 1775884975.800475 trunc from 0x02f61f to 0x00a028 (0x255f6 bytes) 000944 1775884975.812288 write 0x017320 thru 0x01dc24 (0x6905 bytes) 000945 1775884975.825106 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1dc25): Mirror 7: 0xf4c070c2 (stale) Mirror 12: 0xf4c070c2 (stale) Mirror 20: 0xf4c070c2 (stale) Mirror 22: 0xf4c070c2 (stale) Mirror 28: 0xf4c070c2 (stale) Mirror 30: 0xf4c070c2 (stale) Mirror 31: 0xf4c070c2 (stale) Mirror 32: 0xf4c070c2 (stale) Mirror 33: 0x6d9b63c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dc25): Mirror 7: 0xf4c070c2 (stale) Mirror 12: 0xf4c070c2 (stale) Mirror 20: 0xf4c070c2 (stale) Mirror 22: 0xf4c070c2 (stale) Mirror 28: 0xf4c070c2 (stale) Mirror 30: 0xf4c070c2 (stale) Mirror 31: 0xf4c070c2 (stale) Mirror 32: 0xf4c070c2 (stale) Mirror 33: 0x6d9b63c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dc25: skipped 000947 1775884976.215101 fallocate 0x023176 thru 0x02b9e7 (0x8872 bytes) 000948 1775884976.219669 fallocate 0x00251d thru 0x00d117 (0xabfb bytes) 000949 1775884976.224563 write 0x008faf thru 0x0098eb (0x93d bytes) 000950 1775884976.235760 mapread 0x023937 thru 0x026d2d (0x33f7 bytes) 000951 1775884976.249300 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b9e8): Mirror 12: 0x89f9e438 (stale) Mirror 20: 0x89f9e438 (stale) Mirror 22: 0x89f9e438 (stale) Mirror 28: 0x89f9e438 (stale) Mirror 30: 0x89f9e438 (stale) Mirror 31: 0x89f9e438 (stale) Mirror 32: 0x89f9e438 (stale) Mirror 33: 0xb5ecffd6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b9e8): Mirror 12: 0x89f9e438 (stale) Mirror 20: 0x89f9e438 (stale) Mirror 22: 0x89f9e438 (stale) Mirror 28: 0x89f9e438 (stale) Mirror 30: 0x89f9e438 (stale) Mirror 31: 0x89f9e438 (stale) Mirror 32: 0x89f9e438 (stale) Mirror 33: 0xb5ecffd6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b9e8: skipped 000952 1775884976.655200 punch from 0x01394b to 0x015026 (0x16dc bytes) 000953 1775884976.660395 write 0x03cbf5 thru 0x03d15a (0x566 bytes) 000954 1775884976.664557 trunc from 0x03d15b to 0x02b255 (0x11f05 bytes) 000955 1775884976.677618 fallocate 0x0245db thru 0x02aeb0 (0x68d6 bytes) 000956 1775884976.681660 read 0x01e95c thru 0x022634 (0x3cd9 bytes) 000957 1775884976.689858 punch from 0x021765 to 0x029957 (0x81f3 bytes) 000958 1775884976.695231 mapread 0x00934b thru 0x009702 (0x3b8 bytes) 000959 1775884976.699772 read 0x02648f thru 0x02b255 (0x4dc7 bytes) 000960 1775884976.714948 punch from 0x01eeed to 0x02b255 (0xc369 bytes) 000961 1775884976.721532 punch from 0x00fddc to 0x012826 (0x2a4b bytes) 000962 1775884976.726777 read 0x02141f thru 0x02a11c (0x8cfe bytes) 000963 1775884976.733238 trunc from 0x000bae to 0x00f53e (0xe991 bytes) 000965 1775884976.737818 trunc from 0x0158bc to 0x016850 (0xf95 bytes) 000966 1775884976.751710 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b256): Mirror 20: 0xc10ad7f9 (stale) Mirror 22: 0xc10ad7f9 (stale) Mirror 28: 0xc10ad7f9 (stale) Mirror 30: 0xc10ad7f9 (stale) Mirror 31: 0xc10ad7f9 (stale) Mirror 32: 0xc10ad7f9 (stale) Mirror 33: 0x7a6e705 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b256): Mirror 20: 0xc10ad7f9 (stale) Mirror 22: 0xc10ad7f9 (stale) Mirror 28: 0xc10ad7f9 (stale) Mirror 30: 0xc10ad7f9 (stale) Mirror 31: 0xc10ad7f9 (stale) Mirror 32: 0xc10ad7f9 (stale) Mirror 33: 0x7a6e705 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b256: skipped 000967 1775884977.112385 punch from 0x025a03 to 0x02b255 (0x5853 bytes) 000968 1775884977.116354 read 0x014fea thru 0x019cdf (0x4cf6 bytes) 000969 1775884977.126162 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x2da:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b256): Mirror 20: 0xc10ad7f9 (stale) Mirror 22: 0xc10ad7f9 (stale) Mirror 28: 0xc10ad7f9 (stale) Mirror 31: 0xc10ad7f9 (stale) Mirror 32: 0xc10ad7f9 (stale) Mirror 33: 0x7a6e705 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b256): Mirror 20: 0xc10ad7f9 (stale) Mirror 22: 0xc10ad7f9 (stale) Mirror 28: 0xc10ad7f9 (stale) Mirror 31: 0xc10ad7f9 (stale) Mirror 32: 0xc10ad7f9 (stale) Mirror 33: 0x7a6e705 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b256: skipped 000970 1775884977.460267 read 0x007698 thru 0x00b12c (0x3a95 bytes) 000971 1775884977.467557 mapwrite 0x02cf7c thru 0x030b52 (0x3bd7 bytes) 000972 1775884977.509324 mapread 0x00023d thru 0x005dbf (0x5b83 bytes) 000973 1775884977.521737 trunc from 0x012066 to 0x01d998 (0xb933 bytes) 000974 1775884977.526932 read 0x010013 thru 0x012c0b (0x2bf9 bytes) 000975 1775884977.539482 trunc from 0x02bff9 to 0x030b52 (0x4b5a bytes) 000976 1775884977.544112 mapwrite 0x0367ea thru 0x03c4fa (0x5d11 bytes) 000977 1775884977.571543 trunc from 0x0020fa to 0x00a523 (0x842a bytes) 000978 1775884977.575624 write 0x02d709 thru 0x037f93 (0xa88b bytes) 000979 1775884977.581811 fallocate 0x000267 thru 0x007ca8 (0x7a42 bytes) 000980 1775884977.587834 trunc from 0x03c4fb to 0x012c15 (0x298e5 bytes) 000981 1775884977.607498 MIRROR_RESYNC 7 mirrors 000984 1775884978.568355 trunc from 0x012c16 to 0x030933 (0x1dd1e bytes) 000985 1775884978.596576 trunc from 0x030934 to 0x01d7cc (0x13167 bytes) 000986 1775884978.612748 trunc from 0x019784 to 0x01d7cc (0x4049 bytes) 000987 1775884978.616977 mapwrite 0x030e1d thru 0x03db65 (0xcd49 bytes) 000988 1775884978.674818 fallocate 0x033344 thru 0x035c91 (0x294e bytes) 000989 1775884978.682896 MIRROR_RESYNC 7 mirrors 000990 1775884979.555710 punch from 0x004261 to 0x00fbe5 (0xb985 bytes) 000991 1775884979.585330 mapread 0x001d8f thru 0x00f0b0 (0xd322 bytes) 000992 1775884979.605485 mapread 0x033d9c thru 0x03db65 (0x9dca bytes) 000993 1775884979.633628 MIRROR_RESYNC 7 mirrors 000994 1775884980.601736 fallocate 0x038090 thru 0x0391fc (0x116d bytes) 000995 1775884980.624486 mapwrite 0x0385c4 thru 0x03ffff (0x7a3c bytes) 000996 1775884980.682247 punch from 0x018635 to 0x01e38e (0x5d5a bytes) 000998 1775884980.688479 punch from 0x004189 to 0x00b7b5 (0x762d bytes) 000999 1775884980.693326 write 0x027a9d thru 0x028229 (0x78d bytes) 001000 1775884980.700520 mapread 0x014d87 thru 0x022b88 (0xde02 bytes) All operations completed A-OK! PASS 70a (87s) == sanity-flr test 71: check mirror extend parent fid ==== 01:23:03 (1775884983) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x32d:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (6s) == sanity-flr test 72: check append on FLR file ========== 01:23:09 (1775884989) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x96:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131687 s, 15.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x96:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x97:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506711 s, 20.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x96:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x97:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x96:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x96:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x97:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x96:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x98:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (4s) == sanity-flr test 200a: stress test ===================== 01:23:13 (1775884993) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5438528 .. Extending file size to 6366560 .. done Extending file size to 6159008 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7367616 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 7734272 .. Extending file size to 8240800 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8282880 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9155200 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8976704 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9246016 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9012352 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 61062 61064 61066 61068 61070 done 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) 192.168.202.128@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (68s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 01:24:21 (1775885061) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3688288 .. Extending file size to 7983168 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7066784 .. Extending file size to 8748608 .. Extending file size to 8409344 .. Extending file size to 9045632 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8902432 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9144576 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 63005 63006 63008 63010 63012 done 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) 192.168.202.128@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (68s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 01:25:29 (1775885129) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2348 1268956 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2532 1268772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17752 3589268 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25704 3581316 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43456 7170584 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2348 1268956 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2532 1268772 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17752 3589268 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25704 3581316 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43456 7170584 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0554587 s, 37.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0626389 s, 33.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xd19ffcf9 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) PASS 200c (7s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 01:25:37 (1775885137) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0638424 s, 32.8 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 01:25:40 (1775885140) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x62:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 01:25:43 (1775885143) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0478285 s, 43.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3ff:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x400:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x401:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (4s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 01:25:47 (1775885147) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0402517 s, 52.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 01:25:50 (1775885150) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0378645 s, 55.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb1:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 01:26:15 (1775885175) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg228-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245989 s, 42.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x400:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x403:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x402:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x400:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x403:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x402:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x401:0x0] } PASS 204d (36s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 01:26:51 (1775885211) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg228-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0350302 s, 59.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (36s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 01:27:27 (1775885247) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg228-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg228-server: Pool lustre.test_204f created Adding targets to pool oleg228-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg228-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0567402 s, 37.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x404:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x402:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x404:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x402:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg228-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg228-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg228-server: Pool lustre.test_204f destroyed PASS 204f (50s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 01:28:17 (1775885297) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 01:28:20 (1775885300) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 01:28:23 (1775885303) oleg228-server: Pool lustre.test_206 created Adding targets to pool oleg228-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg228-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg228-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg228-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg228-server: Pool lustre.test_206 destroyed PASS 206 (15s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 01:28:38 (1775885318) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307774 s, 34.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157449 s, 66.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x80a97716 Mirror 3: 0x80a97716 PASS 207 (4s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 01:28:42 (1775885322) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 01:28:43 (1775885323) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 01:28:44 (1775885324) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0229252 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2130000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xd30000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xd30000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xd30000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xd80000 Read this (4 bytes): BBBB 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) PASS 209a (19s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 01:29:03 (1775885343) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00591579 s, 1.4 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 01:29:05 (1775885345) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236961 s, 44.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x81 lmm_fid: [0x200000405:0x81:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x81 lmm_fid: [0x200000405:0x81:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbd:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611265 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-11 01:29:06.000000000 -0400 Modify: 2026-04-11 01:29:06.000000000 -0400 Change: 2026-04-11 01:29:07.000000000 -0400 Birth: 2026-04-11 01:29:07.000000000 -0400 PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 01:29:09 (1775885349) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166678 s, 62.9 MB/s oleg228-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg228-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg228-server: O/280000401/d30/be: File not found by ext2_lookup PASS 210b (15s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 01:29:24 (1775885364) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0889221 s, 461 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0145021 s, 282 kB/s size after second write -rw-r--r-- 1 root root 4096 Apr 11 01:29 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x86:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Apr 11 01:29 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 01:29:28 (1775885368) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (3s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 01:29:30 (1775885370) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (3s) == sanity-flr test complete, duration 1121 sec =========== 01:29:33 (1775885373) === sanity-flr: start cleanup 01:29:34 (1775885374) === === sanity-flr: finish cleanup 01:29:37 (1775885377) ===