-----============= acceptance-small: sanity-flr ============----- Mon Jun 29 14:03:00 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg424-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 14:03:16 (1782756196) === oleg424-client.virtnet: executing check_config_client /mnt/lustre oleg424-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg424-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b3d03eeb000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b3d03eeb000.idle_timeout=debug disable quota as required oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 14:03:45 (1782756225) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8229] == sanity-flr test 0a: lfs mirror create with -N option == 14:03:47 (1782756227) PASS 0a (13s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 14:04:01 (1782756241) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 14:04:04 (1782756244) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 14:04:07 (1782756247) PASS 0d (19s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 14:04:26 (1782756266) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 14:04:29 (1782756269) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 14:04:31 (1782756271) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9b3d03eeb000.stats=clear osc.lustre-OST0001-osc-ffff9b3d03eeb000.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 14:04:37 (1782756277) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (8s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 14:04:45 (1782756285) PASS 0j (6s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 14:04:51 (1782756291) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 14:04:53 (1782756293) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (5s) == sanity-flr test 1: create components with setstripe options ========================================================== 14:04:58 (1782756298) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (6s) == sanity-flr test 2: create components from existing files ========================================================== 14:05:04 (1782756304) PASS 2 (5s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 14:05:09 (1782756309) PASS 3 (5s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 14:05:14 (1782756314) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 14:05:20 (1782756320) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (5s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 14:05:27 (1782756327) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 14:05:32 (1782756332) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.135932 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861084 s, 12.2 MB/s PASS 21 (5s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 14:05:37 (1782756337) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.157269 s, 46.7 MB/s PASS 22 (6s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 14:05:43 (1782756343) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (7s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 14:05:51 (1782756351) Stopping /mnt/lustre-ost1 (opts:) on oleg424-server oleg424-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b3d03eeb000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b3d03eeb000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg424-server: oleg424-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg424-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg424-server oleg424-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b3d03eeb000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b3d03eeb000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg424-server: oleg424-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg424-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (87s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 14:07:19 (1782756439) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.24995 s, 16.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg424-server oleg424-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b3d03eeb000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b3d03eeb000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg424-server: oleg424-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg424-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 14:08:03 (1782756483) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg424-server oleg424-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b3d03eeb000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b3d03eeb000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg424-server: oleg424-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg424-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg424-server oleg424-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b3d03eeb000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b3d03eeb000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg424-server: oleg424-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg424-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (89s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 14:09:32 (1782756572) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.1106 s, 16.9 MB/s -rw-r--r-- 1 root root 52428800 Jun 29 14:09 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.912s user 0m0.002s sys 0m0.150s Stopping /mnt/lustre-ost1 (opts:) on oleg424-server oleg424-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b3d03eeb000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b3d03eeb000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.172s user 0m0.001s sys 0m0.179s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg424-server: oleg424-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg424-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg424-server oleg424-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b3d03eeb000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b3d03eeb000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.393s user 0m0.001s sys 0m0.140s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg424-server: oleg424-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg424-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg424-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b3d03eeb000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b3d03eeb000.ost_server_uuid in FULL state after 0 sec PASS 33b (93s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 14:11:06 (1782756666) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 14:11:09 (1782756669) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 14:11:13 (1782756673) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 14:11:16 (1782756676) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 14:11:24 (1782756684) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.444707 s, 28.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 14:11:38 (1782756698) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333693 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380949 s, 27.5 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 14:11:45 (1782756705) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 14:11:55 (1782756715) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00314815 s, 325 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00163422 s, 627 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0119449 s, 85.7 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00258333 s, 396 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 14:12:05 (1782756725) create test files with size 15746089 .. e29256d136f38105f1fc1fb7f829943a - 72ecf27fefaa0f871a8deab0b284eafa - 67f1f7ee6c5a7d1fffaf9f26e9a7e72c - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg424-client.virtnet /mnt/lustre (opts:) Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 14:12:28 (1782756748) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 14:12:43 (1782756763) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 14:12:50 (1782756770) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115055 s, 9.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775537 s, 13.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbf:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (13s) == sanity-flr test 41: lfs mirror resync check =========== 14:13:03 (1782756783) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.318501 s, 6.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.47351 s, 8.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.441652 s, 4.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (24s) == sanity-flr test 42: lfs mirror verify ================= 14:13:27 (1782756807) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 14:13:31 (1782756811) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (6s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 14:13:37 (1782756817) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542921 s, 19.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 14:13:46 (1782756826) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.53805 s, 5.8 MB/s PASS 44a (18s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 14:14:04 (1782756844) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc8:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (12s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 14:14:18 (1782756858) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.299672 s, 35.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg424-client.virtnet /mnt/lustre (opts:) Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 14:14:31 (1782756871) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9b3d20492800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.318141 s, 33.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg424-client.virtnet /mnt/lustre (opts:) Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9b3d20489800.xattr_cache=1 PASS 44e (18s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 14:14:49 (1782756889) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (48s) == sanity-flr test 46: Verify setstripe --copy option ==== 14:15:37 (1782756937) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (39s) == sanity-flr test 47: Verify mirror obj alloc =========== 14:16:16 (1782756976) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 14:16:19 (1782756979) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.219982 s, 14.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.180226 s, 17.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x4a5bc2e2 (nosync,stale) Mirror 2: 0x4a5bc2e2 (nosync,stale) ** verify mirror 2 doesn't change original checksum: db4acd3366ab32df8b4f35316ee8028a - mirror 1 checksum: 76268bfce83c2e48ff24ed609a38d506 - mirror 2 checksum: db4acd3366ab32df8b4f35316ee8028a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-29T14:16:22-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-29T14:16:22-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: db4acd3366ab32df8b4f35316ee8028a - mirror 1 checksum: 76268bfce83c2e48ff24ed609a38d506 - mirror 2 checksum: 76268bfce83c2e48ff24ed609a38d506 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-29T14:16:22-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-29T14:16:22-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 14:16:29 (1782756989) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291337 s, 36.0 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 14:16:36 (1782756996) Failing mds1 on oleg424-server Stopping /mnt/lustre-mds1 (opts:) on oleg424-server 14:16:48 (1782757008) shut down facet: mds1 facet_host: oleg424-server facet_failover_host: oleg424-server Failover mds1 to oleg424-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0000 14:17:11 (1782757031) targets are mounted 14:17:11 (1782757031) facet_failover done oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372078 s, 28.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } PASS 50A (53s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 14:17:29 (1782757049) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0513293 s, 399 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0216139 s, 948 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (11s) == sanity-flr test 50b: mirror rsync handles sparseness == 14:17:40 (1782757060) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0617125 s, 332 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0249912 s, 819 kB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 14:17:51 (1782757071) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0132025 s, 1.2 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 14:17:58 (1782757078) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0678572 s, 302 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 14:18:07 (1782757087) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00741157 s, 553 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0385458 s, 106 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 14:18:16 (1782757096) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x10:0x0]: Invalid argument (22) PASS 60b (26s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 14:18:41 (1782757121) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (32s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 14:19:13 (1782757153) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (28s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 14:19:41 (1782757181) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 14:20:06 (1782757206) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 14:20:15 (1782757215) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 57383 mirror split pid 57384 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7768: 57383 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7768: 57384 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (72s) == sanity-flr test 70a: flr mode fsx test ================ 14:21:27 (1782757287) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 7291 000001 1782757291.986536 mapwrite 0x0100e0 thru 0x018ab8 (0x89d9 bytes) 000002 1782757292.085835 punch from 0x003e3c to 0x010bf4 (0xcdb9 bytes) 000003 1782757292.093000 mapwrite 0x03c395 thru 0x03cfc3 (0xc2f bytes) fallocating to largest ever: 0x40000 000004 1782757292.142688 fallocate 0x035009 thru 0x03ffff (0xaff7 bytes) 000005 1782757292.149954 read 0x006734 thru 0x013946 (0xd213 bytes) 000006 1782757292.165890 fallocate 0x00c2ae thru 0x011367 (0x50ba bytes) 000007 1782757292.178133 fallocate 0x03ee62 thru 0x03ffff (0x119e bytes) 000008 1782757292.183834 trunc from 0x040000 to 0x024fbc (0x1b043 bytes) 000009 1782757292.203779 mapwrite 0x007b14 thru 0x00d257 (0x5744 bytes) 000010 1782757292.226462 write 0x037550 thru 0x039621 (0x20d2 bytes) 000012 1782757292.229651 punch from 0x009141 to 0x013a0e (0xa8ce bytes) 000013 1782757292.236535 trunc from 0x039622 to 0x00d29c (0x2c385 bytes) 000014 1782757292.254858 read 0x00b9d8 thru 0x00d29c (0x18c5 bytes) 000015 1782757292.268040 mapread 0x00a28e thru 0x00d29c (0x300f bytes) 000016 1782757292.269894 punch from 0x00ccfb to 0x00d29c (0x5a2 bytes) 000017 1782757292.277405 read 0x00bc17 thru 0x00d29c (0x1686 bytes) 000018 1782757292.283603 mapwrite 0x00775c thru 0x0154b6 (0xdd5b bytes) 000019 1782757292.345956 trunc from 0x015223 to 0x0154b6 (0x294 bytes) 000021 1782757292.354633 trunc from 0x0154b7 to 0x006490 (0xf026 bytes) 000023 1782757292.382004 read 0x004325 thru 0x006490 (0x216c bytes) 000025 1782757292.385872 write 0x03538b thru 0x03d9e8 (0x865e bytes) 000026 1782757292.388636 read 0x03d402 thru 0x03d9e8 (0x5e7 bytes) 000027 1782757292.389691 read 0x0061ac thru 0x0068d4 (0x729 bytes) 000028 1782757292.390677 fallocate 0x031e75 thru 0x03439a (0x2526 bytes) 000029 1782757292.395879 trunc from 0x01f8a8 to 0x021d52 (0x24ab bytes) 000030 1782757292.401616 mapread 0x0300b8 thru 0x0372c7 (0x7210 bytes) 000031 1782757292.447890 MIRROR_EXTEND to 2 mirrors 000033 1782757293.655151 trunc from 0x03d9e9 to 0x007201 (0x367e7 bytes) 000034 1782757293.731040 mapwrite 0x02fd0f thru 0x03230a (0x25fc bytes) 000035 1782757293.829517 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3230b): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3230b): Mirror 2: 0xbaf9992d (stale) CRC-32 checksum value for chunk [0, 0x3230b): Mirror 1: 0xda27c188 Mirror 3: 0xda27c188 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3230b: skipped 000036 1782757295.177901 read 0x02ab46 thru 0x02d32d (0x27e8 bytes) 000037 1782757295.187724 write 0x01096b thru 0x01b6ac (0xad42 bytes) 000038 1782757295.294757 trunc from 0x03230b to 0x032204 (0x106 bytes) 000039 1782757295.327925 mapread 0x0288bf thru 0x032204 (0x9946 bytes) 000040 1782757295.364036 MIRROR_RESYNC 3 mirrors 000041 1782757296.633006 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x156:0x0]: Invalid argument (22) 000042 1782757297.081499 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x156:0x0]: Invalid argument (22) 000043 1782757297.294350 punch from 0x01284d to 0x0149d4 (0x2188 bytes) 000044 1782757297.331459 punch from 0x0142fd to 0x01c757 (0x845b bytes) 000045 1782757297.345895 punch from 0x01cc67 to 0x02b190 (0xe52a bytes) 000047 1782757297.372345 mapread 0x00e9e6 thru 0x010d30 (0x234b bytes) 000048 1782757297.410001 trunc from 0x032205 to 0x029409 (0x8dfb bytes) 000049 1782757297.437920 punch from 0x004e10 to 0x0147f1 (0xf9e2 bytes) 000050 1782757297.449688 trunc from 0x00015e to 0x00a81f (0xa6c2 bytes) 000051 1782757297.457673 mapwrite 0x0039ab thru 0x0066dc (0x2d32 bytes) 000052 1782757297.502822 fallocate 0x038a28 thru 0x03ffff (0x75d8 bytes) 000053 1782757297.512642 read 0x01e58b thru 0x02a014 (0xba8a bytes) 000054 1782757297.535888 punch from 0x022030 to 0x02a711 (0x86e2 bytes) 000057 1782757297.551474 fallocate 0x015cff thru 0x01a2f7 (0x45f9 bytes) 000058 1782757297.562533 punch from 0x014039 to 0x01482d (0x7f5 bytes) 000059 1782757297.570730 punch from 0x007a59 to 0x015324 (0xd8cc bytes) 000060 1782757297.581457 write 0x00ed38 thru 0x01a980 (0xbc49 bytes) 000061 1782757297.621417 punch from 0x02d748 to 0x034f8e (0x7847 bytes) 000062 1782757297.641489 fallocate 0x03d721 thru 0x03ffff (0x28df bytes) 000063 1782757297.657341 mapread 0x005874 thru 0x009a46 (0x41d3 bytes) 000065 1782757297.680737 punch from 0x014dec to 0x024b4e (0xfd63 bytes) 000067 1782757297.716357 mapwrite 0x0063ce thru 0x010328 (0x9f5b bytes) 000069 1782757297.814523 MIRROR_EXTEND to 2 mirrors 000070 1782757298.973928 mapread 0x032a66 thru 0x03b0f1 (0x868c bytes) 000071 1782757299.024431 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x156:0x0]: Invalid argument (22) 000072 1782757299.296458 write 0x00ce9b thru 0x014367 (0x74cd bytes) 000073 1782757299.350475 write 0x00151c thru 0x0099ab (0x8490 bytes) 000074 1782757299.372627 trunc from 0x040000 to 0x00a2bc (0x35d43 bytes) 000075 1782757299.402869 read 0x0007db thru 0x006495 (0x5cbb bytes) 000076 1782757299.417400 fallocate 0x03a106 thru 0x03ffff (0x5efa bytes) 000077 1782757299.440448 MIRROR_EXTEND to 2 mirrors 000078 1782757301.117867 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x156:0x0]: Invalid argument (22) 000079 1782757301.483455 read 0x01c0db thru 0x020f12 (0x4e38 bytes) 000080 1782757301.510898 write 0x01b324 thru 0x028f66 (0xdc43 bytes) 000081 1782757301.574167 fallocate 0x02eb84 thru 0x031466 (0x28e3 bytes) 000082 1782757301.581736 punch from 0x00db0a to 0x00f30e (0x1805 bytes) 000084 1782757301.636582 trunc from 0x040000 to 0x03d854 (0x27ab bytes) 000085 1782757301.694366 mapwrite 0x000db6 thru 0x005472 (0x46bd bytes) 000086 1782757301.821554 read 0x023190 thru 0x02ff4f (0xcdc0 bytes) 000087 1782757301.832802 read 0x00e534 thru 0x016351 (0x7e1e bytes) 000088 1782757301.833668 mapread 0x02bd65 thru 0x0337f8 (0x7a94 bytes) 000089 1782757301.835875 mapread 0x02953e thru 0x02fe21 (0x68e4 bytes) 000090 1782757301.837315 trunc from 0x02c68e to 0x0341f2 (0x7b65 bytes) 000091 1782757301.843263 mapread 0x014690 thru 0x02437a (0xfceb bytes) 000092 1782757301.844764 write 0x0241d2 thru 0x024a05 (0x834 bytes) 000093 1782757301.845812 write 0x034c0f thru 0x03b06b (0x645d bytes) 000094 1782757301.866489 trunc from 0x03d855 to 0x00dd21 (0x2fb33 bytes) 000095 1782757301.906233 mapread 0x0026b9 thru 0x00712c (0x4a74 bytes) 000096 1782757301.907758 mapread 0x006f5e thru 0x00dd21 (0x6dc4 bytes) 000097 1782757301.915608 mapread 0x00d824 thru 0x00dd21 (0x4fe bytes) 000098 1782757301.916692 mapwrite 0x01e6a5 thru 0x0272aa (0x8c06 bytes) 000099 1782757302.088244 read 0x01b082 thru 0x021392 (0x6311 bytes) 000100 1782757302.122301 mapread 0x0060c0 thru 0x00955a (0x349b bytes) 000101 1782757302.123614 trunc from 0x0272ab to 0x01b659 (0xbc51 bytes) 000102 1782757302.162876 MIRROR_EXTEND to 2 mirrors 000103 1782757303.344908 write 0x01d11d thru 0x01d7ae (0x692 bytes) 000104 1782757303.398617 mapread 0x019fe8 thru 0x01a4de (0x4f7 bytes) 000105 1782757303.416302 write 0x032de5 thru 0x03ba71 (0x8c8d bytes) 000106 1782757303.418856 trunc from 0x0084c6 to 0x00d1a5 (0x4ce0 bytes) 000107 1782757303.425289 mapwrite 0x036119 thru 0x03ffff (0x9ee7 bytes) 000108 1782757303.500882 fallocate 0x02e1e8 thru 0x036aa4 (0x88bd bytes) 000109 1782757303.516370 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xfe9fdd6f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfdc69b97 Mirror 3: 0xfdc69b97 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000110 1782757304.908572 punch from 0x033a6a to 0x03ffff (0xc596 bytes) 000111 1782757305.009416 mapread 0x0172b2 thru 0x01ba5e (0x47ad bytes) 000112 1782757305.073594 trunc from 0x01e012 to 0x025f26 (0x7f15 bytes) 000114 1782757305.087139 mapwrite 0x000d8b thru 0x00b6a9 (0xa91f bytes) 000115 1782757305.169383 read 0x03f69e thru 0x03ffff (0x962 bytes) 000116 1782757305.187969 mapwrite 0x013c0b thru 0x01c232 (0x8628 bytes) 000117 1782757305.245613 read 0x038e33 thru 0x03b123 (0x22f1 bytes) 000118 1782757305.263020 punch from 0x038dfb to 0x03ffff (0x7205 bytes) 000119 1782757305.268533 fallocate 0x031118 thru 0x03550f (0x43f8 bytes) 000120 1782757305.288665 mapread 0x013f3e thru 0x01b7ef (0x78b2 bytes) 000121 1782757305.293876 trunc from 0x03ffbb to 0x03ffff (0x45 bytes) 000124 1782757305.308277 fallocate 0x033a9a thru 0x03e6aa (0xac11 bytes) 000125 1782757305.317562 write 0x012783 thru 0x01faad (0xd32b bytes) 000126 1782757305.326198 trunc from 0x00f2e6 to 0x01a9d9 (0xb6f4 bytes) 000127 1782757305.372843 MIRROR_RESYNC 3 mirrors 000128 1782757306.769472 trunc from 0x040000 to 0x014eb6 (0x2b149 bytes) 000129 1782757306.841428 read 0x0026f3 thru 0x00b797 (0x90a5 bytes) 000130 1782757306.876881 fallocate 0x021d0c thru 0x027c6a (0x5f5f bytes) 000131 1782757306.884185 mapread 0x00f883 thru 0x01d5d1 (0xdd4f bytes) 000132 1782757306.939293 punch from 0x014109 to 0x023bbb (0xfab3 bytes) 000134 1782757306.947862 read 0x01fa1d thru 0x01fc8e (0x272 bytes) 000135 1782757306.962363 trunc from 0x027c6b to 0x004a0b (0x2325f bytes) 000136 1782757307.014362 fallocate 0x03f4f3 thru 0x03ffff (0xb0d bytes) 000137 1782757307.037815 mapwrite 0x005611 thru 0x011ccd (0xc6bd bytes) 000138 1782757307.173324 mapwrite 0x010280 thru 0x012407 (0x2188 bytes) 000139 1782757307.192742 trunc from 0x040000 to 0x011a27 (0x2e5d8 bytes) 000140 1782757307.216515 mapwrite 0x024a37 thru 0x02714a (0x2714 bytes) 000141 1782757307.300330 punch from 0x011c79 to 0x019c0d (0x7f95 bytes) 000142 1782757307.315289 fallocate 0x028c49 thru 0x034336 (0xb6ee bytes) 000143 1782757307.327182 fallocate 0x03b712 thru 0x03ffff (0x48ee bytes) 000144 1782757307.342611 mapwrite 0x036a17 thru 0x03ffff (0x95e9 bytes) 000145 1782757307.431575 punch from 0x036cb3 to 0x03ffff (0x934d bytes) 000146 1782757307.444395 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x156:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x74c397f4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x74c397f4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000147 1782757307.885688 trunc from 0x040000 to 0x013c5d (0x2c3a2 bytes) 000148 1782757307.912868 read 0x00249c thru 0x008076 (0x5bdb bytes) 000149 1782757307.924831 fallocate 0x034312 thru 0x03cb88 (0x8877 bytes) 000152 1782757307.933485 trunc from 0x013c5e to 0x000057 (0x13c06 bytes) 000153 1782757307.951680 write 0x03d07c thru 0x03ffff (0x2f84 bytes) 000154 1782757307.953222 write 0x038f3d thru 0x03ffff (0x70c3 bytes) 000155 1782757307.969967 read 0x0033fd thru 0x004e8e (0x1a92 bytes) 000156 1782757307.995103 mapread 0x012443 thru 0x020ed4 (0xea92 bytes) 000157 1782757308.016922 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x156:0x0]: Invalid argument (22) 000158 1782757308.246790 mapread 0x032db4 thru 0x0340f8 (0x1345 bytes) 000159 1782757308.269455 mapwrite 0x02e744 thru 0x03a68d (0xbf4a bytes) 000160 1782757308.296713 trunc from 0x00eb1e to 0x016aa0 (0x7f83 bytes) 000162 1782757308.303964 punch from 0x037d21 to 0x03ffff (0x82df bytes) 000163 1782757308.310294 trunc from 0x040000 to 0x016229 (0x29dd6 bytes) 000164 1782757308.336861 fallocate 0x0172fd thru 0x0227cb (0xb4cf bytes) 000165 1782757308.343622 fallocate 0x01a79b thru 0x02737b (0xcbe1 bytes) 000166 1782757308.350465 punch from 0x01102e to 0x01d5c3 (0xc596 bytes) 000167 1782757308.362984 mapwrite 0x028c9d thru 0x03531e (0xc682 bytes) 000168 1782757308.425758 read 0x008f9c thru 0x014d1b (0xbd80 bytes) 000169 1782757308.435629 trunc from 0x004b40 to 0x0095c8 (0x4a89 bytes) 000171 1782757308.442518 trunc from 0x02af30 to 0x031929 (0x69fa bytes) 000172 1782757308.447644 trunc from 0x03531f to 0x02146c (0x13eb2 bytes) 000173 1782757308.466658 read 0x01cf84 thru 0x02146c (0x44e9 bytes) 000174 1782757308.478630 trunc from 0x02146d to 0x00ab89 (0x168e3 bytes) 000175 1782757308.497485 mapwrite 0x01e44f thru 0x021e32 (0x39e4 bytes) 000176 1782757308.573485 mapwrite 0x00aacd thru 0x0195ce (0xeb02 bytes) 000177 1782757308.744329 read 0x007073 thru 0x0107e9 (0x9777 bytes) 000178 1782757308.745188 write 0x031fc9 thru 0x03b081 (0x90b9 bytes) 000179 1782757308.746966 write 0x03a5b5 thru 0x03ffff (0x5a4b bytes) 000181 1782757308.749420 trunc from 0x02a56d to 0x0365ce (0xc062 bytes) 000182 1782757308.786487 trunc from 0x040000 to 0x02f966 (0x10699 bytes) 000183 1782757308.819651 write 0x01bd2e thru 0x01cab0 (0xd83 bytes) 000184 1782757308.832813 mapread 0x024a5a thru 0x02d7f3 (0x8d9a bytes) 000186 1782757308.835927 trunc from 0x027cbd to 0x02f966 (0x7caa bytes) 000187 1782757308.845387 mapwrite 0x033c87 thru 0x0377d7 (0x3b51 bytes) 000188 1782757308.940748 mapread 0x0017ae thru 0x00b21e (0x9a71 bytes) 000189 1782757308.943766 mapwrite 0x012ed1 thru 0x01ebde (0xbd0e bytes) 000190 1782757308.979392 mapwrite 0x00e6c1 thru 0x0173cb (0x8d0b bytes) 000191 1782757309.006992 write 0x0267b6 thru 0x02d764 (0x6faf bytes) 000192 1782757309.023624 fallocate 0x01e9f8 thru 0x02636f (0x7978 bytes) 000194 1782757309.044335 MIRROR_EXTEND to 2 mirrors 000195 1782757310.122280 fallocate 0x03ea52 thru 0x03ffff (0x15ae bytes) 000196 1782757310.179139 mapwrite 0x02fab8 thru 0x03c7ab (0xccf4 bytes) 000197 1782757310.331512 read 0x03c3a9 thru 0x03c7ab (0x403 bytes) 000198 1782757310.339323 write 0x019cb7 thru 0x0230b5 (0x93ff bytes) 000199 1782757310.365857 fallocate 0x03a8ae thru 0x03bc63 (0x13b6 bytes) 000200 1782757310.374078 read 0x03b298 thru 0x03c7ab (0x1514 bytes) 000201 1782757310.382115 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x156:0x0]: Invalid argument (22) 000202 1782757310.613739 trunc from 0x027ae8 to 0x028173 (0x68c bytes) 000203 1782757310.627987 mapwrite 0x005e3e thru 0x014a96 (0xec59 bytes) 000204 1782757310.711230 fallocate 0x02acac thru 0x02de81 (0x31d6 bytes) 000205 1782757310.717879 punch from 0x01a9c9 to 0x0228eb (0x7f23 bytes) 000206 1782757310.724768 write 0x009851 thru 0x010285 (0x6a35 bytes) 000209 1782757310.736949 write 0x02e848 thru 0x036abf (0x8278 bytes) 000210 1782757310.751334 fallocate 0x01a34c thru 0x01d64a (0x32ff bytes) 000211 1782757310.758618 trunc from 0x00edf1 to 0x019a37 (0xac47 bytes) 000212 1782757310.783520 write 0x03c3c1 thru 0x03ffff (0x3c3f bytes) 000213 1782757310.798203 MIRROR_EXTEND to 2 mirrors 000214 1782757312.092256 trunc from 0x034dab to 0x038562 (0x37b8 bytes) 000215 1782757312.140534 mapwrite 0x018266 thru 0x022fad (0xad48 bytes) 000216 1782757312.199207 mapread 0x020029 thru 0x02a687 (0xa65f bytes) 000218 1782757312.215214 punch from 0x023800 to 0x02f373 (0xbb74 bytes) 000219 1782757312.224651 write 0x00d171 thru 0x013482 (0x6312 bytes) 000220 1782757312.238464 fallocate 0x02003e thru 0x02225c (0x221f bytes) 000221 1782757312.249658 mapread 0x006bc7 thru 0x012273 (0xb6ad bytes) 000222 1782757312.272640 trunc from 0x016380 to 0x0220fd (0xbd7e bytes) 000223 1782757312.281316 mapread 0x02fd63 thru 0x03f234 (0xf4d2 bytes) 000224 1782757312.306282 write 0x00df17 thru 0x00e86d (0x957 bytes) 000226 1782757312.309544 trunc from 0x018ab7 to 0x02007c (0x75c6 bytes) 000227 1782757312.314594 read 0x02c095 thru 0x034d9f (0x8d0b bytes) 000228 1782757312.316479 trunc from 0x040000 to 0x00f3d5 (0x30c2a bytes) 000229 1782757312.353254 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x156:0x0]: Invalid argument (22) 000230 1782757312.516939 trunc from 0x00bb92 to 0x00f3d5 (0x3844 bytes) 000231 1782757312.526662 write 0x037815 thru 0x03ffff (0x87eb bytes) 000232 1782757312.534977 trunc from 0x040000 to 0x02ddb4 (0x1224b bytes) 000233 1782757312.557472 mapread 0x001e9d thru 0x007d9e (0x5f02 bytes) 000235 1782757312.587421 read 0x004365 thru 0x005beb (0x1887 bytes) 000236 1782757312.599235 MIRROR_EXTEND to 2 mirrors 000237 1782757313.620665 mapread 0x001f39 thru 0x007266 (0x532e bytes) 000238 1782757313.683368 punch from 0x02995d to 0x02ddb4 (0x4458 bytes) 000239 1782757313.747583 punch from 0x02db2d to 0x02ddb4 (0x288 bytes) 000240 1782757313.760528 read 0x017f2f thru 0x0189c0 (0xa92 bytes) 000241 1782757313.771155 mapwrite 0x02fc48 thru 0x03119e (0x1557 bytes) 000242 1782757313.862740 trunc from 0x01c67b to 0x027597 (0xaf1d bytes) 000243 1782757313.868902 write 0x014d57 thru 0x02342c (0xe6d6 bytes) 000244 1782757313.890951 write 0x03a7ec thru 0x03ffff (0x5814 bytes) 000245 1782757313.905090 mapwrite 0x039f63 thru 0x03ffff (0x609d bytes) 000246 1782757313.992267 read 0x016e42 thru 0x025f9e (0xf15d bytes) 000247 1782757314.007118 trunc from 0x03703d to 0x03c2a7 (0x526b bytes) 000248 1782757314.029117 trunc from 0x005630 to 0x005e71 (0x842 bytes) 000249 1782757314.050986 trunc from 0x040000 to 0x027862 (0x1879d bytes) 000250 1782757314.142355 MIRROR_RESYNC 2 mirrors 000251 1782757315.587454 write 0x00ed97 thru 0x01946c (0xa6d6 bytes) 000252 1782757315.715200 trunc from 0x009c02 to 0x00cf30 (0x332f bytes) 000253 1782757315.737427 trunc from 0x00cf48 to 0x00d67e (0x737 bytes) 000254 1782757315.757908 punch from 0x00126f to 0x009f69 (0x8cfb bytes) 000255 1782757315.778638 read 0x001483 thru 0x0091b4 (0x7d32 bytes) 000257 1782757315.804230 trunc from 0x021ed2 to 0x027862 (0x5991 bytes) 000258 1782757315.819423 punch from 0x014db4 to 0x0151a0 (0x3ed bytes) 000259 1782757315.877986 read 0x021e8f thru 0x027862 (0x59d4 bytes) 000260 1782757315.921904 mapwrite 0x002d65 thru 0x00522a (0x24c6 bytes) 000262 1782757315.986588 write 0x03c79d thru 0x03e038 (0x189c bytes) 000263 1782757315.987772 mapread 0x021e4c thru 0x02b7b6 (0x996b bytes) 000264 1782757316.043406 read 0x03b572 thru 0x03e038 (0x2ac7 bytes) 000265 1782757316.044229 mapwrite 0x0268b0 thru 0x0284b8 (0x1c09 bytes) 000266 1782757316.082696 mapwrite 0x025cb4 thru 0x0322fd (0xc64a bytes) 000267 1782757316.162232 read 0x00bfb6 thru 0x00d829 (0x1874 bytes) 000268 1782757316.176719 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e039): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e039): Mirror 2: 0x1196ea59 (stale) CRC-32 checksum value for chunk [0, 0x3e039): Mirror 1: 0xffd9b22a Mirror 3: 0xffd9b22a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e039: skipped 000269 1782757317.915510 mapwrite 0x029698 thru 0x029b66 (0x4cf bytes) 000271 1782757318.023331 trunc from 0x03e039 to 0x012537 (0x2bb01 bytes) 000272 1782757318.053661 write 0x01df2f thru 0x0246c9 (0x679b bytes) 000273 1782757318.058104 write 0x01f602 thru 0x022d54 (0x3753 bytes) 000274 1782757318.062856 trunc from 0x0246ca to 0x012d83 (0x11946 bytes) 000275 1782757318.089933 fallocate 0x01a1ab thru 0x01ca41 (0x2897 bytes) 000276 1782757318.096286 punch from 0x0096a5 to 0x00ea74 (0x53d0 bytes) 000277 1782757318.103165 write 0x00ef7b thru 0x01c831 (0xd8b7 bytes) 000278 1782757318.115319 trunc from 0x01ca42 to 0x03ae70 (0x1e42f bytes) 000279 1782757318.156321 punch from 0x03934a to 0x03ae70 (0x1b27 bytes) 000280 1782757318.166854 MIRROR_RESYNC 3 mirrors 000281 1782757320.111386 trunc from 0x03ae71 to 0x0195b0 (0x218c0 bytes) 000282 1782757320.284413 trunc from 0x0080a8 to 0x01766f (0xf5c8 bytes) 000283 1782757320.297973 read 0x00cae2 thru 0x0195b0 (0xcacf bytes) 000284 1782757320.330585 trunc from 0x00804c to 0x009f9b (0x1f50 bytes) 000285 1782757320.348770 mapread 0x00c5a5 thru 0x0195b0 (0xd00c bytes) 000286 1782757320.357649 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x156:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x195b1): Mirror 2: 0xe366458c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x195b1): Mirror 2: 0xe366458c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x195b1: skipped 000287 1782757320.732898 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x156:0x0]: Invalid argument (22) 000288 1782757320.927979 punch from 0x009731 to 0x00d6af (0x3f7f bytes) 000289 1782757320.935465 trunc from 0x00dac0 to 0x0177e2 (0x9d23 bytes) 000290 1782757320.942807 trunc from 0x0195b1 to 0x00496f (0x14c41 bytes) 000291 1782757320.965478 write 0x000ba1 thru 0x002bf8 (0x2058 bytes) 000292 1782757321.006118 MIRROR_EXTEND to 2 mirrors 000293 1782757322.179190 read 0x0043d4 thru 0x00496f (0x59c bytes) 000294 1782757322.186857 trunc from 0x00046e to 0x00496f (0x4502 bytes) 000295 1782757322.255918 fallocate 0x02c255 thru 0x02dd04 (0x1ab0 bytes) 000296 1782757322.267634 punch from 0x0223d4 to 0x02dd04 (0xb931 bytes) 000297 1782757322.275414 trunc from 0x02dd05 to 0x01546f (0x18895 bytes) 000298 1782757322.348511 mapwrite 0x00aeee thru 0x019443 (0xe556 bytes) 000299 1782757322.501223 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x19444): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19444): Mirror 2: 0x6852375f (stale) CRC-32 checksum value for chunk [0, 0x19444): Mirror 1: 0xaba0c390 Mirror 3: 0xaba0c390 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19444: skipped 000300 1782757323.807490 write 0x01b99b thru 0x0293c6 (0xda2c bytes) 000301 1782757323.853220 read 0x01eb9f thru 0x0293c6 (0xa828 bytes) 000302 1782757323.854146 mapread 0x025cf9 thru 0x0293c6 (0x36ce bytes) 000303 1782757323.859321 mapread 0x0081bf thru 0x010d74 (0x8bb6 bytes) 000304 1782757323.899389 fallocate 0x023ea1 thru 0x0290f5 (0x5255 bytes) 000305 1782757323.925320 punch from 0x00d6ad to 0x0167a0 (0x90f4 bytes) 000306 1782757323.938535 mapread 0x0004d8 thru 0x0081a5 (0x7cce bytes) 000307 1782757323.972206 read 0x010979 thru 0x012fbf (0x2647 bytes) 000308 1782757323.989773 punch from 0x01326c to 0x021dd8 (0xeb6d bytes) 000309 1782757324.013502 trunc from 0x022eb9 to 0x02454e (0x1696 bytes) 000310 1782757324.034010 read 0x00f135 thru 0x010f4e (0x1e1a bytes) 000311 1782757324.036075 fallocate 0x0259c5 thru 0x027f4f (0x258b bytes) 000312 1782757324.051549 MIRROR_RESYNC 3 mirrors 000314 1782757325.420916 read 0x018e1d thru 0x0218fc (0x8ae0 bytes) 000315 1782757325.434003 mapwrite 0x0331c6 thru 0x03355e (0x399 bytes) 000316 1782757325.549451 punch from 0x00d03b to 0x0135b3 (0x6579 bytes) 000317 1782757325.564172 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3355f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3355f): Mirror 2: 0x980f4841 (stale) Mirror 3: 0x980f4841 (stale) CRC-32 checksum value for chunk [0, 0x3355f): Mirror 1: 0x25881140 Mirror 4: 0x25881140 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3355f: skipped 000318 1782757327.498749 mapread 0x025ff1 thru 0x02e6df (0x86ef bytes) 000319 1782757327.538724 trunc from 0x03355f to 0x032d88 (0x7d6 bytes) 000320 1782757327.655368 punch from 0x01ddc2 to 0x01e023 (0x262 bytes) 000321 1782757327.676510 fallocate 0x02f1a7 thru 0x03d1fe (0xe058 bytes) 000322 1782757327.698727 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x156:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d1ff): Mirror 3: 0x7a591edd (stale) Mirror 4: 0x9dd76f1f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d1ff): Mirror 3: 0x7a591edd (stale) Mirror 4: 0x9dd76f1f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d1ff: skipped 000323 1782757328.481913 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3d1ff): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d1ff): Mirror 3: 0x7a591edd (stale) Mirror 4: 0x9dd76f1f (stale) CRC-32 checksum value for chunk [0, 0x3d1ff): Mirror 1: 0x47796fd2 Mirror 5: 0x47796fd2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d1ff: skipped 000324 1782757330.450024 punch from 0x021356 to 0x02dd92 (0xca3d bytes) 000325 1782757330.580338 fallocate 0x02a509 thru 0x02b591 (0x1089 bytes) 000326 1782757330.606313 write 0x0121f4 thru 0x015dea (0x3bf7 bytes) 000327 1782757330.643871 mapread 0x02329e thru 0x026fbb (0x3d1e bytes) 000328 1782757330.697871 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3d1ff): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d1ff): Mirror 3: 0x7a591edd (stale) Mirror 4: 0x9dd76f1f (stale) Mirror 5: 0x47796fd2 (stale) CRC-32 checksum value for chunk [0, 0x3d1ff): Mirror 1: 0x227b95b0 Mirror 6: 0x227b95b0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d1ff: skipped 000329 1782757332.135903 fallocate 0x031f36 thru 0x035804 (0x38cf bytes) 000330 1782757332.237001 mapwrite 0x00096f thru 0x009dbf (0x9451 bytes) 000331 1782757332.345642 read 0x00fcbc thru 0x0148f4 (0x4c39 bytes) 000332 1782757332.360847 fallocate 0x01a71a thru 0x021c01 (0x74e8 bytes) 000333 1782757332.376804 trunc from 0x0073b5 to 0x013d96 (0xc9e2 bytes) 000334 1782757332.393025 fallocate 0x024e86 thru 0x028d36 (0x3eb1 bytes) 000335 1782757332.409927 mapwrite 0x01beac thru 0x02410b (0x8260 bytes) 000336 1782757332.459596 read 0x02b90f thru 0x0341d3 (0x88c5 bytes) 000337 1782757332.477078 trunc from 0x0117bf to 0x020fd9 (0xf81b bytes) 000339 1782757332.489045 read 0x028b7c thru 0x0336b3 (0xab38 bytes) 000340 1782757332.498638 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3d1ff): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d1ff): Mirror 3: 0x7a591edd (stale) Mirror 4: 0x9dd76f1f (stale) Mirror 5: 0x47796fd2 (stale) Mirror 6: 0x227b95b0 (stale) CRC-32 checksum value for chunk [0, 0x3d1ff): Mirror 1: 0x73287822 Mirror 7: 0x73287822 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d1ff: skipped 000342 1782757333.641883 mapwrite 0x01ed4f thru 0x0264bc (0x776e bytes) 000343 1782757333.781788 write 0x00f40f thru 0x012683 (0x3275 bytes) 000344 1782757333.813586 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3d1ff): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d1ff): Mirror 3: 0x7a591edd (stale) Mirror 4: 0x9dd76f1f (stale) Mirror 5: 0x47796fd2 (stale) Mirror 6: 0x227b95b0 (stale) Mirror 7: 0x73287822 (stale) CRC-32 checksum value for chunk [0, 0x3d1ff): Mirror 1: 0xc8dfe644 Mirror 8: 0xc8dfe644 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d1ff: skipped 000345 1782757335.632947 trunc from 0x03d1ff to 0x03b7e6 (0x1a18 bytes) 000347 1782757335.707479 trunc from 0x0237c5 to 0x030622 (0xce5e bytes) 000348 1782757335.716468 trunc from 0x003497 to 0x0058f6 (0x2460 bytes) 000349 1782757335.725303 write 0x03b265 thru 0x03ffff (0x4d9b bytes) 000350 1782757335.738005 punch from 0x03a23e to 0x03cae8 (0x28ab bytes) 000351 1782757335.766652 trunc from 0x0085c2 to 0x00d18f (0x4bce bytes) 000352 1782757335.776269 mapread 0x01bded thru 0x026a10 (0xac24 bytes) 000353 1782757335.839686 mapwrite 0x005417 thru 0x013084 (0xdc6e bytes) 000355 1782757335.903006 mapread 0x02c01b thru 0x0334a7 (0x748d bytes) 000357 1782757335.917632 fallocate 0x021c54 thru 0x0303cb (0xe778 bytes) 000358 1782757335.924228 trunc from 0x017190 to 0x018eee (0x1d5f bytes) 000359 1782757335.931929 read 0x01bddf thru 0x01f012 (0x3234 bytes) 000360 1782757335.934929 trunc from 0x03f139 to 0x03ffff (0xec7 bytes) 000362 1782757335.944462 mapread 0x01d901 thru 0x0215b6 (0x3cb6 bytes) 000363 1782757335.958166 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfa31e038 (stale) Mirror 4: 0xdc79e7d1 (stale) Mirror 5: 0x805e1e12 (stale) Mirror 6: 0x73b52887 (stale) Mirror 7: 0x5e589b70 (stale) Mirror 8: 0x35785b14 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf5a3e890 Mirror 9: 0xf5a3e890 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000364 1782757337.966181 fallocate 0x0180fa thru 0x01ab7f (0x2a86 bytes) 000365 1782757338.065074 write 0x01519c thru 0x018bf4 (0x3a59 bytes) 000366 1782757338.113418 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x156:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfa31e038 (stale) Mirror 4: 0xdc79e7d1 (stale) Mirror 5: 0x805e1e12 (stale) Mirror 6: 0x73b52887 (stale) Mirror 8: 0x35785b14 (stale) Mirror 9: 0xf5a3e890 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfa31e038 (stale) Mirror 4: 0xdc79e7d1 (stale) Mirror 5: 0x805e1e12 (stale) Mirror 6: 0x73b52887 (stale) Mirror 8: 0x35785b14 (stale) Mirror 9: 0xf5a3e890 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000367 1782757338.866988 read 0x00fb8a thru 0x015674 (0x5aeb bytes) 000368 1782757338.881444 read 0x002487 thru 0x0078e0 (0x545a bytes) 000369 1782757338.896307 write 0x036b1e thru 0x03ffff (0x94e2 bytes) 000370 1782757338.907974 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfa31e038 (stale) Mirror 4: 0xdc79e7d1 (stale) Mirror 5: 0x805e1e12 (stale) Mirror 6: 0x73b52887 (stale) Mirror 8: 0x35785b14 (stale) Mirror 9: 0xf5a3e890 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x969ab261 Mirror 10: 0x969ab261 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000371 1782757340.471182 trunc from 0x005924 to 0x00d44a (0x7b27 bytes) 000372 1782757340.531817 punch from 0x00e893 to 0x018506 (0x9c74 bytes) 000373 1782757340.538160 write 0x0371ba thru 0x03f075 (0x7ebc bytes) 000374 1782757340.560636 write 0x0203de thru 0x026020 (0x5c43 bytes) 000375 1782757340.580687 trunc from 0x0322da to 0x03c040 (0x9d67 bytes) 000376 1782757340.616403 trunc from 0x03bc83 to 0x03fff9 (0x4377 bytes) 000377 1782757340.623902 mapread 0x00ab35 thru 0x01431d (0x97e9 bytes) 000378 1782757340.675482 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfa31e038 (stale) Mirror 4: 0xdc79e7d1 (stale) Mirror 5: 0x805e1e12 (stale) Mirror 6: 0x73b52887 (stale) Mirror 8: 0x35785b14 (stale) Mirror 9: 0xf5a3e890 (stale) Mirror 10: 0x969ab261 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x23866d7c Mirror 11: 0x23866d7c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000379 1782757342.380037 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x156:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfa31e038 (stale) Mirror 4: 0xdc79e7d1 (stale) Mirror 6: 0x73b52887 (stale) Mirror 8: 0x35785b14 (stale) Mirror 9: 0xf5a3e890 (stale) Mirror 10: 0x969ab261 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfa31e038 (stale) Mirror 4: 0xdc79e7d1 (stale) Mirror 6: 0x73b52887 (stale) Mirror 8: 0x35785b14 (stale) Mirror 9: 0xf5a3e890 (stale) Mirror 10: 0x969ab261 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x23866d7c Mirror 11: 0x23866d7c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000380 1782757343.448923 write 0x027714 thru 0x02cf93 (0x5880 bytes) 000381 1782757343.531708 MIRROR_RESYNC 8 mirrors 000382 1782757347.921538 mapread 0x025bb2 thru 0x031c7c (0xc0cb bytes) 000383 1782757347.960891 mapread 0x00e10f thru 0x00e29b (0x18d bytes) 000384 1782757347.974792 trunc from 0x014562 to 0x02261a (0xe0b9 bytes) 000385 1782757348.029333 trunc from 0x0020af to 0x008470 (0x63c2 bytes) 000386 1782757348.037775 mapread 0x02e34d thru 0x038fe0 (0xac94 bytes) 000387 1782757348.071712 trunc from 0x0157a4 to 0x024860 (0xf0bd bytes) 000388 1782757348.081536 read 0x02e363 thru 0x036057 (0x7cf5 bytes) 000389 1782757348.082916 mapwrite 0x00e960 thru 0x0185bb (0x9c5c bytes) 000390 1782757348.177782 trunc from 0x00ad5e to 0x00ce86 (0x2129 bytes) 000391 1782757348.192457 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x72f954b2 (stale) Mirror 4: 0x72f954b2 (stale) Mirror 6: 0x72f954b2 (stale) Mirror 8: 0x72f954b2 (stale) Mirror 9: 0x72f954b2 (stale) Mirror 10: 0x72f954b2 (stale) Mirror 11: 0x72f954b2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7b1d746a Mirror 12: 0x7b1d746a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000392 1782757349.811805 MIRROR_RESYNC 9 mirrors 000393 1782757353.128430 mapread 0x000747 thru 0x008d65 (0x861f bytes) 000394 1782757353.167841 mapwrite 0x0099a7 thru 0x011a05 (0x805f bytes) 000395 1782757353.268825 MIRROR_RESYNC 9 mirrors 000397 1782757356.075586 mapread 0x03a391 thru 0x03ffff (0x5c6f bytes) 000398 1782757356.100237 mapwrite 0x017f04 thru 0x0264d0 (0xe5cd bytes) 000400 1782757356.243896 trunc from 0x031ec1 to 0x03ae83 (0x8fc3 bytes) 000401 1782757356.254647 mapwrite 0x02fa01 thru 0x039572 (0x9b72 bytes) 000402 1782757356.374852 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x445603c3 (stale) Mirror 4: 0x445603c3 (stale) Mirror 6: 0x445603c3 (stale) Mirror 8: 0x445603c3 (stale) Mirror 9: 0x445603c3 (stale) Mirror 10: 0x445603c3 (stale) Mirror 11: 0x445603c3 (stale) Mirror 12: 0x445603c3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2fb3236c Mirror 13: 0x2fb3236c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000403 1782757357.806919 read 0x038067 thru 0x03dbda (0x5b74 bytes) 000404 1782757357.829951 mapwrite 0x0367b6 thru 0x03bcbd (0x5508 bytes) 000405 1782757357.976112 punch from 0x036331 to 0x03cc38 (0x6908 bytes) 000406 1782757357.997647 punch from 0x03291e to 0x03fa09 (0xd0ec bytes) 000407 1782757358.011392 trunc from 0x02b7b4 to 0x030616 (0x4e63 bytes) 000408 1782757358.026357 read 0x029799 thru 0x02ec4a (0x54b2 bytes) 000409 1782757358.065474 mapread 0x035918 thru 0x0383c0 (0x2aa9 bytes) 000410 1782757358.072159 trunc from 0x040000 to 0x03e32f (0x1cd0 bytes) 000411 1782757358.122112 mapread 0x027fdd thru 0x0366fd (0xe721 bytes) 000413 1782757358.130542 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3e330): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e330): Mirror 3: 0xe0487238 (stale) Mirror 4: 0xe0487238 (stale) Mirror 6: 0xe0487238 (stale) Mirror 8: 0xe0487238 (stale) Mirror 9: 0xe0487238 (stale) Mirror 10: 0xe0487238 (stale) Mirror 11: 0xe0487238 (stale) Mirror 12: 0xe0487238 (stale) Mirror 13: 0x504e0cf4 (stale) CRC-32 checksum value for chunk [0, 0x3e330): Mirror 1: 0x315db175 Mirror 14: 0x315db175 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e330: skipped 000415 1782757359.365147 trunc from 0x03e330 to 0x012953 (0x2b9dc bytes) 000416 1782757359.443005 punch from 0x000e20 to 0x00b277 (0xa458 bytes) 000417 1782757359.448295 mapread 0x003d6a thru 0x007827 (0x3abe bytes) 000418 1782757359.468837 mapwrite 0x020dd5 thru 0x02f73e (0xe96a bytes) 000419 1782757359.569449 fallocate 0x01f720 thru 0x02ebfc (0xf4dd bytes) 000420 1782757359.577154 fallocate 0x03cd4e thru 0x03ffff (0x32b2 bytes) 000421 1782757359.590641 mapwrite 0x024bac thru 0x02d4f4 (0x8949 bytes) 000422 1782757359.652680 mapwrite 0x027b59 thru 0x0327a7 (0xac4f bytes) 000423 1782757359.732327 write 0x01f3dc thru 0x022e39 (0x3a5e bytes) 000425 1782757359.734967 trunc from 0x0327a8 to 0x010fec (0x217bb bytes) 000426 1782757359.759992 mapwrite 0x026256 thru 0x029a24 (0x37cf bytes) 000427 1782757359.843173 MIRROR_RESYNC 11 mirrors 000428 1782757362.771633 mapread 0x0056f6 thru 0x009cf8 (0x4603 bytes) 000429 1782757362.848687 mapread 0x018779 thru 0x01e019 (0x58a1 bytes) 000430 1782757362.875765 mapread 0x0012ca thru 0x00537f (0x40b6 bytes) 000431 1782757362.876792 mapwrite 0x031ec9 thru 0x03e815 (0xc94d bytes) 000432 1782757363.017431 mapread 0x00f691 thru 0x01150d (0x1e7d bytes) 000434 1782757363.056469 mapwrite 0x007f6f thru 0x016a70 (0xeb02 bytes) 000435 1782757363.130609 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3e816): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e816): Mirror 3: 0xaf30c8f5 (stale) Mirror 4: 0xaf30c8f5 (stale) Mirror 6: 0xaf30c8f5 (stale) Mirror 8: 0xaf30c8f5 (stale) Mirror 9: 0xaf30c8f5 (stale) Mirror 10: 0xaf30c8f5 (stale) Mirror 11: 0xaf30c8f5 (stale) Mirror 12: 0xaf30c8f5 (stale) Mirror 13: 0xaf30c8f5 (stale) Mirror 14: 0xaf30c8f5 (stale) CRC-32 checksum value for chunk [0, 0x3e816): Mirror 1: 0xfbf25194 Mirror 15: 0xfbf25194 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e816: skipped 000436 1782757364.539254 mapwrite 0x012618 thru 0x01ed60 (0xc749 bytes) 000437 1782757364.683105 MIRROR_RESYNC 12 mirrors 000439 1782757369.323344 fallocate 0x0015b2 thru 0x00d23a (0xbc89 bytes) 000440 1782757369.465019 MIRROR_RESYNC 12 mirrors 000441 1782757373.341128 trunc from 0x03e816 to 0x019e43 (0x249d2 bytes) 000442 1782757373.405620 read 0x0088bf thru 0x0102be (0x7a00 bytes) 000443 1782757373.426668 fallocate 0x013985 thru 0x022d5d (0xf3d9 bytes) 000444 1782757373.434020 mapwrite 0x030af4 thru 0x03929b (0x87a8 bytes) 000445 1782757373.569787 mapread 0x0115bd thru 0x02099a (0xf3de bytes) 000446 1782757373.598016 trunc from 0x03929c to 0x022d60 (0x1653b bytes) 000447 1782757373.616992 mapread 0x0155a2 thru 0x022d60 (0xd7bf bytes) 000448 1782757373.619227 trunc from 0x014011 to 0x015fc8 (0x1fb8 bytes) 000449 1782757373.636425 mapwrite 0x01b118 thru 0x024935 (0x981e bytes) 000450 1782757373.724234 punch from 0x01673e to 0x024935 (0xe1f8 bytes) 000451 1782757373.744472 MIRROR_SPLIT mirror 6 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x156:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24936): Mirror 3: 0x2c5390cf (stale) Mirror 4: 0x2c5390cf (stale) Mirror 6: 0x2c5390cf (stale) Mirror 8: 0x2c5390cf (stale) Mirror 9: 0x2c5390cf (stale) Mirror 11: 0x2c5390cf (stale) Mirror 12: 0x2c5390cf (stale) Mirror 13: 0x2c5390cf (stale) Mirror 14: 0x2c5390cf (stale) Mirror 15: 0x2c5390cf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24936): Mirror 3: 0x2c5390cf (stale) Mirror 4: 0x2c5390cf (stale) Mirror 6: 0x2c5390cf (stale) Mirror 8: 0x2c5390cf (stale) Mirror 9: 0x2c5390cf (stale) Mirror 11: 0x2c5390cf (stale) Mirror 12: 0x2c5390cf (stale) Mirror 13: 0x2c5390cf (stale) Mirror 14: 0x2c5390cf (stale) Mirror 15: 0x2c5390cf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24936: skipped 000452 1782757374.684239 read 0x01387d thru 0x021911 (0xe095 bytes) 000453 1782757374.697838 read 0x008a3a thru 0x009093 (0x65a bytes) 000454 1782757374.709402 write 0x016a31 thru 0x017f1d (0x14ed bytes) 000455 1782757374.719858 mapwrite 0x01b609 thru 0x01f3b3 (0x3dab bytes) 000456 1782757374.742037 fallocate 0x00aea3 thru 0x0144d7 (0x9635 bytes) 000457 1782757374.751970 fallocate 0x02f1e1 thru 0x038cbc (0x9adc bytes) 000458 1782757374.763479 fallocate 0x038357 thru 0x03e266 (0x5f10 bytes) 000459 1782757374.771748 write 0x035276 thru 0x03ffff (0xad8a bytes) 000460 1782757374.803879 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x450cb2c2 (stale) Mirror 4: 0x450cb2c2 (stale) Mirror 6: 0x450cb2c2 (stale) Mirror 8: 0x450cb2c2 (stale) Mirror 9: 0x450cb2c2 (stale) Mirror 11: 0x450cb2c2 (stale) Mirror 12: 0x450cb2c2 (stale) Mirror 13: 0x450cb2c2 (stale) Mirror 14: 0x450cb2c2 (stale) Mirror 15: 0x450cb2c2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf8d0bf5c Mirror 16: 0xf8d0bf5c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000461 1782757376.423014 write 0x01ef54 thru 0x02db1f (0xebcc bytes) 000462 1782757376.478966 punch from 0x0211b1 to 0x02d692 (0xc4e2 bytes) 000463 1782757376.494341 mapread 0x012428 thru 0x017765 (0x533e bytes) 000464 1782757376.519378 trunc from 0x01f259 to 0x029f1f (0xacc7 bytes) 000465 1782757376.533517 write 0x01ba7d thru 0x02924d (0xd7d1 bytes) 000466 1782757376.542882 mapwrite 0x01b6b4 thru 0x026f9c (0xb8e9 bytes) 000467 1782757376.606366 write 0x014b87 thru 0x01d778 (0x8bf2 bytes) 000468 1782757376.609686 read 0x01139e thru 0x019969 (0x85cc bytes) 000469 1782757376.612692 mapwrite 0x02bc8d thru 0x038c24 (0xcf98 bytes) 000470 1782757376.721357 write 0x029cf6 thru 0x0361df (0xc4ea bytes) 000471 1782757376.724913 fallocate 0x029acb thru 0x02d986 (0x3ebc bytes) 000472 1782757376.732912 mapwrite 0x001773 thru 0x006513 (0x4da1 bytes) 000473 1782757376.761335 write 0x002902 thru 0x003480 (0xb7f bytes) 000474 1782757376.767804 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x450cb2c2 (stale) Mirror 4: 0x450cb2c2 (stale) Mirror 6: 0x450cb2c2 (stale) Mirror 8: 0x450cb2c2 (stale) Mirror 9: 0x450cb2c2 (stale) Mirror 11: 0x450cb2c2 (stale) Mirror 12: 0x450cb2c2 (stale) Mirror 13: 0x450cb2c2 (stale) Mirror 14: 0x450cb2c2 (stale) Mirror 15: 0x450cb2c2 (stale) Mirror 16: 0xf8d0bf5c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbdde93bc Mirror 17: 0xbdde93bc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000475 1782757378.262526 mapread 0x02b08b thru 0x03acdf (0xfc55 bytes) 000476 1782757378.307551 fallocate 0x001867 thru 0x00ce99 (0xb633 bytes) 000478 1782757378.363998 write 0x006276 thru 0x00c0a6 (0x5e31 bytes) 000479 1782757378.389373 fallocate 0x021c5a thru 0x029e6c (0x8213 bytes) 000481 1782757378.399508 trunc from 0x040000 to 0x001b02 (0x3e4fd bytes) 000482 1782757378.433905 mapwrite 0x012e68 thru 0x01a3b0 (0x7549 bytes) 000483 1782757378.527775 mapwrite 0x01acd4 thru 0x01deb5 (0x31e2 bytes) 000484 1782757378.627712 mapread 0x019085 thru 0x01deb5 (0x4e31 bytes) 000485 1782757378.636694 mapread 0x01189b thru 0x01b24c (0x99b2 bytes) 000486 1782757378.669593 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x1deb6): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1deb6): Mirror 3: 0x41c71ca4 (stale) Mirror 4: 0x41c71ca4 (stale) Mirror 6: 0x41c71ca4 (stale) Mirror 8: 0x41c71ca4 (stale) Mirror 9: 0x41c71ca4 (stale) Mirror 11: 0x41c71ca4 (stale) Mirror 12: 0x41c71ca4 (stale) Mirror 13: 0x41c71ca4 (stale) Mirror 14: 0x41c71ca4 (stale) Mirror 15: 0x41c71ca4 (stale) Mirror 16: 0x24a75a0 (stale) Mirror 17: 0x2546fe08 (stale) CRC-32 checksum value for chunk [0, 0x1deb6): Mirror 1: 0x201edeac Mirror 18: 0x201edeac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1deb6: skipped 000488 1782757380.299848 mapwrite 0x02898a thru 0x02cd6c (0x43e3 bytes) 000489 1782757380.542411 mapwrite 0x03d7ef thru 0x03ffff (0x2811 bytes) 000490 1782757380.677442 mapwrite 0x02d743 thru 0x03828f (0xab4d bytes) 000491 1782757380.776355 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x450cb2c2 (stale) Mirror 4: 0x450cb2c2 (stale) Mirror 6: 0x450cb2c2 (stale) Mirror 8: 0x450cb2c2 (stale) Mirror 9: 0x450cb2c2 (stale) Mirror 11: 0x450cb2c2 (stale) Mirror 12: 0x450cb2c2 (stale) Mirror 13: 0x450cb2c2 (stale) Mirror 14: 0x450cb2c2 (stale) Mirror 15: 0x450cb2c2 (stale) Mirror 16: 0xf8d0bf5c (stale) Mirror 17: 0xbdde93bc (stale) Mirror 18: 0xdbe57627 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6651a931 Mirror 19: 0x6651a931 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000492 1782757382.472645 write 0x0268cd thru 0x02e4db (0x7c0f bytes) 000493 1782757382.583722 MIRROR_SPLIT mirror 5 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x156:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x450cb2c2 (stale) Mirror 4: 0x450cb2c2 (stale) Mirror 6: 0x450cb2c2 (stale) Mirror 8: 0x450cb2c2 (stale) Mirror 11: 0x450cb2c2 (stale) Mirror 12: 0x450cb2c2 (stale) Mirror 13: 0x450cb2c2 (stale) Mirror 14: 0x450cb2c2 (stale) Mirror 15: 0x450cb2c2 (stale) Mirror 16: 0xf8d0bf5c (stale) Mirror 17: 0xbdde93bc (stale) Mirror 18: 0xdbe57627 (stale) Mirror 19: 0x6651a931 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x450cb2c2 (stale) Mirror 4: 0x450cb2c2 (stale) Mirror 6: 0x450cb2c2 (stale) Mirror 8: 0x450cb2c2 (stale) Mirror 11: 0x450cb2c2 (stale) Mirror 12: 0x450cb2c2 (stale) Mirror 13: 0x450cb2c2 (stale) Mirror 14: 0x450cb2c2 (stale) Mirror 15: 0x450cb2c2 (stale) Mirror 16: 0xf8d0bf5c (stale) Mirror 17: 0xbdde93bc (stale) Mirror 18: 0xdbe57627 (stale) Mirror 19: 0x6651a931 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000494 1782757383.922241 mapread 0x019312 thru 0x01fa16 (0x6705 bytes) 000495 1782757383.974627 trunc from 0x040000 to 0x0149ab (0x2b654 bytes) 000496 1782757384.032886 trunc from 0x0149ac to 0x039540 (0x24b95 bytes) 000497 1782757384.077044 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x39541): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39541): Mirror 3: 0x3fd37aa9 (stale) Mirror 4: 0x3fd37aa9 (stale) Mirror 6: 0x3fd37aa9 (stale) Mirror 8: 0x3fd37aa9 (stale) Mirror 11: 0x3fd37aa9 (stale) Mirror 12: 0x3fd37aa9 (stale) Mirror 13: 0x3fd37aa9 (stale) Mirror 14: 0x3fd37aa9 (stale) Mirror 15: 0x3fd37aa9 (stale) Mirror 16: 0x82926ae4 (stale) Mirror 17: 0xd59d58d7 (stale) Mirror 18: 0x21bc1a90 (stale) Mirror 19: 0x67535089 (stale) CRC-32 checksum value for chunk [0, 0x39541): Mirror 1: 0xcc35b05b Mirror 20: 0xcc35b05b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39541: skipped 000498 1782757385.589453 trunc from 0x039541 to 0x0269b9 (0x12b87 bytes) 000499 1782757385.713984 punch from 0x00ba20 to 0x016c5e (0xb23f bytes) 000500 1782757385.725476 mapwrite 0x035008 thru 0x03b3e7 (0x63e0 bytes) 000501 1782757385.903275 trunc from 0x03b3e8 to 0x032b8e (0x8859 bytes) 000502 1782757385.945883 trunc from 0x004419 to 0x012cc0 (0xe8a8 bytes) 000503 1782757385.952534 write 0x03be3a thru 0x03f3ee (0x35b5 bytes) 000504 1782757385.958415 trunc from 0x03f3ef to 0x01fe0c (0x1f5e2 bytes) 000505 1782757386.009181 trunc from 0x006722 to 0x0084a7 (0x1d86 bytes) 000506 1782757386.016818 read 0x00b7d2 thru 0x00bac8 (0x2f7 bytes) 000507 1782757386.032912 punch from 0x01b3bf to 0x01d06c (0x1cae bytes) 000508 1782757386.041217 read 0x01acc9 thru 0x01fb85 (0x4ebd bytes) 000509 1782757386.074880 fallocate 0x034bef thru 0x036923 (0x1d35 bytes) 000510 1782757386.082416 mapread 0x014d17 thru 0x01c85e (0x7b48 bytes) 000511 1782757386.091613 mapread 0x00e312 thru 0x00fe11 (0x1b00 bytes) 000512 1782757386.096691 trunc from 0x036924 to 0x008a29 (0x2defa bytes) 000514 1782757386.144260 mapwrite 0x02d6eb thru 0x0364e3 (0x8df9 bytes) 000515 1782757386.271989 MIRROR_SPLIT mirror 1 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x156:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x364e4): Mirror 4: 0x22020a4e (stale) Mirror 6: 0x22020a4e (stale) Mirror 8: 0x22020a4e (stale) Mirror 11: 0x22020a4e (stale) Mirror 12: 0x22020a4e (stale) Mirror 13: 0x22020a4e (stale) Mirror 14: 0x22020a4e (stale) Mirror 15: 0x22020a4e (stale) Mirror 16: 0x6791bf9b (stale) Mirror 17: 0xcf9f9e80 (stale) Mirror 18: 0xba0ae54b (stale) Mirror 19: 0x9b77084d (stale) Mirror 20: 0x65c931d7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x364e4): Mirror 4: 0x22020a4e (stale) Mirror 6: 0x22020a4e (stale) Mirror 8: 0x22020a4e (stale) Mirror 11: 0x22020a4e (stale) Mirror 12: 0x22020a4e (stale) Mirror 13: 0x22020a4e (stale) Mirror 14: 0x22020a4e (stale) Mirror 15: 0x22020a4e (stale) Mirror 16: 0x6791bf9b (stale) Mirror 17: 0xcf9f9e80 (stale) Mirror 18: 0xba0ae54b (stale) Mirror 19: 0x9b77084d (stale) Mirror 20: 0x65c931d7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x364e4: skipped 000517 1782757387.417201 read 0x032f14 thru 0x0364e3 (0x35d0 bytes) 000519 1782757387.430348 read 0x027642 thru 0x03297b (0xb33a bytes) 000520 1782757387.448025 trunc from 0x0364e4 to 0x00cfc9 (0x2951a bytes) 000521 1782757387.489763 mapwrite 0x01677e thru 0x01a82b (0x40ae bytes) 000522 1782757387.648848 trunc from 0x01a82c to 0x03eb80 (0x24355 bytes) 000523 1782757387.680034 mapread 0x009fa8 thru 0x01523b (0xb294 bytes) 000524 1782757387.819650 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x3eb81): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eb81): Mirror 4: 0x8e2ad723 (stale) Mirror 6: 0x8e2ad723 (stale) Mirror 8: 0x8e2ad723 (stale) Mirror 11: 0x8e2ad723 (stale) Mirror 12: 0x8e2ad723 (stale) Mirror 13: 0x8e2ad723 (stale) Mirror 14: 0x8e2ad723 (stale) Mirror 15: 0x8e2ad723 (stale) Mirror 16: 0x62a7db2d (stale) Mirror 17: 0xd493dc55 (stale) Mirror 18: 0xd432ed6a (stale) Mirror 19: 0x407bebbc (stale) Mirror 20: 0x8d5ed505 (stale) CRC-32 checksum value for chunk [0, 0x3eb81): Mirror 1: 0xce6d827a Mirror 21: 0xce6d827a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eb81: skipped 000525 1782757390.135957 trunc from 0x03eb81 to 0x012503 (0x2c67d bytes) 000526 1782757390.263580 mapread 0x00985d thru 0x012503 (0x8ca7 bytes) 000527 1782757390.289903 read 0x00fdf5 thru 0x012503 (0x270f bytes) 000528 1782757390.291134 fallocate 0x030a37 thru 0x03a5c1 (0x9b8b bytes) 000530 1782757390.315920 fallocate 0x03f26f thru 0x03ffff (0xd91 bytes) 000531 1782757390.347182 mapread 0x037ccc thru 0x03fc68 (0x7f9d bytes) 000533 1782757390.375663 mapread 0x039c69 thru 0x03ffff (0x6397 bytes) 000534 1782757390.385470 write 0x009f39 thru 0x00f212 (0x52da bytes) 000535 1782757390.393639 fallocate 0x03f933 thru 0x03ffff (0x6cd bytes) 000536 1782757390.425354 write 0x00fcab thru 0x01a8b6 (0xac0c bytes) 000537 1782757390.449768 read 0x012e00 thru 0x021e03 (0xf004 bytes) 000539 1782757390.490636 write 0x01257f thru 0x02065e (0xe0e0 bytes) 000541 1782757390.497200 trunc from 0x040000 to 0x0101b7 (0x2fe48 bytes) 000542 1782757390.555620 trunc from 0x0101b8 to 0x00a916 (0x58a1 bytes) 000543 1782757390.606471 mapread 0x0059c6 thru 0x00a916 (0x4f51 bytes) 000544 1782757390.611905 punch from 0x002eaf to 0x009c98 (0x6dea bytes) 000545 1782757390.633975 trunc from 0x00a44e to 0x00a916 (0x4c9 bytes) 000546 1782757390.654270 MIRROR_RESYNC 15 mirrors 000547 1782757394.387812 MIRROR_SPLIT mirror 8 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x156:0x0]: Invalid argument (22) 000548 1782757395.073917 write 0x030030 thru 0x03bfb0 (0xbf81 bytes) 000549 1782757395.133041 trunc from 0x0285d4 to 0x033542 (0xaf6f bytes) 000550 1782757395.156512 MIRROR_RESYNC 14 mirrors 000551 1782757398.355224 trunc from 0x03bfb1 to 0x018560 (0x23a50 bytes) 000552 1782757398.410801 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x18561): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18561): Mirror 4: 0xaafc86f4 (stale) Mirror 6: 0xaafc86f4 (stale) Mirror 8: 0xaafc86f4 (stale) Mirror 11: 0xaafc86f4 (stale) Mirror 12: 0xaafc86f4 (stale) Mirror 13: 0xaafc86f4 (stale) Mirror 14: 0xaafc86f4 (stale) Mirror 16: 0xaafc86f4 (stale) Mirror 17: 0xaafc86f4 (stale) Mirror 18: 0xaafc86f4 (stale) Mirror 19: 0xaafc86f4 (stale) Mirror 20: 0xaafc86f4 (stale) Mirror 21: 0xaafc86f4 (stale) CRC-32 checksum value for chunk [0, 0x18561): Mirror 1: 0xaafc86f4 Mirror 22: 0xaafc86f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18561: skipped 000553 1782757400.168471 read 0x00a63d thru 0x012e8f (0x8853 bytes) 000554 1782757400.233579 mapwrite 0x02a987 thru 0x02c9c1 (0x203b bytes) 000555 1782757400.522919 mapwrite 0x02b1f7 thru 0x037b63 (0xc96d bytes) 000556 1782757400.692959 read 0x0108dc thru 0x01cb2a (0xc24f bytes) 000557 1782757400.758538 trunc from 0x037b64 to 0x019d9f (0x1ddc4 bytes) 000558 1782757400.800486 trunc from 0x019da0 to 0x031c6b (0x17ecc bytes) 000559 1782757400.834983 punch from 0x01109c to 0x01ffd0 (0xef35 bytes) 000560 1782757400.844260 fallocate 0x033de0 thru 0x03ffff (0xc220 bytes) 000561 1782757400.851344 punch from 0x01672c to 0x01d08a (0x695f bytes) 000562 1782757400.856532 write 0x028f2a thru 0x03611b (0xd1f2 bytes) 000565 1782757400.876310 mapwrite 0x01986d thru 0x026c77 (0xd40b bytes) 000566 1782757401.005864 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xbd243a1e (stale) Mirror 6: 0xbd243a1e (stale) Mirror 8: 0xbd243a1e (stale) Mirror 11: 0xbd243a1e (stale) Mirror 12: 0xbd243a1e (stale) Mirror 13: 0xbd243a1e (stale) Mirror 14: 0xbd243a1e (stale) Mirror 16: 0xbd243a1e (stale) Mirror 17: 0xbd243a1e (stale) Mirror 18: 0xbd243a1e (stale) Mirror 19: 0xbd243a1e (stale) Mirror 20: 0xbd243a1e (stale) Mirror 21: 0xbd243a1e (stale) Mirror 22: 0x45cbcc4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb9c53e5c Mirror 23: 0xb9c53e5c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000568 1782757402.631280 write 0x001ebd thru 0x00d4f2 (0xb636 bytes) 000569 1782757402.712558 punch from 0x0051b5 to 0x010857 (0xb6a3 bytes) 000570 1782757402.734946 punch from 0x01a3d1 to 0x01e6dc (0x430c bytes) 000571 1782757402.747208 MIRROR_RESYNC 16 mirrors 000572 1782757409.386111 read 0x0213e2 thru 0x02aad8 (0x96f7 bytes) 000573 1782757409.408278 write 0x036b31 thru 0x03ffff (0x94cf bytes) 000574 1782757409.478352 write 0x0037a7 thru 0x00fd56 (0xc5b0 bytes) 000575 1782757409.495692 read 0x00ce1e thru 0x0180ad (0xb290 bytes) 000576 1782757409.507864 punch from 0x01aa24 to 0x0221e2 (0x77bf bytes) 000577 1782757409.518754 fallocate 0x0052db thru 0x012ce6 (0xda0c bytes) 000578 1782757409.524549 mapwrite 0x022c6d thru 0x02624c (0x35e0 bytes) 000579 1782757409.554431 read 0x0253c0 thru 0x0309d0 (0xb611 bytes) 000580 1782757409.562606 MIRROR_RESYNC 16 mirrors 000581 1782757414.822381 read 0x03a9e0 thru 0x03c2d0 (0x18f1 bytes) 000582 1782757414.829764 punch from 0x00c162 to 0x016031 (0x9ed0 bytes) 000583 1782757414.921565 trunc from 0x0335e2 to 0x03ae95 (0x78b4 bytes) 000584 1782757414.928533 trunc from 0x02b7da to 0x038ca6 (0xd4cd bytes) 000585 1782757414.938640 read 0x01b9a5 thru 0x01fe54 (0x44b0 bytes) 000586 1782757415.007204 mapread 0x02e790 thru 0x031aa8 (0x3319 bytes) 000587 1782757415.015825 MIRROR_RESYNC 16 mirrors 000588 1782757421.270038 write 0x03d0bc thru 0x03ffff (0x2f44 bytes) 000589 1782757421.341570 write 0x01d039 thru 0x01e463 (0x142b bytes) 000590 1782757421.355998 read 0x000413 thru 0x00782c (0x741a bytes) 000591 1782757421.381035 trunc from 0x01e4ad to 0x027580 (0x90d4 bytes) 000592 1782757421.415205 mapread 0x0085f5 thru 0x00b3a3 (0x2daf bytes) 000593 1782757421.442403 mapwrite 0x016184 thru 0x019d0d (0x3b8a bytes) 000594 1782757421.485421 punch from 0x00d8f5 to 0x016f9c (0x96a8 bytes) 000595 1782757421.494204 trunc from 0x02fcb0 to 0x0345d3 (0x4924 bytes) 000596 1782757421.517814 MIRROR_RESYNC 16 mirrors 000597 1782757427.466732 read 0x0036e2 thru 0x011252 (0xdb71 bytes) 000598 1782757427.478403 mapwrite 0x009006 thru 0x010880 (0x787b bytes) 000599 1782757427.616807 punch from 0x011c44 to 0x016bd2 (0x4f8f bytes) 000600 1782757427.633007 trunc from 0x040000 to 0x006726 (0x398d9 bytes) 000601 1782757427.665167 mapread 0x001a7c thru 0x006726 (0x4cab bytes) 000602 1782757427.671658 read 0x0052d7 thru 0x006726 (0x1450 bytes) 000603 1782757427.672444 read 0x0029ea thru 0x006726 (0x3d3d bytes) 000605 1782757427.674041 write 0x0277f4 thru 0x0282a2 (0xaaf bytes) 000606 1782757427.676071 mapread 0x002d6f thru 0x00faa1 (0xcd33 bytes) 000607 1782757427.685222 write 0x0064ff thru 0x011c89 (0xb78b bytes) 000609 1782757427.688185 mapread 0x00fee2 thru 0x01597b (0x5a9a bytes) 000610 1782757427.689520 trunc from 0x024307 to 0x0282a2 (0x3f9c bytes) 000612 1782757427.703906 mapwrite 0x0173fc thru 0x019fc6 (0x2bcb bytes) 000613 1782757427.722005 MIRROR_RESYNC 16 mirrors 000614 1782757431.602539 fallocate 0x02294e thru 0x0328cf (0xff82 bytes) 000615 1782757431.654572 trunc from 0x0328d0 to 0x030dd2 (0x1afd bytes) 000616 1782757431.681569 trunc from 0x030dd3 to 0x0244a5 (0xc92d bytes) 000617 1782757431.700311 write 0x027c29 thru 0x03606e (0xe446 bytes) 000618 1782757431.703009 trunc from 0x00dc94 to 0x00e6b7 (0xa24 bytes) 000619 1782757431.708710 mapwrite 0x0248d6 thru 0x02f2bd (0xa9e8 bytes) 000620 1782757431.747400 trunc from 0x03606f to 0x03ae90 (0x4e22 bytes) 000621 1782757431.769259 MIRROR_RESYNC 16 mirrors 000622 1782757436.708272 punch from 0x004087 to 0x006bd0 (0x2b4a bytes) 000623 1782757436.766214 mapread 0x0129a5 thru 0x01c5b7 (0x9c13 bytes) 000624 1782757436.787240 trunc from 0x03ae91 to 0x03b9a6 (0xb16 bytes) 000625 1782757436.810748 trunc from 0x03b9a7 to 0x004f3e (0x36a68 bytes) 000626 1782757436.833982 write 0x015142 thru 0x01d31f (0x81de bytes) 000627 1782757436.837299 punch from 0x01aa0a to 0x01c00f (0x1606 bytes) 000628 1782757436.854898 read 0x0186c3 thru 0x01931f (0xc5d bytes) 000630 1782757436.856524 mapwrite 0x01b916 thru 0x0294ed (0xdbd8 bytes) 000631 1782757436.960924 MIRROR_RESYNC 16 mirrors 000632 1782757440.068106 mapread 0x02670a thru 0x0294ed (0x2de4 bytes) 000633 1782757440.115154 trunc from 0x01965d to 0x022027 (0x89cb bytes) 000634 1782757440.186779 fallocate 0x021198 thru 0x028b4c (0x79b5 bytes) 000635 1782757440.197402 write 0x03ed52 thru 0x03ffff (0x12ae bytes) 000638 1782757440.209782 trunc from 0x0044ab to 0x009461 (0x4fb7 bytes) 000639 1782757440.217470 trunc from 0x02171a to 0x024b6c (0x3453 bytes) 000640 1782757440.224028 trunc from 0x040000 to 0x002c14 (0x3d3eb bytes) 000641 1782757440.252007 mapwrite 0x0110fb thru 0x01c49b (0xb3a1 bytes) 000642 1782757440.320622 mapread 0x014761 thru 0x01aef9 (0x6799 bytes) 000643 1782757440.323756 trunc from 0x01c49c to 0x01e9a0 (0x2505 bytes) 000644 1782757440.348860 read 0x014404 thru 0x01e9a0 (0xa59d bytes) 000645 1782757440.360630 mapwrite 0x00440d thru 0x01110c (0xcd00 bytes) 000646 1782757440.415623 trunc from 0x007005 to 0x011d45 (0xad41 bytes) 000647 1782757440.425082 mapwrite 0x031d98 thru 0x03309d (0x1306 bytes) 000648 1782757440.502817 punch from 0x013672 to 0x01c584 (0x8f13 bytes) 000649 1782757440.521920 punch from 0x02cb2d to 0x03309d (0x6571 bytes) 000650 1782757440.534757 trunc from 0x03309e to 0x015a5e (0x1d63f bytes) 000651 1782757440.556705 trunc from 0x0096cf to 0x00ad43 (0x1675 bytes) 000652 1782757440.563150 mapread 0x00282f thru 0x00c295 (0x9a67 bytes) 000654 1782757440.583794 write 0x02672a thru 0x029925 (0x31fc bytes) 000658 1782757440.588641 mapwrite 0x03c413 thru 0x03ffff (0x3bed bytes) 000660 1782757440.686611 write 0x001f99 thru 0x006b92 (0x4bfa bytes) 000661 1782757440.689670 fallocate 0x01ca7e thru 0x02bc08 (0xf18b bytes) 000662 1782757440.701618 read 0x00f367 thru 0x012f2b (0x3bc5 bytes) 000663 1782757440.717689 mapwrite 0x00c4dc thru 0x0149da (0x84ff bytes) 000664 1782757440.748046 mapread 0x0234da thru 0x025d1a (0x2841 bytes) 000665 1782757440.749757 trunc from 0x0392c1 to 0x03ffff (0x6d3f bytes) 000666 1782757440.756785 mapwrite 0x0183c2 thru 0x01adc1 (0x2a00 bytes) 000667 1782757440.773901 mapwrite 0x012fe3 thru 0x022528 (0xf546 bytes) 000668 1782757440.827508 MIRROR_RESYNC 16 mirrors 000671 1782757446.375509 MIRROR_RESYNC 16 mirrors 000672 1782757447.595960 punch from 0x01de7a to 0x0204d9 (0x2660 bytes) 000673 1782757447.657368 write 0x032772 thru 0x03ffff (0xd88e bytes) 000674 1782757447.664783 read 0x01bd75 thru 0x01fb62 (0x3dee bytes) 000675 1782757447.679786 fallocate 0x01a51a thru 0x02298e (0x8475 bytes) 000676 1782757447.696570 MIRROR_RESYNC 16 mirrors 000677 1782757454.313355 read 0x017e7a thru 0x01c1dc (0x4363 bytes) 000678 1782757454.331156 write 0x03840d thru 0x03ffff (0x7bf3 bytes) 000679 1782757454.424898 write 0x01e6bc thru 0x0243d0 (0x5d15 bytes) 000681 1782757454.450748 read 0x01190f thru 0x012066 (0x758 bytes) 000684 1782757454.486156 mapwrite 0x03036a thru 0x0364d6 (0x616d bytes) 000685 1782757454.523725 trunc from 0x012a43 to 0x0151c4 (0x2782 bytes) 000686 1782757454.531299 write 0x003be6 thru 0x012eef (0xf30a bytes) 000687 1782757454.539398 trunc from 0x03f01b to 0x03ffff (0xfe5 bytes) 000688 1782757454.548844 read 0x039748 thru 0x03f194 (0x5a4d bytes) 000689 1782757454.566702 punch from 0x00fe47 to 0x0173ca (0x7584 bytes) 000690 1782757454.609840 trunc from 0x040000 to 0x0206b2 (0x1f94d bytes) 000691 1782757454.666743 trunc from 0x00fae3 to 0x00ff70 (0x48e bytes) 000692 1782757454.686924 MIRROR_RESYNC 16 mirrors 000693 1782757458.107895 write 0x012d3b thru 0x022b21 (0xfde7 bytes) 000694 1782757458.179672 mapwrite 0x018fd9 thru 0x021809 (0x8831 bytes) 000695 1782757458.206881 punch from 0x0212e6 to 0x022b21 (0x183c bytes) 000696 1782757458.220683 MIRROR_SPLIT mirror 13 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x156:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22b22): Mirror 4: 0x7b83397f (stale) Mirror 6: 0x7b83397f (stale) Mirror 8: 0x7b83397f (stale) Mirror 11: 0x7b83397f (stale) Mirror 12: 0x7b83397f (stale) Mirror 13: 0x7b83397f (stale) Mirror 14: 0x7b83397f (stale) Mirror 16: 0x7b83397f (stale) Mirror 17: 0x7b83397f (stale) Mirror 18: 0x7b83397f (stale) Mirror 19: 0x7b83397f (stale) Mirror 20: 0x7b83397f (stale) Mirror 22: 0x7b83397f (stale) Mirror 23: 0x7b83397f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22b22): Mirror 4: 0x7b83397f (stale) Mirror 6: 0x7b83397f (stale) Mirror 8: 0x7b83397f (stale) Mirror 11: 0x7b83397f (stale) Mirror 12: 0x7b83397f (stale) Mirror 13: 0x7b83397f (stale) Mirror 14: 0x7b83397f (stale) Mirror 16: 0x7b83397f (stale) Mirror 17: 0x7b83397f (stale) Mirror 18: 0x7b83397f (stale) Mirror 19: 0x7b83397f (stale) Mirror 20: 0x7b83397f (stale) Mirror 22: 0x7b83397f (stale) Mirror 23: 0x7b83397f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22b22: skipped 000697 1782757458.943990 trunc from 0x022b22 to 0x01e419 (0x4708 bytes) 000698 1782757458.973332 trunc from 0x001a34 to 0x005fd1 (0x459e bytes) 000700 1782757458.980754 write 0x0284a2 thru 0x03098d (0x84ec bytes) 000701 1782757458.984309 mapread 0x00bf85 thru 0x012e48 (0x6ec4 bytes) 000702 1782757459.013760 mapwrite 0x0207a1 thru 0x023f4c (0x37ac bytes) 000704 1782757459.049480 read 0x01482e thru 0x01fa0b (0xb1de bytes) 000705 1782757459.056417 trunc from 0x03098e to 0x00fdce (0x20bbf bytes) 000706 1782757459.073203 punch from 0x007147 to 0x00e917 (0x77d1 bytes) 000707 1782757459.079139 mapread 0x003abb thru 0x006d50 (0x3296 bytes) 000708 1782757459.089900 trunc from 0x00fdcf to 0x00d4f9 (0x28d5 bytes) 000709 1782757459.116793 MIRROR_RESYNC 15 mirrors 000710 1782757462.370801 mapwrite 0x00d226 thru 0x01cbb4 (0xf98f bytes) 000711 1782757462.608541 mapwrite 0x02abb6 thru 0x036f16 (0xc361 bytes) 000712 1782757462.722870 read 0x02de50 thru 0x036f16 (0x90c7 bytes) 000713 1782757462.723871 mapwrite 0x031ce7 thru 0x03ffff (0xe319 bytes) 000714 1782757462.799334 trunc from 0x012fb6 to 0x01ba35 (0x8a80 bytes) 000715 1782757462.808432 trunc from 0x00d08b to 0x014cdd (0x7c53 bytes) 000716 1782757462.817986 MIRROR_SPLIT mirror 4 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x156:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xa89a5d3d (stale) Mirror 6: 0xa89a5d3d (stale) Mirror 8: 0xa89a5d3d (stale) Mirror 12: 0xa89a5d3d (stale) Mirror 13: 0xa89a5d3d (stale) Mirror 14: 0xa89a5d3d (stale) Mirror 16: 0xa89a5d3d (stale) Mirror 17: 0xa89a5d3d (stale) Mirror 18: 0xa89a5d3d (stale) Mirror 19: 0xa89a5d3d (stale) Mirror 20: 0xa89a5d3d (stale) Mirror 22: 0xa89a5d3d (stale) Mirror 23: 0xa89a5d3d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xa89a5d3d (stale) Mirror 6: 0xa89a5d3d (stale) Mirror 8: 0xa89a5d3d (stale) Mirror 12: 0xa89a5d3d (stale) Mirror 13: 0xa89a5d3d (stale) Mirror 14: 0xa89a5d3d (stale) Mirror 16: 0xa89a5d3d (stale) Mirror 17: 0xa89a5d3d (stale) Mirror 18: 0xa89a5d3d (stale) Mirror 19: 0xa89a5d3d (stale) Mirror 20: 0xa89a5d3d (stale) Mirror 22: 0xa89a5d3d (stale) Mirror 23: 0xa89a5d3d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000717 1782757463.763229 mapwrite 0x00a9dd thru 0x014102 (0x9726 bytes) 000718 1782757463.881450 mapread 0x00e0cf thru 0x019948 (0xb87a bytes) 000719 1782757463.924482 read 0x013391 thru 0x01ffcb (0xcc3b bytes) 000720 1782757463.935687 mapread 0x03b87f thru 0x03f946 (0x40c8 bytes) 000721 1782757463.961323 trunc from 0x040000 to 0x01313f (0x2cec0 bytes) 000722 1782757464.009687 trunc from 0x013140 to 0x02d30e (0x1a1cf bytes) 000723 1782757464.046248 mapwrite 0x02db31 thru 0x035186 (0x7656 bytes) 000724 1782757464.151932 fallocate 0x006de0 thru 0x00e857 (0x7a78 bytes) 000725 1782757464.159374 trunc from 0x035187 to 0x02204a (0x1313c bytes) 000726 1782757464.187503 write 0x02af33 thru 0x02bcfc (0xdca bytes) 000727 1782757464.194656 write 0x027f36 thru 0x030b5e (0x8c29 bytes) 000730 1782757464.214632 mapwrite 0x01a93c thru 0x01dc56 (0x331b bytes) 000731 1782757464.260461 fallocate 0x03ad74 thru 0x03fdf8 (0x5085 bytes) 000732 1782757464.280997 trunc from 0x030b5f to 0x01d218 (0x13946 bytes) 000733 1782757464.324031 mapwrite 0x00f4d2 thru 0x012479 (0x2fa8 bytes) 000735 1782757464.352774 read 0x0029ea thru 0x003f26 (0x153d bytes) 000736 1782757464.369600 trunc from 0x008b1f to 0x00c57c (0x3a5e bytes) 000738 1782757464.394962 mapread 0x018d11 thru 0x01d218 (0x4508 bytes) 000739 1782757464.403616 MIRROR_RESYNC 14 mirrors 000740 1782757468.919360 trunc from 0x01d219 to 0x00e34e (0xeeca bytes) 000741 1782757469.018585 trunc from 0x00be72 to 0x00e34e (0x24dd bytes) 000743 1782757469.025318 mapread 0x000cac thru 0x0068bf (0x5c14 bytes) 000745 1782757469.082197 trunc from 0x00e250 to 0x00e34e (0xff bytes) 000746 1782757469.087479 trunc from 0x006231 to 0x00e34e (0x811e bytes) 000747 1782757469.096866 trunc from 0x00e34f to 0x01514b (0x6dfd bytes) 000748 1782757469.159203 read 0x00a500 thru 0x013e20 (0x9921 bytes) 000749 1782757469.183758 punch from 0x002e9a to 0x007a31 (0x4b98 bytes) 000750 1782757469.207758 write 0x00a045 thru 0x00c1aa (0x2166 bytes) 000751 1782757469.209419 punch from 0x000247 to 0x0018a7 (0x1661 bytes) 000752 1782757469.215595 trunc from 0x01514c to 0x02decf (0x18d84 bytes) 000753 1782757469.236601 trunc from 0x00c30f to 0x014b5d (0x884f bytes) 000754 1782757469.270848 mapwrite 0x013be6 thru 0x01cbcc (0x8fe7 bytes) 000755 1782757469.362586 MIRROR_RESYNC 14 mirrors 000756 1782757473.910124 trunc from 0x02ded0 to 0x0258ba (0x8615 bytes) 000757 1782757474.017768 read 0x01ae4e thru 0x01dd86 (0x2f39 bytes) 000758 1782757474.031754 read 0x012725 thru 0x01649b (0x3d77 bytes) 000759 1782757474.067672 trunc from 0x0258bb to 0x02a187 (0x48cd bytes) 000760 1782757474.124301 mapread 0x019b9a thru 0x025881 (0xbce8 bytes) 000761 1782757474.125746 fallocate 0x01fd57 thru 0x022242 (0x24ec bytes) 000762 1782757474.133356 punch from 0x003e97 to 0x007f95 (0x40ff bytes) 000763 1782757474.143256 trunc from 0x00175e to 0x001f25 (0x7c8 bytes) 000764 1782757474.152269 write 0x03707e thru 0x03a8c9 (0x384c bytes) 000765 1782757474.153870 punch from 0x007be4 to 0x016b44 (0xef61 bytes) 000766 1782757474.162940 read 0x02509c thru 0x0272a1 (0x2206 bytes) 000767 1782757474.175123 mapread 0x028d2f thru 0x02c600 (0x38d2 bytes) 000768 1782757474.176983 read 0x010b9a thru 0x019c43 (0x90aa bytes) 000769 1782757474.178473 trunc from 0x03a8ca to 0x01a215 (0x206b4 bytes) 000771 1782757474.204026 trunc from 0x01a216 to 0x03580a (0x1b5f5 bytes) 000772 1782757474.221861 mapwrite 0x01999b thru 0x01cc0b (0x3271 bytes) 000774 1782757474.249344 trunc from 0x03580b to 0x01399e (0x21e6c bytes) 000775 1782757474.273937 fallocate 0x01d261 thru 0x02beed (0xec8d bytes) 000776 1782757474.288820 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x1399f): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1399f): Mirror 4: 0x8c78be41 (stale) Mirror 6: 0x8c78be41 (stale) Mirror 8: 0x8c78be41 (stale) Mirror 12: 0x8c78be41 (stale) Mirror 13: 0x8c78be41 (stale) Mirror 14: 0x8c78be41 (stale) Mirror 16: 0x8c78be41 (stale) Mirror 17: 0x8c78be41 (stale) Mirror 18: 0x8c78be41 (stale) Mirror 19: 0x8c78be41 (stale) Mirror 20: 0x8c78be41 (stale) Mirror 22: 0x8c78be41 (stale) Mirror 23: 0x8c78be41 (stale) CRC-32 checksum value for chunk [0, 0x1399f): Mirror 1: 0xdcc3a1b Mirror 24: 0xdcc3a1b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1399f: skipped 000777 1782757475.655798 mapread 0x00378d thru 0x009049 (0x58bd bytes) 000778 1782757475.752407 mapwrite 0x03dced thru 0x03ffff (0x2313 bytes) 000779 1782757475.972718 write 0x00a5ed thru 0x016df4 (0xc808 bytes) 000780 1782757475.990820 mapwrite 0x0133b2 thru 0x022ee3 (0xfb32 bytes) 000781 1782757476.047902 trunc from 0x040000 to 0x01f803 (0x207fc bytes) 000782 1782757476.102911 mapread 0x00a8ad thru 0x0134f0 (0x8c44 bytes) 000783 1782757476.104437 mapread 0x006d5c thru 0x011749 (0xa9ee bytes) 000784 1782757476.135236 fallocate 0x0221f3 thru 0x022650 (0x45e bytes) 000785 1782757476.142525 mapwrite 0x00c4ae thru 0x013f72 (0x7ac5 bytes) 000787 1782757476.171917 MIRROR_SPLIT mirror 14 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x156:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f804): Mirror 4: 0x2b4eaa8a (stale) Mirror 6: 0x2b4eaa8a (stale) Mirror 8: 0x2b4eaa8a (stale) Mirror 12: 0x2b4eaa8a (stale) Mirror 13: 0x2b4eaa8a (stale) Mirror 14: 0x2b4eaa8a (stale) Mirror 16: 0x2b4eaa8a (stale) Mirror 17: 0x2b4eaa8a (stale) Mirror 18: 0x2b4eaa8a (stale) Mirror 19: 0x2b4eaa8a (stale) Mirror 20: 0x2b4eaa8a (stale) Mirror 22: 0x2b4eaa8a (stale) Mirror 23: 0x2b4eaa8a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f804): Mirror 4: 0x2b4eaa8a (stale) Mirror 6: 0x2b4eaa8a (stale) Mirror 8: 0x2b4eaa8a (stale) Mirror 12: 0x2b4eaa8a (stale) Mirror 13: 0x2b4eaa8a (stale) Mirror 14: 0x2b4eaa8a (stale) Mirror 16: 0x2b4eaa8a (stale) Mirror 17: 0x2b4eaa8a (stale) Mirror 18: 0x2b4eaa8a (stale) Mirror 19: 0x2b4eaa8a (stale) Mirror 20: 0x2b4eaa8a (stale) Mirror 22: 0x2b4eaa8a (stale) Mirror 23: 0x2b4eaa8a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f804: skipped 000788 1782757477.218944 punch from 0x01022a to 0x01850c (0x82e3 bytes) 000789 1782757477.235366 fallocate 0x005d69 thru 0x0128ed (0xcb85 bytes) 000790 1782757477.253832 MIRROR_SPLIT mirror 6 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x156:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f804): Mirror 4: 0x2b4eaa8a (stale) Mirror 6: 0x2b4eaa8a (stale) Mirror 8: 0x2b4eaa8a (stale) Mirror 12: 0x2b4eaa8a (stale) Mirror 13: 0x2b4eaa8a (stale) Mirror 16: 0x2b4eaa8a (stale) Mirror 17: 0x2b4eaa8a (stale) Mirror 18: 0x2b4eaa8a (stale) Mirror 19: 0x2b4eaa8a (stale) Mirror 20: 0x2b4eaa8a (stale) Mirror 22: 0x2b4eaa8a (stale) Mirror 23: 0x2b4eaa8a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f804): Mirror 4: 0x2b4eaa8a (stale) Mirror 6: 0x2b4eaa8a (stale) Mirror 8: 0x2b4eaa8a (stale) Mirror 12: 0x2b4eaa8a (stale) Mirror 13: 0x2b4eaa8a (stale) Mirror 16: 0x2b4eaa8a (stale) Mirror 17: 0x2b4eaa8a (stale) Mirror 18: 0x2b4eaa8a (stale) Mirror 19: 0x2b4eaa8a (stale) Mirror 20: 0x2b4eaa8a (stale) Mirror 22: 0x2b4eaa8a (stale) Mirror 23: 0x2b4eaa8a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f804: skipped 000791 1782757478.100473 trunc from 0x01f804 to 0x01bf53 (0x38b0 bytes) 000792 1782757478.129832 mapwrite 0x00f825 thru 0x01a0ce (0xa8aa bytes) 000793 1782757478.179708 read 0x007d12 thru 0x010d2a (0x9019 bytes) 000794 1782757478.186994 punch from 0x003e55 to 0x011aa0 (0xdc4c bytes) 000795 1782757478.193267 mapwrite 0x011431 thru 0x01db49 (0xc719 bytes) 000797 1782757478.255996 read 0x008ff2 thru 0x015737 (0xc746 bytes) 000798 1782757478.269876 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x1db4a): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1db4a): Mirror 4: 0x574a2910 (stale) Mirror 6: 0x574a2910 (stale) Mirror 8: 0x574a2910 (stale) Mirror 12: 0x574a2910 (stale) Mirror 13: 0x574a2910 (stale) Mirror 16: 0x574a2910 (stale) Mirror 17: 0x574a2910 (stale) Mirror 18: 0x574a2910 (stale) Mirror 19: 0x574a2910 (stale) Mirror 20: 0x574a2910 (stale) Mirror 22: 0x574a2910 (stale) Mirror 23: 0x574a2910 (stale) CRC-32 checksum value for chunk [0, 0x1db4a): Mirror 1: 0x69ef1152 Mirror 24: 0x69ef1152 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1db4a: skipped 000799 1782757479.744138 write 0x00d285 thru 0x012f9b (0x5d17 bytes) 000800 1782757479.898841 trunc from 0x003121 to 0x00c701 (0x95e1 bytes) 000801 1782757479.910307 punch from 0x0056b6 to 0x01261e (0xcf69 bytes) 000802 1782757479.942690 mapread 0x00379c thru 0x007593 (0x3df8 bytes) 000803 1782757479.987250 read 0x0194d7 thru 0x01ae37 (0x1961 bytes) 000804 1782757480.000258 mapread 0x0047de thru 0x00ccb7 (0x84da bytes) 000805 1782757480.005883 trunc from 0x01abc0 to 0x01db49 (0x2f8a bytes) 000806 1782757480.013218 write 0x005ac1 thru 0x010adc (0xb01c bytes) 000808 1782757480.018774 punch from 0x004a4a to 0x007cb1 (0x3268 bytes) 000809 1782757480.037022 trunc from 0x01db4a to 0x0331c9 (0x15680 bytes) 000810 1782757480.100425 trunc from 0x029b15 to 0x0331c9 (0x96b5 bytes) 000811 1782757480.105399 write 0x03f1c5 thru 0x03ffff (0xe3b bytes) 000813 1782757480.107972 mapread 0x015493 thru 0x01d4f0 (0x805e bytes) 000814 1782757480.122432 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xc403cf10 (stale) Mirror 6: 0xc403cf10 (stale) Mirror 8: 0xc403cf10 (stale) Mirror 12: 0xc403cf10 (stale) Mirror 13: 0xc403cf10 (stale) Mirror 16: 0xc403cf10 (stale) Mirror 17: 0xc403cf10 (stale) Mirror 18: 0xc403cf10 (stale) Mirror 19: 0xc403cf10 (stale) Mirror 20: 0xc403cf10 (stale) Mirror 22: 0xc403cf10 (stale) Mirror 23: 0xc403cf10 (stale) Mirror 24: 0x5f8d6f40 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x721bfca3 Mirror 25: 0x721bfca3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000815 1782757481.727728 fallocate 0x01eaea thru 0x023270 (0x4787 bytes) 000816 1782757481.829900 trunc from 0x040000 to 0x01b0b9 (0x24f46 bytes) 000817 1782757481.897321 write 0x02c043 thru 0x030007 (0x3fc5 bytes) 000818 1782757481.904032 MIRROR_RESYNC 15 mirrors 000819 1782757487.514533 write 0x033f8e thru 0x03ffff (0xc072 bytes) 000820 1782757487.551147 fallocate 0x010f82 thru 0x01a0b8 (0x9137 bytes) 000821 1782757487.558181 read 0x01a216 thru 0x01a779 (0x564 bytes) 000822 1782757487.563884 mapwrite 0x000c4f thru 0x00ef51 (0xe303 bytes) 000823 1782757487.644537 read 0x01dfd3 thru 0x01ead6 (0xb04 bytes) 000824 1782757487.705238 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1ace5041 (stale) Mirror 6: 0x1ace5041 (stale) Mirror 8: 0x1ace5041 (stale) Mirror 12: 0x1ace5041 (stale) Mirror 13: 0x1ace5041 (stale) Mirror 16: 0x1ace5041 (stale) Mirror 17: 0x1ace5041 (stale) Mirror 18: 0x1ace5041 (stale) Mirror 19: 0x1ace5041 (stale) Mirror 20: 0x1ace5041 (stale) Mirror 22: 0x1ace5041 (stale) Mirror 23: 0x1ace5041 (stale) Mirror 24: 0x1ace5041 (stale) Mirror 25: 0x1ace5041 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1d6b0058 Mirror 26: 0x1d6b0058 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000825 1782757489.742689 MIRROR_RESYNC 16 mirrors 000826 1782757495.523949 trunc from 0x016d38 to 0x022dee (0xc0b7 bytes) 000827 1782757495.594242 read 0x00779f thru 0x00c340 (0x4ba2 bytes) 000828 1782757495.609782 read 0x036c7a thru 0x03c0ab (0x5432 bytes) 000829 1782757495.625587 mapread 0x02066f thru 0x02f685 (0xf017 bytes) 000830 1782757495.628450 mapread 0x02d8e5 thru 0x03386e (0x5f8a bytes) 000831 1782757495.629839 punch from 0x0055a1 to 0x00aeeb (0x594b bytes) 000832 1782757495.640184 trunc from 0x040000 to 0x0076e6 (0x38919 bytes) 000833 1782757495.685809 read 0x0063cb thru 0x0076e6 (0x131c bytes) 000834 1782757495.696344 fallocate 0x030197 thru 0x03a7ae (0xa618 bytes) 000835 1782757495.701596 punch from 0x002283 to 0x010511 (0xe28f bytes) 000840 1782757495.713781 mapread 0x0133bb thru 0x01957b (0x61c1 bytes) 000841 1782757495.728507 trunc from 0x03a7af to 0x02bb28 (0xec86 bytes) 000842 1782757495.757220 read 0x023fd8 thru 0x02bb28 (0x7b51 bytes) 000843 1782757495.758342 trunc from 0x02bb29 to 0x017795 (0x14393 bytes) 000844 1782757495.804027 mapread 0x002a9f thru 0x009cfb (0x725d bytes) 000845 1782757495.824700 mapread 0x014fba thru 0x017795 (0x27dc bytes) 000846 1782757495.826684 trunc from 0x017796 to 0x01a2d0 (0x2b3b bytes) 000847 1782757495.846580 trunc from 0x01a2d1 to 0x007976 (0x1295a bytes) 000848 1782757495.870806 trunc from 0x007977 to 0x02331d (0x1b9a7 bytes) 000849 1782757495.888304 punch from 0x0072ca to 0x00c2b7 (0x4fee bytes) 000850 1782757495.894317 mapwrite 0x01f14e thru 0x022a7e (0x3931 bytes) 000851 1782757495.923172 trunc from 0x01c0ae to 0x0227bb (0x670e bytes) 000852 1782757495.929529 trunc from 0x01abf8 to 0x02331d (0x8726 bytes) 000854 1782757495.935795 read 0x01ce9f thru 0x02331d (0x647f bytes) 000855 1782757495.947153 trunc from 0x01e464 to 0x02331d (0x4eba bytes) 000856 1782757495.952672 mapread 0x016c76 thru 0x01940a (0x2795 bytes) 000857 1782757495.954336 trunc from 0x02331e to 0x00c576 (0x16da7 bytes) 000858 1782757495.972941 fallocate 0x01f4ed thru 0x023c29 (0x473d bytes) 000859 1782757495.981110 trunc from 0x00a76e to 0x00b760 (0xff3 bytes) 000860 1782757495.986564 trunc from 0x007628 to 0x0096f5 (0x20ce bytes) 000861 1782757495.991663 mapwrite 0x015b48 thru 0x016185 (0x63e bytes) 000862 1782757496.035164 mapwrite 0x036615 thru 0x03ffff (0x99eb bytes) 000863 1782757496.110036 fallocate 0x01bfc4 thru 0x01d2f5 (0x1332 bytes) 000864 1782757496.121544 trunc from 0x040000 to 0x035834 (0xa7cb bytes) 000866 1782757496.141514 fallocate 0x00ca2e thru 0x01b48c (0xea5f bytes) 000868 1782757496.148013 trunc from 0x035835 to 0x012ac1 (0x22d73 bytes) 000869 1782757496.169446 punch from 0x00956d to 0x00dcc0 (0x4754 bytes) 000871 1782757496.175702 trunc from 0x012ac2 to 0x007fd6 (0xaaeb bytes) 000872 1782757496.194150 trunc from 0x007fd7 to 0x0310f9 (0x29123 bytes) 000873 1782757496.218142 mapwrite 0x00f844 thru 0x00ff6b (0x728 bytes) 000874 1782757496.251415 MIRROR_SPLIT mirror 11 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x156:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x310fa): Mirror 4: 0x85e49789 (stale) Mirror 6: 0x85e49789 (stale) Mirror 8: 0x85e49789 (stale) Mirror 12: 0x85e49789 (stale) Mirror 13: 0x85e49789 (stale) Mirror 16: 0x85e49789 (stale) Mirror 17: 0x85e49789 (stale) Mirror 18: 0x85e49789 (stale) Mirror 19: 0x85e49789 (stale) Mirror 20: 0x85e49789 (stale) Mirror 23: 0x85e49789 (stale) Mirror 24: 0x85e49789 (stale) Mirror 25: 0x85e49789 (stale) Mirror 26: 0x85e49789 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x310fa): Mirror 4: 0x85e49789 (stale) Mirror 6: 0x85e49789 (stale) Mirror 8: 0x85e49789 (stale) Mirror 12: 0x85e49789 (stale) Mirror 13: 0x85e49789 (stale) Mirror 16: 0x85e49789 (stale) Mirror 17: 0x85e49789 (stale) Mirror 18: 0x85e49789 (stale) Mirror 19: 0x85e49789 (stale) Mirror 20: 0x85e49789 (stale) Mirror 23: 0x85e49789 (stale) Mirror 24: 0x85e49789 (stale) Mirror 25: 0x85e49789 (stale) Mirror 26: 0x85e49789 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x310fa: skipped 000875 1782757497.138000 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x310fa): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x310fa): Mirror 4: 0x85e49789 (stale) Mirror 6: 0x85e49789 (stale) Mirror 8: 0x85e49789 (stale) Mirror 12: 0x85e49789 (stale) Mirror 13: 0x85e49789 (stale) Mirror 16: 0x85e49789 (stale) Mirror 17: 0x85e49789 (stale) Mirror 18: 0x85e49789 (stale) Mirror 19: 0x85e49789 (stale) Mirror 20: 0x85e49789 (stale) Mirror 23: 0x85e49789 (stale) Mirror 24: 0x85e49789 (stale) Mirror 25: 0x85e49789 (stale) Mirror 26: 0x85e49789 (stale) CRC-32 checksum value for chunk [0, 0x310fa): Mirror 1: 0xdfaf29f Mirror 27: 0xdfaf29f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x310fa: skipped 000876 1782757498.966862 trunc from 0x0310fa to 0x0303c6 (0xd33 bytes) 000877 1782757499.089986 trunc from 0x0303c7 to 0x03b35a (0xaf94 bytes) 000880 1782757499.138098 trunc from 0x03b35b to 0x031cf8 (0x9662 bytes) 000881 1782757499.174994 trunc from 0x00dc08 to 0x012fbe (0x53b7 bytes) 000882 1782757499.182509 punch from 0x00cab5 to 0x01bde3 (0xf32f bytes) 000883 1782757499.199824 write 0x020b72 thru 0x0263fe (0x588d bytes) 000884 1782757499.231815 trunc from 0x031cf9 to 0x002a2d (0x2f2cb bytes) 000885 1782757499.280900 write 0x03b631 thru 0x03ffff (0x49cf bytes) 000888 1782757499.291352 read 0x0277ae thru 0x02f643 (0x7e96 bytes) 000889 1782757499.311676 mapread 0x015af6 thru 0x023338 (0xd843 bytes) 000890 1782757499.339918 read 0x01431c thru 0x020bef (0xc8d4 bytes) 000891 1782757499.341015 punch from 0x004d02 to 0x013a55 (0xed54 bytes) 000892 1782757499.346845 trunc from 0x040000 to 0x03f449 (0xbb6 bytes) 000893 1782757499.364821 mapwrite 0x007ffd thru 0x00d791 (0x5795 bytes) 000894 1782757499.420560 read 0x01b38c thru 0x01bfda (0xc4f bytes) 000895 1782757499.421556 mapwrite 0x00a20c thru 0x013afb (0x98f0 bytes) 000899 1782757499.445674 trunc from 0x0184d0 to 0x021895 (0x93c6 bytes) 000900 1782757499.453913 trunc from 0x03f44a to 0x02bfda (0x1346f bytes) 000901 1782757499.477662 trunc from 0x02bfdb to 0x039ad9 (0xdaff bytes) 000902 1782757499.501575 punch from 0x02626d to 0x034799 (0xe52d bytes) 000903 1782757499.508583 write 0x01d0ff thru 0x025b73 (0x8a75 bytes) 000905 1782757499.523396 fallocate 0x037253 thru 0x03ffff (0x8dad bytes) 000907 1782757499.530886 trunc from 0x040000 to 0x022902 (0x1d6fd bytes) 000908 1782757499.560148 mapread 0x017032 thru 0x022902 (0xb8d1 bytes) 000910 1782757499.566460 trunc from 0x004831 to 0x00db36 (0x9306 bytes) 000911 1782757499.579967 punch from 0x01d7f6 to 0x0214c9 (0x3cd4 bytes) 000912 1782757499.597772 mapwrite 0x027815 thru 0x032d78 (0xb564 bytes) 000913 1782757499.727807 read 0x01f465 thru 0x020d22 (0x18be bytes) 000914 1782757499.749432 mapwrite 0x01a243 thru 0x01f33f (0x50fd bytes) 000916 1782757499.771922 punch from 0x02823f to 0x032d78 (0xab3a bytes) 000917 1782757499.778299 mapread 0x00bd2d thru 0x011d44 (0x6018 bytes) 000918 1782757499.780838 trunc from 0x002174 to 0x002dc0 (0xc4d bytes) 000919 1782757499.787319 write 0x01878d thru 0x01abfc (0x2470 bytes) 000920 1782757499.789466 punch from 0x005183 to 0x011f5f (0xcddd bytes) 000921 1782757499.795310 mapread 0x031ce6 thru 0x032d78 (0x1093 bytes) 000922 1782757499.807606 read 0x01b45c thru 0x01dbbf (0x2764 bytes) 000923 1782757499.808606 mapwrite 0x019f09 thru 0x025209 (0xb301 bytes) 000924 1782757499.829938 read 0x019435 thru 0x027a53 (0xe61f bytes) 000925 1782757499.831010 write 0x03d3c9 thru 0x03ffff (0x2c37 bytes) 000926 1782757499.837911 MIRROR_SPLIT mirror 10 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x156:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1d6b0058 (stale) Mirror 6: 0x1d6b0058 (stale) Mirror 8: 0x1d6b0058 (stale) Mirror 12: 0x1d6b0058 (stale) Mirror 13: 0x1d6b0058 (stale) Mirror 16: 0x1d6b0058 (stale) Mirror 17: 0x1d6b0058 (stale) Mirror 18: 0x1d6b0058 (stale) Mirror 19: 0x1d6b0058 (stale) Mirror 23: 0x1d6b0058 (stale) Mirror 24: 0x1d6b0058 (stale) Mirror 25: 0x1d6b0058 (stale) Mirror 26: 0x1d6b0058 (stale) Mirror 27: 0x1698a3c3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1d6b0058 (stale) Mirror 6: 0x1d6b0058 (stale) Mirror 8: 0x1d6b0058 (stale) Mirror 12: 0x1d6b0058 (stale) Mirror 13: 0x1d6b0058 (stale) Mirror 16: 0x1d6b0058 (stale) Mirror 17: 0x1d6b0058 (stale) Mirror 18: 0x1d6b0058 (stale) Mirror 19: 0x1d6b0058 (stale) Mirror 23: 0x1d6b0058 (stale) Mirror 24: 0x1d6b0058 (stale) Mirror 25: 0x1d6b0058 (stale) Mirror 26: 0x1d6b0058 (stale) Mirror 27: 0x1698a3c3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000927 1782757500.933683 trunc from 0x03cfdf to 0x03ffff (0x3021 bytes) 000928 1782757500.939376 fallocate 0x0004fa thru 0x0041da (0x3ce1 bytes) 000929 1782757500.948485 fallocate 0x037011 thru 0x03cfa7 (0x5f97 bytes) 000930 1782757500.954410 write 0x00cf75 thru 0x0161d7 (0x9263 bytes) 000931 1782757500.974662 mapread 0x023e1e thru 0x027f8e (0x4171 bytes) 000932 1782757501.001681 trunc from 0x040000 to 0x02727f (0x18d80 bytes) 000933 1782757501.026753 mapread 0x006355 thru 0x0161a8 (0xfe54 bytes) 000934 1782757501.051227 MIRROR_RESYNC 15 mirrors 000935 1782757505.763732 punch from 0x01d0fe to 0x02664e (0x9551 bytes) 000936 1782757505.819766 punch from 0x006158 to 0x00ceb2 (0x6d5b bytes) 000937 1782757505.827620 write 0x03188c thru 0x036970 (0x50e5 bytes) 000938 1782757505.829437 fallocate 0x023597 thru 0x02c783 (0x91ed bytes) 000939 1782757505.836653 trunc from 0x001f69 to 0x0063dc (0x4474 bytes) 000940 1782757505.844167 mapwrite 0x0314e2 thru 0x036108 (0x4c27 bytes) 000941 1782757505.871506 trunc from 0x029108 to 0x031e9a (0x8d93 bytes) 000942 1782757505.896363 MIRROR_RESYNC 15 mirrors 000943 1782757511.617277 trunc from 0x036971 to 0x008086 (0x2e8ea bytes) 000944 1782757511.727411 punch from 0x006e71 to 0x008086 (0x1216 bytes) 000945 1782757511.738331 fallocate 0x032b23 thru 0x03e28a (0xb768 bytes) 000946 1782757511.756815 fallocate 0x000a00 thru 0x00d5cd (0xcbce bytes) 000947 1782757511.792132 mapread 0x00086b thru 0x008086 (0x781c bytes) 000948 1782757511.853647 punch from 0x006120 to 0x008086 (0x1f67 bytes) 000949 1782757511.864917 trunc from 0x0007fd to 0x00520e (0x4a12 bytes) 000950 1782757511.882945 punch from 0x00116f to 0x008086 (0x6f18 bytes) 000951 1782757511.903330 trunc from 0x004446 to 0x008086 (0x3c41 bytes) 000952 1782757511.909544 fallocate 0x01c625 thru 0x021117 (0x4af3 bytes) 000953 1782757511.925563 trunc from 0x008087 to 0x034a24 (0x2c99e bytes) 000954 1782757511.975904 fallocate 0x02a134 thru 0x02e5be (0x448b bytes) 000955 1782757511.999701 trunc from 0x016f56 to 0x024f44 (0xdfef bytes) 000956 1782757512.013396 read 0x015e0b thru 0x021240 (0xb436 bytes) 000957 1782757512.039348 fallocate 0x024608 thru 0x026550 (0x1f49 bytes) 000958 1782757512.047292 read 0x001dc3 thru 0x00ef58 (0xd196 bytes) 000959 1782757512.069000 punch from 0x023cd2 to 0x02eb78 (0xaea7 bytes) 000960 1782757512.085528 trunc from 0x02c6cb to 0x034a24 (0x835a bytes) 000961 1782757512.098936 punch from 0x027d56 to 0x02ffb7 (0x8262 bytes) 000963 1782757512.133793 mapwrite 0x026cb7 thru 0x02e459 (0x77a3 bytes) 000964 1782757512.307233 fallocate 0x017ebf thru 0x01d4ed (0x562f bytes) 000965 1782757512.313720 trunc from 0x030091 to 0x03437e (0x42ee bytes) 000966 1782757512.321635 mapread 0x0039f1 thru 0x004eb7 (0x14c7 bytes) 000967 1782757512.327939 MIRROR_RESYNC 15 mirrors 000968 1782757515.823713 mapwrite 0x012e3e thru 0x01ed31 (0xbef4 bytes) 000969 1782757516.025080 mapread 0x0119a8 thru 0x018de0 (0x7439 bytes) 000970 1782757516.030767 write 0x015c68 thru 0x0201a9 (0xa542 bytes) 000971 1782757516.034546 write 0x01149b thru 0x017854 (0x63ba bytes) 000972 1782757516.046976 mapwrite 0x03eb67 thru 0x03ffff (0x1499 bytes) 000973 1782757516.140772 read 0x0011cc thru 0x003247 (0x207c bytes) 000974 1782757516.153776 mapwrite 0x0084e8 thru 0x00b2d5 (0x2dee bytes) 000975 1782757516.183522 punch from 0x0348b2 to 0x03883e (0x3f8d bytes) 000976 1782757516.189947 mapwrite 0x008678 thru 0x01357e (0xaf07 bytes) 000977 1782757516.222038 mapwrite 0x0324f6 thru 0x03c825 (0xa330 bytes) 000979 1782757516.277571 MIRROR_RESYNC 15 mirrors 000980 1782757522.128719 mapread 0x03729f thru 0x03cc6d (0x59cf bytes) 000981 1782757522.156229 MIRROR_EXTEND to 16 mirrors 000982 1782757523.432773 punch from 0x01d5f5 to 0x0201dd (0x2be9 bytes) 000983 1782757523.522703 trunc from 0x01a5ab to 0x01b629 (0x107f bytes) 000984 1782757523.530245 mapwrite 0x02782c thru 0x03580f (0xdfe4 bytes) 000985 1782757523.610743 write 0x02747d thru 0x02b48d (0x4011 bytes) 000986 1782757523.639213 MIRROR_RESYNC 16 mirrors 000987 1782757529.637714 write 0x037d73 thru 0x03ffff (0x828d bytes) 000988 1782757529.704594 trunc from 0x040000 to 0x02d571 (0x12a8e bytes) 000989 1782757529.755410 read 0x02bcc3 thru 0x02d571 (0x18af bytes) 000990 1782757529.764339 trunc from 0x010e2e to 0x01fd72 (0xef45 bytes) 000991 1782757529.770948 trunc from 0x008596 to 0x016b7c (0xe5e7 bytes) 000993 1782757529.783923 read 0x01fe57 thru 0x02116f (0x1319 bytes) 000994 1782757529.808808 write 0x032af0 thru 0x03c123 (0x9634 bytes) 000995 1782757529.812680 mapwrite 0x03167c thru 0x03de85 (0xc80a bytes) 000996 1782757529.887447 trunc from 0x03de86 to 0x03361d (0xa868 bytes) 000997 1782757529.915835 fallocate 0x03b3b0 thru 0x03ffff (0x4c50 bytes) 000998 1782757529.923852 mapread 0x0087b2 thru 0x010c64 (0x84b3 bytes) 000999 1782757529.925445 mapwrite 0x02d7f2 thru 0x02ed77 (0x1586 bytes) 001000 1782757529.941635 punch from 0x003b7e to 0x00fe9c (0xc31f bytes) All operations completed A-OK! PASS 70a (247s) == sanity-flr test 71: check mirror extend parent fid ==== 14:25:34 (1782757534) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x198:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 14:25:46 (1782757546) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.245847 s, 8.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156485 s, 6.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 14:25:56 (1782757556) Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre2 Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 1476608 .. Extending file size to 2831648 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6588128 .. Extending file size to 3202272 .. Extending file size to 5235904 .. Extending file size to 5666176 .. Extending file size to 6749984 .. Extending file size to 7835808 .. Extending file size to 8245888 .. Extending file size to 8728352 .. Extending file size to 8892096 .. Extending file size to 9028448 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8430688 .. Extending file size to 8639488 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61019 61020 61022 61024 61025 failed 192.168.204.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg424-client.virtnet /mnt/lustre2 (opts:) 192.168.204.124@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg424-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (91s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 14:27:27 (1782757647) Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre2 Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3093024 .. Extending file size to 5341792 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5927264 .. Extending file size to 5721632 .. Extending file size to 7590272 .. Extending file size to 8218912 .. Extending file size to 7876160 .. done Extending file size to 8015104 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9069856 .. Extending file size to 8329568 .. Extending file size to 8403744 .. Waiting 62397 62398 62399 62402 62404 done 192.168.204.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg424-client.virtnet /mnt/lustre2 (opts:) 192.168.204.124@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg424-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (82s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 14:28:49 (1782757729) Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2392 1268912 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17680 3589340 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25336 3581684 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43016 7171024 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2392 1268912 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17680 3589340 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25336 3581684 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43016 7171024 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.163679 s, 12.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.168986 s, 12.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x809d5c1b (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg424-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 14:29:11 (1782757751) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.136648 s, 15.3 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 14:29:20 (1782757760) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x39:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x121:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x121:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x122:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x123:0x0] } PASS 203 (12s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 14:29:32 (1782757772) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0942283 s, 22.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1cb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1cc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 14:29:42 (1782757782) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.10272 s, 20.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x123:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x124:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x124:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x123:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x124:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x124:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x125:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 14:29:50 (1782757790) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.144772 s, 14.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x125:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x126:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x125:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x126:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x127:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 14:30:25 (1782757825) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg424-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932337 s, 11.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1cb:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1cd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1cb:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1cd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1cc:0x0] } PASS 204d (54s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 14:31:19 (1782757879) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg424-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ce:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.129376 s, 16.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ce:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1cf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ce:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1cf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (57s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 14:32:16 (1782757936) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg424-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg424-server: Pool lustre.test_204f created Adding targets to pool oleg424-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg424-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.163998 s, 12.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x126:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x128:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x126:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x128:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg424-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg424-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg424-server: Pool lustre.test_204f destroyed PASS 204f (79s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 14:33:36 (1782758016) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 14:33:44 (1782758024) PASS 205b (11s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 14:33:56 (1782758036) oleg424-server: Pool lustre.test_206 created Adding targets to pool oleg424-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg424-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg424-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg424-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg424-server: Pool lustre.test_206 destroyed PASS 206 (38s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 14:34:34 (1782758074) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749298 s, 14.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065947 s, 15.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x4d52c031 Mirror 3: 0x4d52c031 PASS 207 (15s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 14:34:49 (1782758089) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 14:34:53 (1782758093) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 14:34:58 (1782758098) Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.123651 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x660000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2280000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2280000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2280000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x9e0000 Read this (4 bytes): BBBB 192.168.204.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg424-client.virtnet /mnt/lustre2 (opts:) PASS 209a (50s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 14:35:48 (1782758148) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0241903 s, 339 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 14:35:57 (1782758157) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412711 s, 25.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x58 lmm_fid: [0x200000405:0x58:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x58 lmm_fid: [0x200000405:0x58:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611224 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-29 14:35:59.000000000 -0400 Modify: 2026-06-29 14:35:59.000000000 -0400 Change: 2026-06-29 14:36:01.000000000 -0400 Birth: 2026-06-29 14:35:59.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 14:36:08 (1782758168) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813779 s, 12.9 MB/s oleg424-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg424-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg424-server: O/280000401/d16/130: File not found by ext2_lookup PASS 210b (32s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 14:36:39 (1782758199) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.568251 s, 72.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.10529 s, 38.9 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 29 14:36 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 29 14:36 /mnt/lustre/f211.sanity-flr PASS 211 (13s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 14:36:52 (1782758212) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 14:37:01 (1782758221) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 2049 sec =========== 14:37:10 (1782758230) === sanity-flr: start cleanup 14:37:12 (1782758232) === === sanity-flr: finish cleanup 14:37:21 (1782758241) ===