-----============= acceptance-small: sanity-flr ============----- Mon Sep 7 12:04:33 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg650-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 12:04:50 (1788797090) === oleg650-client.virtnet: executing check_config_client /mnt/lustre oleg650-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg650-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9ee35065e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9ee35065e800.idle_timeout=debug disable quota as required oleg650-server: oleg650-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 12:05:29 (1788797129) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8254] == sanity-flr test 0a: lfs mirror create with -N option == 12:05:32 (1788797132) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 12:05:48 (1788797148) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 12:05:50 (1788797150) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 12:05:53 (1788797153) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 12:06:16 (1788797176) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 12:06:19 (1788797179) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 12:06:23 (1788797183) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9ee35065e800.stats=clear osc.lustre-OST0001-osc-ffff9ee35065e800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 12:06:30 (1788797190) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 12:06:40 (1788797200) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 12:06:47 (1788797207) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 12:06:50 (1788797210) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 12:06:58 (1788797218) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 12:07:07 (1788797227) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 12:07:14 (1788797234) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 12:07:22 (1788797242) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 12:07:29 (1788797249) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (9s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 12:07:41 (1788797261) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 12:07:48 (1788797268) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110211 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104076 s, 10.1 MB/s PASS 21 (6s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 12:07:55 (1788797275) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.233959 s, 22.4 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 12:08:02 (1788797282) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 12:08:12 (1788797292) Stopping /mnt/lustre-ost1 (opts:) on oleg650-server oleg650-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ee35065e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ee35065e800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg650-server: oleg650-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg650-client: oleg650-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg650-server: oleg650-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg650-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg650-server oleg650-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ee35065e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ee35065e800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg650-server: oleg650-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg650-client: oleg650-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg650-server: oleg650-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg650-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (60s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 12:09:12 (1788797352) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.497055 s, 23.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg650-server oleg650-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ee35065e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ee35065e800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg650-server: oleg650-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg650-client: oleg650-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg650-server: oleg650-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg650-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (34s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 12:09:46 (1788797386) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg650-server oleg650-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ee35065e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ee35065e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg650-server: oleg650-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg650-client: oleg650-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg650-server: oleg650-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg650-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg650-server oleg650-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ee35065e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ee35065e800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg650-server: oleg650-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg650-client: oleg650-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg650-server: oleg650-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg650-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (66s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 12:10:52 (1788797452) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.96177 s, 17.7 MB/s -rw-r--r-- 1 root root 52428800 Sep 7 12:10 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.879s user 0m0.002s sys 0m0.181s Stopping /mnt/lustre-ost1 (opts:) on oleg650-server oleg650-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ee35065e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ee35065e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.024s user 0m0.001s sys 0m0.181s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg650-server: oleg650-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg650-client: oleg650-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg650-server: oleg650-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg650-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg650-server oleg650-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ee35065e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ee35065e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.925s user 0m0.002s sys 0m0.170s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg650-server: oleg650-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg650-client: oleg650-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg650-server: oleg650-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg650-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg650-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ee35065e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ee35065e800.ost_server_uuid in FULL state after 0 sec PASS 33b (77s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 12:12:09 (1788797529) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 12:12:12 (1788797532) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 12:12:15 (1788797535) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 12:12:18 (1788797538) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 12:12:24 (1788797544) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.43871 s, 28.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (9s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 12:12:33 (1788797553) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380924 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023298 s, 45.0 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 12:12:39 (1788797559) PASS 36c (6s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 12:12:45 (1788797565) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00158888 s, 644 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00126409 s, 810 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00170343 s, 601 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00157839 s, 649 kB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 12:12:53 (1788797573) create test files with size 15745029 .. 1e4dd6788b5645a6999313ec21bbde3b - a4a4958c13d4c68e95ba791e8706648c - 4f47d00c56286e097541e657e2573c5e - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg650-client.virtnet /mnt/lustre (opts:) Starting client: oleg650-client.virtnet: -o user_xattr,flock 192.168.206.150@tcp:/lustre /mnt/lustre PASS 37 (19s) == sanity-flr test 38: resync ============================ 12:13:12 (1788797592) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 12:13:26 (1788797606) PASS 39 (5s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 12:13:32 (1788797612) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862244 s, 12.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc4:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175356 s, 59.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc5:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc6:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 12:13:40 (1788797620) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0992529 s, 21.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.451105 s, 9.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.119003 s, 17.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (13s) == sanity-flr test 42: lfs mirror verify ================= 12:13:53 (1788797633) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 12:13:56 (1788797636) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 12:13:59 (1788797639) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293058 s, 35.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 44a: lfs mirror split check =========== 12:14:05 (1788797645) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.153299 s, 20.5 MB/s PASS 44a (8s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 12:14:13 (1788797653) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd1:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (6s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 12:14:20 (1788797660) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.246534 s, 42.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg650-client.virtnet /mnt/lustre (opts:) Starting client: oleg650-client.virtnet: -o user_xattr,flock 192.168.206.150@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 12:14:30 (1788797670) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9ee3452a7000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.262858 s, 39.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg650-client.virtnet /mnt/lustre (opts:) Starting client: oleg650-client.virtnet: -o user_xattr,flock 192.168.206.150@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9ee343e5f800.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 12:14:44 (1788797684) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (31s) == sanity-flr test 46: Verify setstripe --copy option ==== 12:15:15 (1788797715) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (31s) == sanity-flr test 47: Verify mirror obj alloc =========== 12:15:46 (1788797746) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 12:15:49 (1788797749) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.209451 s, 15.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.100585 s, 31.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x4937fe9f (nosync,stale) Mirror 2: 0x4937fe9f (nosync,stale) ** verify mirror 2 doesn't change original checksum: 60bb8dc16a04d6309ec87f106af48490 - mirror 1 checksum: b027200c7f6490b5e69848b93f3b7cac - mirror 2 checksum: 60bb8dc16a04d6309ec87f106af48490 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-07T12:15:52-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-07T12:15:52-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 60bb8dc16a04d6309ec87f106af48490 - mirror 1 checksum: b027200c7f6490b5e69848b93f3b7cac - mirror 2 checksum: b027200c7f6490b5e69848b93f3b7cac - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-07T12:15:52-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-07T12:15:52-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } PASS 48 (8s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 12:15:58 (1788797758) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601054 s, 17.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 12:16:04 (1788797764) Failing mds1 on oleg650-server Stopping /mnt/lustre-mds1 (opts:) on oleg650-server 12:16:16 (1788797776) shut down facet: mds1 facet_host: oleg650-server facet_failover_host: oleg650-server Failover mds1 to oleg650-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg650-server: oleg650-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg650-client: oleg650-server: ssh exited with exit code 1 Started lustre-MDT0000 12:16:39 (1788797799) targets are mounted 12:16:39 (1788797799) facet_failover done oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319648 s, 32.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } PASS 50A (50s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 12:16:55 (1788797815) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.035315 s, 580 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0146024 s, 1.4 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (8s) == sanity-flr test 50b: mirror rsync handles sparseness == 12:17:03 (1788797823) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0595067 s, 344 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00803351 s, 2.5 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (8s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 12:17:11 (1788797831) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0190634 s, 859 kB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 12:17:19 (1788797839) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0415921 s, 492 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (6s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 12:17:26 (1788797846) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00760254 s, 539 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0365239 s, 112 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 12:17:32 (1788797852) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 12:17:52 (1788797872) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 12:18:19 (1788797899) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (21s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 12:18:41 (1788797921) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 12:19:04 (1788797944) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 12:19:10 (1788797950) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57566 mirror split pid 57567 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57566 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57567 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 12:20:17 (1788798017) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 8018 000001 1788798019.086086 write 0x01bd9a thru 0x020e79 (0x50e0 bytes) 000002 1788798019.103080 trunc from 0x00074c to 0x005623 (0x4ed8 bytes) 000003 1788798019.117087 write 0x02f025 thru 0x03865b (0x9637 bytes) 000004 1788798019.120398 mapwrite 0x03df46 thru 0x03eaeb (0xba6 bytes) 000005 1788798019.184373 mapwrite 0x03dac0 thru 0x03ffff (0x2540 bytes) truncating to largest ever: 0x21155 000007 1788798019.230619 trunc from 0x040000 to 0x021154 (0x1eeab bytes) 000008 1788798019.253564 mapwrite 0x020d90 thru 0x02d3e0 (0xc651 bytes) 000009 1788798019.311341 mapread 0x00508f thru 0x009ee9 (0x4e5b bytes) 000010 1788798019.328223 mapwrite 0x00b6f3 thru 0x014e89 (0x9797 bytes) fallocating to largest ever: 0x38ae4 000011 1788798019.366316 fallocate 0x034307 thru 0x038ae3 (0x47dd bytes) 000013 1788798019.374071 fallocate 0x00fa31 thru 0x01f9f6 (0xffc6 bytes) 000014 1788798019.382261 mapwrite 0x03c7aa thru 0x03ffff (0x3856 bytes) 000017 1788798019.429506 fallocate 0x00de8c thru 0x016abf (0x8c34 bytes) 000018 1788798019.434118 mapwrite 0x03841e thru 0x03c5f6 (0x41d9 bytes) 000019 1788798019.457358 trunc from 0x040000 to 0x02e3f9 (0x11c06 bytes) 000020 1788798019.496338 read 0x00b091 thru 0x017c34 (0xcba4 bytes) 000021 1788798019.497177 punch from 0x00ae53 to 0x00ec21 (0x3dcf bytes) 000023 1788798019.509493 trunc from 0x02e3fa to 0x005442 (0x28fb7 bytes) 000024 1788798019.530509 mapread 0x000a92 thru 0x005161 (0x46d0 bytes) fallocating to largest ever: 0x3f0b8 000025 1788798019.540459 fallocate 0x03894e thru 0x03f0b7 (0x676a bytes) 000026 1788798019.546827 trunc from 0x033cc5 to 0x03f0b7 (0xb3f3 bytes) 000027 1788798019.555335 trunc from 0x02b22c to 0x0377b6 (0xc58b bytes) 000028 1788798019.561885 punch from 0x001f75 to 0x00d4e4 (0xb570 bytes) 000029 1788798019.568642 write 0x00780e thru 0x00d815 (0x6008 bytes) 000030 1788798019.581570 punch from 0x02edc8 to 0x03d7da (0xea13 bytes) 000031 1788798019.587050 write 0x03baeb thru 0x03ffff (0x4515 bytes) 000032 1788798019.591850 write 0x029d14 thru 0x0374fd (0xd7ea bytes) 000033 1788798019.612015 trunc from 0x040000 to 0x01f3c7 (0x20c38 bytes) 000034 1788798019.651637 mapwrite 0x038e7d thru 0x03ffff (0x7183 bytes) 000035 1788798019.706356 read 0x01a6cc thru 0x02294d (0x8282 bytes) 000036 1788798019.719730 write 0x032040 thru 0x033385 (0x1346 bytes) zero_range to largest ever: 0x40000 000037 1788798019.722349 trunc from 0x03a41d to 0x03ffff (0x5be3 bytes) 000038 1788798019.728772 mapwrite 0x019dd3 thru 0x01eead (0x50db bytes) 000039 1788798019.747303 trunc from 0x02feb0 to 0x031cef (0x1e40 bytes) 000041 1788798019.753464 trunc from 0x040000 to 0x026e03 (0x191fc bytes) 000042 1788798019.769089 fallocate 0x010bb3 thru 0x0203c1 (0xf80f bytes) 000043 1788798019.777660 mapwrite 0x00e895 thru 0x01084a (0x1fb6 bytes) 000044 1788798019.798499 trunc from 0x014582 to 0x01980e (0x528d bytes) 000045 1788798019.805591 write 0x011ce4 thru 0x020e2b (0xf148 bytes) 000046 1788798019.807539 fallocate 0x0056d2 thru 0x00bea9 (0x67d8 bytes) 000047 1788798019.812322 read 0x00a2ef thru 0x0171ee (0xcf00 bytes) 000048 1788798019.813791 mapread 0x005dbb thru 0x007ffa (0x2240 bytes) 000049 1788798019.816615 write 0x014219 thru 0x01c064 (0x7e4c bytes) 000050 1788798019.820451 read 0x024a7d thru 0x026e03 (0x2387 bytes) 000051 1788798019.822532 read 0x001f60 thru 0x00f5cb (0xd66c bytes) 000052 1788798019.823876 mapwrite 0x012655 thru 0x01ec45 (0xc5f1 bytes) 000053 1788798019.851563 trunc from 0x000799 to 0x00a1bd (0x9a25 bytes) 000054 1788798019.864545 mapread 0x00e354 thru 0x0143f2 (0x609f bytes) 000055 1788798019.871170 punch from 0x0219dd to 0x026e03 (0x5427 bytes) 000056 1788798019.882164 mapread 0x004ac6 thru 0x00726f (0x27aa bytes) 000057 1788798019.912091 trunc from 0x01de4b to 0x02475b (0x6911 bytes) 000058 1788798019.921127 read 0x011d51 thru 0x01f6d6 (0xd986 bytes) 000059 1788798019.947273 trunc from 0x009dfb to 0x012de5 (0x8feb bytes) 000060 1788798019.962695 fallocate 0x0159dc thru 0x0203bd (0xa9e2 bytes) 000062 1788798019.972453 punch from 0x005a75 to 0x00f792 (0x9d1e bytes) 000063 1788798019.981489 fallocate 0x03fd85 thru 0x03ffff (0x27b bytes) 000064 1788798019.995066 mapread 0x010838 thru 0x01e10a (0xd8d3 bytes) 000065 1788798020.031686 mapread 0x00733e thru 0x00feaa (0x8b6d bytes) 000066 1788798020.120533 trunc from 0x01aae0 to 0x022f50 (0x8471 bytes) 000067 1788798020.126724 fallocate 0x0305ea thru 0x034c3d (0x4654 bytes) 000068 1788798020.132161 trunc from 0x026e04 to 0x034927 (0xdb24 bytes) 000069 1788798020.147694 trunc from 0x00e250 to 0x01ce6b (0xec1c bytes) 000070 1788798020.153156 trunc from 0x031a9e to 0x03468a (0x2bed bytes) 000072 1788798020.158546 mapwrite 0x021684 thru 0x02de0f (0xc78c bytes) 000073 1788798020.202318 mapwrite 0x010844 thru 0x01fc49 (0xf406 bytes) 000074 1788798020.316207 mapread 0x0336e0 thru 0x034927 (0x1248 bytes) 000075 1788798020.317214 write 0x009c7a thru 0x010239 (0x65c0 bytes) 000076 1788798020.320372 punch from 0x016f7f to 0x0194a6 (0x2528 bytes) 000078 1788798020.328356 read 0x016de1 thru 0x01ba08 (0x4c28 bytes) 000079 1788798020.349646 MIRROR_EXTEND to 2 mirrors 000080 1788798020.988937 punch from 0x011586 to 0x012be6 (0x1661 bytes) 000081 1788798021.039679 trunc from 0x034928 to 0x023825 (0x11102 bytes) 000082 1788798021.103652 trunc from 0x023826 to 0x03130c (0xdae7 bytes) 000083 1788798021.140083 write 0x00efa7 thru 0x014d59 (0x5db3 bytes) 000084 1788798021.158641 fallocate 0x01c565 thru 0x01f40c (0x2ea8 bytes) 000085 1788798021.170031 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000086 1788798021.352283 read 0x011c0b thru 0x012517 (0x90d bytes) 000087 1788798021.357093 mapread 0x01042f thru 0x01ea49 (0xe61b bytes) 000088 1788798021.387324 read 0x020dca thru 0x030170 (0xf3a7 bytes) 000090 1788798021.403879 mapread 0x01abfc thru 0x023ede (0x92e3 bytes) 000091 1788798021.404797 trunc from 0x02e5bf to 0x02fc75 (0x16b7 bytes) 000092 1788798021.419390 read 0x018cde thru 0x02704c (0xe36f bytes) 000093 1788798021.420321 trunc from 0x02653a to 0x02a677 (0x413e bytes) 000094 1788798021.433254 write 0x0146db thru 0x015468 (0xd8e bytes) 000095 1788798021.441586 fallocate 0x025ec9 thru 0x031bed (0xbd25 bytes) 000096 1788798021.452642 read 0x0093fe thru 0x016f46 (0xdb49 bytes) 000097 1788798021.454121 punch from 0x012e5a to 0x017679 (0x4820 bytes) 000098 1788798021.465219 punch from 0x0111b5 to 0x015d0c (0x4b58 bytes) 000099 1788798021.474672 write 0x036ba6 thru 0x03dc1f (0x707a bytes) 000100 1788798021.476056 punch from 0x03a912 to 0x03c49f (0x1b8e bytes) 000101 1788798021.490576 trunc from 0x03dc20 to 0x01bf55 (0x21cca bytes) 000102 1788798021.517144 mapread 0x00d3b3 thru 0x011bf6 (0x4844 bytes) 000103 1788798021.527623 trunc from 0x01bf56 to 0x02f957 (0x13a02 bytes) 000104 1788798021.550694 read 0x02558d thru 0x02f957 (0xa3cb bytes) 000105 1788798021.561998 punch from 0x00d2f1 to 0x019224 (0xbf34 bytes) 000106 1788798021.567066 fallocate 0x025d1b thru 0x02b442 (0x5728 bytes) 000107 1788798021.572482 punch from 0x029a31 to 0x02f957 (0x5f27 bytes) 000108 1788798021.577375 punch from 0x00b8f6 to 0x01342a (0x7b35 bytes) 000109 1788798021.583171 mapwrite 0x0147e2 thru 0x016234 (0x1a53 bytes) 000110 1788798021.619111 punch from 0x029fa1 to 0x02e545 (0x45a5 bytes) 000111 1788798021.626137 mapread 0x00af9f thru 0x00e7f1 (0x3853 bytes) 000113 1788798021.653733 fallocate 0x03447c thru 0x035d3e (0x18c3 bytes) 000115 1788798021.657845 trunc from 0x02f958 to 0x023878 (0xc0df bytes) 000116 1788798021.671868 punch from 0x0074b5 to 0x0140a0 (0xcbec bytes) 000118 1788798021.683620 mapwrite 0x036211 thru 0x03ffff (0x9def bytes) 000119 1788798021.744314 read 0x008eda thru 0x011699 (0x87c0 bytes) 000120 1788798021.757336 trunc from 0x008fff to 0x00a9eb (0x19ed bytes) 000121 1788798021.762013 punch from 0x000cf5 to 0x006e85 (0x6191 bytes) 000122 1788798021.768500 punch from 0x01e6ba to 0x02b5cd (0xcf14 bytes) 000123 1788798021.774967 trunc from 0x016fd5 to 0x01d10f (0x613b bytes) 000124 1788798021.796436 write 0x0317b0 thru 0x033f30 (0x2781 bytes) 000129 1788798021.801679 trunc from 0x040000 to 0x030d68 (0xf297 bytes) 000130 1788798021.818549 fallocate 0x00d423 thru 0x015154 (0x7d32 bytes) 000132 1788798021.828253 mapwrite 0x008f92 thru 0x012901 (0x9970 bytes) 000133 1788798021.865086 mapwrite 0x03be6c thru 0x03e584 (0x2719 bytes) 000134 1788798021.923479 MIRROR_EXTEND to 2 mirrors 000136 1788798022.596658 write 0x00c99e thru 0x00cc2d (0x290 bytes) 000137 1788798022.651016 mapwrite 0x01c001 thru 0x01db1a (0x1b1a bytes) 000138 1788798022.687698 write 0x0394af thru 0x03ffff (0x6b51 bytes) 000141 1788798022.697653 mapread 0x0203cd thru 0x0214b3 (0x10e7 bytes) 000142 1788798022.718660 read 0x0242dc thru 0x02d11f (0x8e44 bytes) 000143 1788798022.744009 mapwrite 0x017a45 thru 0x01ca7d (0x5039 bytes) 000144 1788798022.807033 MIRROR_RESYNC 2 mirrors 000145 1788798023.515018 mapread 0x0128bf thru 0x01f34b (0xca8d bytes) 000146 1788798023.553182 MIRROR_EXTEND to 3 mirrors 000147 1788798024.346896 read 0x020236 thru 0x0203e2 (0x1ad bytes) 000148 1788798024.358687 mapwrite 0x0327c5 thru 0x03ffff (0xd83b bytes) 000149 1788798024.524610 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x325b5c5b (stale) Mirror 3: 0x325b5c5b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8955e24e Mirror 4: 0x8955e24e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000150 1788798025.561291 write 0x01c6c7 thru 0x0200a8 (0x39e2 bytes) 000151 1788798025.639722 punch from 0x027fdb to 0x029d04 (0x1d2a bytes) 000152 1788798025.647280 punch from 0x028d5a to 0x02b3f7 (0x269e bytes) 000153 1788798025.657260 write 0x02964e thru 0x031776 (0x8129 bytes) 000154 1788798025.686969 fallocate 0x0245d4 thru 0x02bd4a (0x7777 bytes) 000155 1788798025.693530 read 0x03a986 thru 0x03ffff (0x567a bytes) 000157 1788798025.706285 mapwrite 0x03ebff thru 0x03ffff (0x1401 bytes) 000158 1788798025.729354 fallocate 0x03a3c9 thru 0x03ffff (0x5c37 bytes) 000159 1788798025.736225 fallocate 0x02086d thru 0x02920f (0x89a3 bytes) 000160 1788798025.742482 trunc from 0x008862 to 0x010623 (0x7dc2 bytes) 000161 1788798025.747808 mapwrite 0x00726b thru 0x00d6cc (0x6462 bytes) 000162 1788798025.794519 mapwrite 0x0111df thru 0x015df8 (0x4c1a bytes) 000163 1788798025.876599 trunc from 0x040000 to 0x01afc4 (0x2503b bytes) 000164 1788798025.931388 punch from 0x010a2b to 0x01a0d7 (0x96ad bytes) 000165 1788798025.942480 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1afc5): Mirror 2: 0xa5b4e88b (stale) Mirror 4: 0xa5b4e88b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1afc5): Mirror 2: 0xa5b4e88b (stale) Mirror 4: 0xa5b4e88b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1afc5: skipped 000166 1788798026.382965 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1afc5): Mirror 2: 0xa5b4e88b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1afc5): Mirror 2: 0xa5b4e88b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1afc5: skipped 000167 1788798026.740993 punch from 0x0131a0 to 0x01afc4 (0x7e25 bytes) 000168 1788798026.768924 fallocate 0x021eea thru 0x029677 (0x778e bytes) 000169 1788798026.779720 trunc from 0x01574d to 0x025215 (0xfac9 bytes) 000170 1788798026.796069 trunc from 0x00c047 to 0x018aa2 (0xca5c bytes) 000171 1788798026.811882 MIRROR_RESYNC 2 mirrors 000172 1788798027.432822 fallocate 0x00e571 thru 0x00f8ea (0x137a bytes) 000173 1788798027.471130 mapwrite 0x03f7ac thru 0x03ffff (0x854 bytes) 000174 1788798027.509360 punch from 0x016b3b to 0x0214b0 (0xa976 bytes) 000176 1788798027.515217 punch from 0x036db8 to 0x03ffff (0x9248 bytes) 000177 1788798027.520628 fallocate 0x032558 thru 0x03b9f7 (0x94a0 bytes) 000178 1788798027.529802 MIRROR_RESYNC 2 mirrors 000179 1788798028.202096 fallocate 0x0086a0 thru 0x0106ad (0x800e bytes) 000180 1788798028.262123 MIRROR_RESYNC 2 mirrors 000181 1788798028.944710 punch from 0x037504 to 0x037ca6 (0x7a3 bytes) 000182 1788798029.034464 fallocate 0x019831 thru 0x02350e (0x9cde bytes) 000183 1788798029.049595 trunc from 0x02749b to 0x03488c (0xd3f2 bytes) 000184 1788798029.060344 trunc from 0x040000 to 0x02466b (0x1b994 bytes) 000185 1788798029.082687 write 0x00bd79 thru 0x00f4b2 (0x373a bytes) 000187 1788798029.095315 mapwrite 0x0274e7 thru 0x02d8f0 (0x640a bytes) 000188 1788798029.152424 fallocate 0x01b78e thru 0x01c78d (0x1000 bytes) 000189 1788798029.178713 MIRROR_RESYNC 2 mirrors 000190 1788798029.856525 mapwrite 0x015494 thru 0x01dc6f (0x87dc bytes) 000191 1788798029.972469 mapread 0x0115b3 thru 0x015ceb (0x4739 bytes) 000193 1788798029.975295 read 0x013524 thru 0x020820 (0xd2fd bytes) 000194 1788798029.986508 mapwrite 0x0013b1 thru 0x00e1e6 (0xce36 bytes) 000195 1788798030.042711 trunc from 0x02d8f1 to 0x00dac4 (0x1fe2c bytes) 000197 1788798030.091192 punch from 0x0037f6 to 0x00d16d (0x9978 bytes) 000198 1788798030.099337 mapwrite 0x01b249 thru 0x021088 (0x5e40 bytes) 000199 1788798030.142321 mapread 0x011e77 thru 0x01433e (0x24c8 bytes) 000200 1788798030.152054 mapwrite 0x0042b5 thru 0x013b65 (0xf8b1 bytes) 000201 1788798030.180232 fallocate 0x03c598 thru 0x03ffff (0x3a68 bytes) 000202 1788798030.190607 write 0x021b9f thru 0x02d719 (0xbb7b bytes) 000203 1788798030.200582 mapread 0x01f9ba thru 0x02ad47 (0xb38e bytes) 000204 1788798030.203604 write 0x0384b5 thru 0x03ffff (0x7b4b bytes) 000205 1788798030.217175 MIRROR_RESYNC 2 mirrors 000206 1788798031.007973 punch from 0x01f34d to 0x01ff08 (0xbbc bytes) 000207 1788798031.036251 trunc from 0x00da68 to 0x0108cc (0x2e65 bytes) 000208 1788798031.039843 mapwrite 0x030be3 thru 0x0375c2 (0x69e0 bytes) 000209 1788798031.075072 MIRROR_RESYNC 2 mirrors 000210 1788798031.731640 punch from 0x00b004 to 0x01150a (0x6507 bytes) 000211 1788798031.772242 trunc from 0x01885d to 0x0263c8 (0xdb6c bytes) 000212 1788798031.786351 write 0x015a84 thru 0x01ec9e (0x921b bytes) 000213 1788798031.811434 mapwrite 0x0154de thru 0x0165a7 (0x10ca bytes) 000214 1788798031.842556 mapwrite 0x020834 thru 0x026b65 (0x6332 bytes) 000215 1788798031.910538 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000216 1788798032.102191 punch from 0x01c3b5 to 0x02865d (0xc2a9 bytes) 000217 1788798032.113272 mapwrite 0x00a8c7 thru 0x010185 (0x58bf bytes) 000218 1788798032.160544 read 0x032f83 thru 0x03b0fb (0x8179 bytes) 000219 1788798032.176930 mapwrite 0x019d97 thru 0x023d2d (0x9f97 bytes) 000220 1788798032.238932 trunc from 0x040000 to 0x023172 (0x1ce8d bytes) 000221 1788798032.289919 trunc from 0x023173 to 0x03bdb9 (0x18c47 bytes) 000222 1788798032.305390 trunc from 0x03bdba to 0x03b509 (0x8b0 bytes) 000223 1788798032.331496 trunc from 0x01a19d to 0x01bc11 (0x1a75 bytes) 000224 1788798032.341083 trunc from 0x032ae0 to 0x03734f (0x4870 bytes) 000226 1788798032.347336 read 0x02572b thru 0x0288ac (0x3182 bytes) 000227 1788798032.357085 trunc from 0x002835 to 0x008d50 (0x651c bytes) 000228 1788798032.364251 write 0x035043 thru 0x03ffff (0xafbd bytes) 000229 1788798032.366374 trunc from 0x03fb65 to 0x03ffff (0x49b bytes) 000231 1788798032.381332 fallocate 0x03ce51 thru 0x03fcb0 (0x2e60 bytes) 000233 1788798032.390349 punch from 0x011f45 to 0x01a60e (0x86ca bytes) 000234 1788798032.399628 write 0x01bbff thru 0x0225bf (0x69c1 bytes) 000236 1788798032.401323 fallocate 0x011ff9 thru 0x012fcc (0xfd4 bytes) 000237 1788798032.409635 mapread 0x022a31 thru 0x02b8ce (0x8e9e bytes) 000238 1788798032.417229 fallocate 0x009d1c thru 0x019040 (0xf325 bytes) 000242 1788798032.437182 trunc from 0x01c7bb to 0x0289fc (0xc242 bytes) 000243 1788798032.457888 trunc from 0x040000 to 0x00cd27 (0x332d8 bytes) 000244 1788798032.481830 trunc from 0x0052f5 to 0x00cd27 (0x7a33 bytes) 000245 1788798032.487223 read 0x00479f thru 0x004c4e (0x4b0 bytes) 000246 1788798032.497984 trunc from 0x00bb1c to 0x00cd27 (0x120c bytes) 000247 1788798032.503869 write 0x0289aa thru 0x02a899 (0x1ef0 bytes) 000249 1788798032.506830 fallocate 0x01a42c thru 0x026162 (0xbd37 bytes) 000250 1788798032.512158 punch from 0x023c6e to 0x02a899 (0x6c2c bytes) 000251 1788798032.529086 read 0x012053 thru 0x021858 (0xf806 bytes) 000252 1788798032.563320 read 0x0098f3 thru 0x016c47 (0xd355 bytes) 000253 1788798032.564143 fallocate 0x0033f4 thru 0x005ee5 (0x2af2 bytes) 000255 1788798032.570089 mapwrite 0x00018c thru 0x00840b (0x8280 bytes) 000256 1788798032.588516 fallocate 0x014bf8 thru 0x01ef14 (0xa31d bytes) 000258 1788798032.595317 trunc from 0x02a89a to 0x014558 (0x16341 bytes) 000259 1788798032.614249 mapread 0x005ad3 thru 0x01328e (0xd7bc bytes) 000260 1788798032.615448 trunc from 0x014559 to 0x01ebaf (0xa657 bytes) 000262 1788798032.646077 MIRROR_EXTEND to 2 mirrors 000263 1788798033.373147 fallocate 0x01775c thru 0x01e757 (0x6ffc bytes) 000264 1788798033.415436 mapread 0x00ae95 thru 0x011a8b (0x6bf7 bytes) 000265 1788798033.433078 mapwrite 0x01a003 thru 0x01d308 (0x3306 bytes) 000266 1788798033.460466 mapwrite 0x020929 thru 0x02f7c4 (0xee9c bytes) 000267 1788798033.535095 fallocate 0x015f8e thru 0x01e481 (0x84f4 bytes) 000269 1788798033.547787 punch from 0x0124e1 to 0x014693 (0x21b3 bytes) 000271 1788798033.552381 punch from 0x00081b to 0x00da49 (0xd22f bytes) 000272 1788798033.558403 fallocate 0x034741 thru 0x03ffff (0xb8bf bytes) 000273 1788798033.563244 trunc from 0x01b4b6 to 0x02880a (0xd355 bytes) 000274 1788798033.568369 trunc from 0x00f1c1 to 0x01d153 (0xdf93 bytes) 000275 1788798033.575527 fallocate 0x026dbd thru 0x02ac66 (0x3eaa bytes) 000276 1788798033.580485 fallocate 0x031325 thru 0x0379fd (0x66d9 bytes) 000277 1788798033.586055 trunc from 0x0379fe to 0x011f84 (0x25a79 bytes) 000279 1788798033.607609 punch from 0x005590 to 0x008cca (0x373b bytes) 000280 1788798033.613221 mapwrite 0x009a9c thru 0x018607 (0xeb6c bytes) 000281 1788798033.687057 MIRROR_RESYNC 2 mirrors 000282 1788798034.615406 MIRROR_EXTEND to 3 mirrors 000283 1788798035.402201 fallocate 0x030fbb thru 0x034c84 (0x3cca bytes) 000284 1788798035.438885 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34c85): Mirror 3: 0x60b68b82 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34c85): Mirror 3: 0x60b68b82 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34c85: skipped 000286 1788798035.732879 fallocate 0x03345c thru 0x0360c3 (0x2c68 bytes) 000288 1788798035.740484 trunc from 0x005dc0 to 0x01059a (0xa7db bytes) 000289 1788798035.745381 punch from 0x01f83e to 0x0274e0 (0x7ca3 bytes) 000290 1788798035.753238 trunc from 0x031fd9 to 0x0360c3 (0x40eb bytes) 000291 1788798035.758430 trunc from 0x034296 to 0x0360c3 (0x1e2e bytes) 000293 1788798035.763853 mapwrite 0x01fa9a thru 0x02cbbc (0xd123 bytes) 000294 1788798035.820516 punch from 0x006c96 to 0x01524f (0xe5ba bytes) 000295 1788798035.825191 punch from 0x00883d to 0x0116df (0x8ea3 bytes) 000296 1788798035.829932 mapwrite 0x02d024 thru 0x032077 (0x5054 bytes) 000297 1788798035.844267 write 0x03b288 thru 0x03ffff (0x4d78 bytes) 000299 1788798035.849962 fallocate 0x0342f5 thru 0x034b83 (0x88f bytes) 000300 1788798035.854441 trunc from 0x02db08 to 0x03850e (0xaa07 bytes) 000301 1788798035.859824 mapwrite 0x038524 thru 0x03ffff (0x7adc bytes) 000302 1788798035.882067 mapread 0x027a79 thru 0x02dafc (0x6084 bytes) 000303 1788798035.888400 trunc from 0x02ff0f to 0x0373c3 (0x74b5 bytes) 000304 1788798035.892625 write 0x00abfb thru 0x017d87 (0xd18d bytes) 000305 1788798035.905536 fallocate 0x00a5b9 thru 0x00edb8 (0x4800 bytes) 000306 1788798035.910485 mapwrite 0x001dcf thru 0x00a53f (0x8771 bytes) 000307 1788798035.948572 fallocate 0x026675 thru 0x031065 (0xa9f1 bytes) 000308 1788798035.953600 write 0x037ac9 thru 0x03ffff (0x8537 bytes) 000309 1788798035.958510 trunc from 0x040000 to 0x0110f8 (0x2ef07 bytes) 000310 1788798035.974309 fallocate 0x0231d2 thru 0x031d7d (0xebac bytes) 000311 1788798035.980678 mapwrite 0x02854d thru 0x036b5c (0xe610 bytes) 000313 1788798036.034146 trunc from 0x0262a1 to 0x02f1c7 (0x8f27 bytes) 000314 1788798036.041393 read 0x0270c9 thru 0x030d0d (0x9c45 bytes) 000315 1788798036.051982 mapread 0x01b4d6 thru 0x01fa0b (0x4536 bytes) 000316 1788798036.067455 mapwrite 0x016471 thru 0x01b35f (0x4eef bytes) 000317 1788798036.085634 read 0x0172ce thru 0x021044 (0x9d77 bytes) 000318 1788798036.086656 trunc from 0x0277fe to 0x0324b4 (0xacb7 bytes) 000319 1788798036.093737 read 0x02fafc thru 0x036b5c (0x7061 bytes) 000320 1788798036.108138 trunc from 0x036b5d to 0x008332 (0x2e82a bytes) 000321 1788798036.122314 trunc from 0x0024a3 to 0x008332 (0x5e90 bytes) 000322 1788798036.128070 fallocate 0x029950 thru 0x03623d (0xc8ee bytes) 000323 1788798036.134716 mapread 0x000261 thru 0x008332 (0x80d2 bytes) 000324 1788798036.167026 write 0x02cffc thru 0x03b514 (0xe519 bytes) 000325 1788798036.169793 trunc from 0x03b515 to 0x01a4c9 (0x2104b bytes) 000326 1788798036.184372 write 0x0208cb thru 0x027661 (0x6d97 bytes) 000327 1788798036.185882 write 0x0172d3 thru 0x017933 (0x661 bytes) 000328 1788798036.191447 punch from 0x0043cd to 0x006c67 (0x289b bytes) 000329 1788798036.196505 mapread 0x00fb59 thru 0x0156c5 (0x5b6d bytes) 000331 1788798036.226402 write 0x01987f thru 0x01f2d8 (0x5a5a bytes) 000332 1788798036.239026 read 0x0116e8 thru 0x01a5d7 (0x8ef0 bytes) 000333 1788798036.251008 mapwrite 0x005482 thru 0x0125e3 (0xd162 bytes) 000334 1788798036.278073 mapread 0x01fb9a thru 0x027661 (0x7ac8 bytes) 000335 1788798036.284476 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x27662): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27662): Mirror 3: 0x50dda22 (stale) CRC-32 checksum value for chunk [0, 0x27662): Mirror 1: 0xb59f96d7 Mirror 4: 0xb59f96d7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27662: skipped 000336 1788798036.955602 trunc from 0x027662 to 0x03552e (0xdecd bytes) 000337 1788798037.016142 read 0x0071a3 thru 0x00d415 (0x6273 bytes) 000338 1788798037.028074 fallocate 0x030dd9 thru 0x032a3c (0x1c64 bytes) 000339 1788798037.032500 mapread 0x00a4f3 thru 0x00e216 (0x3d24 bytes) 000340 1788798037.037313 fallocate 0x02bb3a thru 0x0335aa (0x7a71 bytes) 000341 1788798037.042139 fallocate 0x003958 thru 0x00453c (0xbe5 bytes) 000342 1788798037.047096 mapwrite 0x02041a thru 0x029e20 (0x9a07 bytes) 000343 1788798037.082576 read 0x010f14 thru 0x019be1 (0x8cce bytes) 000344 1788798037.095410 mapread 0x01bce5 thru 0x01ce7d (0x1199 bytes) 000345 1788798037.096528 write 0x008357 thru 0x00b682 (0x332c bytes) 000346 1788798037.098304 punch from 0x02e38d to 0x03552e (0x71a2 bytes) 000347 1788798037.103311 read 0x02c3aa thru 0x0303ea (0x4041 bytes) 000348 1788798037.111922 mapwrite 0x02e9a5 thru 0x02f6f8 (0xd54 bytes) 000349 1788798037.129179 punch from 0x02e7a8 to 0x03552e (0x6d87 bytes) 000350 1788798037.135879 trunc from 0x00505d to 0x00a532 (0x54d6 bytes) 000351 1788798037.141635 trunc from 0x00aa6a to 0x00e285 (0x381c bytes) 000352 1788798037.146220 punch from 0x027c48 to 0x02dccd (0x6086 bytes) 000353 1788798037.155184 MIRROR_RESYNC 3 mirrors 000354 1788798038.567474 trunc from 0x01d68d to 0x02d2e0 (0xfc54 bytes) 000355 1788798038.636350 trunc from 0x03552f to 0x02afdb (0xa553 bytes) 000356 1788798038.661274 mapwrite 0x0003d0 thru 0x00e26e (0xde9f bytes) 000357 1788798038.721926 read 0x0002c8 thru 0x008da0 (0x8ad9 bytes) 000358 1788798038.723202 trunc from 0x02afdc to 0x03209d (0x70c2 bytes) 000359 1788798038.742468 trunc from 0x03194d to 0x031a80 (0x134 bytes) 000360 1788798038.755581 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3209e): Mirror 3: 0xc64d807d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3209e): Mirror 3: 0xc64d807d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3209e: skipped 000361 1788798039.061199 fallocate 0x015064 thru 0x02078c (0xb729 bytes) 000362 1788798039.071374 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3209e): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3209e): Mirror 3: 0xc64d807d (stale) CRC-32 checksum value for chunk [0, 0x3209e): Mirror 1: 0x5fc2e005 Mirror 4: 0x5fc2e005 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3209e: skipped 000364 1788798040.172540 punch from 0x00511f to 0x00d00a (0x7eec bytes) 000365 1788798040.259449 mapread 0x022275 thru 0x030fc0 (0xed4c bytes) 000366 1788798040.327739 punch from 0x026dfc to 0x02923c (0x2441 bytes) 000367 1788798040.337927 MIRROR_RESYNC 3 mirrors 000368 1788798041.526576 fallocate 0x01f376 thru 0x022f5e (0x3be9 bytes) 000369 1788798041.581833 fallocate 0x039817 thru 0x03ffff (0x67e9 bytes) 000370 1788798041.589148 fallocate 0x005398 thru 0x0152fb (0xff64 bytes) 000371 1788798041.598405 MIRROR_RESYNC 3 mirrors 000372 1788798043.200587 write 0x0131e3 thru 0x01bac8 (0x88e6 bytes) 000373 1788798043.316876 write 0x002943 thru 0x010c22 (0xe2e0 bytes) 000374 1788798043.340655 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9dd79b2c (stale) Mirror 4: 0x9dd79b2c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc216e102 Mirror 5: 0xc216e102 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000375 1788798044.782387 trunc from 0x040000 to 0x02a32f (0x15cd0 bytes) 000376 1788798044.903210 punch from 0x01ab4b to 0x023a79 (0x8f2f bytes) 000377 1788798044.925004 write 0x006562 thru 0x007f38 (0x19d7 bytes) 000378 1788798044.941903 fallocate 0x02a729 thru 0x02c99d (0x2275 bytes) 000379 1788798044.947287 fallocate 0x021d97 thru 0x0254b2 (0x371c bytes) 000380 1788798044.953816 punch from 0x014656 to 0x01bef3 (0x789e bytes) 000381 1788798044.963571 mapwrite 0x003656 thru 0x006e13 (0x37be bytes) 000382 1788798045.006868 trunc from 0x01cae8 to 0x02a32f (0xd848 bytes) 000383 1788798045.011834 read 0x00d643 thru 0x01981f (0xc1dd bytes) 000384 1788798045.024396 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a330): Mirror 3: 0x83b0775d (stale) Mirror 4: 0x83b0775d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a330): Mirror 3: 0x83b0775d (stale) Mirror 4: 0x83b0775d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a330: skipped 000385 1788798045.399908 read 0x001385 thru 0x00dd43 (0xc9bf bytes) 000386 1788798045.412942 fallocate 0x005104 thru 0x011a0b (0xc908 bytes) 000387 1788798045.424230 trunc from 0x02a330 to 0x034878 (0xa549 bytes) 000388 1788798045.450799 read 0x034413 thru 0x034878 (0x466 bytes) 000389 1788798045.471446 trunc from 0x015cce to 0x01e553 (0x8886 bytes) 000391 1788798045.477374 mapread 0x01c9ff thru 0x0218a0 (0x4ea2 bytes) 000392 1788798045.501510 write 0x033261 thru 0x03709a (0x3e3a bytes) 000393 1788798045.503513 mapwrite 0x02f531 thru 0x035c46 (0x6716 bytes) 000395 1788798045.528718 mapread 0x00a0d8 thru 0x00bf06 (0x1e2f bytes) 000396 1788798045.530590 mapwrite 0x03ca38 thru 0x03ffff (0x35c8 bytes) 000397 1788798045.575395 write 0x027a91 thru 0x02fbb4 (0x8124 bytes) 000398 1788798045.576780 read 0x00492d thru 0x0132a1 (0xe975 bytes) 000399 1788798045.580797 mapread 0x01d066 thru 0x022104 (0x509f bytes) 000400 1788798045.583424 punch from 0x02de60 to 0x03bf6f (0xe110 bytes) 000401 1788798045.600031 read 0x01ffd7 thru 0x0252a2 (0x52cc bytes) 000402 1788798045.601493 trunc from 0x040000 to 0x027f7b (0x18084 bytes) 000403 1788798045.628375 fallocate 0x023313 thru 0x02350c (0x1fa bytes) 000404 1788798045.641569 trunc from 0x027f7c to 0x00f027 (0x18f54 bytes) 000405 1788798045.684297 MIRROR_RESYNC 3 mirrors 000406 1788798046.601098 fallocate 0x013124 thru 0x0225d9 (0xf4b6 bytes) 000407 1788798046.640127 punch from 0x00e50c to 0x0138a1 (0x5396 bytes) 000408 1788798046.646826 trunc from 0x0225da to 0x005b53 (0x1ca86 bytes) 000409 1788798046.665571 punch from 0x001a18 to 0x005b53 (0x413c bytes) 000410 1788798046.671506 write 0x022942 thru 0x025164 (0x2823 bytes) 000411 1788798046.674143 trunc from 0x025165 to 0x02081c (0x4948 bytes) 000412 1788798046.690370 fallocate 0x037513 thru 0x03df31 (0x6a1f bytes) 000414 1788798046.697414 trunc from 0x004c70 to 0x01280d (0xdb9e bytes) 000415 1788798046.702834 fallocate 0x017bd8 thru 0x0180fa (0x523 bytes) 000417 1788798046.708461 trunc from 0x013724 to 0x016439 (0x2d16 bytes) 000418 1788798046.713401 write 0x033626 thru 0x0339e1 (0x3bc bytes) 000419 1788798046.724208 read 0x02e8c9 thru 0x03a2b3 (0xb9eb bytes) 000420 1788798046.745794 trunc from 0x03df32 to 0x00be1a (0x32117 bytes) 000422 1788798046.809741 mapread 0x00722a thru 0x00be1a (0x4bf1 bytes) 000424 1788798046.872079 write 0x02e838 thru 0x03d225 (0xe9ee bytes) 000425 1788798046.877356 trunc from 0x03d226 to 0x013b5b (0x296ca bytes) 000426 1788798046.930493 trunc from 0x0045a1 to 0x005dd4 (0x1834 bytes) 000427 1788798046.944846 mapread 0x008a4e thru 0x01362f (0xabe2 bytes) 000428 1788798046.979445 trunc from 0x0103f6 to 0x013b5b (0x3766 bytes) 000429 1788798046.998419 read 0x012420 thru 0x013b5b (0x173c bytes) 000431 1788798047.052536 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x13b5c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13b5c): Mirror 3: 0xb630f2e (stale) Mirror 4: 0xb630f2e (stale) CRC-32 checksum value for chunk [0, 0x13b5c): Mirror 1: 0xe45e3edc Mirror 5: 0xe45e3edc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13b5c: skipped 000433 1788798048.081201 punch from 0x006b80 to 0x00dcb7 (0x7138 bytes) 000436 1788798048.145242 fallocate 0x0376a2 thru 0x03ffff (0x895e bytes) 000437 1788798048.154033 read 0x00ed94 thru 0x013b5b (0x4dc8 bytes) 000439 1788798048.164914 write 0x00a62a thru 0x01640d (0xbde4 bytes) 000440 1788798048.174864 fallocate 0x00b06f thru 0x01a3b0 (0xf342 bytes) 000441 1788798048.179693 trunc from 0x004a80 to 0x0097ac (0x4d2d bytes) 000442 1788798048.184723 mapread 0x00a45a thru 0x01640d (0xbfb4 bytes) 000443 1788798048.187174 fallocate 0x025478 thru 0x02ce3b (0x79c4 bytes) 000446 1788798048.198132 trunc from 0x02ce3c to 0x0077a9 (0x25692 bytes) 000447 1788798048.221157 fallocate 0x01f362 thru 0x022f89 (0x3c28 bytes) 000449 1788798048.230824 read 0x00fd6a thru 0x01efb9 (0xf250 bytes) 000450 1788798048.251481 fallocate 0x008a7a thru 0x00fe71 (0x73f8 bytes) 000451 1788798048.258666 mapread 0x004052 thru 0x00af1e (0x6ecd bytes) 000452 1788798048.261854 trunc from 0x022f8a to 0x030449 (0xd4c0 bytes) 000453 1788798048.301926 punch from 0x010463 to 0x015550 (0x50ee bytes) 000454 1788798048.311061 read 0x02cc59 thru 0x030449 (0x37f1 bytes) 000455 1788798048.330289 mapread 0x009111 thru 0x017022 (0xdf12 bytes) 000457 1788798048.332348 read 0x013b2b thru 0x015b0e (0x1fe4 bytes) 000458 1788798048.333305 mapwrite 0x03667b thru 0x039683 (0x3009 bytes) 000459 1788798048.407471 mapread 0x01b60b thru 0x028379 (0xcd6f bytes) 000460 1788798048.408806 write 0x033f73 thru 0x03e0fc (0xa18a bytes) 000462 1788798048.413014 write 0x027de3 thru 0x0315f5 (0x9813 bytes) 000463 1788798048.414895 read 0x0347c9 thru 0x035f77 (0x17af bytes) 000464 1788798048.423833 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e0fd): Mirror 3: 0xe232b533 (stale) Mirror 4: 0xe232b533 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e0fd): Mirror 3: 0xe232b533 (stale) Mirror 4: 0xe232b533 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e0fd: skipped 000465 1788798048.803791 mapwrite 0x022296 thru 0x02da17 (0xb782 bytes) 000466 1788798048.897494 mapwrite 0x007a52 thru 0x009063 (0x1612 bytes) 000467 1788798048.925099 trunc from 0x03e0fd to 0x022fd1 (0x1b12b bytes) 000468 1788798048.952896 MIRROR_RESYNC 3 mirrors 000469 1788798049.963354 read 0x007864 thru 0x00bc81 (0x441e bytes) 000470 1788798049.987952 mapread 0x0165e2 thru 0x01773e (0x115d bytes) 000472 1788798050.001396 MIRROR_EXTEND to 4 mirrors 000473 1788798051.080427 mapwrite 0x017f37 thru 0x0249c2 (0xca8c bytes) 000474 1788798051.194956 trunc from 0x0249c3 to 0x02c162 (0x77a0 bytes) 000475 1788798051.248244 MIRROR_RESYNC 4 mirrors 000476 1788798052.836556 mapread 0x006041 thru 0x00dc7a (0x7c3a bytes) 000477 1788798052.861930 MIRROR_EXTEND to 5 mirrors 000478 1788798053.763422 punch from 0x005039 to 0x007869 (0x2831 bytes) 000479 1788798053.831415 write 0x00f94c thru 0x018fb8 (0x966d bytes) 000480 1788798053.853158 trunc from 0x0282ae to 0x02c162 (0x3eb5 bytes) 000481 1788798053.859547 mapwrite 0x019af9 thru 0x02927f (0xf787 bytes) 000482 1788798053.945093 punch from 0x01330e to 0x022012 (0xed05 bytes) 000483 1788798053.950892 mapread 0x011201 thru 0x01be8c (0xac8c bytes) 000484 1788798053.973932 trunc from 0x0044d0 to 0x00e5d7 (0xa108 bytes) 000486 1788798053.979636 mapwrite 0x036d2c thru 0x037ff2 (0x12c7 bytes) 000487 1788798054.025553 mapwrite 0x020f40 thru 0x02925f (0x8320 bytes) 000488 1788798054.058408 fallocate 0x027884 thru 0x02e8db (0x7058 bytes) 000489 1788798054.066432 fallocate 0x01c048 thru 0x022187 (0x6140 bytes) 000490 1788798054.076398 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37ff3): Mirror 4: 0x227258f3 (stale) Mirror 5: 0x227258f3 (stale) Mirror 6: 0x227258f3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ff3): Mirror 4: 0x227258f3 (stale) Mirror 5: 0x227258f3 (stale) Mirror 6: 0x227258f3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ff3: skipped 000491 1788798054.571203 punch from 0x01f68b to 0x02e53f (0xeeb5 bytes) 000492 1788798054.582243 trunc from 0x0044d9 to 0x00fde6 (0xb90e bytes) 000493 1788798054.596798 fallocate 0x02eca1 thru 0x03aa30 (0xbd90 bytes) 000494 1788798054.604732 mapread 0x0334f2 thru 0x0389e6 (0x54f5 bytes) 000495 1788798054.641396 trunc from 0x03aa31 to 0x0232e8 (0x17748 bytes) 000497 1788798054.688783 fallocate 0x0055df thru 0x008681 (0x30a3 bytes) 000498 1788798054.709685 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x232e9): Mirror 4: 0x1081bef2 (stale) Mirror 6: 0x1081bef2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x232e9): Mirror 4: 0x1081bef2 (stale) Mirror 6: 0x1081bef2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x232e9: skipped 000499 1788798055.113991 read 0x016775 thru 0x0169c3 (0x24f bytes) 000501 1788798055.129791 read 0x0088ba thru 0x01017c (0x78c3 bytes) 000502 1788798055.154432 mapwrite 0x01578d thru 0x02353c (0xddb0 bytes) 000503 1788798055.249817 punch from 0x00d60d to 0x00f75c (0x2150 bytes) 000504 1788798055.260913 punch from 0x010be0 to 0x01326b (0x268c bytes) 000505 1788798055.285510 write 0x025f24 thru 0x033d82 (0xde5f bytes) 000506 1788798055.291408 punch from 0x01905a to 0x025f46 (0xceed bytes) 000507 1788798055.366259 write 0x022a81 thru 0x030b7c (0xe0fc bytes) 000508 1788798055.391612 mapwrite 0x039c18 thru 0x03e4e2 (0x48cb bytes) 000509 1788798055.561272 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e4e3): Mirror 4: 0x49d59005 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e4e3): Mirror 4: 0x49d59005 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e4e3: skipped 000510 1788798055.956976 read 0x025f66 thru 0x02b41b (0x54b6 bytes) 000512 1788798055.972523 read 0x02be00 thru 0x02e592 (0x2793 bytes) 000513 1788798056.014636 read 0x021bfd thru 0x02d6ee (0xbaf2 bytes) 000514 1788798056.015908 punch from 0x01d5f2 to 0x01f9a3 (0x23b2 bytes) 000515 1788798056.023813 mapread 0x00692d thru 0x008812 (0x1ee6 bytes) 000516 1788798056.025829 mapwrite 0x03896e thru 0x03f3ff (0x6a92 bytes) 000517 1788798056.080838 mapwrite 0x0198a9 thru 0x02761b (0xdd73 bytes) 000518 1788798056.106016 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f400): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f400): Mirror 4: 0xfac87226 (stale) CRC-32 checksum value for chunk [0, 0x3f400): Mirror 1: 0xf72dc42f Mirror 5: 0xf72dc42f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f400: skipped 000519 1788798057.122031 MIRROR_RESYNC 3 mirrors 000520 1788798057.928099 MIRROR_RESYNC 3 mirrors 000521 1788798058.387717 read 0x01c6c9 thru 0x024e66 (0x879e bytes) 000522 1788798058.395673 read 0x032532 thru 0x03f3ff (0xcece bytes) 000523 1788798058.406388 trunc from 0x00fb85 to 0x019234 (0x96b0 bytes) 000524 1788798058.453410 punch from 0x00a72e to 0x00c43b (0x1d0e bytes) 000525 1788798058.458438 trunc from 0x03f400 to 0x03f8f4 (0x4f5 bytes) 000526 1788798058.528141 mapread 0x034711 thru 0x038803 (0x40f3 bytes) 000527 1788798058.569086 trunc from 0x014898 to 0x015351 (0xaba bytes) 000528 1788798058.576512 mapwrite 0x0354a6 thru 0x03962c (0x4187 bytes) 000529 1788798058.610096 mapread 0x00ea75 thru 0x017ea0 (0x942c bytes) 000530 1788798058.660191 fallocate 0x01d70b thru 0x029d67 (0xc65d bytes) 000532 1788798058.665367 fallocate 0x0312f7 thru 0x036ecd (0x5bd7 bytes) 000533 1788798058.692464 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f8f5): Mirror 5: 0x767ed8e5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f8f5): Mirror 5: 0x767ed8e5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f8f5: skipped 000534 1788798059.028948 trunc from 0x025e66 to 0x02f311 (0x94ac bytes) 000536 1788798059.035962 trunc from 0x03f8f5 to 0x020b40 (0x1edb4 bytes) 000537 1788798059.078786 read 0x01c021 thru 0x020b40 (0x4b20 bytes) 000538 1788798059.087707 punch from 0x016262 to 0x019678 (0x3417 bytes) 000539 1788798059.100619 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x20b41): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20b41): Mirror 5: 0xe898f95e (stale) CRC-32 checksum value for chunk [0, 0x20b41): Mirror 1: 0xa62e9425 Mirror 6: 0xa62e9425 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20b41: skipped 000540 1788798060.068740 trunc from 0x020b41 to 0x02278a (0x1c4a bytes) 000541 1788798060.167881 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2278b): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2278b): Mirror 5: 0x752db9fd (stale) Mirror 6: 0x630307b (stale) CRC-32 checksum value for chunk [0, 0x2278b): Mirror 1: 0x630307b Mirror 7: 0x630307b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2278b: skipped 000542 1788798061.514079 mapwrite 0x026462 thru 0x033b99 (0xd738 bytes) 000543 1788798061.754503 MIRROR_RESYNC 4 mirrors 000544 1788798063.268286 mapread 0x01eaa2 thru 0x0267d7 (0x7d36 bytes) 000545 1788798063.298363 trunc from 0x00dbf8 to 0x010b79 (0x2f82 bytes) 000546 1788798063.340048 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x33b9a): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33b9a): Mirror 5: 0x168782de (stale) Mirror 6: 0x168782de (stale) Mirror 7: 0x168782de (stale) CRC-32 checksum value for chunk [0, 0x33b9a): Mirror 1: 0x168782de Mirror 8: 0x168782de lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33b9a: skipped 000548 1788798064.425126 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x33b9a): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33b9a): Mirror 5: 0x168782de (stale) Mirror 6: 0x168782de (stale) Mirror 7: 0x168782de (stale) CRC-32 checksum value for chunk [0, 0x33b9a): Mirror 1: 0x168782de Mirror 8: 0x168782de Mirror 9: 0x168782de lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33b9a: skipped 000549 1788798065.490587 punch from 0x00d79a to 0x01cf44 (0xf7ab bytes) 000550 1788798065.550821 mapread 0x0175fc thru 0x024028 (0xca2d bytes) 000551 1788798065.582049 mapwrite 0x028e5e thru 0x02cfbb (0x415e bytes) 000552 1788798065.621921 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33b9a): Mirror 6: 0x168782de (stale) Mirror 7: 0x168782de (stale) Mirror 8: 0x168782de (stale) Mirror 9: 0x168782de (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33b9a): Mirror 6: 0x168782de (stale) Mirror 7: 0x168782de (stale) Mirror 8: 0x168782de (stale) Mirror 9: 0x168782de (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33b9a: skipped 000553 1788798066.103924 punch from 0x0106ac to 0x01b12d (0xaa82 bytes) 000555 1788798066.117659 punch from 0x004370 to 0x0122f8 (0xdf89 bytes) 000556 1788798066.132964 punch from 0x02dbd8 to 0x032bce (0x4ff7 bytes) 000557 1788798066.143654 mapwrite 0x00d146 thru 0x012a50 (0x590b bytes) 000558 1788798066.194395 trunc from 0x0332e9 to 0x033b99 (0x8b1 bytes) 000559 1788798066.200338 trunc from 0x001a88 to 0x00f0bb (0xd634 bytes) 000561 1788798066.206366 fallocate 0x01efd8 thru 0x0221a6 (0x31cf bytes) 000562 1788798066.212295 punch from 0x014a44 to 0x02006e (0xb62b bytes) 000563 1788798066.217729 punch from 0x02c5cb to 0x031a13 (0x5449 bytes) 000564 1788798066.229194 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33b9a): Mirror 7: 0x168782de (stale) Mirror 8: 0x168782de (stale) Mirror 9: 0x168782de (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33b9a): Mirror 7: 0x168782de (stale) Mirror 8: 0x168782de (stale) Mirror 9: 0x168782de (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33b9a: skipped 000565 1788798066.789673 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33b9a): Mirror 7: 0x168782de (stale) Mirror 9: 0x168782de (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33b9a): Mirror 7: 0x168782de (stale) Mirror 9: 0x168782de (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33b9a: skipped 000567 1788798067.245916 trunc from 0x033b9a to 0x031057 (0x2b42 bytes) 000568 1788798067.279500 punch from 0x0283e2 to 0x029005 (0xc24 bytes) 000569 1788798067.298581 MIRROR_RESYNC 3 mirrors 000571 1788798069.083280 fallocate 0x0091c9 thru 0x00dc27 (0x4a5f bytes) 000572 1788798069.135920 fallocate 0x00255b thru 0x00bb31 (0x95d7 bytes) 000574 1788798069.149709 read 0x02005f thru 0x028cf6 (0x8c98 bytes) 000575 1788798069.167601 fallocate 0x024dee thru 0x02c6f6 (0x7909 bytes) 000576 1788798069.186108 fallocate 0x009a10 thru 0x011f75 (0x8566 bytes) 000577 1788798069.195810 fallocate 0x0090ed thru 0x00efb9 (0x5ecd bytes) 000578 1788798069.216890 MIRROR_RESYNC 3 mirrors 000579 1788798070.977338 write 0x02ad37 thru 0x03338e (0x8658 bytes) 000580 1788798071.013388 read 0x024dc1 thru 0x03170f (0xc94f bytes) 000582 1788798071.048455 mapwrite 0x025321 thru 0x02f1b1 (0x9e91 bytes) 000583 1788798071.145221 trunc from 0x02a093 to 0x03338e (0x92fc bytes) 000584 1788798071.152947 fallocate 0x01623a thru 0x01c6a0 (0x6467 bytes) 000585 1788798071.161634 mapwrite 0x0200f9 thru 0x02ad11 (0xac19 bytes) 000587 1788798071.210544 punch from 0x01cbc7 to 0x0223e8 (0x5822 bytes) 000588 1788798071.219410 fallocate 0x004633 thru 0x006011 (0x19df bytes) 000589 1788798071.227597 trunc from 0x01e24d to 0x02a1a2 (0xbf56 bytes) 000591 1788798071.235419 read 0x005363 thru 0x012021 (0xccbf bytes) 000592 1788798071.250599 punch from 0x000e0f to 0x00125b (0x44d bytes) 000593 1788798071.260879 fallocate 0x033116 thru 0x039b54 (0x6a3f bytes) 000594 1788798071.271872 mapread 0x02ec21 thru 0x03338e (0x476e bytes) 000595 1788798071.273266 trunc from 0x03338f to 0x024409 (0xef85 bytes) 000596 1788798071.301073 mapwrite 0x01f027 thru 0x022063 (0x303d bytes) 000597 1788798071.319347 fallocate 0x03a64b thru 0x03d7ab (0x3161 bytes) 000598 1788798071.325054 write 0x00c527 thru 0x00c83f (0x319 bytes) 000599 1788798071.334270 mapwrite 0x03f47d thru 0x03ffff (0xb83 bytes) 000600 1788798071.429650 read 0x0238e1 thru 0x0301ed (0xc90d bytes) 000601 1788798071.456121 MIRROR_RESYNC 3 mirrors 000602 1788798073.310365 write 0x011abe thru 0x01d78e (0xbcd1 bytes) 000603 1788798073.359109 write 0x003adf thru 0x01380d (0xfd2f bytes) 000604 1788798073.372051 mapread 0x026a33 thru 0x028eb8 (0x2486 bytes) 000605 1788798073.404358 trunc from 0x022da3 to 0x028d80 (0x5fde bytes) 000606 1788798073.410787 punch from 0x03318b to 0x03cabb (0x9931 bytes) 000607 1788798073.416280 punch from 0x032b69 to 0x039d3b (0x71d3 bytes) 000608 1788798073.423801 write 0x00b238 thru 0x012d70 (0x7b39 bytes) 000610 1788798073.425481 mapwrite 0x035742 thru 0x03e1ac (0x8a6b bytes) 000611 1788798073.462941 mapwrite 0x0047a1 thru 0x00deb0 (0x9710 bytes) 000612 1788798073.499552 read 0x02b187 thru 0x03544f (0xa2c9 bytes) 000615 1788798073.506555 mapwrite 0x02e5d1 thru 0x03026a (0x1c9a bytes) 000616 1788798073.524948 trunc from 0x01fb9a to 0x024102 (0x4569 bytes) 000617 1788798073.533707 write 0x01690d thru 0x01e7f7 (0x7eeb bytes) 000619 1788798073.548033 read 0x02ad4f thru 0x0303d1 (0x5683 bytes) 000620 1788798073.549370 mapwrite 0x01116e thru 0x01f550 (0xe3e3 bytes) 000623 1788798073.589434 punch from 0x007653 to 0x00c18c (0x4b3a bytes) 000624 1788798073.596506 punch from 0x007d82 to 0x016511 (0xe790 bytes) 000625 1788798073.606880 mapwrite 0x0162f2 thru 0x01b22f (0x4f3e bytes) 000626 1788798073.639120 fallocate 0x01cc41 thru 0x02c107 (0xf4c7 bytes) 000628 1788798073.650178 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x244012d7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x244012d7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000629 1788798074.018628 trunc from 0x011fe4 to 0x01eb25 (0xcb42 bytes) 000630 1788798074.041318 mapread 0x02fd7e thru 0x03d452 (0xd6d5 bytes) 000631 1788798074.077299 fallocate 0x03d023 thru 0x03ffff (0x2fdd bytes) 000632 1788798074.094404 mapwrite 0x036dc1 thru 0x038871 (0x1ab1 bytes) 000633 1788798074.124143 read 0x03e269 thru 0x03ffff (0x1d97 bytes) 000634 1788798074.124686 punch from 0x0171c9 to 0x01a323 (0x315b bytes) 000635 1788798074.131271 trunc from 0x040000 to 0x0238ae (0x1c751 bytes) 000636 1788798074.154997 trunc from 0x0238af to 0x00d34c (0x16562 bytes) 000637 1788798074.175299 punch from 0x007079 to 0x00c7f4 (0x577c bytes) 000638 1788798074.199232 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000639 1788798074.350641 trunc from 0x00143f to 0x0091c5 (0x7d87 bytes) 000640 1788798074.358312 mapwrite 0x030c5e thru 0x0377bc (0x6b5f bytes) 000641 1788798074.420034 read 0x003865 thru 0x00fe80 (0xc61c bytes) 000642 1788798074.437568 fallocate 0x018b69 thru 0x023449 (0xa8e1 bytes) 000643 1788798074.446057 fallocate 0x034060 thru 0x03ffff (0xbfa0 bytes) 000645 1788798074.451475 mapwrite 0x03ea52 thru 0x03ffff (0x15ae bytes) 000646 1788798074.490970 MIRROR_EXTEND to 2 mirrors 000647 1788798075.058345 mapread 0x01fd82 thru 0x02d3d6 (0xd655 bytes) 000648 1788798075.107363 trunc from 0x036cd1 to 0x03c45e (0x578e bytes) 000649 1788798075.194969 mapwrite 0x02f4f0 thru 0x03d16a (0xdc7b bytes) 000650 1788798075.237446 trunc from 0x028ce6 to 0x033efe (0xb219 bytes) 000651 1788798075.243206 punch from 0x02deeb to 0x033580 (0x5696 bytes) 000652 1788798075.250537 punch from 0x017851 to 0x025356 (0xdb06 bytes) 000653 1788798075.255841 mapwrite 0x00c82d thru 0x019f2b (0xd6ff bytes) 000654 1788798075.305917 MIRROR_RESYNC 2 mirrors 000655 1788798075.836117 fallocate 0x03dbb6 thru 0x03ffff (0x244a bytes) 000656 1788798075.887325 punch from 0x0158ef to 0x01c2a2 (0x69b4 bytes) 000657 1788798075.903253 MIRROR_RESYNC 2 mirrors 000658 1788798076.610650 trunc from 0x040000 to 0x00d115 (0x32eea bytes) 000659 1788798076.664536 mapwrite 0x0043b6 thru 0x00a4de (0x6129 bytes) 000660 1788798076.703328 trunc from 0x00d116 to 0x03d93e (0x30829 bytes) 000661 1788798076.721668 mapread 0x02a4f7 thru 0x038517 (0xe021 bytes) 000662 1788798076.745656 mapread 0x001d2a thru 0x008656 (0x692d bytes) 000663 1788798076.747556 punch from 0x0109ef to 0x0173e8 (0x69fa bytes) 000664 1788798076.752663 trunc from 0x00a1e0 to 0x017536 (0xd357 bytes) 000666 1788798076.759561 mapwrite 0x0116aa thru 0x0136a0 (0x1ff7 bytes) 000669 1788798076.783546 write 0x026457 thru 0x033ce1 (0xd88b bytes) 000670 1788798076.785972 trunc from 0x03d93f to 0x00b3b7 (0x32587 bytes) 000671 1788798076.806355 mapread 0x00478b thru 0x00b3b7 (0x6c2d bytes) 000673 1788798076.809033 read 0x002c03 thru 0x00782a (0x4c28 bytes) 000674 1788798076.811832 mapread 0x003380 thru 0x00b3b7 (0x8038 bytes) 000676 1788798076.814715 trunc from 0x00b3b8 to 0x00a65d (0xd5a bytes) 000677 1788798076.846245 mapread 0x0058b2 thru 0x00a65d (0x4dac bytes) 000678 1788798076.847405 fallocate 0x004d0b thru 0x0064ec (0x17e2 bytes) 000679 1788798076.852985 mapwrite 0x023a10 thru 0x0297bf (0x5db0 bytes) 000680 1788798076.915649 punch from 0x00633b to 0x00f5d5 (0x929b bytes) 000681 1788798076.926638 write 0x00f39e thru 0x011b5d (0x27c0 bytes) 000682 1788798076.943399 trunc from 0x016f75 to 0x025bca (0xec56 bytes) 000683 1788798076.957136 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x297c0): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x297c0): Mirror 2: 0x2cb5f792 (stale) CRC-32 checksum value for chunk [0, 0x297c0): Mirror 1: 0xc3f8b25c Mirror 3: 0xc3f8b25c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x297c0: skipped 000685 1788798078.383686 write 0x038281 thru 0x03ffff (0x7d7f bytes) 000686 1788798078.454980 fallocate 0x005213 thru 0x012e00 (0xdbee bytes) 000687 1788798078.461652 read 0x025136 thru 0x02e312 (0x91dd bytes) 000688 1788798078.469540 mapwrite 0x02b4b9 thru 0x035c41 (0xa789 bytes) 000689 1788798078.516692 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xdb19da30 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xdb19da30 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000690 1788798079.064208 trunc from 0x040000 to 0x0171dc (0x28e23 bytes) 000691 1788798079.090209 trunc from 0x0171dd to 0x03bc79 (0x24a9d bytes) 000692 1788798079.105663 mapread 0x031a46 thru 0x036c8a (0x5245 bytes) 000693 1788798079.137734 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3bc7a): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bc7a): Mirror 3: 0xf8b08cf2 (stale) CRC-32 checksum value for chunk [0, 0x3bc7a): Mirror 1: 0xb9f7c19d Mirror 4: 0xb9f7c19d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bc7a: skipped 000694 1788798080.158605 mapread 0x038cd7 thru 0x03bc79 (0x2fa3 bytes) 000695 1788798080.191654 mapwrite 0x01bed4 thru 0x01c197 (0x2c4 bytes) 000696 1788798080.324012 trunc from 0x03bc7a to 0x025e7c (0x15dfd bytes) 000697 1788798080.397247 trunc from 0x025e7d to 0x01f2d2 (0x6baa bytes) 000698 1788798080.421556 trunc from 0x01f2d3 to 0x01f502 (0x230 bytes) 000699 1788798080.445856 mapread 0x007bcb thru 0x00dcde (0x6114 bytes) 000700 1788798080.461031 mapread 0x0167e2 thru 0x01a504 (0x3d23 bytes) 000701 1788798080.462960 mapread 0x0188a7 thru 0x01b19d (0x28f7 bytes) 000702 1788798080.464628 mapread 0x017bd1 thru 0x01f502 (0x7932 bytes) 000703 1788798080.466356 punch from 0x010be1 to 0x01a3d4 (0x97f4 bytes) 000704 1788798080.475089 read 0x001e10 thru 0x005858 (0x3a49 bytes) 000705 1788798080.476091 read 0x015f58 thru 0x01f502 (0x95ab bytes) 000706 1788798080.491114 write 0x0268c9 thru 0x031e0e (0xb546 bytes) 000707 1788798080.500733 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31e0f): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31e0f): Mirror 3: 0x2a5fd3d7 (stale) Mirror 4: 0x43e469de (stale) CRC-32 checksum value for chunk [0, 0x31e0f): Mirror 1: 0x1655975f Mirror 5: 0x1655975f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31e0f: skipped 000708 1788798081.600200 punch from 0x00c931 to 0x01518a (0x885a bytes) 000709 1788798081.659872 trunc from 0x031e0f to 0x01f85f (0x125af bytes) 000710 1788798081.768400 mapwrite 0x02c9f8 thru 0x034303 (0x790c bytes) 000711 1788798081.835287 read 0x00914e thru 0x00a6cd (0x1580 bytes) 000712 1788798081.846017 trunc from 0x034304 to 0x0385a8 (0x42a5 bytes) 000713 1788798081.871267 fallocate 0x01eace thru 0x023cb0 (0x51e3 bytes) 000714 1788798081.884399 fallocate 0x02c080 thru 0x03383b (0x77bc bytes) 000715 1788798081.909637 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x385a9): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x385a9): Mirror 3: 0x4acc385a (stale) Mirror 4: 0x9f15ded (stale) Mirror 5: 0x2d1fa801 (stale) CRC-32 checksum value for chunk [0, 0x385a9): Mirror 1: 0xe6b8d1d6 Mirror 6: 0xe6b8d1d6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x385a9: skipped 000716 1788798083.177470 punch from 0x0107e5 to 0x016144 (0x5960 bytes) 000718 1788798083.233080 fallocate 0x013826 thru 0x01aafb (0x72d6 bytes) 000720 1788798083.242081 punch from 0x02e8f7 to 0x0316af (0x2db9 bytes) 000721 1788798083.248004 punch from 0x033474 to 0x0385a8 (0x5135 bytes) 000722 1788798083.262470 mapwrite 0x014cd4 thru 0x02492c (0xfc59 bytes) 000723 1788798083.438012 fallocate 0x035303 thru 0x0389da (0x36d8 bytes) 000724 1788798083.445615 trunc from 0x00ac90 to 0x00eee6 (0x4257 bytes) 000725 1788798083.459278 mapwrite 0x02bf6d thru 0x034f16 (0x8faa bytes) 000726 1788798083.569101 fallocate 0x01a4a7 thru 0x020751 (0x62ab bytes) 000727 1788798083.575628 write 0x013e35 thru 0x01e7ce (0xa99a bytes) 000728 1788798083.584633 mapread 0x001aed thru 0x01098e (0xeea2 bytes) 000729 1788798083.600374 punch from 0x00d91b to 0x01ad53 (0xd439 bytes) 000730 1788798083.649905 punch from 0x01253c to 0x012ef3 (0x9b8 bytes) 000731 1788798083.657044 mapread 0x00c0f2 thru 0x0182ae (0xc1bd bytes) 000732 1788798083.713212 trunc from 0x03857c to 0x0389da (0x45f bytes) 000733 1788798083.723179 fallocate 0x0313d4 thru 0x0381f9 (0x6e26 bytes) 000734 1788798083.740802 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x389db): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x389db): Mirror 3: 0x70586912 (stale) Mirror 4: 0x3d35c4e4 (stale) Mirror 5: 0x6c7d4f7a (stale) Mirror 6: 0xe1bca084 (stale) CRC-32 checksum value for chunk [0, 0x389db): Mirror 1: 0x69a12f77 Mirror 7: 0x69a12f77 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x389db: skipped 000735 1788798085.219626 write 0x019ace thru 0x01ba7d (0x1fb0 bytes) 000736 1788798085.336969 read 0x021c29 thru 0x0287e9 (0x6bc1 bytes) 000738 1788798085.352629 trunc from 0x0389db to 0x0326b6 (0x6324 bytes) 000739 1788798085.377801 punch from 0x0055a9 to 0x012471 (0xcec9 bytes) 000740 1788798085.385946 read 0x00f806 thru 0x016ca2 (0x749d bytes) 000741 1788798085.424584 write 0x023f85 thru 0x02c81f (0x889b bytes) 000742 1788798085.427917 mapwrite 0x00af02 thru 0x015391 (0xa490 bytes) 000743 1788798085.464100 read 0x0013f0 thru 0x00a40f (0x9020 bytes) 000744 1788798085.465022 mapwrite 0x032706 thru 0x03cb73 (0xa46e bytes) 000745 1788798085.541934 read 0x023c3b thru 0x02fe04 (0xc1ca bytes) 000746 1788798085.544000 write 0x015bb9 thru 0x0178f9 (0x1d41 bytes) 000747 1788798085.545915 trunc from 0x020279 to 0x02ea28 (0xe7b0 bytes) 000748 1788798085.553047 write 0x014f25 thru 0x015185 (0x261 bytes) 000749 1788798085.563950 trunc from 0x034a1d to 0x03cb73 (0x8157 bytes) 000751 1788798085.577572 trunc from 0x03cb74 to 0x011c01 (0x2af72 bytes) 000752 1788798085.615567 trunc from 0x011c02 to 0x037f9a (0x26399 bytes) 000753 1788798085.648859 punch from 0x037b98 to 0x037f9a (0x403 bytes) 000754 1788798085.659980 mapwrite 0x03560e thru 0x03ffff (0xa9f2 bytes) 000755 1788798085.764659 trunc from 0x0167a9 to 0x0227a6 (0xbffe bytes) 000756 1788798085.775693 read 0x01193a thru 0x016363 (0x4a2a bytes) 000757 1788798085.787366 mapwrite 0x010d61 thru 0x016b49 (0x5de9 bytes) 000758 1788798085.813498 mapwrite 0x00216b thru 0x002c98 (0xb2e bytes) 000760 1788798085.827242 write 0x0035c8 thru 0x006e82 (0x38bb bytes) 000761 1788798085.828638 mapwrite 0x01cf08 thru 0x020260 (0x3359 bytes) 000762 1788798085.850196 mapread 0x032a94 thru 0x0341fe (0x176b bytes) 000763 1788798085.851651 trunc from 0x01a243 to 0x022dab (0x8b69 bytes) 000764 1788798085.861325 mapwrite 0x00abfd thru 0x01434e (0x9752 bytes) 000765 1788798085.880745 mapwrite 0x000318 thru 0x006b95 (0x687e bytes) 000766 1788798085.897543 read 0x02a62f thru 0x03486e (0xa240 bytes) 000767 1788798085.898300 punch from 0x035d68 to 0x0395d8 (0x3871 bytes) 000768 1788798085.905374 mapwrite 0x03e94a thru 0x03ffff (0x16b6 bytes) 000770 1788798085.922430 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xdb19da30 (stale) Mirror 4: 0xec35a385 (stale) Mirror 5: 0x5e23a0f8 (stale) Mirror 6: 0x77588505 (stale) Mirror 7: 0x282f1377 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdfb35e03 Mirror 8: 0xdfb35e03 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000771 1788798087.640701 read 0x00fa8c thru 0x014eba (0x542f bytes) 000772 1788798087.663284 mapwrite 0x00c18f thru 0x00ce28 (0xc9a bytes) 000773 1788798087.791796 trunc from 0x040000 to 0x00e85a (0x317a5 bytes) 000774 1788798087.831995 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xe85b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe85b): Mirror 3: 0x524d4835 (stale) Mirror 4: 0x524d4835 (stale) Mirror 5: 0x524d4835 (stale) Mirror 6: 0x524d4835 (stale) Mirror 7: 0x524d4835 (stale) Mirror 8: 0x7a68c529 (stale) CRC-32 checksum value for chunk [0, 0xe85b): Mirror 1: 0xb42311 Mirror 9: 0xb42311 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe85b: skipped 000775 1788798089.327334 mapread 0x00d6b3 thru 0x00e85a (0x11a8 bytes) 000776 1788798089.371315 punch from 0x00732a to 0x00e85a (0x7531 bytes) 000777 1788798089.445714 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0xe85b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe85b): Mirror 3: 0x524d4835 (stale) Mirror 4: 0x524d4835 (stale) Mirror 5: 0x524d4835 (stale) Mirror 6: 0x524d4835 (stale) Mirror 7: 0x524d4835 (stale) Mirror 8: 0x7a68c529 (stale) Mirror 9: 0xb42311 (stale) CRC-32 checksum value for chunk [0, 0xe85b): Mirror 1: 0x9ecab118 Mirror 10: 0x9ecab118 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe85b: skipped 000778 1788798090.694387 mapread 0x002d5a thru 0x0098cf (0x6b76 bytes) 000779 1788798090.809345 write 0x0092a4 thru 0x0146dc (0xb439 bytes) 000780 1788798090.865397 punch from 0x00e58f to 0x00f2c1 (0xd33 bytes) 000781 1788798090.884577 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x146dd): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x146dd): Mirror 3: 0xd31a1253 (stale) Mirror 4: 0xd31a1253 (stale) Mirror 5: 0x84beff8c (stale) Mirror 6: 0xc9b5ff41 (stale) Mirror 7: 0xc9b5ff41 (stale) Mirror 8: 0x538aa83f (stale) Mirror 9: 0xb56b514b (stale) Mirror 10: 0xd28b5b98 (stale) CRC-32 checksum value for chunk [0, 0x146dd): Mirror 1: 0xf534d775 Mirror 11: 0xf534d775 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x146dd: skipped 000782 1788798092.224247 trunc from 0x000945 to 0x00e4fd (0xdbb9 bytes) 000784 1788798092.291931 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x146dd): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x146dd): Mirror 3: 0xd31a1253 (stale) Mirror 4: 0xd31a1253 (stale) Mirror 5: 0x84beff8c (stale) Mirror 6: 0xc9b5ff41 (stale) Mirror 7: 0xc9b5ff41 (stale) Mirror 8: 0x538aa83f (stale) Mirror 9: 0xb56b514b (stale) Mirror 10: 0xd28b5b98 (stale) Mirror 11: 0xf534d775 (stale) CRC-32 checksum value for chunk [0, 0x146dd): Mirror 1: 0x620c506 Mirror 12: 0x620c506 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x146dd: skipped 000785 1788798093.565069 mapread 0x014487 thru 0x0146dc (0x256 bytes) 000786 1788798093.593666 write 0x0182ff thru 0x026002 (0xdd04 bytes) 000787 1788798093.642450 write 0x018f3e thru 0x01d972 (0x4a35 bytes) 000788 1788798093.645679 punch from 0x01bc5e to 0x01ebd8 (0x2f7b bytes) 000789 1788798093.674318 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x26003): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26003): Mirror 3: 0x9f87b2a6 (stale) Mirror 4: 0x807b8492 (stale) Mirror 5: 0x5073cee0 (stale) Mirror 6: 0x8fd728e7 (stale) Mirror 7: 0xe2aeaea0 (stale) Mirror 8: 0x98a4c7aa (stale) Mirror 9: 0x21c03abb (stale) Mirror 10: 0xaee5d0dd (stale) Mirror 11: 0xa7c4b4a9 (stale) Mirror 12: 0x9d6fb1b7 (stale) CRC-32 checksum value for chunk [0, 0x26003): Mirror 1: 0x6669abdd Mirror 13: 0x6669abdd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26003: skipped 000790 1788798095.197211 trunc from 0x019f44 to 0x01d36d (0x342a bytes) 000791 1788798095.302178 trunc from 0x026003 to 0x001573 (0x24a8f bytes) 000792 1788798095.327725 mapread 0x000979 thru 0x001573 (0xbfb bytes) 000793 1788798095.350945 punch from 0x00004a to 0x001573 (0x152a bytes) 000794 1788798095.356834 trunc from 0x001574 to 0x004f5b (0x39e8 bytes) 000795 1788798095.379362 read 0x0001ce thru 0x004f5b (0x4d8e bytes) 000796 1788798095.395345 mapread 0x0030c3 thru 0x004f5b (0x1e99 bytes) 000797 1788798095.401423 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x4f5c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4f5c): Mirror 3: 0x4164964b (stale) Mirror 4: 0x4164964b (stale) Mirror 5: 0x4164964b (stale) Mirror 6: 0x4164964b (stale) Mirror 7: 0x4164964b (stale) Mirror 8: 0xe18b7aec (stale) Mirror 9: 0xe18b7aec (stale) Mirror 10: 0xe18b7aec (stale) Mirror 11: 0xe18b7aec (stale) Mirror 12: 0x91378acc (stale) Mirror 13: 0x91378acc (stale) CRC-32 checksum value for chunk [0, 0x4f5c): Mirror 1: 0xce549381 Mirror 14: 0xce549381 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4f5c: skipped 000798 1788798096.518652 mapread 0x003003 thru 0x004f5b (0x1f59 bytes) 000799 1788798096.563525 mapwrite 0x034bb7 thru 0x03c07d (0x74c7 bytes) 000801 1788798096.705556 punch from 0x01d7fc to 0x0200f4 (0x28f9 bytes) 000802 1788798096.711590 mapread 0x00b4db thru 0x016ea8 (0xb9ce bytes) 000804 1788798096.745148 fallocate 0x03570d thru 0x03ffff (0xa8f3 bytes) 000805 1788798096.767155 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xdb19da30 (stale) Mirror 4: 0xec35a385 (stale) Mirror 5: 0x5e23a0f8 (stale) Mirror 6: 0x77588505 (stale) Mirror 7: 0x282f1377 (stale) Mirror 8: 0xdfb35e03 (stale) Mirror 9: 0x4dd6456 (stale) Mirror 10: 0x81ac580a (stale) Mirror 11: 0x87f6dc52 (stale) Mirror 12: 0xe6b21db5 (stale) Mirror 13: 0x6bcc2930 (stale) Mirror 14: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa4889f3f Mirror 15: 0xa4889f3f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000806 1788798098.332990 trunc from 0x040000 to 0x027364 (0x18c9b bytes) 000807 1788798098.437687 punch from 0x015fe4 to 0x02455b (0xe578 bytes) 000808 1788798098.450064 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x27365): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27365): Mirror 3: 0xeb3f5b85 (stale) Mirror 4: 0xaa3d2d54 (stale) Mirror 5: 0x41ad6824 (stale) Mirror 6: 0x99aa7cbd (stale) Mirror 7: 0x15fd07ee (stale) Mirror 8: 0x469e19af (stale) Mirror 9: 0x64835e0f (stale) Mirror 10: 0x68d4ecba (stale) Mirror 11: 0x1acac0da (stale) Mirror 12: 0x2626e07a (stale) Mirror 13: 0x92b842d8 (stale) Mirror 14: 0xb71215e4 (stale) Mirror 15: 0xb71215e4 (stale) CRC-32 checksum value for chunk [0, 0x27365): Mirror 1: 0xb71215e4 Mirror 16: 0xb71215e4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27365: skipped 000809 1788798099.734191 mapread 0x010e0f thru 0x01c115 (0xb307 bytes) 000810 1788798099.788291 mapwrite 0x03928f thru 0x03ffff (0x6d71 bytes) 000811 1788798099.948659 fallocate 0x01e20d thru 0x02d411 (0xf205 bytes) 000812 1788798099.959184 write 0x01e250 thru 0x027d1e (0x9acf bytes) 000813 1788798099.982993 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xdb19da30 (stale) Mirror 4: 0xec35a385 (stale) Mirror 5: 0x5e23a0f8 (stale) Mirror 6: 0x77588505 (stale) Mirror 7: 0x282f1377 (stale) Mirror 8: 0xdfb35e03 (stale) Mirror 9: 0x4dd6456 (stale) Mirror 10: 0x81ac580a (stale) Mirror 11: 0x87f6dc52 (stale) Mirror 12: 0xe6b21db5 (stale) Mirror 13: 0x6bcc2930 (stale) Mirror 14: 0xe20eea22 (stale) Mirror 15: 0xa4889f3f (stale) Mirror 16: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa91bd422 Mirror 17: 0xa91bd422 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000814 1788798101.877925 trunc from 0x040000 to 0x006ef6 (0x39109 bytes) 000815 1788798102.018871 mapread 0x000555 thru 0x006ef6 (0x69a2 bytes) 000816 1788798102.049260 fallocate 0x00a194 thru 0x00d8b5 (0x3722 bytes) 000817 1788798102.060262 trunc from 0x001760 to 0x00b57f (0x9e20 bytes) 000818 1788798102.083684 mapread 0x0069d7 thru 0x00d8b5 (0x6edf bytes) 000819 1788798102.094892 fallocate 0x02242a thru 0x0257c2 (0x3399 bytes) 000820 1788798102.103875 trunc from 0x007ea0 to 0x00d8b5 (0x5a16 bytes) 000821 1788798102.111187 fallocate 0x030e92 thru 0x035077 (0x41e6 bytes) 000822 1788798102.117466 trunc from 0x035078 to 0x022e4e (0x12229 bytes) 000823 1788798102.162665 mapwrite 0x03bb9a thru 0x03ffff (0x4466 bytes) 000824 1788798102.242182 trunc from 0x012800 to 0x0201f0 (0xd9f1 bytes) 000825 1788798102.255455 write 0x032950 thru 0x03b0dc (0x878d bytes) 000826 1788798102.276414 read 0x009b03 thru 0x00e0d3 (0x45d1 bytes) 000828 1788798102.291727 write 0x015340 thru 0x020452 (0xb113 bytes) 000829 1788798102.307958 write 0x0028ad thru 0x00714b (0x489f bytes) 000830 1788798102.315249 trunc from 0x040000 to 0x01cec7 (0x23138 bytes) 000831 1788798102.342495 MIRROR_SPLIT mirror 6 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1cec8): Mirror 3: 0x38b7e873 (stale) Mirror 4: 0x38b7e873 (stale) Mirror 5: 0xa225c640 (stale) Mirror 6: 0x8b87b5fd (stale) Mirror 7: 0x2870907d (stale) Mirror 9: 0x99449450 (stale) Mirror 10: 0xa6818295 (stale) Mirror 11: 0xcd667bda (stale) Mirror 12: 0xac1f6711 (stale) Mirror 13: 0x8fd5dfcf (stale) Mirror 14: 0xbaaf2dfb (stale) Mirror 15: 0xbaaf2dfb (stale) Mirror 16: 0xbaaf2dfb (stale) Mirror 17: 0xbaaf2dfb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cec8): Mirror 3: 0x38b7e873 (stale) Mirror 4: 0x38b7e873 (stale) Mirror 5: 0xa225c640 (stale) Mirror 6: 0x8b87b5fd (stale) Mirror 7: 0x2870907d (stale) Mirror 9: 0x99449450 (stale) Mirror 10: 0xa6818295 (stale) Mirror 11: 0xcd667bda (stale) Mirror 12: 0xac1f6711 (stale) Mirror 13: 0x8fd5dfcf (stale) Mirror 14: 0xbaaf2dfb (stale) Mirror 15: 0xbaaf2dfb (stale) Mirror 16: 0xbaaf2dfb (stale) Mirror 17: 0xbaaf2dfb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cec8: skipped 000832 1788798103.408594 fallocate 0x012330 thru 0x014980 (0x2651 bytes) 000833 1788798103.424355 MIRROR_RESYNC 15 mirrors 000834 1788798107.127196 trunc from 0x009134 to 0x00ce85 (0x3d52 bytes) 000836 1788798107.185429 fallocate 0x03be5c thru 0x03ffff (0x41a4 bytes) 000837 1788798107.205697 write 0x0316ec thru 0x03d8f8 (0xc20d bytes) 000838 1788798107.228246 trunc from 0x040000 to 0x022287 (0x1dd78 bytes) 000839 1788798107.269594 punch from 0x0048d0 to 0x0089f8 (0x4129 bytes) 000840 1788798107.276528 fallocate 0x014249 thru 0x0198ea (0x56a2 bytes) 000841 1788798107.284685 read 0x020e9c thru 0x022287 (0x13ec bytes) 000842 1788798107.319399 MIRROR_RESYNC 15 mirrors 000844 1788798111.873011 mapread 0x01de96 thru 0x022287 (0x43f2 bytes) 000846 1788798111.901527 fallocate 0x006609 thru 0x0098d2 (0x32ca bytes) 000847 1788798111.948165 mapread 0x01842a thru 0x022287 (0x9e5e bytes) 000848 1788798111.982295 MIRROR_SPLIT mirror 2 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22288): Mirror 3: 0x4ff1a8c9 (stale) Mirror 5: 0x4ff1a8c9 (stale) Mirror 6: 0x4ff1a8c9 (stale) Mirror 7: 0x4ff1a8c9 (stale) Mirror 9: 0x4ff1a8c9 (stale) Mirror 10: 0x4ff1a8c9 (stale) Mirror 11: 0x4ff1a8c9 (stale) Mirror 12: 0x4ff1a8c9 (stale) Mirror 13: 0x4ff1a8c9 (stale) Mirror 14: 0x4ff1a8c9 (stale) Mirror 15: 0x4ff1a8c9 (stale) Mirror 16: 0x4ff1a8c9 (stale) Mirror 17: 0x4ff1a8c9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22288): Mirror 3: 0x4ff1a8c9 (stale) Mirror 5: 0x4ff1a8c9 (stale) Mirror 6: 0x4ff1a8c9 (stale) Mirror 7: 0x4ff1a8c9 (stale) Mirror 9: 0x4ff1a8c9 (stale) Mirror 10: 0x4ff1a8c9 (stale) Mirror 11: 0x4ff1a8c9 (stale) Mirror 12: 0x4ff1a8c9 (stale) Mirror 13: 0x4ff1a8c9 (stale) Mirror 14: 0x4ff1a8c9 (stale) Mirror 15: 0x4ff1a8c9 (stale) Mirror 16: 0x4ff1a8c9 (stale) Mirror 17: 0x4ff1a8c9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22288: skipped 000849 1788798112.896319 trunc from 0x0115cb to 0x01c28c (0xacc2 bytes) 000851 1788798112.905056 trunc from 0x022288 to 0x0165b9 (0xbcce bytes) 000852 1788798112.931884 punch from 0x0047d9 to 0x00f232 (0xaa5a bytes) 000853 1788798112.942293 punch from 0x00cb99 to 0x014e9a (0x8302 bytes) 000854 1788798112.955116 mapread 0x007d85 thru 0x009eb7 (0x2133 bytes) 000855 1788798112.992059 mapwrite 0x0294c3 thru 0x033308 (0x9e46 bytes) 000858 1788798113.064992 read 0x0089c8 thru 0x015932 (0xcf6b bytes) 000859 1788798113.077798 mapread 0x009192 thru 0x019115 (0xff84 bytes) 000860 1788798113.117830 trunc from 0x033309 to 0x006201 (0x2d107 bytes) 000861 1788798113.149742 mapwrite 0x027c09 thru 0x036bcb (0xefc3 bytes) 000862 1788798113.263577 read 0x01e343 thru 0x026c1f (0x88dd bytes) 000865 1788798113.277114 mapread 0x0097ba thru 0x00f2c0 (0x5b07 bytes) 000866 1788798113.278575 punch from 0x025f45 to 0x02db6b (0x7c27 bytes) 000867 1788798113.286724 read 0x021d97 thru 0x023c18 (0x1e82 bytes) 000868 1788798113.287722 mapread 0x001434 thru 0x00a872 (0x943f bytes) 000869 1788798113.289283 mapwrite 0x004ebe thru 0x0109f1 (0xbb34 bytes) 000870 1788798113.312662 mapwrite 0x021ae7 thru 0x027e28 (0x6342 bytes) 000871 1788798113.348080 MIRROR_SPLIT mirror 5 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36bcc): Mirror 3: 0x6cf4a8f1 (stale) Mirror 5: 0x6cf4a8f1 (stale) Mirror 6: 0x6cf4a8f1 (stale) Mirror 7: 0x6cf4a8f1 (stale) Mirror 10: 0x6cf4a8f1 (stale) Mirror 11: 0x6cf4a8f1 (stale) Mirror 12: 0x6cf4a8f1 (stale) Mirror 13: 0x6cf4a8f1 (stale) Mirror 14: 0x6cf4a8f1 (stale) Mirror 15: 0x6cf4a8f1 (stale) Mirror 16: 0x6cf4a8f1 (stale) Mirror 17: 0x6cf4a8f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36bcc): Mirror 3: 0x6cf4a8f1 (stale) Mirror 5: 0x6cf4a8f1 (stale) Mirror 6: 0x6cf4a8f1 (stale) Mirror 7: 0x6cf4a8f1 (stale) Mirror 10: 0x6cf4a8f1 (stale) Mirror 11: 0x6cf4a8f1 (stale) Mirror 12: 0x6cf4a8f1 (stale) Mirror 13: 0x6cf4a8f1 (stale) Mirror 14: 0x6cf4a8f1 (stale) Mirror 15: 0x6cf4a8f1 (stale) Mirror 16: 0x6cf4a8f1 (stale) Mirror 17: 0x6cf4a8f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36bcc: skipped 000872 1788798113.933707 trunc from 0x036bcc to 0x02966f (0xd55c bytes) 000873 1788798113.955696 trunc from 0x003561 to 0x00fdb5 (0xc855 bytes) 000874 1788798113.961504 read 0x018da3 thru 0x022cb6 (0x9f14 bytes) 000875 1788798113.970737 trunc from 0x029670 to 0x007844 (0x21e2b bytes) 000876 1788798113.995031 mapwrite 0x031121 thru 0x03dcc6 (0xcba6 bytes) 000877 1788798114.055386 trunc from 0x03a2a0 to 0x03dcc6 (0x3a27 bytes) 000878 1788798114.060650 punch from 0x035097 to 0x03dcc6 (0x8c30 bytes) 000879 1788798114.066716 punch from 0x031044 to 0x03dcc6 (0xcc83 bytes) 000880 1788798114.073199 read 0x032ad6 thru 0x03dcc6 (0xb1f1 bytes) 000881 1788798114.089718 trunc from 0x01d69f to 0x02bb1e (0xe480 bytes) 000882 1788798114.095362 write 0x0159d9 thru 0x01897c (0x2fa4 bytes) 000883 1788798114.097018 fallocate 0x032b28 thru 0x03f701 (0xcbda bytes) 000884 1788798114.102098 write 0x00ec34 thru 0x015dc0 (0x718d bytes) 000885 1788798114.104217 read 0x02cd35 thru 0x02f892 (0x2b5e bytes) 000886 1788798114.105038 fallocate 0x03f683 thru 0x03ffff (0x97d bytes) 000887 1788798114.110072 punch from 0x01b987 to 0x025239 (0x98b3 bytes) 000888 1788798114.119044 MIRROR_RESYNC 13 mirrors 000889 1788798118.338180 write 0x0390ce thru 0x03dbf1 (0x4b24 bytes) 000890 1788798118.401053 trunc from 0x0395cf to 0x03dcc6 (0x46f8 bytes) 000891 1788798118.425636 trunc from 0x03dcc7 to 0x029b61 (0x14165 bytes) 000892 1788798118.450607 fallocate 0x01eeb3 thru 0x02c8db (0xda29 bytes) 000893 1788798118.457683 mapwrite 0x018be3 thru 0x025569 (0xc987 bytes) 000894 1788798118.518285 read 0x01351f thru 0x01e0c2 (0xaba4 bytes) 000895 1788798118.526635 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x29b62): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29b62): Mirror 3: 0xefd75fcb (stale) Mirror 5: 0xefd75fcb (stale) Mirror 6: 0xefd75fcb (stale) Mirror 7: 0xefd75fcb (stale) Mirror 10: 0xefd75fcb (stale) Mirror 11: 0xefd75fcb (stale) Mirror 12: 0xefd75fcb (stale) Mirror 13: 0xefd75fcb (stale) Mirror 14: 0xefd75fcb (stale) Mirror 15: 0xefd75fcb (stale) Mirror 16: 0xefd75fcb (stale) Mirror 17: 0xefd75fcb (stale) CRC-32 checksum value for chunk [0, 0x29b62): Mirror 1: 0x6103209d Mirror 18: 0x6103209d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29b62: skipped 000896 1788798120.267566 MIRROR_SPLIT mirror 5 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29b62): Mirror 3: 0xefd75fcb (stale) Mirror 5: 0xefd75fcb (stale) Mirror 6: 0xefd75fcb (stale) Mirror 7: 0xefd75fcb (stale) Mirror 11: 0xefd75fcb (stale) Mirror 12: 0xefd75fcb (stale) Mirror 13: 0xefd75fcb (stale) Mirror 14: 0xefd75fcb (stale) Mirror 15: 0xefd75fcb (stale) Mirror 16: 0xefd75fcb (stale) Mirror 17: 0xefd75fcb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29b62): Mirror 3: 0xefd75fcb (stale) Mirror 5: 0xefd75fcb (stale) Mirror 6: 0xefd75fcb (stale) Mirror 7: 0xefd75fcb (stale) Mirror 11: 0xefd75fcb (stale) Mirror 12: 0xefd75fcb (stale) Mirror 13: 0xefd75fcb (stale) Mirror 14: 0xefd75fcb (stale) Mirror 15: 0xefd75fcb (stale) Mirror 16: 0xefd75fcb (stale) Mirror 17: 0xefd75fcb (stale) CRC-32 checksum value for chunk [0, 0x29b62): Mirror 1: 0x6103209d Mirror 18: 0x6103209d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29b62: skipped 000898 1788798121.638291 fallocate 0x03e1a0 thru 0x03ffff (0x1e60 bytes) 000899 1788798121.747300 write 0x02e9d5 thru 0x0331ac (0x47d8 bytes) 000900 1788798121.776119 trunc from 0x0331ad to 0x0027c1 (0x309eb bytes) 000901 1788798121.826509 trunc from 0x000944 to 0x0027c1 (0x1e7e bytes) 000902 1788798121.839505 trunc from 0x0027c2 to 0x0219e6 (0x1f225 bytes) 000903 1788798121.867092 punch from 0x001efe to 0x00361e (0x1721 bytes) 000904 1788798121.877651 mapwrite 0x00bdae thru 0x00e2ea (0x253d bytes) 000905 1788798121.948841 mapwrite 0x00a63f thru 0x017d42 (0xd704 bytes) 000906 1788798122.010118 mapread 0x009a8f thru 0x012d77 (0x92e9 bytes) 000907 1788798122.013727 mapwrite 0x00eef3 thru 0x017956 (0x8a64 bytes) 000908 1788798122.094249 punch from 0x009aa4 to 0x00f975 (0x5ed2 bytes) 000909 1788798122.099318 trunc from 0x0219e7 to 0x009537 (0x184af bytes) 000910 1788798122.126551 mapwrite 0x00b04c thru 0x01626f (0xb224 bytes) 000911 1788798122.214686 mapwrite 0x015e92 thru 0x0214c8 (0xb637 bytes) 000912 1788798122.267182 write 0x011ca1 thru 0x0141de (0x253e bytes) 000913 1788798122.268721 mapwrite 0x034fd5 thru 0x03ffff (0xb02b bytes) 000916 1788798122.327073 trunc from 0x040000 to 0x011879 (0x2e786 bytes) 000917 1788798122.350155 trunc from 0x003715 to 0x011879 (0xe165 bytes) 000918 1788798122.357153 read 0x00980d thru 0x00f3b0 (0x5ba4 bytes) 000919 1788798122.367345 mapwrite 0x01db01 thru 0x02572c (0x7c2c bytes) 000920 1788798122.410561 trunc from 0x02572d to 0x0169ca (0xed62 bytes) 000921 1788798122.431092 punch from 0x007c05 to 0x00f766 (0x7b62 bytes) 000922 1788798122.436809 fallocate 0x035d10 thru 0x038cc1 (0x2fb2 bytes) 000923 1788798122.441538 mapwrite 0x03f5ae thru 0x03ffff (0xa52 bytes) 000924 1788798122.480836 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1d6f2fa0 (stale) Mirror 5: 0x1d6f2fa0 (stale) Mirror 6: 0x1d6f2fa0 (stale) Mirror 7: 0x1d6f2fa0 (stale) Mirror 11: 0x1d6f2fa0 (stale) Mirror 12: 0x1d6f2fa0 (stale) Mirror 13: 0x1d6f2fa0 (stale) Mirror 14: 0x1d6f2fa0 (stale) Mirror 15: 0x1d6f2fa0 (stale) Mirror 16: 0x1d6f2fa0 (stale) Mirror 17: 0x1d6f2fa0 (stale) Mirror 18: 0xcc94e8e5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc80f2a29 Mirror 19: 0xc80f2a29 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000925 1788798123.674734 read 0x013501 thru 0x02200d (0xeb0d bytes) 000926 1788798123.689269 mapread 0x035209 thru 0x03ffff (0xadf7 bytes) 000927 1788798123.709243 write 0x000dce thru 0x005235 (0x4468 bytes) 000928 1788798123.772511 trunc from 0x01078f to 0x01ab75 (0xa3e7 bytes) 000929 1788798123.777935 mapread 0x030882 thru 0x03cbc8 (0xc347 bytes) 000930 1788798123.795048 mapread 0x032356 thru 0x032a27 (0x6d2 bytes) 000931 1788798123.800515 MIRROR_SPLIT mirror 12 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1d6f2fa0 (stale) Mirror 5: 0x1d6f2fa0 (stale) Mirror 6: 0x1d6f2fa0 (stale) Mirror 7: 0x1d6f2fa0 (stale) Mirror 11: 0x1d6f2fa0 (stale) Mirror 12: 0x1d6f2fa0 (stale) Mirror 13: 0x1d6f2fa0 (stale) Mirror 14: 0x1d6f2fa0 (stale) Mirror 15: 0x1d6f2fa0 (stale) Mirror 16: 0x1d6f2fa0 (stale) Mirror 17: 0x1d6f2fa0 (stale) Mirror 19: 0xc80f2a29 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1d6f2fa0 (stale) Mirror 5: 0x1d6f2fa0 (stale) Mirror 6: 0x1d6f2fa0 (stale) Mirror 7: 0x1d6f2fa0 (stale) Mirror 11: 0x1d6f2fa0 (stale) Mirror 12: 0x1d6f2fa0 (stale) Mirror 13: 0x1d6f2fa0 (stale) Mirror 14: 0x1d6f2fa0 (stale) Mirror 15: 0x1d6f2fa0 (stale) Mirror 16: 0x1d6f2fa0 (stale) Mirror 17: 0x1d6f2fa0 (stale) Mirror 19: 0xc80f2a29 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000933 1788798124.801950 read 0x018d44 thru 0x020fe9 (0x82a6 bytes) 000934 1788798124.821433 read 0x0383f5 thru 0x03ffff (0x7c0b bytes) 000935 1788798124.849941 mapread 0x00ae20 thru 0x013b78 (0x8d59 bytes) 000936 1788798124.857009 mapread 0x0076ae thru 0x012b11 (0xb464 bytes) 000937 1788798124.860700 trunc from 0x02cd89 to 0x0321c6 (0x543e bytes) 000938 1788798124.881404 MIRROR_SPLIT mirror 7 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1d6f2fa0 (stale) Mirror 5: 0x1d6f2fa0 (stale) Mirror 6: 0x1d6f2fa0 (stale) Mirror 7: 0x1d6f2fa0 (stale) Mirror 11: 0x1d6f2fa0 (stale) Mirror 12: 0x1d6f2fa0 (stale) Mirror 14: 0x1d6f2fa0 (stale) Mirror 15: 0x1d6f2fa0 (stale) Mirror 16: 0x1d6f2fa0 (stale) Mirror 17: 0x1d6f2fa0 (stale) Mirror 19: 0xc80f2a29 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1d6f2fa0 (stale) Mirror 5: 0x1d6f2fa0 (stale) Mirror 6: 0x1d6f2fa0 (stale) Mirror 7: 0x1d6f2fa0 (stale) Mirror 11: 0x1d6f2fa0 (stale) Mirror 12: 0x1d6f2fa0 (stale) Mirror 14: 0x1d6f2fa0 (stale) Mirror 15: 0x1d6f2fa0 (stale) Mirror 16: 0x1d6f2fa0 (stale) Mirror 17: 0x1d6f2fa0 (stale) Mirror 19: 0xc80f2a29 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000939 1788798125.662361 read 0x00fff3 thru 0x01ea07 (0xea15 bytes) 000940 1788798125.687404 read 0x00187d thru 0x0033c7 (0x1b4b bytes) 000941 1788798125.702371 write 0x01fb8c thru 0x0250ac (0x5521 bytes) 000942 1788798125.714981 MIRROR_RESYNC 12 mirrors 000943 1788798129.228302 punch from 0x02691c to 0x034ab3 (0xe198 bytes) 000944 1788798129.275223 trunc from 0x040000 to 0x0219c3 (0x1e63c bytes) 000945 1788798129.317275 read 0x003cab thru 0x00cbf5 (0x8f4b bytes) 000946 1788798129.361876 trunc from 0x0219c4 to 0x02ead7 (0xd114 bytes) 000947 1788798129.393162 trunc from 0x0037c3 to 0x00f314 (0xbb52 bytes) 000948 1788798129.399795 fallocate 0x02b07c thru 0x02f527 (0x44ac bytes) 000949 1788798129.419104 MIRROR_RESYNC 12 mirrors 000950 1788798131.951349 read 0x0150cc thru 0x01f564 (0xa499 bytes) 000951 1788798131.969081 fallocate 0x007041 thru 0x0077ef (0x7af bytes) 000952 1788798132.007651 mapread 0x02b9ec thru 0x02ead7 (0x30ec bytes) 000953 1788798132.027556 write 0x024f8b thru 0x02b041 (0x60b7 bytes) 000954 1788798132.035932 trunc from 0x02ead8 to 0x03d847 (0xed70 bytes) 000955 1788798132.070625 fallocate 0x001d50 thru 0x0080dc (0x638d bytes) 000956 1788798132.081449 read 0x01e046 thru 0x023a4d (0x5a08 bytes) 000957 1788798132.095690 read 0x02c553 thru 0x035abd (0x956b bytes) 000958 1788798132.098028 fallocate 0x01b5f6 thru 0x0288dd (0xd2e8 bytes) 000959 1788798132.107229 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3d848): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d848): Mirror 3: 0x842bf3aa (stale) Mirror 5: 0x842bf3aa (stale) Mirror 6: 0x842bf3aa (stale) Mirror 7: 0x842bf3aa (stale) Mirror 11: 0x842bf3aa (stale) Mirror 12: 0x842bf3aa (stale) Mirror 14: 0x842bf3aa (stale) Mirror 15: 0x842bf3aa (stale) Mirror 16: 0x842bf3aa (stale) Mirror 17: 0x842bf3aa (stale) Mirror 19: 0x842bf3aa (stale) CRC-32 checksum value for chunk [0, 0x3d848): Mirror 1: 0xbe317db6 Mirror 20: 0xbe317db6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d848: skipped 000960 1788798133.559683 mapwrite 0x01ea9c thru 0x02d3fb (0xe960 bytes) 000961 1788798133.770670 write 0x00c409 thru 0x018dcd (0xc9c5 bytes) 000962 1788798133.797673 punch from 0x0246a8 to 0x029a69 (0x53c2 bytes) 000963 1788798133.805531 write 0x03b950 thru 0x03ffff (0x46b0 bytes) 000964 1788798133.826931 mapwrite 0x01236e thru 0x0222da (0xff6d bytes) 000965 1788798133.899593 fallocate 0x02792e thru 0x02a907 (0x2fda bytes) 000966 1788798133.910840 read 0x008a8d thru 0x009f66 (0x14da bytes) 000967 1788798133.936741 MIRROR_SPLIT mirror 12 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x76f52021 (stale) Mirror 5: 0x76f52021 (stale) Mirror 6: 0x76f52021 (stale) Mirror 7: 0x76f52021 (stale) Mirror 11: 0x76f52021 (stale) Mirror 12: 0x76f52021 (stale) Mirror 14: 0x76f52021 (stale) Mirror 15: 0x76f52021 (stale) Mirror 16: 0x76f52021 (stale) Mirror 17: 0x76f52021 (stale) Mirror 19: 0x76f52021 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x76f52021 (stale) Mirror 5: 0x76f52021 (stale) Mirror 6: 0x76f52021 (stale) Mirror 7: 0x76f52021 (stale) Mirror 11: 0x76f52021 (stale) Mirror 12: 0x76f52021 (stale) Mirror 14: 0x76f52021 (stale) Mirror 15: 0x76f52021 (stale) Mirror 16: 0x76f52021 (stale) Mirror 17: 0x76f52021 (stale) Mirror 19: 0x76f52021 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000968 1788798134.731823 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x76f52021 (stale) Mirror 5: 0x76f52021 (stale) Mirror 6: 0x76f52021 (stale) Mirror 7: 0x76f52021 (stale) Mirror 11: 0x76f52021 (stale) Mirror 12: 0x76f52021 (stale) Mirror 14: 0x76f52021 (stale) Mirror 15: 0x76f52021 (stale) Mirror 16: 0x76f52021 (stale) Mirror 17: 0x76f52021 (stale) Mirror 19: 0x76f52021 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe159fa63 Mirror 20: 0xe159fa63 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000969 1788798136.186364 mapread 0x03d5fd thru 0x03ffff (0x2a03 bytes) 000970 1788798136.228372 mapread 0x0243ce thru 0x02ea6f (0xa6a2 bytes) 000971 1788798136.282018 read 0x0059b1 thru 0x00c11a (0x676a bytes) 000972 1788798136.301449 mapread 0x00af1e thru 0x011b73 (0x6c56 bytes) 000973 1788798136.329951 punch from 0x00b780 to 0x019c96 (0xe517 bytes) 000974 1788798136.453638 mapread 0x02ea71 thru 0x02f658 (0xbe8 bytes) 000975 1788798136.465024 fallocate 0x0170f9 thru 0x017147 (0x4f bytes) 000976 1788798136.500941 write 0x0248e2 thru 0x02ca35 (0x8154 bytes) 000977 1788798136.515162 trunc from 0x040000 to 0x00a338 (0x35cc7 bytes) 000978 1788798136.541395 read 0x001c52 thru 0x003b5d (0x1f0c bytes) 000979 1788798136.553414 fallocate 0x029c1f thru 0x035523 (0xb905 bytes) 000980 1788798136.565089 trunc from 0x00a339 to 0x032741 (0x28409 bytes) 000981 1788798136.607126 MIRROR_RESYNC 13 mirrors 000983 1788798138.827560 MIRROR_EXTEND to 14 mirrors 000984 1788798139.728409 punch from 0x004d85 to 0x01115f (0xc3db bytes) 000985 1788798139.770233 punch from 0x01fc3d to 0x022cc6 (0x308a bytes) 000986 1788798139.777507 mapread 0x031335 thru 0x032741 (0x140d bytes) 000987 1788798139.795923 write 0x002aba thru 0x00655d (0x3aa4 bytes) 000988 1788798139.807599 punch from 0x026a7e to 0x032741 (0xbcc4 bytes) 000989 1788798139.813286 punch from 0x02e22d to 0x032741 (0x4515 bytes) 000992 1788798139.819000 mapread 0x02a934 thru 0x032268 (0x7935 bytes) 000993 1788798139.828605 punch from 0x0077aa to 0x01536f (0xdbc6 bytes) 000995 1788798139.833926 punch from 0x01361b to 0x01caaf (0x9495 bytes) 000996 1788798139.838504 mapwrite 0x031883 thru 0x03f333 (0xdab1 bytes) 000997 1788798139.947171 read 0x03d911 thru 0x03f333 (0x1a23 bytes) 000998 1788798139.947828 mapwrite 0x00a0ff thru 0x017819 (0xd71b bytes) 001000 1788798140.005597 punch from 0x02c008 to 0x02efd0 (0x2fc9 bytes) All operations completed A-OK! PASS 70a (127s) == sanity-flr test 71: check mirror extend parent fid ==== 12:22:25 (1788798145) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1ca:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (10s) == sanity-flr test 72: check append on FLR file ========== 12:22:35 (1788798155) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.273968 s, 7.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129547 s, 8.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 12:22:43 (1788798163) Starting client: oleg650-client.virtnet: -o user_xattr,flock 192.168.206.150@tcp:/lustre /mnt/lustre2 Starting client: oleg650-client.virtnet: -o user_xattr,flock 192.168.206.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 870016 .. Extending file size to 4545728 .. failed Extending file size to 1231904 .. Extending file size to 2998912 .. Extending file size to 6378880 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6056160 .. Extending file size to 7581376 .. Extending file size to 8136544 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8705792 .. Extending file size to 8771648 .. Extending file size to 8739680 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9218368 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8813216 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61449 61451 61452 61454 61455 done 192.168.206.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg650-client.virtnet /mnt/lustre2 (opts:) 192.168.206.150@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg650-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (84s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 12:24:07 (1788798247) Starting client: oleg650-client.virtnet: -o user_xattr,flock 192.168.206.150@tcp:/lustre /mnt/lustre2 Starting client: oleg650-client.virtnet: -o user_xattr,flock 192.168.206.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6235808 .. Extending file size to 5415936 .. done Extending file size to 6472064 .. Extending file size to 5918016 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8492640 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8766400 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8756512 .. Extending file size to 8817248 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8901568 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8817568 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9071808 .. Waiting 63111 63113 63114 63116 63118 failed 192.168.206.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg650-client.virtnet /mnt/lustre2 (opts:) 192.168.206.150@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg650-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (76s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 12:25:23 (1788798323) Starting client: oleg650-client.virtnet: -o user_xattr,flock 192.168.206.150@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2460 1268844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2096 1269208 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17780 3589240 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25556 3581464 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43336 7170704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2460 1268844 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2096 1269208 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17780 3589240 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25556 3581464 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43336 7170704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0896171 s, 23.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.221356 s, 9.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x17ad5a62 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg650-client.virtnet /mnt/lustre2 (opts:) PASS 200c (15s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 12:25:40 (1788798340) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0660538 s, 31.7 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 12:25:46 (1788798346) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x7b:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14a:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14a:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14b:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12b:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 12:25:54 (1788798354) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.170146 s, 12.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 12:26:02 (1788798362) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.12571 s, 16.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23b:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 12:26:10 (1788798370) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.129581 s, 16.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23d:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 12:26:44 (1788798404) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg650-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936909 s, 11.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23f:0x0] } PASS 204d (48s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 12:27:32 (1788798452) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg650-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.125059 s, 16.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 12:28:18 (1788798498) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg650-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg650-server: Pool lustre.test_204f created Adding targets to pool oleg650-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg650-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.170773 s, 12.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x240:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x240:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x240:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x240:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg650-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg650-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg650-server: Pool lustre.test_204f destroyed PASS 204f (65s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 12:29:24 (1788798564) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 12:29:30 (1788798570) PASS 205b (5s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 12:29:35 (1788798575) oleg650-server: Pool lustre.test_206 created Adding targets to pool oleg650-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg650-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg650-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg650-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg650-server: Pool lustre.test_206 destroyed PASS 206 (24s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 12:29:59 (1788798599) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540084 s, 19.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665091 s, 15.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xabcd79d2 Mirror 3: 0xabcd79d2 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 12:30:09 (1788798609) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 12:30:12 (1788798612) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 12:30:15 (1788798615) Starting client: oleg650-client.virtnet: -o user_xattr,flock 192.168.206.150@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0459028 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x13e0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x860000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x860000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x860000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1890000 Read this (4 bytes): BBBB 192.168.206.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg650-client.virtnet /mnt/lustre2 (opts:) PASS 209a (31s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 12:30:46 (1788798646) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.016731 s, 490 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 12:30:53 (1788798653) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689233 s, 15.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x98 lmm_fid: [0x200000405:0x98:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x98 lmm_fid: [0x200000405:0x98:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x158:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611288 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 12:30:54.000000000 -0400 Modify: 2026-09-07 12:30:54.000000000 -0400 Change: 2026-09-07 12:30:56.000000000 -0400 Birth: 2026-09-07 12:30:55.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 12:31:01 (1788798661) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357822 s, 29.3 MB/s oleg650-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg650-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg650-server: O/280000401/d25/159: File not found by ext2_lookup PASS 210b (22s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 12:31:22 (1788798682) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.243182 s, 168 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0286164 s, 143 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 7 12:31 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 7 12:31 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 12:31:29 (1788798689) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 12:31:36 (1788798696) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1630 sec =========== 12:31:44 (1788798704) === sanity-flr: start cleanup 12:31:46 (1788798706) === === sanity-flr: finish cleanup 12:31:54 (1788798714) ===