-----============= acceptance-small: sanity-flr ============----- Wed Jul 15 10:00:17 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg323-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 10:00:26 (1784124026) === oleg323-client.virtnet: executing check_config_client /mnt/lustre oleg323-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg323-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8ad188f33800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8ad188f33800.idle_timeout=debug disable quota as required oleg323-server: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 10:00:42 (1784124042) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8208] == sanity-flr test 0a: lfs mirror create with -N option == 10:00:43 (1784124043) PASS 0a (8s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:00:51 (1784124051) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:00:53 (1784124053) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 10:00:55 (1784124055) PASS 0d (11s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:01:06 (1784124066) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:01:07 (1784124067) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 10:01:09 (1784124069) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8ad188f33800.stats=clear osc.lustre-OST0001-osc-ffff8ad188f33800.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:01:13 (1784124073) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (5s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:01:18 (1784124078) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:01:22 (1784124082) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 10:01:24 (1784124084) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:01:28 (1784124088) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 10:01:32 (1784124092) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:01:36 (1784124096) PASS 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:01:39 (1784124099) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:01:43 (1784124103) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:01:48 (1784124108) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:01:51 (1784124111) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 0.759736 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048922 s, 21.4 MB/s PASS 21 (5s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:01:56 (1784124116) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 0.258642 s, 56.8 MB/s PASS 22 (4s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 10:02:00 (1784124120) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (6s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:02:06 (1784124126) Stopping /mnt/lustre-ost1 (opts:) on oleg323-server oleg323-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ad188f33800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ad188f33800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg323-server: oleg323-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg323-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg323-server oleg323-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ad188f33800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ad188f33800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg323-server: oleg323-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg323-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (38s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:02:44 (1784124164) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0940527 s, 44.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg323-server oleg323-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ad188f33800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ad188f33800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg323-server: oleg323-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg323-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (19s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:03:03 (1784124183) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg323-server oleg323-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ad188f33800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ad188f33800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg323-server: oleg323-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg323-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg323-server oleg323-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ad188f33800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ad188f33800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg323-server: oleg323-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg323-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (32s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:03:36 (1784124216) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.22118 s, 42.9 MB/s -rw-r--r-- 1 root root 52428800 Jul 15 10:03 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.381s user 0m0.002s sys 0m0.164s Stopping /mnt/lustre-ost1 (opts:) on oleg323-server oleg323-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ad188f33800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ad188f33800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.505s user 0m0.002s sys 0m0.172s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg323-server: oleg323-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg323-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg323-server oleg323-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ad188f33800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ad188f33800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.443s user 0m0.000s sys 0m0.158s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg323-server: oleg323-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg323-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg323-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ad188f33800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ad188f33800.ost_server_uuid in FULL state after 0 sec PASS 33b (42s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:04:18 (1784124258) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:04:19 (1784124259) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:04:21 (1784124261) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 10:04:23 (1784124263) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 10:04:27 (1784124267) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.241208 s, 52.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (6s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:04:32 (1784124272) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136768 s, 76.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141932 s, 73.9 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:04:35 (1784124275) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:04:39 (1784124279) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000769674 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000687431 s, 1.5 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000746493 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000757538 s, 1.4 MB/s flv=10 olv=10 PASS 36d (5s) == sanity-flr test 37: mirror I/O API verification ======= 10:04:44 (1784124284) create test files with size 15735738 .. 93ea8079411489716bdec594ad07e5ae - b4b27667f4d1227e19684e06e80fb7d3 - 5874f7ef46ad6cf45f756a969dc6ee61 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg323-client.virtnet /mnt/lustre (opts:) Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre PASS 37 (7s) == sanity-flr test 38: resync ============================ 10:04:51 (1784124291) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (5s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:04:56 (1784124296) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:04:59 (1784124299) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380591 s, 27.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185564 s, 56.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 10:05:02 (1784124302) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0393736 s, 53.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.133531 s, 31.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0564761 s, 37.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (6s) == sanity-flr test 42: lfs mirror verify ================= 10:05:08 (1784124308) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 10:05:09 (1784124309) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:05:10 (1784124310) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016172 s, 64.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 10:05:14 (1784124314) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0992179 s, 31.7 MB/s PASS 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:05:17 (1784124317) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:05:21 (1784124321) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.119033 s, 88.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg323-client.virtnet /mnt/lustre (opts:) Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (15s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:05:37 (1784124337) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8ad18929c000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.120649 s, 86.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg323-client.virtnet /mnt/lustre (opts:) Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8ad189556000.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:05:42 (1784124342) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (15s) == sanity-flr test 46: Verify setstripe --copy option ==== 10:05:57 (1784124357) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (15s) == sanity-flr test 47: Verify mirror obj alloc =========== 10:06:12 (1784124372) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 10:06:13 (1784124373) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.105761 s, 29.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0717415 s, 43.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x94a9234c (nosync,stale) Mirror 2: 0x94a9234c (nosync,stale) ** verify mirror 2 doesn't change original checksum: 3ede88789ca75aade0bd5b5bfba6a8a8 - mirror 1 checksum: 152865efcc046cc3eb6b0308c3cec67f - mirror 2 checksum: 3ede88789ca75aade0bd5b5bfba6a8a8 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-15T10:06:13-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-15T10:06:13-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 3ede88789ca75aade0bd5b5bfba6a8a8 - mirror 1 checksum: 152865efcc046cc3eb6b0308c3cec67f - mirror 2 checksum: 152865efcc046cc3eb6b0308c3cec67f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-15T10:06:13-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-15T10:06:13-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbd:0x0] } PASS 48 (4s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 10:06:17 (1784124377) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169039 s, 62.0 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (4s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:06:21 (1784124381) Failing mds1 on oleg323-server Stopping /mnt/lustre-mds1 (opts:) on oleg323-server 10:06:30 (1784124390) shut down facet: mds1 facet_host: oleg323-server facet_failover_host: oleg323-server Failover mds1 to oleg323-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg323-server: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-MDT0000 10:06:45 (1784124405) targets are mounted 10:06:45 (1784124405) facet_failover done oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159603 s, 65.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbf:0x0] } PASS 50A (34s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 10:06:55 (1784124415) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0183139 s, 1.1 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00678077 s, 3.0 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 10:06:59 (1784124419) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0176318 s, 1.2 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00775453 s, 2.6 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:07:03 (1784124423) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00656569 s, 2.5 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 10:07:06 (1784124426) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0215852 s, 949 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:07:09 (1784124429) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00383403 s, 1.1 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0179572 s, 228 kB/s PASS 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:07:12 (1784124432) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (9s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:07:21 (1784124441) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (21s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 10:07:42 (1784124462) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x47:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 10:08:04 (1784124484) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 10:08:32 (1784124512) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 10:08:41 (1784124521) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 57465 mirror split pid 57467 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 57465 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 57467 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (65s) == sanity-flr test 70a: flr mode fsx test ================ 10:09:46 (1784124586) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 4587 000001 1784124587.634532 mapwrite 0x03bf2b thru 0x03ffff (0x40d5 bytes) 000002 1784124587.677786 mapwrite 0x002f8c thru 0x0087f1 (0x5866 bytes) 000003 1784124587.699048 mapread 0x03a796 thru 0x03d9b1 (0x321c bytes) 000004 1784124587.700430 read 0x013293 thru 0x01e0de (0xae4c bytes) 000005 1784124587.709110 mapread 0x03d50a thru 0x03ffff (0x2af6 bytes) 000006 1784124587.710976 punch from 0x008a2d to 0x00cf38 (0x450c bytes) 000007 1784124587.715725 mapread 0x023f84 thru 0x02ce57 (0x8ed4 bytes) zero_range to largest ever: 0x17ec9 000008 1784124587.736567 trunc from 0x0086b3 to 0x017ec8 (0xf816 bytes) 000009 1784124587.742446 write 0x001471 thru 0x00cd50 (0xb8e0 bytes) fallocating to largest ever: 0x3a551 000010 1784124587.750459 fallocate 0x0309f2 thru 0x03a550 (0x9b5f bytes) 000012 1784124587.756042 trunc from 0x012829 to 0x021d49 (0xf521 bytes) 000013 1784124587.761136 punch from 0x02062e to 0x021b84 (0x1557 bytes) 000014 1784124587.769548 trunc from 0x040000 to 0x031d9c (0xe263 bytes) 000015 1784124587.789622 trunc from 0x031d9d to 0x00fd54 (0x22048 bytes) 000017 1784124587.813826 trunc from 0x00fd55 to 0x037246 (0x274f2 bytes) 000018 1784124587.826927 write 0x0077f8 thru 0x00b128 (0x3931 bytes) 000019 1784124587.828303 punch from 0x025352 to 0x033403 (0xe0b2 bytes) 000020 1784124587.832653 read 0x01700d thru 0x021ffc (0xaff0 bytes) 000022 1784124587.845655 punch from 0x008af7 to 0x009a41 (0xf4b bytes) fallocating to largest ever: 0x40000 000023 1784124587.857803 fallocate 0x03dfaa thru 0x03ffff (0x2056 bytes) 000024 1784124587.862102 trunc from 0x040000 to 0x009ac4 (0x3653b bytes) 000026 1784124587.876875 write 0x002050 thru 0x003b28 (0x1ad9 bytes) 000029 1784124587.880427 mapwrite 0x03c919 thru 0x03ffff (0x36e7 bytes) 000030 1784124587.928560 MIRROR_EXTEND to 2 mirrors 000031 1784124588.429919 trunc from 0x0180eb to 0x01cfc0 (0x4ed6 bytes) 000032 1784124588.462402 read 0x00b655 thru 0x0192c5 (0xdc71 bytes) 000033 1784124588.468035 trunc from 0x040000 to 0x017e14 (0x281eb bytes) 000034 1784124588.482645 trunc from 0x017e15 to 0x01f487 (0x7673 bytes) 000035 1784124588.498462 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1f488): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f488): Mirror 2: 0xd70f9853 (stale) CRC-32 checksum value for chunk [0, 0x1f488): Mirror 1: 0xd70f9853 Mirror 3: 0xd70f9853 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f488: skipped 000036 1784124588.983700 fallocate 0x034022 thru 0x03dc59 (0x9c38 bytes) 000037 1784124589.019595 MIRROR_RESYNC 3 mirrors 000038 1784124589.589903 mapread 0x02d803 thru 0x030566 (0x2d64 bytes) 000039 1784124589.624549 punch from 0x02ce0f to 0x03c334 (0xf526 bytes) 000040 1784124589.670116 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x217:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dc5a): Mirror 3: 0x852dc076 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc5a): Mirror 3: 0x852dc076 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc5a: skipped 000041 1784124589.903901 mapwrite 0x01cd51 thru 0x0203bf (0x366f bytes) 000042 1784124589.934449 fallocate 0x02a79c thru 0x030c27 (0x648c bytes) 000043 1784124589.940924 trunc from 0x03dc5a to 0x009ed9 (0x33d80 bytes) 000044 1784124589.963356 write 0x01618a thru 0x01d794 (0x760b bytes) 000045 1784124589.965297 trunc from 0x000483 to 0x00cc10 (0xc78e bytes) 000046 1784124589.970450 mapread 0x01ae23 thru 0x01d794 (0x2972 bytes) 000047 1784124589.971522 write 0x027b1a thru 0x02a1c4 (0x26ab bytes) 000048 1784124589.972689 fallocate 0x021810 thru 0x0236bb (0x1eac bytes) 000049 1784124589.977200 write 0x00c371 thru 0x00e7c5 (0x2455 bytes) 000050 1784124589.985889 write 0x02e96a thru 0x0318f0 (0x2f87 bytes) 000051 1784124589.987422 trunc from 0x0318f1 to 0x009ddb (0x27b15 bytes) 000052 1784124590.004620 trunc from 0x003ef4 to 0x009ddb (0x5ee8 bytes) 000053 1784124590.010434 mapread 0x006ed0 thru 0x007630 (0x761 bytes) 000055 1784124590.022661 trunc from 0x00242b to 0x009ddb (0x79b1 bytes) 000056 1784124590.026731 mapwrite 0x01d9ed thru 0x02c394 (0xe9a8 bytes) 000057 1784124590.082083 punch from 0x02265f to 0x02c394 (0x9d36 bytes) 000058 1784124590.089109 trunc from 0x02c395 to 0x012c08 (0x1978c bytes) 000059 1784124590.114435 mapwrite 0x00dc16 thru 0x016453 (0x883e bytes) 000060 1784124590.156165 mapread 0x000e42 thru 0x00f7d5 (0xe994 bytes) 000061 1784124590.165365 punch from 0x005cc7 to 0x00e198 (0x84d2 bytes) 000062 1784124590.170072 write 0x012927 thru 0x01b7e7 (0x8ec1 bytes) 000063 1784124590.172319 fallocate 0x021f85 thru 0x02b052 (0x90ce bytes) 000064 1784124590.181718 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1b7e8): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b7e8): Mirror 3: 0xba71a051 (stale) CRC-32 checksum value for chunk [0, 0x1b7e8): Mirror 1: 0xf5c8baa1 Mirror 4: 0xf5c8baa1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b7e8: skipped 000065 1784124590.703621 fallocate 0x0283be thru 0x032305 (0x9f48 bytes) 000066 1784124590.731919 write 0x023016 thru 0x030685 (0xd670 bytes) 000067 1784124590.740342 trunc from 0x030686 to 0x019fb0 (0x166d5 bytes) 000068 1784124590.775310 trunc from 0x009ad9 to 0x013dfd (0xa325 bytes) 000070 1784124590.779919 fallocate 0x01f3f7 thru 0x026979 (0x7583 bytes) 000071 1784124590.788216 fallocate 0x00577d thru 0x00e2f2 (0x8b76 bytes) 000072 1784124590.791925 trunc from 0x02697a to 0x0158ca (0x110af bytes) 000073 1784124590.815582 mapread 0x00eb35 thru 0x0158ca (0x6d96 bytes) 000074 1784124590.837367 read 0x0026ab thru 0x007f46 (0x589c bytes) 000075 1784124590.845992 punch from 0x0038e3 to 0x00c268 (0x8986 bytes) 000076 1784124590.849277 read 0x006aba thru 0x0145c3 (0xdb0a bytes) 000077 1784124590.860001 write 0x027078 thru 0x02b373 (0x42fc bytes) 000078 1784124590.861601 mapwrite 0x00340d thru 0x008f6f (0x5b63 bytes) 000079 1784124590.881730 trunc from 0x02b374 to 0x01100c (0x1a367 bytes) 000080 1784124590.894818 mapread 0x0032cb thru 0x00b22e (0x7f64 bytes) 000081 1784124590.899324 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1100d): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1100d): Mirror 3: 0xe8e19364 (stale) Mirror 4: 0x4fd4f6cf (stale) CRC-32 checksum value for chunk [0, 0x1100d): Mirror 1: 0x8bf5aaa1 Mirror 5: 0x8bf5aaa1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1100d: skipped 000082 1784124591.580319 trunc from 0x01100d to 0x00be45 (0x51c7 bytes) 000083 1784124591.635326 punch from 0x00b0b4 to 0x00be45 (0xd92 bytes) 000084 1784124591.649070 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x217:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xbe46): Mirror 4: 0x78b8bf2 (stale) Mirror 5: 0x6ee02e91 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbe46): Mirror 4: 0x78b8bf2 (stale) Mirror 5: 0x6ee02e91 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbe46: skipped 000085 1784124591.934356 write 0x007118 thru 0x01061f (0x9508 bytes) 000086 1784124591.951298 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x217:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10620): Mirror 5: 0xd52e51fc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10620): Mirror 5: 0xd52e51fc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10620: skipped 000087 1784124592.190775 punch from 0x005563 to 0x009780 (0x421e bytes) 000088 1784124592.197892 fallocate 0x0329b0 thru 0x032e6f (0x4c0 bytes) 000090 1784124592.205398 mapread 0x01900a thru 0x01b963 (0x295a bytes) 000091 1784124592.230586 trunc from 0x02cdca to 0x032e6f (0x60a6 bytes) 000092 1784124592.235832 fallocate 0x02a6ba thru 0x02b6d9 (0x1020 bytes) 000093 1784124592.244314 MIRROR_RESYNC 2 mirrors 000094 1784124592.860223 fallocate 0x013ad8 thru 0x01d4cd (0x99f6 bytes) 000095 1784124592.887929 mapwrite 0x0121e4 thru 0x01af56 (0x8d73 bytes) 000096 1784124592.928416 punch from 0x02caac to 0x02d135 (0x68a bytes) 000097 1784124592.932888 write 0x024b81 thru 0x032333 (0xd7b3 bytes) 000098 1784124592.948803 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x32e70): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32e70): Mirror 5: 0x7af31b96 (stale) CRC-32 checksum value for chunk [0, 0x32e70): Mirror 1: 0x3d35a724 Mirror 6: 0x3d35a724 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32e70: skipped 000099 1784124593.597699 punch from 0x01ba20 to 0x01bdd1 (0x3b2 bytes) 000100 1784124593.630865 write 0x026d6a thru 0x036cb0 (0xff47 bytes) 000102 1784124593.645858 punch from 0x031a29 to 0x036cb0 (0x5288 bytes) 000103 1784124593.665883 trunc from 0x036cb1 to 0x02cb98 (0xa118 bytes) 000104 1784124593.686291 trunc from 0x02cb99 to 0x013efd (0x18c9b bytes) 000105 1784124593.701345 trunc from 0x013efe to 0x02a66c (0x1676f bytes) 000106 1784124593.717927 trunc from 0x01d8c7 to 0x022949 (0x5083 bytes) 000107 1784124593.722866 write 0x028833 thru 0x029da1 (0x156f bytes) 000108 1784124593.731640 fallocate 0x011b52 thru 0x01f649 (0xdaf8 bytes) 000109 1784124593.741257 mapread 0x003c2f thru 0x011b40 (0xdf12 bytes) 000110 1784124593.771256 mapread 0x00941b thru 0x01835f (0xef45 bytes) 000111 1784124593.774172 read 0x02823a thru 0x02a66c (0x2433 bytes) 000112 1784124593.781325 fallocate 0x000520 thru 0x00f386 (0xee67 bytes) 000113 1784124593.788310 mapread 0x0181b9 thru 0x024f4e (0xcd96 bytes) 000114 1784124593.805377 read 0x011510 thru 0x017490 (0x5f81 bytes) 000115 1784124593.806131 punch from 0x00ad63 to 0x012552 (0x77f0 bytes) 000116 1784124593.819924 punch from 0x014077 to 0x014e3d (0xdc7 bytes) 000117 1784124593.824699 read 0x00df2a thru 0x01ddd6 (0xfead bytes) 000118 1784124593.831675 read 0x014745 thru 0x01c9b9 (0x8275 bytes) 000119 1784124593.832537 trunc from 0x02a66d to 0x0230db (0x7591 bytes) 000120 1784124593.844013 punch from 0x005eaa to 0x00f551 (0x96a8 bytes) 000121 1784124593.848410 read 0x0220e6 thru 0x0230db (0xff6 bytes) 000122 1784124593.849088 trunc from 0x000ea7 to 0x0056ab (0x4805 bytes) 000123 1784124593.856548 MIRROR_RESYNC 3 mirrors 000124 1784124594.398295 fallocate 0x030b76 thru 0x037b40 (0x6fcb bytes) 000125 1784124594.424842 punch from 0x01bf29 to 0x0230db (0x71b3 bytes) 000126 1784124594.433685 trunc from 0x00b88a to 0x00c05c (0x7d3 bytes) 000127 1784124594.438897 trunc from 0x0230dc to 0x0244fc (0x1421 bytes) 000128 1784124594.462580 trunc from 0x0244fd to 0x01cb6e (0x798e bytes) 000129 1784124594.485299 mapwrite 0x03df2e thru 0x03ffff (0x20d2 bytes) 000130 1784124594.547620 trunc from 0x01e4c4 to 0x02858a (0xa0c7 bytes) 000131 1784124594.553236 trunc from 0x015d45 to 0x0188dc (0x2b98 bytes) 000132 1784124594.561198 write 0x03f16d thru 0x03ffff (0xe93 bytes) 000133 1784124594.562463 write 0x00986f thru 0x018651 (0xede3 bytes) 000134 1784124594.584372 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x217:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x10d8c937 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x10d8c937 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000135 1784124594.827494 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x10d8c937 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6044cca5 Mirror 6: 0x6044cca5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000136 1784124595.512913 fallocate 0x00d92e thru 0x01a196 (0xc869 bytes) 000137 1784124595.543809 mapwrite 0x00aeff thru 0x00c508 (0x160a bytes) 000138 1784124595.576957 mapread 0x02b38b thru 0x030c76 (0x58ec bytes) 000139 1784124595.584072 trunc from 0x03d63c to 0x03eed0 (0x1895 bytes) 000140 1784124595.591066 write 0x00df80 thru 0x01d960 (0xf9e1 bytes) 000141 1784124595.598284 trunc from 0x02482a to 0x033cc6 (0xf49d bytes) 000142 1784124595.603399 read 0x011292 thru 0x015c19 (0x4988 bytes) 000143 1784124595.607204 MIRROR_RESYNC 3 mirrors 000145 1784124596.241330 mapread 0x012225 thru 0x0216ef (0xf4cb bytes) 000146 1784124596.269202 trunc from 0x040000 to 0x006d14 (0x392eb bytes) 000147 1784124596.307454 fallocate 0x0148b1 thru 0x023988 (0xf0d8 bytes) 000148 1784124596.312879 trunc from 0x000c91 to 0x003333 (0x26a3 bytes) 000149 1784124596.319422 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x6d15): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6d15): Mirror 5: 0xfcdcad6 (stale) Mirror 6: 0xfcdcad6 (stale) CRC-32 checksum value for chunk [0, 0x6d15): Mirror 1: 0xfcdcad6 Mirror 7: 0xfcdcad6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6d15: skipped 000150 1784124596.916078 fallocate 0x02b175 thru 0x03878a (0xd616 bytes) 000151 1784124596.952042 mapwrite 0x01cbf4 thru 0x01f1b7 (0x25c4 bytes) 000152 1784124597.016390 mapread 0x01bae4 thru 0x01f1b7 (0x36d4 bytes) 000153 1784124597.017908 trunc from 0x001c81 to 0x008813 (0x6b93 bytes) 000154 1784124597.028728 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1f1b8): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f1b8): Mirror 5: 0xf7eba82 (stale) Mirror 6: 0xf7eba82 (stale) Mirror 7: 0x130280ed (stale) CRC-32 checksum value for chunk [0, 0x1f1b8): Mirror 1: 0x25eb7527 Mirror 8: 0x25eb7527 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f1b8: skipped 000155 1784124597.645227 write 0x022bd6 thru 0x028886 (0x5cb1 bytes) 000156 1784124597.685260 fallocate 0x02fb57 thru 0x03aba9 (0xb053 bytes) 000157 1784124597.690716 write 0x0098f4 thru 0x00bd82 (0x248f bytes) 000158 1784124597.704039 write 0x007673 thru 0x0120d7 (0xaa65 bytes) 000161 1784124597.714094 read 0x00e876 thru 0x012337 (0x3ac2 bytes) 000162 1784124597.714768 trunc from 0x03abaa to 0x022c31 (0x17f78 bytes) 000163 1784124597.745042 fallocate 0x022b80 thru 0x02b030 (0x84b1 bytes) 000164 1784124597.754451 mapread 0x01bbbf thru 0x024baa (0x8fec bytes) 000165 1784124597.785379 trunc from 0x02b031 to 0x03f481 (0x14451 bytes) 000166 1784124597.802209 read 0x027743 thru 0x02ff99 (0x8857 bytes) 000168 1784124597.817158 MIRROR_RESYNC 5 mirrors 000169 1784124599.088557 trunc from 0x03f482 to 0x0326f7 (0xcd8a bytes) 000170 1784124599.127372 mapread 0x0018e4 thru 0x009d0e (0x842b bytes) 000171 1784124599.145710 mapread 0x01d3ad thru 0x02cad0 (0xf724 bytes) 000172 1784124599.163606 read 0x023d0d thru 0x02d136 (0x942a bytes) 000173 1784124599.164654 write 0x0009f6 thru 0x00b7ab (0xadb6 bytes) 000174 1784124599.173246 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x217:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x326f8): Mirror 5: 0x1e9fd4a (stale) Mirror 6: 0x1e9fd4a (stale) Mirror 8: 0x1e9fd4a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x326f8): Mirror 5: 0x1e9fd4a (stale) Mirror 6: 0x1e9fd4a (stale) Mirror 8: 0x1e9fd4a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x326f8: skipped 000175 1784124599.471148 trunc from 0x02fb60 to 0x0326f7 (0x2b98 bytes) 000176 1784124599.476345 mapwrite 0x016360 thru 0x023b4d (0xd7ee bytes) 000177 1784124599.528468 trunc from 0x01e3ea to 0x02c9b8 (0xe5cf bytes) 000178 1784124599.535490 read 0x026331 thru 0x0326f7 (0xc3c7 bytes) 000179 1784124599.545767 trunc from 0x0326f8 to 0x026e71 (0xb886 bytes) 000180 1784124599.568332 trunc from 0x026e72 to 0x012888 (0x145e9 bytes) 000181 1784124599.583612 fallocate 0x02daf5 thru 0x039e65 (0xc371 bytes) 000182 1784124599.588228 punch from 0x013a01 to 0x019bf8 (0x61f8 bytes) 000184 1784124599.597085 trunc from 0x039e66 to 0x00051d (0x39948 bytes) 000185 1784124599.616510 trunc from 0x00051e to 0x029623 (0x29106 bytes) 000186 1784124599.638376 mapwrite 0x0081df thru 0x012af6 (0xa918 bytes) 000187 1784124599.675998 mapwrite 0x01b6ab thru 0x02930e (0xdc64 bytes) 000188 1784124599.708310 mapread 0x0258c1 thru 0x029623 (0x3d63 bytes) 000189 1784124599.710012 trunc from 0x00ab60 to 0x0141fc (0x969d bytes) 000190 1784124599.715829 punch from 0x014f05 to 0x01cd3a (0x7e36 bytes) 000192 1784124599.721397 write 0x02a3f3 thru 0x0315d7 (0x71e5 bytes) 000193 1784124599.723598 trunc from 0x012de5 to 0x013d34 (0xf50 bytes) 000194 1784124599.728233 fallocate 0x0006ac thru 0x00fc50 (0xf5a5 bytes) 000196 1784124599.732417 mapwrite 0x010c9e thru 0x0128ac (0x1c0f bytes) 000197 1784124599.767080 MIRROR_RESYNC 4 mirrors 000198 1784124600.785873 fallocate 0x01809a thru 0x018e87 (0xdee bytes) 000199 1784124600.823059 read 0x0046ce thru 0x00b3de (0x6d11 bytes) 000200 1784124600.831187 mapwrite 0x03cc05 thru 0x03ff00 (0x32fc bytes) 000201 1784124600.870627 trunc from 0x03ff01 to 0x010cb3 (0x2f24d bytes) 000203 1784124600.886429 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x217:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10cb4): Mirror 6: 0x8cc52ce4 (stale) Mirror 8: 0x8cc52ce4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10cb4): Mirror 6: 0x8cc52ce4 (stale) Mirror 8: 0x8cc52ce4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10cb4: skipped 000204 1784124601.167694 fallocate 0x019e0d thru 0x027ce7 (0xdedb bytes) 000205 1784124601.178914 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x217:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27ce8): Mirror 8: 0xe175bcc0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27ce8): Mirror 8: 0xe175bcc0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27ce8: skipped 000206 1784124601.409660 write 0x017005 thru 0x0220de (0xb0da bytes) 000207 1784124601.423858 trunc from 0x027ce8 to 0x018372 (0xf975 bytes) 000208 1784124601.448273 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x217:0x0]: Invalid argument (22) 000209 1784124601.568618 mapwrite 0x013d31 thru 0x01e3ab (0xa67b bytes) 000210 1784124601.627836 mapread 0x017fa1 thru 0x01e3ab (0x640b bytes) 000211 1784124601.628958 trunc from 0x01e3ac to 0x039bc3 (0x1b818 bytes) 000212 1784124601.643097 read 0x03618e thru 0x039bc3 (0x3a36 bytes) 000213 1784124601.650071 write 0x031cc1 thru 0x034d32 (0x3072 bytes) 000215 1784124601.660364 write 0x001f05 thru 0x007cf6 (0x5df2 bytes) 000216 1784124601.669789 read 0x00acb8 thru 0x00fdae (0x50f7 bytes) 000217 1784124601.678256 trunc from 0x039bc4 to 0x000c83 (0x38f40 bytes) 000219 1784124601.700455 fallocate 0x02ff9a thru 0x03715f (0x71c6 bytes) 000220 1784124601.707700 fallocate 0x011dbd thru 0x01c6d5 (0xa919 bytes) 000221 1784124601.714693 trunc from 0x0108b8 to 0x017295 (0x69de bytes) 000222 1784124601.720602 mapread 0x031c29 thru 0x03715f (0x5537 bytes) 000223 1784124601.759864 read 0x01ad85 thru 0x0214ad (0x6729 bytes) 000226 1784124601.778291 write 0x003d90 thru 0x00ac2f (0x6ea0 bytes) 000229 1784124601.783083 read 0x025be7 thru 0x025cee (0x108 bytes) 000230 1784124601.783731 mapread 0x020be6 thru 0x02d0d6 (0xc4f1 bytes) 000231 1784124601.786753 write 0x02badd thru 0x035299 (0x97bd bytes) 000232 1784124601.788932 mapread 0x00911b thru 0x00b207 (0x20ed bytes) 000233 1784124601.790081 trunc from 0x037160 to 0x009085 (0x2e0da bytes) 000234 1784124601.808341 mapwrite 0x03435a thru 0x03b6ac (0x7353 bytes) 000235 1784124601.857736 mapread 0x021652 thru 0x0308ee (0xf29d bytes) 000236 1784124601.859776 mapread 0x0127e9 thru 0x01a723 (0x7f3b bytes) 000238 1784124601.867063 trunc from 0x03b6ad to 0x01b4de (0x201ce bytes) 000239 1784124601.889807 mapread 0x00b301 thru 0x00e161 (0x2e61 bytes) 000240 1784124601.891506 punch from 0x00a6af to 0x010c7c (0x65ce bytes) 000242 1784124601.899894 mapwrite 0x03879f thru 0x03ffff (0x7861 bytes) 000243 1784124601.971066 trunc from 0x004cf4 to 0x014417 (0xf724 bytes) 000244 1784124601.977164 mapwrite 0x004727 thru 0x005e11 (0x16eb bytes) 000245 1784124602.003462 trunc from 0x040000 to 0x028133 (0x17ecc bytes) 000246 1784124602.015726 mapwrite 0x02582e thru 0x026fd9 (0x17ac bytes) 000247 1784124602.039868 trunc from 0x01bc8c to 0x01c117 (0x48c bytes) 000248 1784124602.051549 mapread 0x015c19 thru 0x01929b (0x3683 bytes) 000249 1784124602.052811 write 0x0126f8 thru 0x01caef (0xa3f8 bytes) 000251 1784124602.065433 trunc from 0x00ba45 to 0x00e718 (0x2cd4 bytes) 000252 1784124602.070297 fallocate 0x002b24 thru 0x00a8d2 (0x7daf bytes) 000253 1784124602.074908 read 0x022130 thru 0x028133 (0x6004 bytes) 000254 1784124602.084379 punch from 0x0235d7 to 0x028133 (0x4b5d bytes) 000255 1784124602.089509 fallocate 0x00916e thru 0x0179d7 (0xe86a bytes) 000256 1784124602.100574 mapwrite 0x003f45 thru 0x00a61a (0x66d6 bytes) 000257 1784124602.129809 fallocate 0x03bbd4 thru 0x03ffff (0x442c bytes) 000258 1784124602.136249 punch from 0x02c0d4 to 0x02faa8 (0x39d5 bytes) 000259 1784124602.143817 write 0x012320 thru 0x01aad3 (0x87b4 bytes) 000260 1784124602.145343 trunc from 0x040000 to 0x026d1e (0x192e1 bytes) 000261 1784124602.168669 MIRROR_EXTEND to 2 mirrors 000262 1784124602.691062 mapwrite 0x004b1e thru 0x008c22 (0x4105 bytes) 000263 1784124602.772320 read 0x016dba thru 0x01c00f (0x5256 bytes) 000264 1784124602.784871 trunc from 0x026d1f to 0x01a2a4 (0xca7a bytes) 000265 1784124602.828397 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1a2a5): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a2a5): Mirror 2: 0xc4c76949 (stale) CRC-32 checksum value for chunk [0, 0x1a2a5): Mirror 1: 0x71ec0cc8 Mirror 3: 0x71ec0cc8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a2a5: skipped 000266 1784124603.425436 MIRROR_RESYNC 3 mirrors 000267 1784124603.956597 mapread 0x00dbec thru 0x0160c5 (0x84da bytes) 000269 1784124603.981523 read 0x011aba thru 0x018ea3 (0x73ea bytes) 000270 1784124603.986803 write 0x014b0a thru 0x0154cb (0x9c2 bytes) 000271 1784124604.036678 read 0x0045b9 thru 0x007cb2 (0x36fa bytes) 000272 1784124604.047223 mapwrite 0x00c864 thru 0x019415 (0xcbb2 bytes) 000273 1784124604.071848 MIRROR_RESYNC 3 mirrors 000274 1784124604.734882 MIRROR_EXTEND to 4 mirrors 000275 1784124605.349109 mapread 0x003eee thru 0x00bfac (0x80bf bytes) 000276 1784124605.367092 trunc from 0x012f20 to 0x01a1df (0x72c0 bytes) 000277 1784124605.414098 trunc from 0x01a2a5 to 0x00b65f (0xec45 bytes) 000278 1784124605.439980 trunc from 0x00b660 to 0x039b68 (0x2e509 bytes) 000279 1784124605.457556 trunc from 0x039b69 to 0x006906 (0x33262 bytes) 000280 1784124605.474310 trunc from 0x006907 to 0x00798b (0x1085 bytes) 000281 1784124605.497495 trunc from 0x00624c to 0x00798b (0x1740 bytes) 000282 1784124605.510006 read 0x003db6 thru 0x00798b (0x3bd6 bytes) 000283 1784124605.527598 punch from 0x000547 to 0x00798b (0x7445 bytes) 000284 1784124605.543154 trunc from 0x00798c to 0x026ae6 (0x1f15b bytes) 000285 1784124605.569783 fallocate 0x0073c4 thru 0x011da9 (0xa9e6 bytes) 000286 1784124605.580242 fallocate 0x027b76 thru 0x03122e (0x96b9 bytes) 000287 1784124605.590111 write 0x003e42 thru 0x00bdc9 (0x7f88 bytes) 000288 1784124605.607043 mapread 0x022254 thru 0x022599 (0x346 bytes) 000289 1784124605.629039 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3122f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3122f): Mirror 2: 0xa61d11d (stale) Mirror 3: 0xa61d11d (stale) Mirror 4: 0xa61d11d (stale) CRC-32 checksum value for chunk [0, 0x3122f): Mirror 1: 0x7d48ad95 Mirror 5: 0x7d48ad95 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3122f: skipped 000290 1784124606.317793 write 0x03712c thru 0x03ffff (0x8ed4 bytes) 000291 1784124606.387291 MIRROR_RESYNC 5 mirrors 000292 1784124607.326254 mapread 0x009442 thru 0x015f55 (0xcb14 bytes) 000294 1784124607.347766 trunc from 0x040000 to 0x02127e (0x1ed81 bytes) 000295 1784124607.381826 write 0x03af29 thru 0x03ffff (0x50d7 bytes) 000296 1784124607.383382 punch from 0x022f06 to 0x02b4eb (0x85e6 bytes) 000297 1784124607.388983 read 0x01b689 thru 0x02a2f1 (0xec69 bytes) 000298 1784124607.401336 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x217:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x569bf38 (stale) Mirror 3: 0x569bf38 (stale) Mirror 5: 0x569bf38 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x569bf38 (stale) Mirror 3: 0x569bf38 (stale) Mirror 5: 0x569bf38 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000299 1784124607.705971 fallocate 0x03ecbd thru 0x03ffff (0x1343 bytes) 000300 1784124607.714783 mapwrite 0x00676b thru 0x0087b8 (0x204e bytes) 000301 1784124607.744694 fallocate 0x00061d thru 0x001a5d (0x1441 bytes) 000302 1784124607.750284 write 0x03d0eb thru 0x03ffff (0x2f15 bytes) 000303 1784124607.760509 trunc from 0x038b6f to 0x03ffff (0x7491 bytes) 000304 1784124607.779013 read 0x028d76 thru 0x0313e1 (0x866c bytes) 000306 1784124607.788444 read 0x01023a thru 0x0135ab (0x3372 bytes) 000307 1784124607.800217 write 0x021e66 thru 0x023855 (0x19f0 bytes) 000308 1784124607.805635 MIRROR_RESYNC 4 mirrors 000310 1784124608.687320 mapwrite 0x020cf4 thru 0x02740c (0x6719 bytes) 000311 1784124608.780199 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x217:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x239151fe (stale) Mirror 3: 0x239151fe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x239151fe (stale) Mirror 3: 0x239151fe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000312 1784124609.055297 mapwrite 0x037a93 thru 0x03ffff (0x856d bytes) 000313 1784124609.116858 read 0x02d531 thru 0x032509 (0x4fd9 bytes) 000314 1784124609.128635 mapwrite 0x023d09 thru 0x028da8 (0x50a0 bytes) 000315 1784124609.167214 write 0x033cbb thru 0x03ffff (0xc345 bytes) 000316 1784124609.175123 trunc from 0x01ace4 to 0x02477a (0x9a97 bytes) 000317 1784124609.179695 read 0x02a1f8 thru 0x02b5a4 (0x13ad bytes) 000318 1784124609.180457 trunc from 0x012f37 to 0x016742 (0x380c bytes) 000319 1784124609.185091 fallocate 0x019946 thru 0x02799e (0xe059 bytes) 000321 1784124609.190193 mapwrite 0x02e8f7 thru 0x03bda3 (0xd4ad bytes) 000322 1784124609.217786 trunc from 0x040000 to 0x03fa96 (0x569 bytes) 000323 1784124609.234350 mapread 0x03eb59 thru 0x03fa96 (0xf3e bytes) 000324 1784124609.235458 punch from 0x0267d3 to 0x02a9cc (0x41fa bytes) 000325 1784124609.239510 write 0x018ebb thru 0x025d99 (0xcedf bytes) 000326 1784124609.264490 trunc from 0x010936 to 0x0110a4 (0x76f bytes) 000327 1784124609.268321 mapwrite 0x020bdb thru 0x02d0d6 (0xc4fc bytes) 000328 1784124609.283536 mapread 0x030b80 thru 0x031f30 (0x13b1 bytes) 000329 1784124609.284501 mapwrite 0x01d502 thru 0x026945 (0x9444 bytes) 000330 1784124609.302669 fallocate 0x023d58 thru 0x02a316 (0x65bf bytes) 000331 1784124609.309360 trunc from 0x03fa97 to 0x003a7f (0x3c017 bytes) 000332 1784124609.337411 read 0x002e0d thru 0x003a7f (0xc73 bytes) 000333 1784124609.341897 MIRROR_RESYNC 3 mirrors 000334 1784124609.857913 mapwrite 0x00fb84 thru 0x01a3a5 (0xa822 bytes) 000335 1784124609.961635 fallocate 0x027ea8 thru 0x037e6f (0xffc8 bytes) 000336 1784124609.970360 trunc from 0x00999e to 0x016c50 (0xd2b3 bytes) 000337 1784124609.978123 fallocate 0x03a5f0 thru 0x03bb3c (0x154d bytes) 000338 1784124609.984235 read 0x0278ae thru 0x037823 (0xff76 bytes) 000339 1784124609.995321 mapwrite 0x039662 thru 0x03ffff (0x699e bytes) 000340 1784124610.048000 trunc from 0x040000 to 0x01059d (0x2fa62 bytes) 000342 1784124610.060056 trunc from 0x00bda7 to 0x01059d (0x47f7 bytes) 000343 1784124610.068035 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1059e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1059e): Mirror 2: 0xeb35b706 (stale) Mirror 3: 0xeb35b706 (stale) CRC-32 checksum value for chunk [0, 0x1059e): Mirror 1: 0xeb35b706 Mirror 4: 0xeb35b706 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1059e: skipped 000344 1784124610.668327 trunc from 0x01059e to 0x008346 (0x8257 bytes) 000345 1784124610.722645 MIRROR_RESYNC 4 mirrors 000346 1784124611.362737 punch from 0x00265a to 0x008346 (0x5ced bytes) 000347 1784124611.398908 trunc from 0x0060bf to 0x008346 (0x2288 bytes) 000348 1784124611.403634 write 0x0110a1 thru 0x0166fd (0x565d bytes) 000349 1784124611.405772 punch from 0x00ee3a to 0x012b08 (0x3ccf bytes) 000350 1784124611.419012 write 0x03b181 thru 0x03ffff (0x4e7f bytes) 000351 1784124611.421397 read 0x01e84b thru 0x024f04 (0x66ba bytes) 000352 1784124611.434204 write 0x01b072 thru 0x01e707 (0x3696 bytes) 000353 1784124611.441545 trunc from 0x040000 to 0x031ddd (0xe222 bytes) 000354 1784124611.467449 mapwrite 0x031e5c thru 0x03ffff (0xe1a4 bytes) 000355 1784124611.542403 mapwrite 0x03cc32 thru 0x03de48 (0x1217 bytes) 000357 1784124611.565663 trunc from 0x040000 to 0x037874 (0x878b bytes) 000358 1784124611.593689 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x37875): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37875): Mirror 2: 0xebfc2c8a (stale) Mirror 3: 0xebfc2c8a (stale) Mirror 4: 0xebfc2c8a (stale) CRC-32 checksum value for chunk [0, 0x37875): Mirror 1: 0xcc80942e Mirror 5: 0xcc80942e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37875: skipped 000359 1784124612.177135 punch from 0x023380 to 0x0259d4 (0x2655 bytes) 000360 1784124612.213510 read 0x02a4e7 thru 0x02b383 (0xe9d bytes) 000361 1784124612.218614 mapread 0x0342a9 thru 0x037874 (0x35cc bytes) 000362 1784124612.241868 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x217:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37875): Mirror 2: 0xebfc2c8a (stale) Mirror 3: 0xebfc2c8a (stale) Mirror 5: 0xcc80942e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37875): Mirror 2: 0xebfc2c8a (stale) Mirror 3: 0xebfc2c8a (stale) Mirror 5: 0xcc80942e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37875: skipped 000363 1784124612.548340 fallocate 0x017855 thru 0x0224dd (0xac89 bytes) 000364 1784124612.553126 fallocate 0x008210 thru 0x009e1b (0x1c0c bytes) 000365 1784124612.558549 fallocate 0x012d14 thru 0x01acb2 (0x7f9f bytes) 000368 1784124612.570427 mapwrite 0x03b00e thru 0x03b4be (0x4b1 bytes) 000369 1784124612.640751 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3b4bf): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b4bf): Mirror 2: 0x9c78740f (stale) Mirror 3: 0x9c78740f (stale) Mirror 5: 0x2838a475 (stale) CRC-32 checksum value for chunk [0, 0x3b4bf): Mirror 1: 0x3c9f2d38 Mirror 6: 0x3c9f2d38 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b4bf: skipped 000370 1784124613.264203 mapread 0x03a494 thru 0x03b4be (0x102b bytes) 000371 1784124613.290705 mapwrite 0x021099 thru 0x02759a (0x6502 bytes) 000372 1784124613.397730 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b4bf): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b4bf): Mirror 2: 0x9c78740f (stale) Mirror 3: 0x9c78740f (stale) Mirror 5: 0x2838a475 (stale) Mirror 6: 0x3c9f2d38 (stale) CRC-32 checksum value for chunk [0, 0x3b4bf): Mirror 1: 0x646b7451 Mirror 7: 0x646b7451 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b4bf: skipped 000373 1784124614.159842 mapwrite 0x03302f thru 0x038441 (0x5413 bytes) 000374 1784124614.248187 write 0x0074a2 thru 0x014493 (0xcff2 bytes) 000376 1784124614.272600 read 0x01340a thru 0x01b135 (0x7d2c bytes) 000377 1784124614.284851 trunc from 0x01711d to 0x01f9c2 (0x88a6 bytes) 000378 1784124614.296906 mapread 0x010de8 thru 0x01428b (0x34a4 bytes) 000379 1784124614.298165 trunc from 0x03b4bf to 0x02e91a (0xcba4 bytes) 000380 1784124614.317280 mapread 0x005815 thru 0x00e3d6 (0x8bc2 bytes) 000381 1784124614.328488 punch from 0x0203e0 to 0x0298ef (0x9510 bytes) 000382 1784124614.333165 read 0x007901 thru 0x00d8a3 (0x5fa3 bytes) 000384 1784124614.334763 punch from 0x005c03 to 0x00b1cc (0x55ca bytes) 000385 1784124614.347348 trunc from 0x02e91b to 0x03ae4d (0xc533 bytes) 000387 1784124614.359177 mapwrite 0x00218e thru 0x00d5a2 (0xb415 bytes) 000388 1784124614.396333 read 0x02bbf7 thru 0x034db4 (0x91be bytes) 000389 1784124614.398675 mapread 0x033417 thru 0x03ae4d (0x7a37 bytes) 000390 1784124614.402061 trunc from 0x03ae4e to 0x01e212 (0x1cc3b bytes) 000393 1784124614.420339 trunc from 0x01e213 to 0x0087c1 (0x15a51 bytes) 000394 1784124614.434815 read 0x0006be thru 0x0087c1 (0x8104 bytes) 000395 1784124614.438595 trunc from 0x0087c2 to 0x03e658 (0x35e97 bytes) 000396 1784124614.457647 punch from 0x0382e9 to 0x03e658 (0x6370 bytes) 000397 1784124614.466038 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x217:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e659): Mirror 2: 0x7a6cf6c (stale) Mirror 3: 0x7a6cf6c (stale) Mirror 5: 0x5ed6bf6f (stale) Mirror 6: 0x90bb8563 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e659): Mirror 2: 0x7a6cf6c (stale) Mirror 3: 0x7a6cf6c (stale) Mirror 5: 0x5ed6bf6f (stale) Mirror 6: 0x90bb8563 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e659: skipped 000398 1784124614.782885 write 0x01aee9 thru 0x020ca0 (0x5db8 bytes) 000399 1784124614.811208 fallocate 0x01c270 thru 0x02409b (0x7e2c bytes) 000400 1784124614.822133 write 0x027226 thru 0x02ec40 (0x7a1b bytes) 000401 1784124614.840478 trunc from 0x03e659 to 0x007074 (0x375e4 bytes) 000402 1784124614.868629 MIRROR_RESYNC 5 mirrors 000403 1784124615.573467 mapread 0x006e1c thru 0x007074 (0x259 bytes) 000405 1784124615.582525 write 0x030eae thru 0x03d84b (0xc99e bytes) 000406 1784124615.601823 write 0x023505 thru 0x02b566 (0x8062 bytes) 000407 1784124615.608257 punch from 0x01b272 to 0x01cc86 (0x1a15 bytes) 000409 1784124615.612914 mapwrite 0x0310ca thru 0x033e9f (0x2dd6 bytes) 000410 1784124615.630982 fallocate 0x000a81 thru 0x010084 (0xf604 bytes) 000411 1784124615.642426 punch from 0x02a544 to 0x02f407 (0x4ec4 bytes) 000412 1784124615.649798 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x217:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d84c): Mirror 2: 0x27250672 (stale) Mirror 3: 0x27250672 (stale) Mirror 6: 0x27250672 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d84c): Mirror 2: 0x27250672 (stale) Mirror 3: 0x27250672 (stale) Mirror 6: 0x27250672 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d84c: skipped 000413 1784124615.942830 fallocate 0x03acc3 thru 0x03da6d (0x2dab bytes) 000414 1784124615.947432 fallocate 0x0047b4 thru 0x013609 (0xee56 bytes) 000415 1784124615.951501 mapread 0x02ac7a thru 0x037ee3 (0xd26a bytes) 000416 1784124615.987856 fallocate 0x015ece thru 0x023523 (0xd656 bytes) 000417 1784124615.995919 write 0x011aa5 thru 0x01e6ed (0xcc49 bytes) 000418 1784124616.013276 trunc from 0x02524c to 0x032803 (0xd5b8 bytes) 000419 1784124616.019664 punch from 0x01b321 to 0x0219d5 (0x66b5 bytes) 000420 1784124616.037754 MIRROR_RESYNC 4 mirrors 000421 1784124617.034787 trunc from 0x0314cc to 0x03b2e7 (0x9e1c bytes) 000422 1784124617.076886 mapwrite 0x02954c thru 0x02edba (0x586f bytes) 000423 1784124617.131332 read 0x004aa7 thru 0x005093 (0x5ed bytes) 000424 1784124617.146072 trunc from 0x03da6e to 0x007006 (0x36a67 bytes) 000425 1784124617.167076 write 0x01b8de thru 0x021945 (0x6068 bytes) 000426 1784124617.170584 trunc from 0x01c7d6 to 0x021945 (0x5170 bytes) 000427 1784124617.187598 mapwrite 0x026424 thru 0x02a44a (0x4027 bytes) 000428 1784124617.247441 trunc from 0x02a44b to 0x02b8ca (0x1480 bytes) 000429 1784124617.270295 trunc from 0x02b8cb to 0x00c794 (0x1f136 bytes) 000430 1784124617.286894 fallocate 0x03e768 thru 0x03ffff (0x1898 bytes) 000431 1784124617.293789 write 0x03ef5e thru 0x03ffff (0x10a2 bytes) 000432 1784124617.311360 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x2680eef6 (stale) Mirror 3: 0x2680eef6 (stale) Mirror 6: 0x2680eef6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xac66d288 Mirror 7: 0xac66d288 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000433 1784124618.072980 mapwrite 0x014579 thru 0x014728 (0x1b0 bytes) 000434 1784124618.170106 read 0x00cc82 thru 0x011a94 (0x4e13 bytes) 000435 1784124618.180126 read 0x00e51a thru 0x012943 (0x442a bytes) 000436 1784124618.194666 mapread 0x03dade thru 0x03ffff (0x2522 bytes) 000437 1784124618.196028 trunc from 0x03b1d7 to 0x03ffff (0x4e29 bytes) 000438 1784124618.203850 MIRROR_RESYNC 5 mirrors 000439 1784124619.127636 read 0x02b6ac thru 0x02ef84 (0x38d9 bytes) 000441 1784124619.140752 mapread 0x011fcd thru 0x012640 (0x674 bytes) 000442 1784124619.162790 MIRROR_RESYNC 5 mirrors 000443 1784124619.454741 mapread 0x035393 thru 0x03ffff (0xac6d bytes) 000444 1784124619.473403 mapwrite 0x03a6b8 thru 0x03ffff (0x5948 bytes) 000445 1784124619.545326 mapwrite 0x00f269 thru 0x018982 (0x971a bytes) 000446 1784124619.587663 fallocate 0x03beae thru 0x03ffff (0x4152 bytes) 000447 1784124619.592069 punch from 0x01df85 to 0x01f9f8 (0x1a74 bytes) 000448 1784124619.597346 mapwrite 0x014e94 thru 0x01ef87 (0xa0f4 bytes) 000449 1784124619.627242 fallocate 0x02cf49 thru 0x0335cb (0x6683 bytes) 000450 1784124619.631224 read 0x01e230 thru 0x024009 (0x5dda bytes) 000451 1784124619.632026 read 0x004c35 thru 0x008027 (0x33f3 bytes) 000452 1784124619.643664 mapread 0x002d02 thru 0x00892b (0x5c2a bytes) 000453 1784124619.644908 trunc from 0x039141 to 0x03ccd0 (0x3b90 bytes) 000454 1784124619.649683 trunc from 0x017f91 to 0x026f02 (0xef72 bytes) 000455 1784124619.655768 fallocate 0x033dd8 thru 0x036850 (0x2a79 bytes) 000456 1784124619.660048 mapread 0x015130 thru 0x0188f4 (0x37c5 bytes) 000458 1784124619.669266 trunc from 0x00d268 to 0x01bba4 (0xe93d bytes) 000459 1784124619.673787 punch from 0x010dcd to 0x018559 (0x778d bytes) 000460 1784124619.677791 write 0x037be3 thru 0x03d066 (0x5484 bytes) 000461 1784124619.679854 trunc from 0x040000 to 0x01edc0 (0x2123f bytes) 000462 1784124619.723294 mapread 0x00e31f thru 0x019dea (0xbacc bytes) 000463 1784124619.800491 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x217:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1edc1): Mirror 3: 0x40fe677a (stale) Mirror 6: 0x40fe677a (stale) Mirror 7: 0x40fe677a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1edc1): Mirror 3: 0x40fe677a (stale) Mirror 6: 0x40fe677a (stale) Mirror 7: 0x40fe677a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1edc1: skipped 000464 1784124620.061304 read 0x019975 thru 0x01af51 (0x15dd bytes) 000465 1784124620.065336 trunc from 0x01edc1 to 0x005ed9 (0x18ee7 bytes) 000466 1784124620.087129 trunc from 0x005d66 to 0x005ed9 (0x174 bytes) 000467 1784124620.092205 fallocate 0x01fb33 thru 0x020ad2 (0xfa0 bytes) 000468 1784124620.097432 mapread 0x008156 thru 0x00e9d5 (0x6880 bytes) 000469 1784124620.117599 trunc from 0x0082bc to 0x00e986 (0x66cb bytes) 000470 1784124620.122533 fallocate 0x011434 thru 0x0158fd (0x44ca bytes) 000471 1784124620.126640 fallocate 0x026db4 thru 0x03565e (0xe8ab bytes) 000472 1784124620.134222 fallocate 0x006074 thru 0x0086f0 (0x267d bytes) 000473 1784124620.138752 trunc from 0x03565f to 0x021bca (0x13a94 bytes) 000474 1784124620.151350 fallocate 0x003363 thru 0x0118b4 (0xe552 bytes) 000475 1784124620.157176 mapread 0x014b12 thru 0x021bca (0xd0b9 bytes) 000476 1784124620.174788 write 0x00dd92 thru 0x01a6a7 (0xc916 bytes) 000477 1784124620.182403 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x217:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21bcb): Mirror 6: 0x1df3ae5f (stale) Mirror 7: 0x1df3ae5f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21bcb): Mirror 6: 0x1df3ae5f (stale) Mirror 7: 0x1df3ae5f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21bcb: skipped 000478 1784124620.450489 write 0x037a39 thru 0x03ffff (0x85c7 bytes) 000479 1784124620.459688 MIRROR_RESYNC 3 mirrors 000480 1784124621.146873 write 0x0261f6 thru 0x0359bc (0xf7c7 bytes) 000481 1784124621.197660 read 0x027465 thru 0x035368 (0xdf04 bytes) 000484 1784124621.199511 trunc from 0x02a354 to 0x02f96c (0x5619 bytes) 000485 1784124621.215960 fallocate 0x03cd39 thru 0x03ec7f (0x1f47 bytes) 000486 1784124621.219774 write 0x026338 thru 0x02977c (0x3445 bytes) 000487 1784124621.220793 mapwrite 0x00a8f9 thru 0x0172ac (0xc9b4 bytes) 000488 1784124621.264771 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x6022656 (stale) Mirror 7: 0x6022656 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8f5b2e3e Mirror 8: 0x8f5b2e3e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000489 1784124622.142081 fallocate 0x02d5d3 thru 0x03900a (0xba38 bytes) 000490 1784124622.185661 mapread 0x00d3e9 thru 0x01521f (0x7e37 bytes) 000491 1784124622.205919 write 0x0272ac thru 0x0330c7 (0xbe1c bytes) 000492 1784124622.228677 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x217:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x6022656 (stale) Mirror 8: 0x8f5b2e3e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x6022656 (stale) Mirror 8: 0x8f5b2e3e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000493 1784124622.522231 write 0x03addd thru 0x03ffff (0x5223 bytes) 000494 1784124622.531667 fallocate 0x018000 thru 0x025354 (0xd355 bytes) 000495 1784124622.542145 punch from 0x034777 to 0x03abd8 (0x6462 bytes) 000496 1784124622.556378 fallocate 0x00b74a thru 0x01b5e5 (0xfe9c bytes) 000497 1784124622.560443 mapread 0x0107a0 thru 0x01b509 (0xad6a bytes) 000498 1784124622.582657 write 0x008dd3 thru 0x014581 (0xb7af bytes) 000499 1784124622.587617 trunc from 0x02b76e to 0x0368d4 (0xb167 bytes) 000500 1784124622.593131 write 0x0360c1 thru 0x03855e (0x249e bytes) 000501 1784124622.602107 punch from 0x00b6a0 to 0x0141bc (0x8b1d bytes) 000502 1784124622.616745 punch from 0x03940e to 0x039c80 (0x873 bytes) 000503 1784124622.621198 punch from 0x026449 to 0x026c0c (0x7c4 bytes) 000504 1784124622.635089 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x6022656 (stale) Mirror 8: 0x8f5b2e3e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x758f24f1 Mirror 9: 0x758f24f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000506 1784124623.453368 write 0x02df27 thru 0x034426 (0x6500 bytes) 000507 1784124623.517226 trunc from 0x040000 to 0x01645b (0x29ba4 bytes) 000508 1784124623.538086 read 0x0070d4 thru 0x00d78d (0x66ba bytes) 000509 1784124623.549351 mapread 0x013e17 thru 0x01645b (0x2645 bytes) 000510 1784124623.575740 write 0x03d5a5 thru 0x03ffff (0x2a5b bytes) 000511 1784124623.577197 write 0x031050 thru 0x0328a7 (0x1858 bytes) 000512 1784124623.584221 punch from 0x0213bf to 0x025741 (0x4383 bytes) 000513 1784124623.591087 trunc from 0x0336b2 to 0x03a6ab (0x6ffa bytes) 000514 1784124623.597879 mapwrite 0x02bd55 thru 0x036830 (0xaadc bytes) 000515 1784124623.648198 trunc from 0x040000 to 0x03e39c (0x1c63 bytes) 000517 1784124623.671360 mapwrite 0x02bcb7 thru 0x038c76 (0xcfc0 bytes) 000518 1784124623.702623 write 0x0172a3 thru 0x0191c3 (0x1f21 bytes) 000519 1784124623.716074 punch from 0x00854c to 0x011a45 (0x94fa bytes) 000520 1784124623.719909 read 0x03a344 thru 0x03e39c (0x4059 bytes) 000521 1784124623.727413 MIRROR_RESYNC 4 mirrors 000522 1784124624.769596 write 0x031aa7 thru 0x035f76 (0x44d0 bytes) 000523 1784124624.811124 mapwrite 0x0011bf thru 0x0054d2 (0x4314 bytes) 000524 1784124624.851199 fallocate 0x02ce1d thru 0x02f7cc (0x29b0 bytes) 000525 1784124624.855313 mapread 0x002e07 thru 0x00463c (0x1836 bytes) 000526 1784124624.857270 fallocate 0x00f10f thru 0x01587e (0x6770 bytes) 000527 1784124624.862442 trunc from 0x03e39d to 0x039182 (0x521a bytes) 000528 1784124624.890293 MIRROR_RESYNC 4 mirrors 000529 1784124625.632629 trunc from 0x0211b7 to 0x02a445 (0x928f bytes) 000530 1784124625.652623 mapread 0x02f09b thru 0x038ac4 (0x9a2a bytes) 000531 1784124625.665094 write 0x02502c thru 0x02c099 (0x706e bytes) 000532 1784124625.669917 write 0x0342f0 thru 0x03880c (0x451d bytes) 000533 1784124625.671380 write 0x0338b2 thru 0x03ffff (0xc74e bytes) 000534 1784124625.673609 mapwrite 0x01b596 thru 0x022983 (0x73ee bytes) 000535 1784124625.697186 fallocate 0x0330f3 thru 0x03e9c7 (0xb8d5 bytes) 000537 1784124625.702569 trunc from 0x040000 to 0x0250c2 (0x1af3d bytes) 000538 1784124625.721893 punch from 0x01053b to 0x01717b (0x6c41 bytes) 000539 1784124625.727081 mapread 0x001c21 thru 0x001c44 (0x24 bytes) 000540 1784124625.735110 mapread 0x00b9e9 thru 0x01b6fd (0xfd15 bytes) 000541 1784124625.743729 punch from 0x0045ac to 0x00852b (0x3f80 bytes) 000542 1784124625.747735 read 0x00a3dd thru 0x013d81 (0x99a5 bytes) 000543 1784124625.748501 mapwrite 0x01d52c thru 0x0206ef (0x31c4 bytes) 000544 1784124625.763575 punch from 0x013a0e to 0x018798 (0x4d8b bytes) 000545 1784124625.769203 trunc from 0x00cef7 to 0x019571 (0xc67b bytes) 000547 1784124625.774512 mapread 0x011d0c thru 0x01e2a4 (0xc599 bytes) 000548 1784124625.785807 mapwrite 0x037b4d thru 0x03b9c7 (0x3e7b bytes) 000549 1784124625.827364 fallocate 0x034d7a thru 0x03ffff (0xb286 bytes) 000550 1784124625.843823 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3b9c8): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b9c8): Mirror 7: 0x58034551 (stale) Mirror 8: 0x58034551 (stale) Mirror 9: 0x58034551 (stale) CRC-32 checksum value for chunk [0, 0x3b9c8): Mirror 1: 0x5374e166 Mirror 10: 0x5374e166 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b9c8: skipped 000551 1784124626.575735 write 0x0029ed thru 0x004b9a (0x21ae bytes) 000552 1784124626.612464 write 0x031752 thru 0x035807 (0x40b6 bytes) 000553 1784124626.631216 trunc from 0x03b9c8 to 0x02aaf8 (0x10ecf bytes) 000555 1784124626.653253 mapread 0x00b854 thru 0x016030 (0xa7dd bytes) 000558 1784124626.688016 mapwrite 0x018bcd thru 0x02733d (0xe771 bytes) 000559 1784124626.739389 trunc from 0x00fadd to 0x01c039 (0xc55d bytes) 000560 1784124626.745577 punch from 0x021376 to 0x02aaf8 (0x9783 bytes) 000561 1784124626.752392 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2aaf9): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aaf9): Mirror 7: 0xabeba922 (stale) Mirror 8: 0xabeba922 (stale) Mirror 9: 0xabeba922 (stale) Mirror 10: 0xcc1dc03d (stale) CRC-32 checksum value for chunk [0, 0x2aaf9): Mirror 1: 0x312be652 Mirror 11: 0x312be652 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aaf9: skipped 000562 1784124627.570797 mapread 0x017089 thru 0x021066 (0x9fde bytes) 000563 1784124627.596671 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x217:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2aaf9): Mirror 8: 0xabeba922 (stale) Mirror 9: 0xabeba922 (stale) Mirror 10: 0xcc1dc03d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aaf9): Mirror 8: 0xabeba922 (stale) Mirror 9: 0xabeba922 (stale) Mirror 10: 0xcc1dc03d (stale) CRC-32 checksum value for chunk [0, 0x2aaf9): Mirror 1: 0x312be652 Mirror 11: 0x312be652 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aaf9: skipped 000564 1784124628.141685 mapwrite 0x021d47 thru 0x02586a (0x3b24 bytes) 000565 1784124628.249037 write 0x00b4f7 thru 0x01543e (0x9f48 bytes) 000566 1784124628.263813 mapwrite 0x02dabd thru 0x02fe10 (0x2354 bytes) 000567 1784124628.308036 fallocate 0x012249 thru 0x01a665 (0x841d bytes) 000568 1784124628.317932 mapread 0x019d17 thru 0x02659f (0xc889 bytes) 000569 1784124628.327909 trunc from 0x02fe11 to 0x01b347 (0x14ac9 bytes) 000570 1784124628.346302 write 0x014a12 thru 0x014af9 (0xe8 bytes) 000571 1784124628.347794 write 0x0094d5 thru 0x01671a (0xd246 bytes) 000572 1784124628.355488 punch from 0x01544c to 0x01b347 (0x5efc bytes) 000573 1784124628.371344 write 0x02611d thru 0x029406 (0x32ea bytes) 000574 1784124628.372719 trunc from 0x00365d to 0x00367e (0x22 bytes) 000575 1784124628.386330 MIRROR_RESYNC 5 mirrors 000576 1784124629.407539 read 0x00fcf0 thru 0x017e41 (0x8152 bytes) 000577 1784124629.414868 write 0x00734c thru 0x00867e (0x1333 bytes) 000578 1784124629.449595 mapread 0x0008ac thru 0x000db7 (0x50c bytes) 000579 1784124629.458991 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x29407): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29407): Mirror 8: 0x77a88e82 (stale) Mirror 9: 0x77a88e82 (stale) Mirror 10: 0x77a88e82 (stale) Mirror 11: 0x77a88e82 (stale) CRC-32 checksum value for chunk [0, 0x29407): Mirror 1: 0x690ad598 Mirror 12: 0x690ad598 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29407: skipped 000580 1784124630.210583 mapwrite 0x0376aa thru 0x03ffff (0x8956 bytes) 000581 1784124630.352478 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x217:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x508fb34c (stale) Mirror 9: 0x508fb34c (stale) Mirror 11: 0x508fb34c (stale) Mirror 12: 0x770e5acc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x508fb34c (stale) Mirror 9: 0x508fb34c (stale) Mirror 11: 0x508fb34c (stale) Mirror 12: 0x770e5acc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000582 1784124630.710222 write 0x03c0de thru 0x03ffff (0x3f22 bytes) 000583 1784124630.720258 mapread 0x02a2a5 thru 0x034599 (0xa2f5 bytes) 000584 1784124630.741362 MIRROR_RESYNC 5 mirrors 000585 1784124631.861805 MIRROR_EXTEND to 6 mirrors 000586 1784124632.465674 punch from 0x038f93 to 0x03ffff (0x706d bytes) 000588 1784124632.504841 fallocate 0x033da9 thru 0x033e61 (0xb9 bytes) 000590 1784124632.510579 punch from 0x03a355 to 0x03ffff (0x5cab bytes) 000591 1784124632.526846 mapread 0x012095 thru 0x0171bc (0x5128 bytes) 000592 1784124632.546413 read 0x02aa1d thru 0x02b4e4 (0xac8 bytes) 000593 1784124632.553504 punch from 0x00f150 to 0x01c0d0 (0xcf81 bytes) 000594 1784124632.561450 mapread 0x02f8b3 thru 0x03bd64 (0xc4b2 bytes) 000595 1784124632.583510 punch from 0x01eabd to 0x029c89 (0xb1cd bytes) 000596 1784124632.592737 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x217:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xd1293077 (stale) Mirror 11: 0xd1293077 (stale) Mirror 12: 0xd1293077 (stale) Mirror 13: 0xd1293077 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xd1293077 (stale) Mirror 11: 0xd1293077 (stale) Mirror 12: 0xd1293077 (stale) Mirror 13: 0xd1293077 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000597 1784124632.916813 MIRROR_RESYNC 5 mirrors 000599 1784124634.018902 trunc from 0x040000 to 0x01e636 (0x219c9 bytes) 000600 1784124634.060632 write 0x012687 thru 0x015fed (0x3967 bytes) 000601 1784124634.076570 punch from 0x015756 to 0x017cef (0x259a bytes) 000602 1784124634.086698 punch from 0x001517 to 0x00f807 (0xe2f1 bytes) 000603 1784124634.092410 write 0x0396a7 thru 0x03ffff (0x6959 bytes) 000604 1784124634.094703 trunc from 0x01c69f to 0x0290f0 (0xca52 bytes) 000606 1784124634.099790 mapread 0x000907 thru 0x00c530 (0xbc2a bytes) 000607 1784124634.134302 punch from 0x029485 to 0x02e784 (0x5300 bytes) 000609 1784124634.141375 punch from 0x0243bb to 0x0301f5 (0xbe3b bytes) 000610 1784124634.146158 write 0x02eb27 thru 0x03e9b3 (0xfe8d bytes) 000611 1784124634.159333 fallocate 0x02b5b3 thru 0x02bbfa (0x648 bytes) 000612 1784124634.163197 mapread 0x037f51 thru 0x03a32a (0x23da bytes) 000613 1784124634.164245 fallocate 0x01d70a thru 0x02648b (0x8d82 bytes) 000614 1784124634.170406 mapread 0x022b52 thru 0x02e5dc (0xba8b bytes) 000615 1784124634.190023 mapread 0x02abb7 thru 0x02ae96 (0x2e0 bytes) 000617 1784124634.194219 write 0x00a538 thru 0x0196df (0xf1a8 bytes) 000618 1784124634.204152 MIRROR_RESYNC 5 mirrors 000619 1784124634.985134 MIRROR_RESYNC 5 mirrors 000620 1784124635.316168 mapwrite 0x00274d thru 0x002781 (0x35 bytes) 000621 1784124635.374200 trunc from 0x040000 to 0x033cc2 (0xc33d bytes) 000622 1784124635.414827 trunc from 0x033cc3 to 0x00b9e9 (0x282d9 bytes) 000623 1784124635.432318 trunc from 0x0006fd to 0x00b94b (0xb24f bytes) 000624 1784124635.436870 punch from 0x0022d0 to 0x003d99 (0x1aca bytes) 000625 1784124635.445562 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x217:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb9ea): Mirror 8: 0x516157c8 (stale) Mirror 12: 0x516157c8 (stale) Mirror 13: 0x516157c8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb9ea): Mirror 8: 0x516157c8 (stale) Mirror 12: 0x516157c8 (stale) Mirror 13: 0x516157c8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb9ea: skipped 000626 1784124635.708906 mapread 0x0046b8 thru 0x00b423 (0x6d6c bytes) 000629 1784124635.728243 write 0x021e13 thru 0x02540b (0x35f9 bytes) 000630 1784124635.735723 read 0x0185e5 thru 0x01bd40 (0x375c bytes) 000631 1784124635.744691 trunc from 0x02540c to 0x02536d (0x9e bytes) 000632 1784124635.761810 mapwrite 0x013d29 thru 0x018a30 (0x4d08 bytes) 000633 1784124635.802252 write 0x019a13 thru 0x02734d (0xd93b bytes) 000634 1784124635.805843 trunc from 0x02734e to 0x02deba (0x6b6d bytes) 000635 1784124635.824113 punch from 0x00f70c to 0x0174ca (0x7dbf bytes) 000636 1784124635.828434 mapwrite 0x0009ee thru 0x00b422 (0xaa35 bytes) 000637 1784124635.854749 punch from 0x008984 to 0x009746 (0xdc3 bytes) 000638 1784124635.859386 trunc from 0x02debb to 0x028a96 (0x5424 bytes) 000639 1784124635.871865 punch from 0x01b0d0 to 0x026e44 (0xbd75 bytes) 000640 1784124635.878872 MIRROR_RESYNC 4 mirrors 000641 1784124636.670484 punch from 0x00755e to 0x007584 (0x27 bytes) 000643 1784124636.712466 read 0x01addd thru 0x01ae96 (0xba bytes) 000644 1784124636.717071 trunc from 0x028a97 to 0x03d26c (0x147d6 bytes) 000645 1784124636.734958 read 0x023649 thru 0x02c36f (0x8d27 bytes) 000646 1784124636.745046 trunc from 0x019fa0 to 0x0220a1 (0x8102 bytes) 000647 1784124636.751584 trunc from 0x00fb8b to 0x01c6d2 (0xcb48 bytes) 000648 1784124636.756100 trunc from 0x03c34e to 0x03d26c (0xf1f bytes) 000649 1784124636.759986 trunc from 0x03d26d to 0x03288b (0xa9e1 bytes) 000650 1784124636.773268 mapwrite 0x024ee8 thru 0x02a3ed (0x5506 bytes) 000651 1784124636.798483 mapread 0x00299a thru 0x00adcb (0x8432 bytes) 000652 1784124636.800136 mapread 0x006f08 thru 0x00bc01 (0x4cfa bytes) 000654 1784124636.802532 trunc from 0x03288c to 0x00a399 (0x284f2 bytes) 000655 1784124636.817192 write 0x031b07 thru 0x03f991 (0xde8b bytes) 000656 1784124636.819037 fallocate 0x034ed6 thru 0x03ffff (0xb12a bytes) 000657 1784124636.823213 punch from 0x0176ed to 0x01ce8d (0x57a1 bytes) 000658 1784124636.827655 mapread 0x016f5d thru 0x0197dd (0x2881 bytes) 000659 1784124636.841623 punch from 0x0169e7 to 0x01797e (0xf98 bytes) 000661 1784124636.847319 punch from 0x003abe to 0x008fdb (0x551e bytes) 000662 1784124636.852229 trunc from 0x03f992 to 0x010a92 (0x2eeff bytes) 000664 1784124636.867492 mapread 0x00b64a thru 0x010a92 (0x5449 bytes) 000665 1784124636.913136 mapwrite 0x03bb2a thru 0x03ffff (0x44d6 bytes) 000666 1784124636.986866 trunc from 0x040000 to 0x006c50 (0x393af bytes) 000667 1784124637.004291 read 0x001856 thru 0x001948 (0xf3 bytes) 000668 1784124637.004975 write 0x03d67d thru 0x03ffff (0x2983 bytes) 000669 1784124637.006141 mapwrite 0x00cac7 thru 0x00ffbe (0x34f8 bytes) 000670 1784124637.039779 trunc from 0x040000 to 0x016bd9 (0x29426 bytes) 000671 1784124637.055446 write 0x0190ef thru 0x01a9b6 (0x18c8 bytes) 000672 1784124637.066623 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1a9b7): Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a9b7): Mirror 8: 0x3863d719 (stale) Mirror 12: 0x3863d719 (stale) Mirror 13: 0x3863d719 (stale) CRC-32 checksum value for chunk [0, 0x1a9b7): Mirror 1: 0xb0476db Mirror 14: 0xb0476db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a9b7: skipped 000673 1784124637.782405 fallocate 0x03abee thru 0x03c2e8 (0x16fb bytes) 000674 1784124637.813046 write 0x015268 thru 0x01e53d (0x92d6 bytes) 000675 1784124637.823584 mapread 0x00b318 thru 0x011d0e (0x69f7 bytes) 000676 1784124637.840151 read 0x004dbb thru 0x007da2 (0x2fe8 bytes) 000677 1784124637.845236 mapread 0x005624 thru 0x00d724 (0x8101 bytes) 000678 1784124637.847220 punch from 0x00d234 to 0x016798 (0x9565 bytes) 000680 1784124637.860968 fallocate 0x010df7 thru 0x0135e3 (0x27ed bytes) 000681 1784124637.865258 write 0x018ff1 thru 0x026659 (0xd669 bytes) 000682 1784124637.870356 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2665a): Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2665a): Mirror 8: 0xc84d5439 (stale) Mirror 12: 0xc84d5439 (stale) Mirror 13: 0xc84d5439 (stale) Mirror 14: 0x9d24c644 (stale) CRC-32 checksum value for chunk [0, 0x2665a): Mirror 1: 0x4a811f3f Mirror 15: 0x4a811f3f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2665a: skipped 000684 1784124638.587293 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2665a): Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2665a): Mirror 8: 0xc84d5439 (stale) Mirror 12: 0xc84d5439 (stale) Mirror 13: 0xc84d5439 (stale) Mirror 14: 0x9d24c644 (stale) CRC-32 checksum value for chunk [0, 0x2665a): Mirror 1: 0x4a811f3f Mirror 15: 0x4a811f3f Mirror 16: 0x4a811f3f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2665a: skipped 000685 1784124639.308458 mapwrite 0x010ce6 thru 0x01642f (0x574a bytes) 000686 1784124639.440021 mapwrite 0x019a4e thru 0x023fc9 (0xa57c bytes) 000687 1784124639.479470 MIRROR_RESYNC 7 mirrors 000689 1784124640.794826 read 0x000da6 thru 0x00c5fb (0xb856 bytes) 000690 1784124640.801692 punch from 0x01de29 to 0x02105d (0x3235 bytes) 000692 1784124640.831426 read 0x01bd0b thru 0x01cb7d (0xe73 bytes) 000693 1784124640.848337 fallocate 0x027f40 thru 0x02d2a4 (0x5365 bytes) 000694 1784124640.852826 trunc from 0x01b1b9 to 0x0259d9 (0xa821 bytes) 000695 1784124640.857204 trunc from 0x02d2a5 to 0x01374e (0x19b56 bytes) 000696 1784124640.881026 fallocate 0x037bfe thru 0x038c22 (0x1025 bytes) 000697 1784124640.891489 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x217:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38c23): Mirror 8: 0x6238b673 (stale) Mirror 13: 0x6238b673 (stale) Mirror 14: 0x6238b673 (stale) Mirror 15: 0x6238b673 (stale) Mirror 16: 0x6238b673 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38c23): Mirror 8: 0x6238b673 (stale) Mirror 13: 0x6238b673 (stale) Mirror 14: 0x6238b673 (stale) Mirror 15: 0x6238b673 (stale) Mirror 16: 0x6238b673 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c23: skipped 000698 1784124641.349652 trunc from 0x02bded to 0x034fac (0x91c0 bytes) 000699 1784124641.358750 trunc from 0x037ab6 to 0x038c22 (0x116d bytes) 000700 1784124641.365607 write 0x01fd1e thru 0x0228db (0x2bbe bytes) 000701 1784124641.379850 punch from 0x00890c to 0x00928d (0x982 bytes) 000702 1784124641.388318 punch from 0x02df8e to 0x031263 (0x32d6 bytes) 000703 1784124641.394760 read 0x00fad3 thru 0x017ed3 (0x8401 bytes) 000704 1784124641.408487 mapread 0x0052b8 thru 0x011836 (0xc57f bytes) 000706 1784124641.437581 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x217:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38c23): Mirror 8: 0x6238b673 (stale) Mirror 13: 0x6238b673 (stale) Mirror 15: 0x6238b673 (stale) Mirror 16: 0x6238b673 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38c23): Mirror 8: 0x6238b673 (stale) Mirror 13: 0x6238b673 (stale) Mirror 15: 0x6238b673 (stale) Mirror 16: 0x6238b673 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c23: skipped 000707 1784124641.851019 write 0x00eac2 thru 0x011e22 (0x3361 bytes) 000708 1784124641.870827 trunc from 0x00b79d to 0x00ce94 (0x16f8 bytes) 000709 1784124641.883084 punch from 0x02f2e4 to 0x030d0e (0x1a2b bytes) 000711 1784124641.891238 write 0x0187c3 thru 0x01a469 (0x1ca7 bytes) 000712 1784124641.915453 MIRROR_RESYNC 5 mirrors 000714 1784124643.277133 MIRROR_EXTEND to 6 mirrors 000715 1784124643.862212 trunc from 0x038c23 to 0x0327b7 (0x646b bytes) 000717 1784124643.915248 punch from 0x012c2c to 0x0190a5 (0x647a bytes) 000718 1784124643.920904 trunc from 0x0258fe to 0x02dfff (0x8702 bytes) 000719 1784124643.928418 trunc from 0x0327b8 to 0x02d0ef (0x56c8 bytes) 000720 1784124643.956802 read 0x00f2fe thru 0x01dea5 (0xeba8 bytes) 000721 1784124643.979971 trunc from 0x02d0f0 to 0x03f590 (0x124a1 bytes) 000722 1784124644.010593 trunc from 0x02daa8 to 0x03a991 (0xceea bytes) 000723 1784124644.019401 trunc from 0x03f591 to 0x02380c (0x1bd84 bytes) 000724 1784124644.055501 mapread 0x00e091 thru 0x01a5b9 (0xc529 bytes) 000725 1784124644.063798 write 0x00b195 thru 0x016432 (0xb29e bytes) 000726 1784124644.074103 mapwrite 0x000bd6 thru 0x00213b (0x1566 bytes) 000727 1784124644.111362 trunc from 0x00ca7e to 0x012b3b (0x60be bytes) 000729 1784124644.119676 trunc from 0x01d89e to 0x02380c (0x5f6f bytes) 000730 1784124644.127444 fallocate 0x0027ac thru 0x005580 (0x2dd5 bytes) 000732 1784124644.133488 fallocate 0x00a493 thru 0x016631 (0xc19f bytes) 000733 1784124644.139199 fallocate 0x01a63c thru 0x024edf (0xa8a4 bytes) 000734 1784124644.150297 write 0x00b24c thru 0x01032f (0x50e4 bytes) 000735 1784124644.156059 mapread 0x024c52 thru 0x024edf (0x28e bytes) 000736 1784124644.184545 fallocate 0x016c65 thru 0x018cb0 (0x204c bytes) 000738 1784124644.206092 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x24ee0): Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24ee0): Mirror 8: 0x4df48573 (stale) Mirror 13: 0x4df48573 (stale) Mirror 15: 0x4df48573 (stale) Mirror 16: 0x4df48573 (stale) Mirror 17: 0x4df48573 (stale) CRC-32 checksum value for chunk [0, 0x24ee0): Mirror 1: 0x19823731 Mirror 18: 0x19823731 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24ee0: skipped 000739 1784124644.987832 punch from 0x00f76a to 0x010ef4 (0x178b bytes) 000740 1784124645.026133 punch from 0x0176d5 to 0x024edf (0xd80b bytes) 000741 1784124645.030404 mapread 0x01d6ed thru 0x01eb50 (0x1464 bytes) 000742 1784124645.042451 mapread 0x01e567 thru 0x01ef80 (0xa1a bytes) 000743 1784124645.043559 punch from 0x00d3d6 to 0x00e818 (0x1443 bytes) 000744 1784124645.050751 trunc from 0x0085d5 to 0x014b5d (0xc589 bytes) 000745 1784124645.063180 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x24ee0): Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24ee0): Mirror 8: 0x4df48573 (stale) Mirror 13: 0x4df48573 (stale) Mirror 15: 0x4df48573 (stale) Mirror 16: 0x4df48573 (stale) Mirror 17: 0x4df48573 (stale) Mirror 18: 0x19823731 (stale) CRC-32 checksum value for chunk [0, 0x24ee0): Mirror 1: 0xb5a1199b Mirror 19: 0xb5a1199b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24ee0: skipped 000747 1784124645.934777 MIRROR_RESYNC 8 mirrors 000748 1784124647.091180 read 0x001a7d thru 0x0027e3 (0xd67 bytes) 000749 1784124647.098880 MIRROR_EXTEND to 9 mirrors 000750 1784124647.853859 trunc from 0x008f52 to 0x0162d4 (0xd383 bytes) 000751 1784124647.894192 read 0x023ec0 thru 0x024edf (0x1020 bytes) 000752 1784124647.900489 trunc from 0x01a7ab to 0x01bcd9 (0x152f bytes) 000753 1784124647.905091 punch from 0x0125fe to 0x01d3cf (0xadd2 bytes) 000754 1784124647.911725 mapwrite 0x0341e8 thru 0x039211 (0x502a bytes) 000755 1784124647.973521 mapwrite 0x011d3b thru 0x0157f3 (0x3ab9 bytes) 000756 1784124648.000799 fallocate 0x00f8ee thru 0x013437 (0x3b4a bytes) 000757 1784124648.009346 write 0x00a6e7 thru 0x0100b3 (0x59cd bytes) 000758 1784124648.010557 trunc from 0x039212 to 0x03cc81 (0x3a70 bytes) 000759 1784124648.027113 write 0x015ed3 thru 0x018c6f (0x2d9d bytes) 000760 1784124648.029660 punch from 0x020d38 to 0x026fda (0x62a3 bytes) 000761 1784124648.040518 MIRROR_RESYNC 9 mirrors 000762 1784124649.784539 MIRROR_EXTEND to 10 mirrors 000763 1784124650.560734 mapread 0x03a7dc thru 0x03cc81 (0x24a6 bytes) 000765 1784124650.587550 MIRROR_RESYNC 10 mirrors 000766 1784124651.138332 MIRROR_RESYNC 10 mirrors 000767 1784124651.551466 mapread 0x02f07d thru 0x0337ee (0x4772 bytes) 000768 1784124651.576386 trunc from 0x0289ee to 0x035ee0 (0xd4f3 bytes) 000769 1784124651.610162 fallocate 0x0069ca thru 0x00a1e5 (0x381c bytes) 000770 1784124651.618869 trunc from 0x030a19 to 0x030f44 (0x52c bytes) 000771 1784124651.628085 trunc from 0x037150 to 0x03cc81 (0x5b32 bytes) 000772 1784124651.634005 mapread 0x022ff0 thru 0x0300d2 (0xd0e3 bytes) 000773 1784124651.661759 MIRROR_RESYNC 10 mirrors 000774 1784124653.609135 trunc from 0x03cc82 to 0x0072b2 (0x359cf bytes) 000775 1784124653.674281 trunc from 0x004e5d to 0x0072b2 (0x2456 bytes) 000776 1784124653.678588 mapread 0x0009bf thru 0x006484 (0x5ac6 bytes) 000777 1784124653.708695 fallocate 0x0342aa thru 0x03850a (0x4261 bytes) 000778 1784124653.716965 trunc from 0x0002a6 to 0x008e43 (0x8b9e bytes) 000779 1784124653.722145 punch from 0x03447d to 0x03850a (0x408e bytes) 000780 1784124653.728567 fallocate 0x000b53 thru 0x00be1f (0xb2cd bytes) 000781 1784124653.735126 write 0x013049 thru 0x018b76 (0x5b2e bytes) 000782 1784124653.750720 fallocate 0x01ab0c thru 0x01c1a5 (0x169a bytes) 000783 1784124653.760636 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x217:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3850b): Mirror 8: 0x8ec4288 (stale) Mirror 13: 0x8ec4288 (stale) Mirror 15: 0x8ec4288 (stale) Mirror 16: 0x8ec4288 (stale) Mirror 17: 0x8ec4288 (stale) Mirror 18: 0x8ec4288 (stale) Mirror 20: 0x8ec4288 (stale) Mirror 21: 0x8ec4288 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3850b): Mirror 8: 0x8ec4288 (stale) Mirror 13: 0x8ec4288 (stale) Mirror 15: 0x8ec4288 (stale) Mirror 16: 0x8ec4288 (stale) Mirror 17: 0x8ec4288 (stale) Mirror 18: 0x8ec4288 (stale) Mirror 20: 0x8ec4288 (stale) Mirror 21: 0x8ec4288 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3850b: skipped 000784 1784124654.219462 fallocate 0x02f4c8 thru 0x03a8f0 (0xb429 bytes) 000786 1784124654.224659 read 0x008aca thru 0x00a112 (0x1649 bytes) 000787 1784124654.230530 punch from 0x023637 to 0x02c541 (0x8f0b bytes) 000788 1784124654.236116 write 0x033304 thru 0x0372ca (0x3fc7 bytes) 000789 1784124654.253343 trunc from 0x014927 to 0x016b79 (0x2253 bytes) 000790 1784124654.259113 read 0x0326ee thru 0x0361d8 (0x3aeb bytes) 000791 1784124654.267407 mapread 0x001e95 thru 0x00a4e4 (0x8650 bytes) 000792 1784124654.274092 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x217:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3850b): Mirror 8: 0x8ec4288 (stale) Mirror 13: 0x8ec4288 (stale) Mirror 15: 0x8ec4288 (stale) Mirror 16: 0x8ec4288 (stale) Mirror 17: 0x8ec4288 (stale) Mirror 18: 0x8ec4288 (stale) Mirror 21: 0x8ec4288 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3850b): Mirror 8: 0x8ec4288 (stale) Mirror 13: 0x8ec4288 (stale) Mirror 15: 0x8ec4288 (stale) Mirror 16: 0x8ec4288 (stale) Mirror 17: 0x8ec4288 (stale) Mirror 18: 0x8ec4288 (stale) Mirror 21: 0x8ec4288 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3850b: skipped 000793 1784124654.731808 trunc from 0x03850b to 0x036d1a (0x17f0 bytes) 000794 1784124654.757624 read 0x033cf9 thru 0x036d1a (0x3022 bytes) 000795 1784124654.763862 punch from 0x00d2ee to 0x00f442 (0x2155 bytes) 000796 1784124654.768014 trunc from 0x02b53d to 0x030dad (0x5871 bytes) 000797 1784124654.772069 mapwrite 0x016f8a thru 0x0246f4 (0xd76b bytes) 000798 1784124654.819201 write 0x0327a1 thru 0x035b98 (0x33f8 bytes) 000799 1784124654.825731 mapread 0x01c7ae thru 0x02af58 (0xe7ab bytes) 000800 1784124654.834810 fallocate 0x020aea thru 0x024ce3 (0x41fa bytes) 000801 1784124654.839307 read 0x01940b thru 0x01b3c8 (0x1fbe bytes) 000802 1784124654.839858 trunc from 0x036d1b to 0x00d077 (0x29ca3 bytes) 000803 1784124654.857043 mapwrite 0x0106e3 thru 0x01092c (0x24a bytes) 000805 1784124654.906223 mapread 0x005580 thru 0x01092c (0xb3ad bytes) 000806 1784124654.907491 write 0x0176b0 thru 0x0197af (0x2100 bytes) 000808 1784124654.910748 punch from 0x017c46 to 0x0197af (0x1b6a bytes) 000809 1784124654.926268 mapread 0x013af7 thru 0x0197af (0x5cb9 bytes) 000810 1784124654.934558 punch from 0x0085be to 0x00fee8 (0x792b bytes) 000811 1784124654.940360 read 0x0049c4 thru 0x01123a (0xc877 bytes) 000812 1784124654.953163 trunc from 0x00b8e8 to 0x017f2f (0xc648 bytes) 000813 1784124654.958916 mapwrite 0x03788a thru 0x03d319 (0x5a90 bytes) 000814 1784124655.016721 mapwrite 0x0310fb thru 0x03dc12 (0xcb18 bytes) 000815 1784124655.059879 trunc from 0x03732e to 0x03dc12 (0x68e5 bytes) 000816 1784124655.066866 read 0x03cfd2 thru 0x03dc12 (0xc41 bytes) 000817 1784124655.083447 mapwrite 0x02998b thru 0x033865 (0x9edb bytes) 000818 1784124655.133603 MIRROR_RESYNC 8 mirrors 000819 1784124656.908386 fallocate 0x034716 thru 0x03ffff (0xb8ea bytes) 000820 1784124656.948374 mapwrite 0x01493f thru 0x01f181 (0xa843 bytes) 000821 1784124657.014907 trunc from 0x006cb8 to 0x010bdc (0x9f25 bytes) 000822 1784124657.018548 trunc from 0x03dc13 to 0x01b6bb (0x22557 bytes) 000823 1784124657.052123 punch from 0x001269 to 0x004392 (0x312a bytes) 000824 1784124657.056442 trunc from 0x006b34 to 0x00927b (0x2748 bytes) 000825 1784124657.072122 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1b6bc): Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b6bc): Mirror 8: 0xaed0b306 (stale) Mirror 13: 0xaed0b306 (stale) Mirror 15: 0xaed0b306 (stale) Mirror 16: 0xaed0b306 (stale) Mirror 17: 0xaed0b306 (stale) Mirror 18: 0xaed0b306 (stale) Mirror 21: 0xaed0b306 (stale) CRC-32 checksum value for chunk [0, 0x1b6bc): Mirror 1: 0xa7907ba4 Mirror 22: 0xa7907ba4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b6bc: skipped 000826 1784124657.858545 read 0x00ea23 thru 0x01a8c5 (0xbea3 bytes) 000827 1784124657.868460 mapwrite 0x011777 thru 0x01f722 (0xdfac bytes) 000828 1784124657.970984 trunc from 0x01f723 to 0x00ffd1 (0xf751 bytes) 000829 1784124657.992669 read 0x00f69e thru 0x00ffd1 (0x934 bytes) 000830 1784124657.995373 fallocate 0x024077 thru 0x033207 (0xf191 bytes) 000831 1784124658.001238 read 0x001ed6 thru 0x00ffd1 (0xe0fc bytes) 000832 1784124658.009849 mapwrite 0x02c8d0 thru 0x03a8a4 (0xdfd5 bytes) 000834 1784124658.061982 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x217:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a8a5): Mirror 8: 0x77a30307 (stale) Mirror 13: 0x77a30307 (stale) Mirror 15: 0x77a30307 (stale) Mirror 16: 0x77a30307 (stale) Mirror 17: 0x77a30307 (stale) Mirror 18: 0x77a30307 (stale) Mirror 22: 0x444b31aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a8a5): Mirror 8: 0x77a30307 (stale) Mirror 13: 0x77a30307 (stale) Mirror 15: 0x77a30307 (stale) Mirror 16: 0x77a30307 (stale) Mirror 17: 0x77a30307 (stale) Mirror 18: 0x77a30307 (stale) Mirror 22: 0x444b31aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a8a5: skipped 000835 1784124658.471694 mapread 0x0186ee thru 0x024188 (0xba9b bytes) 000836 1784124658.505998 write 0x022438 thru 0x02ac4e (0x8817 bytes) 000837 1784124658.510496 read 0x01959c thru 0x01a876 (0x12db bytes) 000838 1784124658.511390 trunc from 0x03a8a5 to 0x000f61 (0x39943 bytes) 000839 1784124658.524795 write 0x03dfa7 thru 0x03ffff (0x2059 bytes) 000840 1784124658.540757 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xd436019d (stale) Mirror 13: 0xd436019d (stale) Mirror 15: 0xd436019d (stale) Mirror 16: 0xd436019d (stale) Mirror 17: 0xd436019d (stale) Mirror 18: 0xd436019d (stale) Mirror 22: 0xe89106fe (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x55f033ff Mirror 23: 0x55f033ff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000841 1784124659.302428 fallocate 0x024f88 thru 0x027ed0 (0x2f49 bytes) 000842 1784124659.332561 fallocate 0x020ca3 thru 0x023ca4 (0x3002 bytes) 000843 1784124659.338198 mapread 0x028c9b thru 0x029fbf (0x1325 bytes) 000845 1784124659.346348 trunc from 0x027b78 to 0x0293b7 (0x1840 bytes) 000846 1784124659.355515 mapwrite 0x0129b9 thru 0x0167af (0x3df7 bytes) 000848 1784124659.390734 mapwrite 0x009a2e thru 0x013e48 (0xa41b bytes) 000849 1784124659.422195 trunc from 0x040000 to 0x0013e6 (0x3ec19 bytes) 000850 1784124659.481191 trunc from 0x001366 to 0x0013e6 (0x81 bytes) 000851 1784124659.486832 write 0x01d028 thru 0x0265e8 (0x95c1 bytes) 000852 1784124659.488717 trunc from 0x00ba9d to 0x00ec7b (0x31df bytes) 000853 1784124659.494966 write 0x00332b thru 0x010674 (0xd34a bytes) 000854 1784124659.507309 read 0x01500b thru 0x01ad1a (0x5d10 bytes) 000855 1784124659.514025 trunc from 0x0265e9 to 0x023f10 (0x26d8 bytes) 000856 1784124659.525281 fallocate 0x013311 thru 0x0146ac (0x139c bytes) 000858 1784124659.534857 trunc from 0x004a4a to 0x00f86b (0xae22 bytes) 000859 1784124659.549795 read 0x0026a3 thru 0x009831 (0x718f bytes) 000861 1784124659.563788 mapread 0x00145d thru 0x002289 (0xe2d bytes) 000862 1784124659.564980 write 0x00226f thru 0x00d08b (0xae1d bytes) 000863 1784124659.566805 punch from 0x0074d8 to 0x009013 (0x1b3c bytes) 000864 1784124659.582295 mapwrite 0x01a952 thru 0x01d020 (0x26cf bytes) 000866 1784124659.602281 write 0x01d670 thru 0x02bb57 (0xe4e8 bytes) 000867 1784124659.604355 read 0x005faa thru 0x0085e1 (0x2638 bytes) 000868 1784124659.614399 trunc from 0x02bb58 to 0x03e071 (0x1251a bytes) 000869 1784124659.630305 punch from 0x01dd67 to 0x0207c3 (0x2a5d bytes) 000870 1784124659.646441 write 0x03def1 thru 0x03ffff (0x210f bytes) 000871 1784124659.662367 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xd436019d (stale) Mirror 13: 0xd436019d (stale) Mirror 15: 0xd436019d (stale) Mirror 16: 0xd436019d (stale) Mirror 17: 0xd436019d (stale) Mirror 18: 0xd436019d (stale) Mirror 22: 0xe89106fe (stale) Mirror 23: 0x55f033ff (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe86d6af8 Mirror 24: 0xe86d6af8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000872 1784124660.517659 trunc from 0x03cc48 to 0x03ffff (0x33b8 bytes) 000873 1784124660.551164 trunc from 0x040000 to 0x019951 (0x266ae bytes) 000874 1784124660.575744 mapwrite 0x009350 thru 0x010e02 (0x7ab3 bytes) 000875 1784124660.621123 write 0x0242f8 thru 0x02d64d (0x9356 bytes) 000876 1784124660.622925 read 0x02a195 thru 0x02d64d (0x34b9 bytes) 000877 1784124660.623596 punch from 0x003919 to 0x0103fc (0xcae4 bytes) 000878 1784124660.628710 fallocate 0x019eb2 thru 0x01c2a9 (0x23f8 bytes) 000879 1784124660.633420 trunc from 0x022f81 to 0x02d64d (0xa6cd bytes) 000880 1784124660.648866 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2d64e): Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d64e): Mirror 8: 0xd548609b (stale) Mirror 13: 0xd548609b (stale) Mirror 15: 0xd548609b (stale) Mirror 16: 0xd548609b (stale) Mirror 17: 0xd548609b (stale) Mirror 18: 0xd548609b (stale) Mirror 22: 0xfb099515 (stale) Mirror 23: 0x5bbd4101 (stale) Mirror 24: 0x55aabee3 (stale) CRC-32 checksum value for chunk [0, 0x2d64e): Mirror 1: 0xd7f32994 Mirror 25: 0xd7f32994 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d64e: skipped 000881 1784124661.426236 read 0x00360c thru 0x004d12 (0x1707 bytes) 000882 1784124661.444071 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2d64e): Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d64e): Mirror 8: 0xd548609b (stale) Mirror 13: 0xd548609b (stale) Mirror 15: 0xd548609b (stale) Mirror 16: 0xd548609b (stale) Mirror 17: 0xd548609b (stale) Mirror 18: 0xd548609b (stale) Mirror 22: 0xfb099515 (stale) Mirror 23: 0x5bbd4101 (stale) Mirror 24: 0x55aabee3 (stale) CRC-32 checksum value for chunk [0, 0x2d64e): Mirror 1: 0xd7f32994 Mirror 25: 0xd7f32994 Mirror 26: 0xd7f32994 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d64e: skipped 000884 1784124662.333423 fallocate 0x001d6a thru 0x00c0ce (0xa365 bytes) 000886 1784124662.382662 mapread 0x000eb1 thru 0x010e55 (0xffa5 bytes) 000887 1784124662.410932 mapwrite 0x00737d thru 0x016b50 (0xf7d4 bytes) 000888 1784124662.449732 read 0x0226f0 thru 0x027480 (0x4d91 bytes) 000889 1784124662.457516 mapread 0x001e43 thru 0x00f7de (0xd99c bytes) 000890 1784124662.462071 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x2d64e): Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d64e): Mirror 8: 0xd548609b (stale) Mirror 13: 0xd548609b (stale) Mirror 15: 0xd548609b (stale) Mirror 16: 0xd548609b (stale) Mirror 17: 0xd548609b (stale) Mirror 18: 0xd548609b (stale) Mirror 22: 0xfb099515 (stale) Mirror 23: 0x5bbd4101 (stale) Mirror 24: 0x55aabee3 (stale) Mirror 25: 0xd7f32994 (stale) Mirror 26: 0xd7f32994 (stale) CRC-32 checksum value for chunk [0, 0x2d64e): Mirror 1: 0x84d19494 Mirror 27: 0x84d19494 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d64e: skipped 000891 1784124663.318442 write 0x038b0a thru 0x03ffff (0x74f6 bytes) 000892 1784124663.348802 read 0x02d857 thru 0x02f9eb (0x2195 bytes) 000893 1784124663.353084 read 0x01a09b thru 0x0296c7 (0xf62d bytes) 000894 1784124663.373411 write 0x03c5dc thru 0x03ffff (0x3a24 bytes) 000895 1784124663.374607 punch from 0x0079e1 to 0x00c5fb (0x4c1b bytes) 000896 1784124663.380550 fallocate 0x0351c2 thru 0x03ffff (0xae3e bytes) 000897 1784124663.385232 fallocate 0x035dd3 thru 0x03ffff (0xa22d bytes) 000898 1784124663.397183 MIRROR_RESYNC 13 mirrors 000900 1784124666.806233 trunc from 0x040000 to 0x0265d5 (0x19a2a bytes) 000901 1784124666.850901 trunc from 0x0265d6 to 0x017526 (0xf0af bytes) 000902 1784124666.867603 fallocate 0x013234 thru 0x01e9e6 (0xb7b3 bytes) 000903 1784124666.872484 mapwrite 0x031cde thru 0x03d635 (0xb958 bytes) 000904 1784124666.943816 MIRROR_RESYNC 13 mirrors 000905 1784124670.075465 MIRROR_EXTEND to 14 mirrors 000906 1784124670.803121 mapwrite 0x0142b9 thru 0x0157b9 (0x1501 bytes) 000907 1784124670.899160 mapread 0x01846d thru 0x01f9e9 (0x757d bytes) 000908 1784124670.912115 punch from 0x004c17 to 0x012f5a (0xe344 bytes) 000909 1784124670.918647 mapwrite 0x01a370 thru 0x0285dd (0xe26e bytes) 000910 1784124670.964075 fallocate 0x009c2f thru 0x00f061 (0x5433 bytes) 000911 1784124670.971632 read 0x027971 thru 0x031bc4 (0xa254 bytes) 000912 1784124670.985430 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x3d636): Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d636): Mirror 8: 0xc295845d (stale) Mirror 13: 0xc295845d (stale) Mirror 15: 0xc295845d (stale) Mirror 16: 0xc295845d (stale) Mirror 17: 0xc295845d (stale) Mirror 18: 0xc295845d (stale) Mirror 22: 0xc295845d (stale) Mirror 23: 0xc295845d (stale) Mirror 24: 0xc295845d (stale) Mirror 25: 0xc295845d (stale) Mirror 26: 0xc295845d (stale) Mirror 27: 0xc295845d (stale) Mirror 28: 0xc295845d (stale) CRC-32 checksum value for chunk [0, 0x3d636): Mirror 1: 0x46d45f42 Mirror 29: 0x46d45f42 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d636: skipped 000913 1784124672.028994 trunc from 0x03d636 to 0x015732 (0x27f03 bytes) 000914 1784124672.080230 write 0x03ed84 thru 0x03ffff (0x127c bytes) 000915 1784124672.085679 MIRROR_RESYNC 15 mirrors 000916 1784124675.052404 read 0x02057e thru 0x02ac0d (0xa690 bytes) 000917 1784124675.072707 MIRROR_RESYNC 15 mirrors 000918 1784124675.789286 mapwrite 0x004e48 thru 0x00ea9f (0x9c58 bytes) 000919 1784124675.882623 read 0x025bff thru 0x035924 (0xfd26 bytes) 000920 1784124675.897595 fallocate 0x01f4f6 thru 0x020580 (0x108b bytes) 000921 1784124675.906001 mapread 0x008d91 thru 0x011911 (0x8b81 bytes) 000922 1784124675.919048 trunc from 0x040000 to 0x02cad4 (0x1352b bytes) 000924 1784124675.940502 trunc from 0x02cad5 to 0x03244b (0x5977 bytes) 000925 1784124675.955631 mapread 0x023217 thru 0x0238b1 (0x69b bytes) 000926 1784124675.963401 write 0x0181fb thru 0x0235b7 (0xb3bd bytes) 000927 1784124675.966044 mapwrite 0x01f9ea thru 0x025288 (0x589f bytes) 000928 1784124675.985477 punch from 0x0078ab to 0x00bc8a (0x43e0 bytes) 000929 1784124675.992513 write 0x02cbe1 thru 0x038a69 (0xbe89 bytes) 000930 1784124675.999473 MIRROR_SPLIT mirror 10 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x217:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38a6a): Mirror 8: 0xfae86b4b (stale) Mirror 13: 0xfae86b4b (stale) Mirror 15: 0xfae86b4b (stale) Mirror 16: 0xfae86b4b (stale) Mirror 17: 0xfae86b4b (stale) Mirror 18: 0xfae86b4b (stale) Mirror 22: 0xfae86b4b (stale) Mirror 23: 0xfae86b4b (stale) Mirror 24: 0xfae86b4b (stale) Mirror 26: 0xfae86b4b (stale) Mirror 27: 0xfae86b4b (stale) Mirror 28: 0xfae86b4b (stale) Mirror 29: 0xfae86b4b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38a6a): Mirror 8: 0xfae86b4b (stale) Mirror 13: 0xfae86b4b (stale) Mirror 15: 0xfae86b4b (stale) Mirror 16: 0xfae86b4b (stale) Mirror 17: 0xfae86b4b (stale) Mirror 18: 0xfae86b4b (stale) Mirror 22: 0xfae86b4b (stale) Mirror 23: 0xfae86b4b (stale) Mirror 24: 0xfae86b4b (stale) Mirror 26: 0xfae86b4b (stale) Mirror 27: 0xfae86b4b (stale) Mirror 28: 0xfae86b4b (stale) Mirror 29: 0xfae86b4b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38a6a: skipped 000931 1784124676.595909 read 0x037114 thru 0x037668 (0x555 bytes) 000932 1784124676.605735 mapwrite 0x0276b8 thru 0x028962 (0x12ab bytes) 000933 1784124676.626504 trunc from 0x038a6a to 0x013ef7 (0x24b72 bytes) 000934 1784124676.651796 fallocate 0x029b5b thru 0x035bd0 (0xc076 bytes) 000935 1784124676.657749 fallocate 0x03aa56 thru 0x03ffff (0x55aa bytes) 000936 1784124676.661415 trunc from 0x032518 to 0x0395fc (0x70e5 bytes) 000937 1784124676.669603 trunc from 0x040000 to 0x021007 (0x1eff8 bytes) 000938 1784124676.684205 trunc from 0x009cd1 to 0x0195a1 (0xf8d1 bytes) 000939 1784124676.693299 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x21008): Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21008): Mirror 8: 0x1373b9c0 (stale) Mirror 13: 0x1373b9c0 (stale) Mirror 15: 0x1373b9c0 (stale) Mirror 16: 0x1373b9c0 (stale) Mirror 17: 0x1373b9c0 (stale) Mirror 18: 0x1373b9c0 (stale) Mirror 22: 0x1373b9c0 (stale) Mirror 23: 0x1373b9c0 (stale) Mirror 24: 0x1373b9c0 (stale) Mirror 26: 0x1373b9c0 (stale) Mirror 27: 0x1373b9c0 (stale) Mirror 28: 0x1373b9c0 (stale) Mirror 29: 0x1373b9c0 (stale) CRC-32 checksum value for chunk [0, 0x21008): Mirror 1: 0x5a28b612 Mirror 30: 0x5a28b612 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21008: skipped 000940 1784124677.676191 fallocate 0x0141e1 thru 0x015f09 (0x1d29 bytes) 000941 1784124677.721041 fallocate 0x02f0cf thru 0x03b16f (0xc0a1 bytes) 000942 1784124677.725618 trunc from 0x03b170 to 0x034d07 (0x6468 bytes) 000944 1784124677.748794 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x34d08): Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34d08): Mirror 8: 0x76433343 (stale) Mirror 13: 0x76433343 (stale) Mirror 15: 0x76433343 (stale) Mirror 16: 0x76433343 (stale) Mirror 17: 0x76433343 (stale) Mirror 18: 0x76433343 (stale) Mirror 22: 0x76433343 (stale) Mirror 23: 0x76433343 (stale) Mirror 24: 0x76433343 (stale) Mirror 26: 0x76433343 (stale) Mirror 27: 0x76433343 (stale) Mirror 28: 0x76433343 (stale) Mirror 29: 0x76433343 (stale) Mirror 30: 0x27dba385 (stale) CRC-32 checksum value for chunk [0, 0x34d08): Mirror 1: 0x27dba385 Mirror 31: 0x27dba385 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34d08: skipped 000946 1784124678.666510 MIRROR_SPLIT mirror 2 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x217:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34d08): Mirror 8: 0x76433343 (stale) Mirror 15: 0x76433343 (stale) Mirror 16: 0x76433343 (stale) Mirror 17: 0x76433343 (stale) Mirror 18: 0x76433343 (stale) Mirror 22: 0x76433343 (stale) Mirror 23: 0x76433343 (stale) Mirror 24: 0x76433343 (stale) Mirror 26: 0x76433343 (stale) Mirror 27: 0x76433343 (stale) Mirror 28: 0x76433343 (stale) Mirror 29: 0x76433343 (stale) Mirror 30: 0x27dba385 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34d08): Mirror 8: 0x76433343 (stale) Mirror 15: 0x76433343 (stale) Mirror 16: 0x76433343 (stale) Mirror 17: 0x76433343 (stale) Mirror 18: 0x76433343 (stale) Mirror 22: 0x76433343 (stale) Mirror 23: 0x76433343 (stale) Mirror 24: 0x76433343 (stale) Mirror 26: 0x76433343 (stale) Mirror 27: 0x76433343 (stale) Mirror 28: 0x76433343 (stale) Mirror 29: 0x76433343 (stale) Mirror 30: 0x27dba385 (stale) CRC-32 checksum value for chunk [0, 0x34d08): Mirror 1: 0x27dba385 Mirror 31: 0x27dba385 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34d08: skipped 000947 1784124679.679879 mapwrite 0x03bbf6 thru 0x03c6cc (0xad7 bytes) 000948 1784124679.760159 punch from 0x036665 to 0x03c6cc (0x6068 bytes) 000949 1784124679.764491 trunc from 0x03c6cd to 0x03e84c (0x2180 bytes) 000950 1784124679.780049 mapread 0x019e28 thru 0x027971 (0xdb4a bytes) 000951 1784124679.804248 fallocate 0x03cf2b thru 0x03ffff (0x30d5 bytes) 000952 1784124679.811032 write 0x0127ed thru 0x016fe1 (0x47f5 bytes) 000954 1784124679.813461 write 0x03e048 thru 0x03ffff (0x1fb8 bytes) 000955 1784124679.819529 read 0x033271 thru 0x037ac5 (0x4855 bytes) 000956 1784124679.830096 mapread 0x029423 thru 0x033341 (0x9f1f bytes) 000957 1784124679.842240 write 0x00a1af thru 0x01617d (0xbfcf bytes) 000958 1784124679.849346 MIRROR_SPLIT mirror 5 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x217:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xb95a1b38 (stale) Mirror 15: 0xb95a1b38 (stale) Mirror 16: 0xb95a1b38 (stale) Mirror 17: 0xb95a1b38 (stale) Mirror 22: 0xb95a1b38 (stale) Mirror 23: 0xb95a1b38 (stale) Mirror 24: 0xb95a1b38 (stale) Mirror 26: 0xb95a1b38 (stale) Mirror 27: 0xb95a1b38 (stale) Mirror 28: 0xb95a1b38 (stale) Mirror 29: 0xb95a1b38 (stale) Mirror 30: 0x69f7ebcc (stale) Mirror 31: 0x69f7ebcc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xb95a1b38 (stale) Mirror 15: 0xb95a1b38 (stale) Mirror 16: 0xb95a1b38 (stale) Mirror 17: 0xb95a1b38 (stale) Mirror 22: 0xb95a1b38 (stale) Mirror 23: 0xb95a1b38 (stale) Mirror 24: 0xb95a1b38 (stale) Mirror 26: 0xb95a1b38 (stale) Mirror 27: 0xb95a1b38 (stale) Mirror 28: 0xb95a1b38 (stale) Mirror 29: 0xb95a1b38 (stale) Mirror 30: 0x69f7ebcc (stale) Mirror 31: 0x69f7ebcc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000959 1784124680.477318 write 0x0305d7 thru 0x03c170 (0xbb9a bytes) 000960 1784124680.498309 punch from 0x03b445 to 0x03f970 (0x452c bytes) 000961 1784124680.523554 punch from 0x0069ce to 0x013eda (0xd50d bytes) 000962 1784124680.532789 read 0x005241 thru 0x00dfa8 (0x8d68 bytes) 000963 1784124680.553755 punch from 0x026d0a to 0x033c13 (0xcf0a bytes) 000964 1784124680.560757 mapread 0x0003d0 thru 0x00a67e (0xa2af bytes) 000965 1784124680.595923 mapwrite 0x010cf2 thru 0x01d507 (0xc816 bytes) 000966 1784124680.657277 trunc from 0x040000 to 0x0186ef (0x27910 bytes) 000967 1784124680.674127 read 0x006ca1 thru 0x0072ee (0x64e bytes) 000968 1784124680.675078 trunc from 0x0186f0 to 0x00ec92 (0x9a5d bytes) 000969 1784124680.694259 MIRROR_RESYNC 14 mirrors 000970 1784124682.985658 mapread 0x00408d thru 0x00e6ff (0xa673 bytes) 000972 1784124683.017293 MIRROR_RESYNC 14 mirrors 000973 1784124683.576568 fallocate 0x0078ca thru 0x00fd74 (0x84ab bytes) 000974 1784124683.624587 punch from 0x0017cc to 0x00b56d (0x9da2 bytes) 000975 1784124683.629364 punch from 0x00a529 to 0x00fd74 (0x584c bytes) 000976 1784124683.634817 trunc from 0x00bd9d to 0x00fd74 (0x3fd8 bytes) 000977 1784124683.645052 write 0x037898 thru 0x03a065 (0x27ce bytes) 000978 1784124683.646281 punch from 0x027090 to 0x0295d7 (0x2548 bytes) 000980 1784124683.661119 MIRROR_RESYNC 14 mirrors 000981 1784124685.457301 trunc from 0x03a066 to 0x00ae89 (0x2f1dc bytes) 000982 1784124685.522596 trunc from 0x00ae8a to 0x008d26 (0x2163 bytes) 000983 1784124685.570656 trunc from 0x002100 to 0x008d26 (0x6c27 bytes) 000984 1784124685.594715 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x8d27): Mirror 8: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8d27): Mirror 8: 0x1f3a4faf (stale) Mirror 15: 0x1f3a4faf (stale) Mirror 16: 0x1f3a4faf (stale) Mirror 17: 0x1f3a4faf (stale) Mirror 22: 0x1f3a4faf (stale) Mirror 23: 0x1f3a4faf (stale) Mirror 24: 0x1f3a4faf (stale) Mirror 26: 0x1f3a4faf (stale) Mirror 27: 0x1f3a4faf (stale) Mirror 28: 0x1f3a4faf (stale) Mirror 29: 0x1f3a4faf (stale) Mirror 30: 0x1f3a4faf (stale) Mirror 31: 0x1f3a4faf (stale) CRC-32 checksum value for chunk [0, 0x8d27): Mirror 1: 0x1f3a4faf Mirror 32: 0x1f3a4faf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8d27: skipped 000985 1784124686.462183 fallocate 0x003567 thru 0x005426 (0x1ec0 bytes) 000986 1784124686.515321 mapwrite 0x039d99 thru 0x03b17a (0x13e2 bytes) 000987 1784124686.590637 trunc from 0x00f12f to 0x01d8ac (0xe77e bytes) 000988 1784124686.596874 fallocate 0x02df94 thru 0x03b036 (0xd0a3 bytes) 000990 1784124686.603868 read 0x00a7d7 thru 0x019d00 (0xf52a bytes) 000992 1784124686.613790 write 0x003138 thru 0x0074a8 (0x4371 bytes) 000993 1784124686.623125 trunc from 0x009480 to 0x00b54f (0x20d0 bytes) 000994 1784124686.630520 punch from 0x02312d to 0x02e32f (0xb203 bytes) 000995 1784124686.639602 trunc from 0x02a54f to 0x02b591 (0x1043 bytes) 000996 1784124686.653438 fallocate 0x00f9eb thru 0x011df3 (0x2409 bytes) 000997 1784124686.665164 mapwrite 0x02749b thru 0x02d465 (0x5fcb bytes) 000998 1784124686.697133 mapwrite 0x021bec thru 0x03045d (0xe872 bytes) 000999 1784124686.723365 trunc from 0x03b17b to 0x0246e6 (0x16a94 bytes) 001000 1784124686.743355 mapread 0x01e2d1 thru 0x024425 (0x6155 bytes) All operations completed A-OK! PASS 70a (104s) == sanity-flr test 71: check mirror extend parent fid ==== 10:11:30 (1784124690) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x261:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (6s) == sanity-flr test 72: check append on FLR file ========== 10:11:36 (1784124696) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.170403 s, 12.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838796 s, 12.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (6s) == sanity-flr test 200a: stress test ===================== 10:11:42 (1784124702) Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre2 Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5102400 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3037792 .. Extending file size to 8057792 .. Extending file size to 8409408 .. Extending file size to 8791456 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8709184 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9179072 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 61690 61691 61692 61695 61696 done 192.168.203.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg323-client.virtnet /mnt/lustre2 (opts:) 192.168.203.123@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg323-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (85s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 10:13:07 (1784124787) Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre2 Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4297152 .. Extending file size to 8640576 .. Extending file size to 7081856 .. Extending file size to 7290880 .. Extending file size to 8381440 .. Extending file size to 8622400 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8714336 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8822240 .. Waiting 63149 63151 63152 63153 63156 done 192.168.203.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg323-client.virtnet /mnt/lustre2 (opts:) 192.168.203.123@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg323-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (74s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 10:14:21 (1784124861) Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2288 1269016 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2372 1268932 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18544 3588476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24032 3582988 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42576 7171464 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2288 1269016 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2372 1268932 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18544 3588476 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24032 3582988 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42576 7171464 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0790408 s, 26.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.162613 s, 12.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x95fa783d (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg323-client.virtnet /mnt/lustre2 (opts:) PASS 200c (11s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 10:14:33 (1784124873) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0864322 s, 24.3 MB/s PASS 202 (5s) == sanity-flr test 203: mirror file preserve mirror ID === 10:14:38 (1784124878) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x60:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } PASS 203 (4s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 10:14:42 (1784124882) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0762938 s, 27.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x302:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x300:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x301:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (5s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 10:14:48 (1784124888) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0644768 s, 32.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } PASS 204b (4s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 10:14:52 (1784124892) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0727231 s, 28.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } PASS 204c (28s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 10:15:20 (1784124920) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg323-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440166 s, 23.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x303:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x303:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x302:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x303:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x303:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x302:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x304:0x0] } PASS 204d (43s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 10:16:03 (1784124963) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg323-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0723494 s, 29.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (43s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 10:16:46 (1784125006) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg323-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg323-server: Pool lustre.test_204f created Adding targets to pool oleg323-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg323-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0732866 s, 28.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg323-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg323-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg323-server: Pool lustre.test_204f destroyed PASS 204f (55s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 10:17:41 (1784125061) PASS 205a (4s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 10:17:45 (1784125065) PASS 205b (5s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 10:17:50 (1784125070) oleg323-server: Pool lustre.test_206 created Adding targets to pool oleg323-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg323-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg323-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg323-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg323-server: Pool lustre.test_206 destroyed PASS 206 (18s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 10:18:09 (1784125089) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445757 s, 23.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187181 s, 56.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x6f83145b Mirror 3: 0x6f83145b PASS 207 (6s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 10:18:15 (1784125095) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 10:18:17 (1784125097) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 10:18:19 (1784125099) Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0287786 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1ee0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1a50000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1a50000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1a50000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x12a0000 Read this (4 bytes): BBBB 192.168.203.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg323-client.virtnet /mnt/lustre2 (opts:) PASS 209a (24s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 10:18:43 (1784125123) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00753335 s, 1.1 MB/s the very first read read with warm cache PASS 209b (4s) == sanity-flr test 210a: handle broken mirrored lovea ==== 10:18:48 (1784125128) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198653 s, 52.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x81 lmm_fid: [0x200000405:0x81:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x81 lmm_fid: [0x200000405:0x81:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11c:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611265 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-15 10:18:48.000000000 -0400 Modify: 2026-07-15 10:18:49.000000000 -0400 Change: 2026-07-15 10:18:49.000000000 -0400 Birth: 2026-07-15 10:18:47.000000000 -0400 PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 10:18:52 (1784125132) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192749 s, 54.4 MB/s oleg323-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg323-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg323-server: O/280000401/d28/13c: File not found by ext2_lookup PASS 210b (14s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 10:19:06 (1784125146) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.107985 s, 379 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0174769 s, 234 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 15 10:19 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x86:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 15 10:19 /mnt/lustre/f211.sanity-flr PASS 211 (4s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 10:19:10 (1784125150) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (4s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 10:19:14 (1784125154) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (4s) == sanity-flr test complete, duration 1140 sec =========== 10:19:18 (1784125158) === sanity-flr: start cleanup 10:19:19 (1784125159) === === sanity-flr: finish cleanup 10:19:24 (1784125164) ===