-----============= acceptance-small: sanity-flr ============----- Wed Aug 26 19:19:55 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg449-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 19:20:12 (1787786412) === oleg449-client.virtnet: executing check_config_client /mnt/lustre oleg449-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg449-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8f65c395f800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8f65c395f800.idle_timeout=debug disable quota as required oleg449-server: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 19:20:49 (1787786449) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8210] == sanity-flr test 0a: lfs mirror create with -N option == 19:20:51 (1787786451) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 19:21:05 (1787786465) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 19:21:08 (1787786468) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 19:21:10 (1787786470) PASS 0d (21s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 19:21:32 (1787786492) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 19:21:35 (1787786495) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 19:21:39 (1787786499) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8f65c395f800.stats=clear osc.lustre-OST0001-osc-ffff8f65c395f800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 19:21:47 (1787786507) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 19:21:58 (1787786518) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 19:22:06 (1787786526) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 19:22:10 (1787786530) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 19:22:18 (1787786538) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 19:22:27 (1787786547) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 19:22:35 (1787786555) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 19:22:42 (1787786562) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 19:22:50 (1787786570) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 19:23:01 (1787786581) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 19:23:09 (1787786589) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 1.18455 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820906 s, 12.8 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 19:23:19 (1787786599) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.211673 s, 29.7 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 19:23:26 (1787786606) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 19:23:37 (1787786617) Stopping /mnt/lustre-ost1 (opts:) on oleg449-server oleg449-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f65c395f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f65c395f800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg449-server: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg449-server: oleg449-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg449-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg449-server oleg449-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f65c395f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f65c395f800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg449-server: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg449-server: oleg449-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg449-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (74s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 19:24:51 (1787786691) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.164171 s, 12.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg449-server oleg449-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f65c395f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f65c395f800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg449-server: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg449-server: oleg449-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg449-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (39s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 19:25:30 (1787786730) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg449-server oleg449-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f65c395f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f65c395f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg449-server: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg449-server: oleg449-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg449-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg449-server oleg449-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f65c395f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f65c395f800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg449-server: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg449-server: oleg449-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg449-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (72s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 19:26:42 (1787786802) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.21557 s, 16.3 MB/s -rw-r--r-- 1 root root 52428800 Aug 26 19:26 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.084s user 0m0.000s sys 0m0.173s Stopping /mnt/lustre-ost1 (opts:) on oleg449-server oleg449-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f65c395f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f65c395f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.898s user 0m0.001s sys 0m0.196s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg449-server: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg449-server: oleg449-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg449-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg449-server oleg449-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f65c395f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f65c395f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.486s user 0m0.001s sys 0m0.159s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg449-server: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg449-server: oleg449-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg449-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg449-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f65c395f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f65c395f800.ost_server_uuid in FULL state after 0 sec PASS 33b (89s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 19:28:12 (1787786892) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 19:28:17 (1787786897) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (5s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 19:28:23 (1787786903) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 19:28:26 (1787786906) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 19:28:33 (1787786913) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.577162 s, 21.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 19:28:45 (1787786925) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468512 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448642 s, 23.4 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 19:28:52 (1787786932) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 19:29:00 (1787786940) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00820361 s, 125 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00121289 s, 844 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00135655 s, 755 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00116095 s, 882 kB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 19:29:09 (1787786949) create test files with size 15736985 .. e0829419a3cdfefce8aa68fd6bb05d4b - 111b96167731394f2f0605af26637a7c - 28e1d5d3adb85d26025e16cf685ad694 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg449-client.virtnet /mnt/lustre (opts:) Starting client: oleg449-client.virtnet: -o user_xattr,flock 192.168.204.149@tcp:/lustre /mnt/lustre PASS 37 (19s) == sanity-flr test 38: resync ============================ 19:29:29 (1787786969) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 19:29:42 (1787786982) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 19:29:50 (1787786990) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852494 s, 12.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343748 s, 30.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 19:29:59 (1787786999) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100806 s, 20.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.41907 s, 10.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.247224 s, 8.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 19:30:16 (1787787016) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 19:30:19 (1787787019) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 19:30:23 (1787787023) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429144 s, 24.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 43c: read prefer uncompressed mirror == 19:30:32 (1787787032) llite.lustre-ffff8f65d3413000.enable_compression=1 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (8s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 19:30:39 (1787787039) llite.lustre-ffff8f65d3413000.enable_compression=1 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039119 s, 26.8 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046605 s, 22.5 MB/s **verify components PASS 43d (8s) == sanity-flr test 44a: lfs mirror split check =========== 19:30:47 (1787787047) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.291796 s, 10.8 MB/s PASS 44a (9s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 19:30:56 (1787787056) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 19:31:06 (1787787066) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.344485 s, 30.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0xa:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg449-client.virtnet /mnt/lustre (opts:) Starting client: oleg449-client.virtnet: -o user_xattr,flock 192.168.204.149@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 19:31:16 (1787787076) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8f65d01af000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.259947 s, 40.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg449-client.virtnet /mnt/lustre (opts:) Starting client: oleg449-client.virtnet: -o user_xattr,flock 192.168.204.149@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8f65c3959000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 19:31:29 (1787787089) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (35s) == sanity-flr test 46: Verify setstripe --copy option ==== 19:32:05 (1787787125) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (34s) == sanity-flr test 47: Verify mirror obj alloc =========== 19:32:39 (1787787159) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 19:32:42 (1787787162) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.209288 s, 15.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.156429 s, 20.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xb3fb2623 (nosync,stale) Mirror 2: 0xb3fb2623 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 6ea57803caf3006e60a331488c354b3a - mirror 1 checksum: d3f5e6282ea6711ef3a9f1564bbaef19 - mirror 2 checksum: 6ea57803caf3006e60a331488c354b3a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-26T19:32:45-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-26T19:32:45-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 6ea57803caf3006e60a331488c354b3a - mirror 1 checksum: d3f5e6282ea6711ef3a9f1564bbaef19 - mirror 2 checksum: d3f5e6282ea6711ef3a9f1564bbaef19 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-26T19:32:45-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-26T19:32:45-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 19:32:52 (1787787172) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289837 s, 36.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 19:32:59 (1787787179) Failing mds1 on oleg449-server Stopping /mnt/lustre-mds1 (opts:) on oleg449-server 19:33:10 (1787787190) shut down facet: mds1 facet_host: oleg449-server facet_failover_host: oleg449-server Failover mds1 to oleg449-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg449-server: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Started lustre-MDT0000 19:33:43 (1787787223) targets are mounted 19:33:43 (1787787223) facet_failover done oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268476 s, 39.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd8:0x0] } PASS 50A (59s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 19:33:59 (1787787239) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.03005 s, 682 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0257762 s, 795 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 19:34:08 (1787787248) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.081092 s, 253 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0094426 s, 2.2 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 19:34:18 (1787787258) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00925046 s, 1.8 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 19:34:25 (1787787265) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0556745 s, 368 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (6s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 19:34:31 (1787787271) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.034148 s, 120 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0285638 s, 143 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 19:34:38 (1787787278) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 19:35:00 (1787787300) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (25s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 19:35:25 (1787787325) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 19:35:47 (1787787347) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 19:36:10 (1787787370) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 19:36:16 (1787787376) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 58826 mirror split pid 58827 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 58826 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 58827 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 19:37:23 (1787787443) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 7445 truncating to largest ever: 0x330ce 000001 1787787445.531690 trunc from 00000000 to 0x0330cd (0x330ce bytes) 000003 1787787445.546459 punch from 0x00bf4a to 0x018899 (0xc950 bytes) 000004 1787787445.551216 read 0x011804 thru 0x01a5f4 (0x8df1 bytes) 000006 1787787445.559553 mapread 0x000193 thru 0x00c911 (0xc77f bytes) 000007 1787787445.581996 trunc from 0x0330ce to 0x00b3b0 (0x27d1d bytes) 000008 1787787445.595167 write 0x0094f0 thru 0x0194ab (0xffbc bytes) fallocating to largest ever: 0x40000 000009 1787787445.598927 fallocate 0x03bfbd thru 0x03ffff (0x4043 bytes) 000011 1787787445.604275 mapread 0x015b90 thru 0x0194ab (0x391c bytes) 000012 1787787445.605961 write 0x01f61b thru 0x024217 (0x4bfd bytes) 000013 1787787445.607707 read 0x0102d4 thru 0x01fa66 (0xf793 bytes) 000014 1787787445.615140 trunc from 0x024218 to 0x03555e (0x11347 bytes) 000015 1787787445.631220 mapwrite 0x03ff5c thru 0x03ffff (0xa4 bytes) 000016 1787787445.675338 trunc from 0x040000 to 0x01978f (0x26870 bytes) 000017 1787787445.695587 mapwrite 0x00fa11 thru 0x01bb15 (0xc105 bytes) 000019 1787787445.746594 trunc from 0x01bb16 to 0x028968 (0xce53 bytes) 000021 1787787445.765887 fallocate 0x03cfd6 thru 0x03ffff (0x302a bytes) 000022 1787787445.770556 write 0x0009b3 thru 0x009be0 (0x922e bytes) 000023 1787787445.771947 fallocate 0x01fc22 thru 0x020751 (0xb30 bytes) 000024 1787787445.776841 punch from 0x03a7a7 to 0x03d268 (0x2ac2 bytes) 000025 1787787445.780740 fallocate 0x0103e2 thru 0x010d41 (0x960 bytes) 000026 1787787445.784926 mapread 0x00520f thru 0x00ef0a (0x9cfc bytes) 000028 1787787445.786812 read 0x00a4e5 thru 0x01354e (0x906a bytes) 000029 1787787445.787810 read 0x007900 thru 0x016dff (0xf500 bytes) 000030 1787787445.788905 write 0x0241e1 thru 0x02578a (0x15aa bytes) 000031 1787787445.797479 mapread 0x030f1b thru 0x03833e (0x7424 bytes) 000033 1787787445.800690 write 0x020f75 thru 0x02fa94 (0xeb20 bytes) 000034 1787787445.806129 MIRROR_EXTEND to 2 mirrors 000035 1787787446.346624 write 0x014b03 thru 0x0211af (0xc6ad bytes) 000037 1787787446.389011 read 0x01a008 thru 0x01fbf3 (0x5bec bytes) 000038 1787787446.389612 trunc from 0x0174db to 0x018f79 (0x1a9f bytes) 000039 1787787446.406024 trunc from 0x040000 to 0x005d9d (0x3a262 bytes) 000040 1787787446.430312 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) 000041 1787787446.554580 trunc from 0x005d9e to 0x029e39 (0x2409c bytes) 000043 1787787446.573983 trunc from 0x029e3a to 0x01bb0b (0xe32e bytes) 000044 1787787446.590208 fallocate 0x011528 thru 0x0204c6 (0xef9f bytes) 000046 1787787446.598547 read 0x00d254 thru 0x016f5a (0x9d07 bytes) 000047 1787787446.609389 write 0x037781 thru 0x03ffff (0x887f bytes) 000048 1787787446.610810 mapread 0x01876e thru 0x01f9a6 (0x7239 bytes) 000049 1787787446.627156 trunc from 0x040000 to 0x0397e2 (0x681d bytes) 000051 1787787446.644338 fallocate 0x033bd7 thru 0x03ffff (0xc429 bytes) 000052 1787787446.650561 read 0x037473 thru 0x038fa7 (0x1b35 bytes) 000053 1787787446.651542 trunc from 0x02535e to 0x033720 (0xe3c3 bytes) 000054 1787787446.655959 read 0x0298d5 thru 0x033668 (0x9d94 bytes) 000055 1787787446.664938 read 0x02e2da thru 0x032447 (0x416e bytes) 000056 1787787446.665548 mapwrite 0x015dcf thru 0x01824a (0x247c bytes) 000058 1787787446.681160 mapwrite 0x008b0c thru 0x0173f8 (0xe8ed bytes) 000059 1787787446.698930 trunc from 0x025351 to 0x029f90 (0x4c40 bytes) 000060 1787787446.711838 trunc from 0x0314f0 to 0x0397e2 (0x82f3 bytes) 000061 1787787446.719870 punch from 0x022e62 to 0x029b39 (0x6cd8 bytes) 000063 1787787446.728506 read 0x015c97 thru 0x01b10c (0x5476 bytes) 000064 1787787446.729499 write 0x016141 thru 0x01e302 (0x81c2 bytes) 000065 1787787446.734599 write 0x00d473 thru 0x0135e8 (0x6176 bytes) 000066 1787787446.739094 mapread 0x03058b thru 0x034e2b (0x48a1 bytes) 000067 1787787446.777382 MIRROR_EXTEND to 2 mirrors 000069 1787787447.593889 trunc from 0x0397e3 to 0x0043a6 (0x3543c bytes) 000070 1787787447.637524 write 0x03e4d4 thru 0x03ffff (0x1b2c bytes) 000071 1787787447.639999 trunc from 0x040000 to 0x03671b (0x98e4 bytes) 000072 1787787447.661110 mapread 0x01a3ab thru 0x0239a8 (0x95fe bytes) 000073 1787787447.680226 punch from 0x002b07 to 0x00ae27 (0x8321 bytes) skipping zero size read 000075 1787787447.692825 read 0x000283 thru 0x00ebe5 (0xe963 bytes) 000077 1787787447.722106 trunc from 0x0014ba to 0x00986c (0x83b3 bytes) 000079 1787787447.731118 mapwrite 0x017695 thru 0x01d488 (0x5df4 bytes) 000080 1787787447.761563 read 0x020148 thru 0x027d92 (0x7c4b bytes) 000081 1787787447.818895 mapwrite 0x0282c3 thru 0x02e12a (0x5e68 bytes) 000082 1787787447.854883 mapwrite 0x018105 thru 0x018f7b (0xe77 bytes) 000083 1787787447.874530 trunc from 0x03671c to 0x005225 (0x314f6 bytes) 000085 1787787447.896187 read 0x001d09 thru 0x005225 (0x351d bytes) 000086 1787787447.897407 fallocate 0x028bcd thru 0x02a2e9 (0x171d bytes) 000087 1787787447.913548 write 0x027229 thru 0x036ca2 (0xfa7a bytes) 000088 1787787447.929548 trunc from 0x018808 to 0x01c301 (0x3afa bytes) 000089 1787787447.935120 trunc from 0x036ca3 to 0x009f75 (0x2cd2d bytes) 000090 1787787447.949420 mapread 0x0046d8 thru 0x009f75 (0x589e bytes) 000091 1787787447.950391 write 0x03361f thru 0x03c4ee (0x8ed0 bytes) 000092 1787787447.959289 write 0x004ec9 thru 0x01258b (0xd6c3 bytes) 000094 1787787447.970124 mapread 0x02f70e thru 0x03aca7 (0xb59a bytes) 000095 1787787447.971140 fallocate 0x03fe91 thru 0x03ffff (0x16f bytes) 000096 1787787447.976246 fallocate 0x0115b7 thru 0x01bf6f (0xa9b9 bytes) 000097 1787787447.981253 write 0x0361ab thru 0x03ffff (0x9e55 bytes) 000098 1787787447.989978 MIRROR_RESYNC 2 mirrors 000099 1787787448.725194 punch from 0x035e27 to 0x037f9f (0x2179 bytes) 000100 1787787448.776747 mapwrite 0x006f69 thru 0x01379e (0xc836 bytes) 000101 1787787448.834253 mapread 0x0207a2 thru 0x02ead9 (0xe338 bytes) 000102 1787787448.870238 trunc from 0x00a1ea to 0x017399 (0xd1b0 bytes) 000103 1787787448.892052 trunc from 0x028773 to 0x031dd4 (0x9662 bytes) 000104 1787787448.898432 trunc from 0x01a643 to 0x01cd52 (0x2710 bytes) 000105 1787787448.904138 trunc from 0x02383e to 0x027368 (0x3b2b bytes) 000106 1787787448.911004 mapread 0x018227 thru 0x0270bb (0xee95 bytes) 000107 1787787448.936842 write 0x020907 thru 0x02d1cf (0xc8c9 bytes) 000108 1787787448.961036 read 0x001584 thru 0x00e264 (0xcce1 bytes) 000109 1787787448.996666 MIRROR_RESYNC 2 mirrors 000111 1787787449.672752 write 0x01af97 thru 0x028a93 (0xdafd bytes) 000112 1787787449.712140 fallocate 0x0172ad thru 0x020a6e (0x97c2 bytes) 000113 1787787449.722357 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) 000114 1787787449.854235 write 0x02a5ba thru 0x03a1fe (0xfc45 bytes) 000115 1787787449.878511 trunc from 0x040000 to 0x038a80 (0x757f bytes) 000117 1787787449.899979 mapread 0x028641 thru 0x02a081 (0x1a41 bytes) 000118 1787787449.909753 fallocate 0x0097cc thru 0x0167f9 (0xd02e bytes) 000119 1787787449.914519 trunc from 0x038a81 to 0x016400 (0x22680 bytes) 000120 1787787449.936440 trunc from 0x014c6c to 0x016400 (0x1795 bytes) 000121 1787787449.943598 write 0x00d9a2 thru 0x010838 (0x2e97 bytes) 000122 1787787449.957245 trunc from 0x0158b9 to 0x016400 (0xb48 bytes) 000123 1787787449.963203 punch from 0x00b3ae to 0x016400 (0xb053 bytes) 000124 1787787449.981622 MIRROR_EXTEND to 2 mirrors 000125 1787787450.794369 mapread 0x00f2bc thru 0x016400 (0x7145 bytes) 000127 1787787450.813151 trunc from 0x001b71 to 0x007632 (0x5ac2 bytes) 000128 1787787450.853912 punch from 0x00418c to 0x00cd6c (0x8be1 bytes) 000129 1787787450.862823 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x16401): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16401): Mirror 2: 0x4acb6db3 (stale) CRC-32 checksum value for chunk [0, 0x16401): Mirror 1: 0x7a0bc2b9 Mirror 3: 0x7a0bc2b9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16401: skipped 000130 1787787451.522465 trunc from 0x016401 to 0x0146cb (0x1d35 bytes) 000131 1787787451.572104 fallocate 0x0104e7 thru 0x01ae24 (0xa93e bytes) 000132 1787787451.579810 punch from 0x010ee6 to 0x013f6a (0x3085 bytes) 000133 1787787451.584739 fallocate 0x035411 thru 0x039576 (0x4166 bytes) 000134 1787787451.607225 read 0x003bdc thru 0x00c2a3 (0x86c8 bytes) 000135 1787787451.622274 fallocate 0x012099 thru 0x01d48a (0xb3f2 bytes) 000136 1787787451.627487 mapread 0x003ef8 thru 0x00b22e (0x7337 bytes) 000137 1787787451.630906 trunc from 0x01d48b to 0x02d763 (0x102d9 bytes) 000138 1787787451.672249 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d764): Mirror 3: 0xd18bf77f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d764): Mirror 3: 0xd18bf77f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d764: skipped 000139 1787787452.050357 mapread 0x01bbad thru 0x025177 (0x95cb bytes) 000141 1787787452.074487 fallocate 0x02dfdf thru 0x03d119 (0xf13b bytes) 000142 1787787452.078887 fallocate 0x0066fa thru 0x00cebe (0x67c5 bytes) 000143 1787787452.083220 write 0x03ea19 thru 0x03ffff (0x15e7 bytes) 000144 1787787452.090003 trunc from 0x0392d6 to 0x03ffff (0x6d2a bytes) 000145 1787787452.098973 trunc from 0x040000 to 0x00a070 (0x35f8f bytes) 000146 1787787452.116946 mapwrite 0x03b189 thru 0x03ffff (0x4e77 bytes) 000147 1787787452.156942 mapwrite 0x02912b thru 0x02ebb5 (0x5a8b bytes) 000148 1787787452.184349 mapread 0x02cc2a thru 0x038e1b (0xc1f2 bytes) 000149 1787787452.189640 trunc from 0x02f098 to 0x0357fa (0x6763 bytes) 000150 1787787452.194081 write 0x02365c thru 0x0309b5 (0xd35a bytes) 000151 1787787452.200059 trunc from 0x040000 to 0x03b6c8 (0x4937 bytes) 000152 1787787452.216156 mapwrite 0x02b3d1 thru 0x02ed66 (0x3996 bytes) 000153 1787787452.231478 fallocate 0x028506 thru 0x0322a6 (0x9da1 bytes) 000154 1787787452.236168 trunc from 0x019ac5 to 0x0275e8 (0xdb24 bytes) 000155 1787787452.240615 read 0x038f78 thru 0x03b6c8 (0x2751 bytes) 000156 1787787452.241207 mapwrite 0x00fa8e thru 0x016702 (0x6c75 bytes) 000158 1787787452.269969 fallocate 0x021272 thru 0x02af48 (0x9cd7 bytes) 000159 1787787452.276115 write 0x0325c2 thru 0x036242 (0x3c81 bytes) 000160 1787787452.288177 write 0x0204f6 thru 0x02670b (0x6216 bytes) 000161 1787787452.309624 mapwrite 0x0146b5 thru 0x015d70 (0x16bc bytes) 000162 1787787452.325407 trunc from 0x03b6c9 to 0x01b391 (0x20337 bytes) 000163 1787787452.342345 punch from 0x00b293 to 0x014e25 (0x9b93 bytes) 000166 1787787452.350855 trunc from 0x0070bc to 0x009a7f (0x29c4 bytes) 000167 1787787452.355619 mapwrite 0x0273cb thru 0x02d6e7 (0x631d bytes) 000168 1787787452.398626 write 0x01eb4e thru 0x0200d5 (0x1588 bytes) 000169 1787787452.406171 punch from 0x00b9d6 to 0x010465 (0x4a90 bytes) 000170 1787787452.410527 trunc from 0x02d6e8 to 0x002267 (0x2b480 bytes) 000171 1787787452.440613 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2268): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2268): Mirror 3: 0x906d6b02 (stale) CRC-32 checksum value for chunk [0, 0x2268): Mirror 1: 0x906d6b02 Mirror 4: 0x906d6b02 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2268: skipped 000172 1787787453.090932 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) 000173 1787787453.327909 mapread 0x0005a3 thru 0x002267 (0x1cc5 bytes) 000174 1787787453.345144 fallocate 0x0203cb thru 0x028464 (0x809a bytes) 000175 1787787453.389308 trunc from 0x028465 to 0x02fc10 (0x77ac bytes) 000176 1787787453.409222 punch from 0x020ccd to 0x021110 (0x444 bytes) 000177 1787787453.417738 MIRROR_RESYNC 2 mirrors 000178 1787787454.031961 MIRROR_RESYNC 2 mirrors 000179 1787787454.418482 MIRROR_RESYNC 2 mirrors 000180 1787787454.926011 fallocate 0x01a867 thru 0x02829c (0xda36 bytes) 000181 1787787454.989742 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2fc11): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fc11): Mirror 4: 0x21f934bd (stale) CRC-32 checksum value for chunk [0, 0x2fc11): Mirror 1: 0x21f934bd Mirror 5: 0x21f934bd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fc11: skipped 000183 1787787455.826870 mapread 0x00e1ee thru 0x013407 (0x521a bytes) 000184 1787787455.848095 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) 000185 1787787456.115370 trunc from 0x02fc11 to 0x02b27e (0x4992 bytes) 000186 1787787456.226083 punch from 0x001f18 to 0x0032ab (0x1394 bytes) 000187 1787787456.255423 fallocate 0x03c3e7 thru 0x03ffff (0x3c19 bytes) 000188 1787787456.285488 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x11c4fb77 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x11c4fb77 Mirror 6: 0x11c4fb77 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000189 1787787457.017380 trunc from 0x01aff9 to 0x0240b3 (0x90bb bytes) 000190 1787787457.080000 MIRROR_RESYNC 3 mirrors 000191 1787787457.683571 mapwrite 0x02d8d7 thru 0x036252 (0x897c bytes) 000192 1787787457.771598 mapwrite 0x019106 thru 0x01e2a8 (0x51a3 bytes) 000193 1787787457.833144 punch from 0x017f72 to 0x021ade (0x9b6d bytes) 000194 1787787457.838360 punch from 0x03184e to 0x032038 (0x7eb bytes) 000195 1787787457.849495 fallocate 0x01a795 thru 0x022804 (0x8070 bytes) 000197 1787787457.861178 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x11c4fb77 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x11c4fb77 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000198 1787787458.268065 fallocate 0x027611 thru 0x028e02 (0x17f2 bytes) 000199 1787787458.274504 punch from 0x036378 to 0x036af2 (0x77b bytes) 000200 1787787458.279640 mapread 0x032e28 thru 0x0331e6 (0x3bf bytes) 000201 1787787458.288837 mapwrite 0x004a06 thru 0x01447a (0xfa75 bytes) 000202 1787787458.326979 fallocate 0x03f496 thru 0x03ffff (0xb6a bytes) 000203 1787787458.331354 trunc from 0x012d30 to 0x01be93 (0x9164 bytes) 000204 1787787458.339321 read 0x020dc5 thru 0x028de3 (0x801f bytes) 000205 1787787458.348516 punch from 0x037c79 to 0x03b19f (0x3527 bytes) 000206 1787787458.352832 read 0x033a5d thru 0x03ffff (0xc5a3 bytes) 000207 1787787458.361239 mapwrite 0x02f4ef thru 0x035b66 (0x6678 bytes) 000209 1787787458.375225 write 0x000a2f thru 0x00a72e (0x9d00 bytes) 000210 1787787458.380501 trunc from 0x00cd36 to 0x012eea (0x61b5 bytes) 000211 1787787458.397312 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) 000213 1787787458.575401 mapwrite 0x00ed36 thru 0x016e66 (0x8131 bytes) 000214 1787787458.627000 trunc from 0x02305f to 0x02e8ff (0xb8a1 bytes) 000215 1787787458.639399 punch from 0x015811 to 0x01e4ac (0x8c9c bytes) 000217 1787787458.648131 mapread 0x0109f3 thru 0x010ad2 (0xe0 bytes) 000218 1787787458.655227 mapwrite 0x03ed05 thru 0x03fac9 (0xdc5 bytes) 000219 1787787458.692286 mapread 0x003a86 thru 0x00a9bf (0x6f3a bytes) 000220 1787787458.704813 trunc from 0x000bd8 to 0x00fa6c (0xee95 bytes) 000221 1787787458.728652 trunc from 0x040000 to 0x0209d6 (0x1f629 bytes) 000223 1787787458.764080 MIRROR_EXTEND to 2 mirrors 000224 1787787459.433292 fallocate 0x003c1e thru 0x008bcd (0x4fb0 bytes) 000225 1787787459.479266 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x209d7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x209d7): Mirror 2: 0x79230ddf (stale) CRC-32 checksum value for chunk [0, 0x209d7): Mirror 1: 0x79230ddf Mirror 3: 0x79230ddf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x209d7: skipped 000226 1787787460.481517 read 0x00384b thru 0x009493 (0x5c49 bytes) 000227 1787787460.495612 mapwrite 0x004f74 thru 0x008354 (0x33e1 bytes) 000228 1787787460.620477 read 0x01456b thru 0x0209d6 (0xc46c bytes) 000229 1787787460.632617 trunc from 0x0209d7 to 0x02dbea (0xd214 bytes) 000230 1787787460.663696 mapwrite 0x032a1d thru 0x0333fa (0x9de bytes) 000231 1787787460.715190 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x333fb): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x333fb): Mirror 2: 0xb0cbd6a5 (stale) Mirror 3: 0xb0cbd6a5 (stale) CRC-32 checksum value for chunk [0, 0x333fb): Mirror 1: 0x4156b573 Mirror 4: 0x4156b573 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x333fb: skipped 000232 1787787461.666307 mapread 0x02a8ba thru 0x0333fa (0x8b41 bytes) 000233 1787787461.703662 trunc from 0x0333fb to 0x006501 (0x2cef9 bytes) 000234 1787787461.755900 trunc from 0x006502 to 0x003b37 (0x29ca bytes) 000235 1787787461.772810 mapwrite 0x0353fa thru 0x03ffff (0xac06 bytes) 000236 1787787461.820022 punch from 0x032824 to 0x03ffff (0xd7dc bytes) 000237 1787787461.826830 mapwrite 0x032975 thru 0x03beec (0x9578 bytes) 000238 1787787461.875621 read 0x0216e9 thru 0x0277a2 (0x60ba bytes) 000239 1787787461.890690 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x373d140e (stale) Mirror 3: 0x373d140e (stale) Mirror 4: 0x34d6dde4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x352b9d41 Mirror 5: 0x352b9d41 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000240 1787787462.780598 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x373d140e (stale) Mirror 3: 0x373d140e (stale) Mirror 4: 0x34d6dde4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x352b9d41 Mirror 5: 0x352b9d41 Mirror 6: 0x352b9d41 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000241 1787787463.801645 fallocate 0x005db7 thru 0x0097a7 (0x39f1 bytes) 000243 1787787463.860999 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x373d140e (stale) Mirror 3: 0x373d140e (stale) Mirror 4: 0x34d6dde4 (stale) Mirror 5: 0x352b9d41 (stale) Mirror 6: 0x352b9d41 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x352b9d41 Mirror 7: 0x352b9d41 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000244 1787787464.885708 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x373d140e (stale) Mirror 3: 0x373d140e (stale) Mirror 4: 0x34d6dde4 (stale) Mirror 5: 0x352b9d41 (stale) Mirror 6: 0x352b9d41 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x352b9d41 Mirror 7: 0x352b9d41 Mirror 8: 0x352b9d41 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000245 1787787465.895437 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x373d140e (stale) Mirror 3: 0x373d140e (stale) Mirror 4: 0x34d6dde4 (stale) Mirror 5: 0x352b9d41 (stale) Mirror 6: 0x352b9d41 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x352b9d41 Mirror 7: 0x352b9d41 Mirror 8: 0x352b9d41 Mirror 9: 0x352b9d41 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000246 1787787467.051330 read 0x001ecc thru 0x007bbc (0x5cf1 bytes) 000247 1787787467.059942 fallocate 0x00e065 thru 0x015822 (0x77be bytes) 000248 1787787467.109310 punch from 0x0052f8 to 0x012f04 (0xdc0d bytes) 000249 1787787467.116547 trunc from 0x040000 to 0x03ad02 (0x52fd bytes) 000251 1787787467.139561 trunc from 0x03ad03 to 0x00542d (0x358d5 bytes) 000252 1787787467.158152 trunc from 0x00542e to 0x014d8c (0xf95f bytes) 000253 1787787467.171120 fallocate 0x007f0e thru 0x017a00 (0xfaf3 bytes) 000254 1787787467.176431 write 0x00154a thru 0x001d21 (0x7d8 bytes) 000255 1787787467.181933 mapread 0x009807 thru 0x00b719 (0x1f13 bytes) 000256 1787787467.197289 fallocate 0x01aed9 thru 0x023a36 (0x8b5e bytes) 000257 1787787467.202149 read 0x01e679 thru 0x023a36 (0x53be bytes) 000258 1787787467.209413 read 0x008b5f thru 0x00adea (0x228c bytes) 000259 1787787467.210170 mapwrite 0x0246b2 thru 0x026f5a (0x28a9 bytes) 000260 1787787467.243825 fallocate 0x02e398 thru 0x033749 (0x53b2 bytes) 000262 1787787467.249397 read 0x01313d thru 0x0184cd (0x5391 bytes) 000263 1787787467.250323 read 0x00d4fd thru 0x01a2b6 (0xcdba bytes) 000264 1787787467.251046 trunc from 0x03374a to 0x01985a (0x19eef bytes) 000265 1787787467.271188 trunc from 0x01985b to 0x00c105 (0xd755 bytes) 000266 1787787467.285195 trunc from 0x00c106 to 0x01c177 (0x10072 bytes) 000267 1787787467.297586 trunc from 0x01c178 to 0x01b524 (0xc53 bytes) 000269 1787787467.311999 write 0x02e953 thru 0x031136 (0x27e4 bytes) 000270 1787787467.313075 write 0x0125c9 thru 0x01cb0a (0xa542 bytes) 000271 1787787467.324081 punch from 0x02d443 to 0x031136 (0x3cf4 bytes) 000273 1787787467.339173 write 0x02e05b thru 0x03c760 (0xe706 bytes) 000274 1787787467.345506 mapwrite 0x016a45 thru 0x01b493 (0x4a4f bytes) 000275 1787787467.365831 fallocate 0x03119a thru 0x03ffff (0xee66 bytes) 000276 1787787467.370735 write 0x008f1d thru 0x00e51c (0x5600 bytes) 000277 1787787467.371757 mapwrite 0x0173f2 thru 0x01e1de (0x6ded bytes) 000278 1787787467.393143 mapread 0x00f810 thru 0x0130d3 (0x38c4 bytes) 000279 1787787467.394165 read 0x03b2f4 thru 0x03ffff (0x4d0c bytes) 000280 1787787467.402617 trunc from 0x040000 to 0x02d59d (0x12a62 bytes) 000281 1787787467.418261 punch from 0x00715f to 0x00d5de (0x6480 bytes) 000282 1787787467.422800 punch from 0x018eb7 to 0x01e269 (0x53b3 bytes) 000283 1787787467.428395 mapwrite 0x03fc6e thru 0x03ffff (0x392 bytes) 000284 1787787467.459605 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x373d140e (stale) Mirror 3: 0x373d140e (stale) Mirror 4: 0x34d6dde4 (stale) Mirror 6: 0x352b9d41 (stale) Mirror 7: 0x352b9d41 (stale) Mirror 8: 0x352b9d41 (stale) Mirror 9: 0x352b9d41 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x373d140e (stale) Mirror 3: 0x373d140e (stale) Mirror 4: 0x34d6dde4 (stale) Mirror 6: 0x352b9d41 (stale) Mirror 7: 0x352b9d41 (stale) Mirror 8: 0x352b9d41 (stale) Mirror 9: 0x352b9d41 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000285 1787787467.904270 mapread 0x005670 thru 0x011b76 (0xc507 bytes) 000286 1787787467.937052 fallocate 0x03dfc5 thru 0x03ffff (0x203b bytes) 000287 1787787467.941898 mapwrite 0x01186b thru 0x01d311 (0xbaa7 bytes) 000288 1787787467.964836 fallocate 0x031c12 thru 0x03ff41 (0xe330 bytes) 000289 1787787467.969393 fallocate 0x025c56 thru 0x02ab7b (0x4f26 bytes) 000290 1787787467.973581 trunc from 0x040000 to 0x009a6b (0x36594 bytes) 000291 1787787467.989500 read 0x005ad1 thru 0x009a6b (0x3f9b bytes) 000292 1787787467.990170 write 0x03d2e4 thru 0x03ffff (0x2d1c bytes) 000293 1787787467.991455 mapwrite 0x03423e thru 0x03ffff (0xbdc2 bytes) 000294 1787787468.017698 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x373d140e (stale) Mirror 3: 0x373d140e (stale) Mirror 4: 0x34d6dde4 (stale) Mirror 7: 0x352b9d41 (stale) Mirror 8: 0x352b9d41 (stale) Mirror 9: 0x352b9d41 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x373d140e (stale) Mirror 3: 0x373d140e (stale) Mirror 4: 0x34d6dde4 (stale) Mirror 7: 0x352b9d41 (stale) Mirror 8: 0x352b9d41 (stale) Mirror 9: 0x352b9d41 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000295 1787787468.522158 mapwrite 0x014827 thru 0x0164e8 (0x1cc2 bytes) 000296 1787787468.552051 trunc from 0x00d561 to 0x019d0d (0xc7ad bytes) 000297 1787787468.563505 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x373d140e (stale) Mirror 4: 0x34d6dde4 (stale) Mirror 7: 0x352b9d41 (stale) Mirror 8: 0x352b9d41 (stale) Mirror 9: 0x352b9d41 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x373d140e (stale) Mirror 4: 0x34d6dde4 (stale) Mirror 7: 0x352b9d41 (stale) Mirror 8: 0x352b9d41 (stale) Mirror 9: 0x352b9d41 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000298 1787787469.009447 trunc from 0x016ba3 to 0x01e0d7 (0x7535 bytes) 000299 1787787469.015926 trunc from 0x030c8a to 0x03659f (0x5916 bytes) 000300 1787787469.020762 fallocate 0x0293ec thru 0x02e9f7 (0x560c bytes) 000301 1787787469.025382 mapwrite 0x02535e thru 0x02dd2f (0x89d2 bytes) 000302 1787787469.080332 read 0x0144a4 thru 0x021e6d (0xd9ca bytes) 000303 1787787469.108387 trunc from 0x040000 to 0x027ccb (0x18334 bytes) 000304 1787787469.174833 trunc from 0x0112ba to 0x01ccc8 (0xba0f bytes) 000305 1787787469.197591 trunc from 0x00c69e to 0x01c5d7 (0xff3a bytes) 000306 1787787469.202478 write 0x02f066 thru 0x031aed (0x2a88 bytes) 000307 1787787469.203642 fallocate 0x034c67 thru 0x035b51 (0xeeb bytes) 000308 1787787469.213122 trunc from 0x01fa8e to 0x021ec6 (0x2439 bytes) 000309 1787787469.221439 write 0x011c33 thru 0x01de00 (0xc1ce bytes) 000310 1787787469.237499 MIRROR_RESYNC 6 mirrors 000311 1787787470.979749 punch from 0x0106da to 0x01d6d8 (0xcfff bytes) 000312 1787787471.016023 mapwrite 0x00fcee thru 0x01fbf9 (0xff0c bytes) 000313 1787787471.071039 fallocate 0x02db94 thru 0x03b002 (0xd46f bytes) 000314 1787787471.076222 trunc from 0x01f786 to 0x0216e5 (0x1f60 bytes) 000315 1787787471.084243 punch from 0x00c521 to 0x00ff84 (0x3a64 bytes) 000316 1787787471.089185 punch from 0x008af2 to 0x0113a7 (0x88b6 bytes) 000317 1787787471.094641 mapread 0x02ff06 thru 0x03820a (0x8305 bytes) 000318 1787787471.110054 mapread 0x017ff7 thru 0x02663f (0xe649 bytes) 000319 1787787471.119192 write 0x00d376 thru 0x01c1bb (0xee46 bytes) 000320 1787787471.125234 mapread 0x03065e thru 0x03b002 (0xa9a5 bytes) 000321 1787787471.126319 read 0x025c64 thru 0x0343b3 (0xe750 bytes) 000322 1787787471.131576 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3b003): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b003): Mirror 3: 0x1f32676f (stale) Mirror 4: 0x1f32676f (stale) Mirror 7: 0x1f32676f (stale) Mirror 8: 0x1f32676f (stale) Mirror 9: 0x1f32676f (stale) CRC-32 checksum value for chunk [0, 0x3b003): Mirror 1: 0x34f9ca56 Mirror 10: 0x34f9ca56 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b003: skipped 000323 1787787472.324519 trunc from 0x002bab to 0x00968a (0x6ae0 bytes) 000324 1787787472.400005 read 0x010288 thru 0x01ea7d (0xe7f6 bytes) 000325 1787787472.408848 fallocate 0x0087f4 thru 0x0170d7 (0xe8e4 bytes) 000326 1787787472.416213 trunc from 0x036c2d to 0x03b002 (0x43d6 bytes) 000327 1787787472.424204 punch from 0x039a96 to 0x03b002 (0x156d bytes) 000328 1787787472.429078 read 0x004a6f thru 0x012e22 (0xe3b4 bytes) 000329 1787787472.457103 trunc from 0x022180 to 0x02c1a4 (0xa025 bytes) 000330 1787787472.464483 read 0x033a64 thru 0x03b002 (0x759f bytes) 000331 1787787472.465523 write 0x030627 thru 0x0307e3 (0x1bd bytes) 000332 1787787472.481374 trunc from 0x03b003 to 0x037a9c (0x3566 bytes) 000333 1787787472.541808 mapread 0x029d3b thru 0x0324b8 (0x877e bytes) 000334 1787787472.568014 mapwrite 0x006c41 thru 0x0144d1 (0xd891 bytes) 000335 1787787472.597049 mapread 0x011844 thru 0x01326c (0x1a29 bytes) 000336 1787787472.599615 trunc from 0x02a7f2 to 0x033a64 (0x9273 bytes) 000337 1787787472.606662 write 0x01147b thru 0x0125ad (0x1133 bytes) 000338 1787787472.608959 mapread 0x01b2d0 thru 0x025430 (0xa161 bytes) 000340 1787787472.610991 trunc from 0x037a9d to 0x035391 (0x270b bytes) 000342 1787787472.627643 punch from 0x031499 to 0x0333ce (0x1f36 bytes) 000344 1787787472.635548 mapwrite 0x02c778 thru 0x03b044 (0xe8cd bytes) 000345 1787787472.732220 mapread 0x01ae44 thru 0x0284b5 (0xd672 bytes) 000346 1787787472.733588 write 0x00b866 thru 0x00f1c4 (0x395f bytes) 000347 1787787472.737557 punch from 0x005c1e to 0x00fd8d (0xa170 bytes) 000348 1787787472.752508 trunc from 0x01d36d to 0x020fb3 (0x3c47 bytes) 000349 1787787472.758069 trunc from 0x006f78 to 0x013e0a (0xce93 bytes) 000350 1787787472.762410 punch from 0x01df22 to 0x02481c (0x68fb bytes) 000351 1787787472.780832 read 0x028a25 thru 0x02e647 (0x5c23 bytes) 000352 1787787472.797072 trunc from 0x03b045 to 0x01eee7 (0x1c15d bytes) 000353 1787787472.815044 fallocate 0x01dd94 thru 0x02ad07 (0xcf74 bytes) 000354 1787787472.838289 MIRROR_RESYNC 7 mirrors 000355 1787787474.385532 mapread 0x0080a0 thru 0x00b719 (0x367a bytes) 000356 1787787474.402024 trunc from 0x010684 to 0x01eee7 (0xe864 bytes) 000357 1787787474.444207 punch from 0x00841e to 0x013928 (0xb50b bytes) 000358 1787787474.450158 trunc from 0x0043a8 to 0x00c9ec (0x8645 bytes) 000359 1787787474.456763 punch from 0x017efb to 0x01970b (0x1811 bytes) 000360 1787787474.463329 fallocate 0x01a75a thru 0x026442 (0xbce9 bytes) 000361 1787787474.468264 fallocate 0x010ecb thru 0x017429 (0x655f bytes) 000362 1787787474.473925 mapwrite 0x019e5f thru 0x02247d (0x861f bytes) 000363 1787787474.533363 MIRROR_RESYNC 7 mirrors 000365 1787787476.039281 trunc from 0x02247e to 0x00d389 (0x150f4 bytes) 000366 1787787476.095762 fallocate 0x0022f6 thru 0x00ec72 (0xc97d bytes) 000367 1787787476.113067 trunc from 0x00ec73 to 0x014085 (0x5413 bytes) 000368 1787787476.136630 write 0x01de31 thru 0x02759e (0x976e bytes) 000369 1787787476.146209 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2759f): Mirror 3: 0x78dda96d (stale) Mirror 4: 0x78dda96d (stale) Mirror 7: 0x78dda96d (stale) Mirror 9: 0x78dda96d (stale) Mirror 10: 0x78dda96d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2759f): Mirror 3: 0x78dda96d (stale) Mirror 4: 0x78dda96d (stale) Mirror 7: 0x78dda96d (stale) Mirror 9: 0x78dda96d (stale) Mirror 10: 0x78dda96d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2759f: skipped 000370 1787787476.561748 punch from 0x00e7b9 to 0x016d94 (0x85dc bytes) 000371 1787787476.571497 MIRROR_RESYNC 6 mirrors 000372 1787787477.506720 fallocate 0x03d2a8 thru 0x03ffff (0x2d58 bytes) 000373 1787787477.550390 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x7eeaa819 (stale) Mirror 7: 0x7eeaa819 (stale) Mirror 9: 0x7eeaa819 (stale) Mirror 10: 0x7eeaa819 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x7eeaa819 (stale) Mirror 7: 0x7eeaa819 (stale) Mirror 9: 0x7eeaa819 (stale) Mirror 10: 0x7eeaa819 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000374 1787787477.897456 read 0x009dc6 thru 0x010308 (0x6543 bytes) 000375 1787787477.913746 trunc from 0x040000 to 0x02e3c2 (0x11c3d bytes) 000377 1787787477.939502 mapread 0x013ee0 thru 0x022d54 (0xee75 bytes) 000378 1787787477.991365 mapread 0x025a9f thru 0x026481 (0x9e3 bytes) 000379 1787787477.992657 write 0x019df5 thru 0x027b82 (0xdd8e bytes) 000380 1787787478.004389 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e3c3): Mirror 4: 0x67bf68a3 (stale) Mirror 9: 0x67bf68a3 (stale) Mirror 10: 0x67bf68a3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e3c3): Mirror 4: 0x67bf68a3 (stale) Mirror 9: 0x67bf68a3 (stale) Mirror 10: 0x67bf68a3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e3c3: skipped 000381 1787787478.385395 mapread 0x0147a6 thru 0x01a376 (0x5bd1 bytes) 000382 1787787478.443291 fallocate 0x012c48 thru 0x021eab (0xf264 bytes) 000383 1787787478.466347 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e3c3): Mirror 4: 0x67bf68a3 (stale) Mirror 9: 0x67bf68a3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e3c3): Mirror 4: 0x67bf68a3 (stale) Mirror 9: 0x67bf68a3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e3c3: skipped 000384 1787787478.775515 trunc from 0x02e3c3 to 0x03442a (0x6068 bytes) 000385 1787787478.797208 mapread 0x0086f9 thru 0x012fa3 (0xa8ab bytes) 000386 1787787478.822932 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3442b): Mirror 4: 0xb0a12582 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3442b): Mirror 4: 0xb0a12582 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3442b: skipped 000387 1787787479.064454 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) 000388 1787787479.197696 trunc from 0x03442b to 0x001c0b (0x3281f bytes) 000389 1787787479.223291 read 0x0003c1 thru 0x001c0b (0x184b bytes) 000391 1787787479.228973 read 0x000a76 thru 0x001c0b (0x1196 bytes) 000392 1787787479.230273 mapwrite 0x02d060 thru 0x036d1d (0x9cbe bytes) 000394 1787787479.291337 punch from 0x00186b to 0x00bf88 (0xa71e bytes) 000396 1787787479.309909 mapread 0x0271fd thru 0x0312e0 (0xa0e4 bytes) 000397 1787787479.311133 write 0x0069b8 thru 0x01169b (0xace4 bytes) 000398 1787787479.323731 fallocate 0x036901 thru 0x03ffff (0x96ff bytes) 000399 1787787479.331512 write 0x03c5d4 thru 0x03ffff (0x3a2c bytes) 000400 1787787479.334185 fallocate 0x003b07 thru 0x005ad2 (0x1fcc bytes) 000401 1787787479.340672 write 0x00cef6 thru 0x01381e (0x6929 bytes) 000402 1787787479.343944 punch from 0x002983 to 0x007c0d (0x528b bytes) 000403 1787787479.381461 read 0x00cb07 thru 0x00efca (0x24c4 bytes) 000404 1787787479.382575 trunc from 0x040000 to 0x01433d (0x2bcc2 bytes) 000405 1787787479.404813 write 0x00632a thru 0x00c408 (0x60df bytes) 000406 1787787479.423696 MIRROR_EXTEND to 2 mirrors 000407 1787787480.291496 mapread 0x001666 thru 0x0073d3 (0x5d6e bytes) 000408 1787787480.311320 trunc from 0x01433e to 0x03f210 (0x2aed3 bytes) 000409 1787787480.411362 trunc from 0x03f211 to 0x03d5a4 (0x1c6c bytes) 000410 1787787480.439822 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d5a5): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d5a5): Mirror 2: 0x3174d22d (stale) CRC-32 checksum value for chunk [0, 0x3d5a5): Mirror 1: 0x3174d22d Mirror 3: 0x3174d22d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d5a5: skipped 000411 1787787481.296238 mapread 0x00a014 thru 0x015469 (0xb456 bytes) 000412 1787787481.318571 trunc from 0x02d5fa to 0x03aa76 (0xd47d bytes) 000413 1787787481.354101 write 0x02b52f thru 0x0371b4 (0xbc86 bytes) 000414 1787787481.375362 mapread 0x0240ce thru 0x03240b (0xe33e bytes) 000416 1787787481.386611 punch from 0x03626a to 0x03adf6 (0x4b8d bytes) 000417 1787787481.402660 fallocate 0x03a1fa thru 0x03eae9 (0x48f0 bytes) 000418 1787787481.408453 mapwrite 0x0176e0 thru 0x020145 (0x8a66 bytes) 000419 1787787481.440057 trunc from 0x03eaea to 0x026b19 (0x17fd0 bytes) 000420 1787787481.462391 fallocate 0x004cc5 thru 0x006df1 (0x212d bytes) 000421 1787787481.467802 fallocate 0x014e7a thru 0x01a89e (0x5a25 bytes) 000422 1787787481.476864 MIRROR_RESYNC 3 mirrors 000425 1787787482.307088 punch from 0x005cd0 to 0x01067e (0xa9af bytes) 000426 1787787482.342588 mapread 0x006e1c thru 0x00db4f (0x6d34 bytes) 000428 1787787482.373954 fallocate 0x0242b0 thru 0x02794a (0x369b bytes) 000429 1787787482.380465 mapwrite 0x038a71 thru 0x039bd3 (0x1163 bytes) 000430 1787787482.430261 read 0x0374e4 thru 0x0383ed (0xf0a bytes) 000431 1787787482.431292 mapwrite 0x021873 thru 0x03164c (0xfdda bytes) 000432 1787787482.469525 mapread 0x000447 thru 0x005907 (0x54c1 bytes) 000434 1787787482.477459 trunc from 0x0020ba to 0x00ee28 (0xcd6f bytes) 000435 1787787482.483438 punch from 0x000082 to 0x00ea01 (0xe980 bytes) 000436 1787787482.488852 trunc from 0x039bd4 to 0x011e41 (0x27d92 bytes) 000437 1787787482.508513 punch from 0x00c965 to 0x00d0df (0x77b bytes) 000438 1787787482.513524 write 0x03330f thru 0x039796 (0x6488 bytes) 000439 1787787482.516111 mapread 0x0002b1 thru 0x003866 (0x35b6 bytes) 000440 1787787482.536139 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39797): Mirror 3: 0x74b697c7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39797): Mirror 3: 0x74b697c7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39797: skipped 000441 1787787482.801380 write 0x027c7a thru 0x029920 (0x1ca7 bytes) 000442 1787787482.817847 mapwrite 0x0337d7 thru 0x03ffff (0xc829 bytes) 000443 1787787482.883446 mapwrite 0x024892 thru 0x027e48 (0x35b7 bytes) 000444 1787787482.911217 trunc from 0x040000 to 0x00e311 (0x31cee bytes) 000445 1787787482.926423 punch from 0x00db32 to 0x00e311 (0x7e0 bytes) 000447 1787787482.938605 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xe312): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe312): Mirror 3: 0x9cf25b0f (stale) CRC-32 checksum value for chunk [0, 0xe312): Mirror 1: 0xb7009c4e Mirror 4: 0xb7009c4e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe312: skipped 000448 1787787483.906548 trunc from 0x00e312 to 0x02163e (0x1332d bytes) 000449 1787787484.012852 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2163f): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2163f): Mirror 3: 0xb6681507 (stale) Mirror 4: 0xc126420c (stale) CRC-32 checksum value for chunk [0, 0x2163f): Mirror 1: 0xc126420c Mirror 5: 0xc126420c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2163f: skipped 000450 1787787484.942014 write 0x020905 thru 0x02a11c (0x9818 bytes) 000451 1787787484.983703 MIRROR_RESYNC 4 mirrors 000452 1787787485.706480 fallocate 0x02277f thru 0x02d3ca (0xac4c bytes) 000453 1787787485.769369 read 0x011791 thru 0x016d1b (0x558b bytes) 000454 1787787485.778355 mapwrite 0x01e35f thru 0x022e79 (0x4b1b bytes) 000455 1787787485.823151 trunc from 0x02d3cb to 0x032bf5 (0x582b bytes) 000456 1787787485.839551 trunc from 0x015d96 to 0x017261 (0x14cc bytes) 000457 1787787485.844075 trunc from 0x01b294 to 0x01b4fe (0x26b bytes) 000458 1787787485.853249 trunc from 0x0316c7 to 0x032bf5 (0x152f bytes) 000459 1787787485.861804 trunc from 0x032bf6 to 0x011261 (0x21994 bytes) 000460 1787787485.879580 write 0x02682d thru 0x029c27 (0x33fb bytes) 000461 1787787485.884440 MIRROR_RESYNC 4 mirrors 000462 1787787486.679588 trunc from 0x001bb0 to 0x001d2c (0x17d bytes) 000464 1787787486.747077 write 0x028289 thru 0x0349a4 (0xc71c bytes) 000465 1787787486.759626 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x349a5): Mirror 4: 0xe06bcb63 (stale) Mirror 5: 0xe06bcb63 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x349a5): Mirror 4: 0xe06bcb63 (stale) Mirror 5: 0xe06bcb63 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x349a5: skipped 000466 1787787487.139502 fallocate 0x011abe thru 0x01adf5 (0x9338 bytes) 000467 1787787487.148251 MIRROR_RESYNC 3 mirrors 000468 1787787488.011100 MIRROR_RESYNC 3 mirrors 000469 1787787488.397996 trunc from 0x0349a5 to 0x030395 (0x460f bytes) 000470 1787787488.454223 mapread 0x0032ff thru 0x011ab6 (0xe7b8 bytes) 000471 1787787488.495024 mapwrite 0x00b7b0 thru 0x00f20e (0x3a5f bytes) 000472 1787787488.536716 fallocate 0x00aa6f thru 0x014584 (0x9b16 bytes) 000473 1787787488.550074 mapread 0x00b6af thru 0x011d92 (0x66e4 bytes) 000474 1787787488.553497 mapread 0x0124a8 thru 0x018a3c (0x6595 bytes) 000475 1787787488.554935 mapwrite 0x001d42 thru 0x001f32 (0x1f1 bytes) 000476 1787787488.595766 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x30396): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30396): Mirror 4: 0x4638323d (stale) Mirror 5: 0x4638323d (stale) CRC-32 checksum value for chunk [0, 0x30396): Mirror 1: 0xec2152b4 Mirror 6: 0xec2152b4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30396: skipped 000477 1787787489.580398 fallocate 0x02ff5c thru 0x037e06 (0x7eab bytes) 000478 1787787489.616403 fallocate 0x01cd8b thru 0x02646d (0x96e3 bytes) 000479 1787787489.621971 punch from 0x0338a0 to 0x037e06 (0x4567 bytes) 000480 1787787489.634113 mapread 0x010851 thru 0x0172fd (0x6aad bytes) 000481 1787787489.672128 mapwrite 0x01a0ac thru 0x027d6a (0xdcbf bytes) 000482 1787787489.727068 punch from 0x0239a0 to 0x0316d9 (0xdd3a bytes) 000483 1787787489.732130 mapread 0x033e2e thru 0x037e06 (0x3fd9 bytes) 000484 1787787489.739041 write 0x02d525 thru 0x03c310 (0xedec bytes) 000485 1787787489.745025 punch from 0x01adc4 to 0x01c2e4 (0x1521 bytes) 000486 1787787489.750466 mapwrite 0x0274e4 thru 0x02b93b (0x4458 bytes) 000488 1787787489.775441 trunc from 0x028e35 to 0x030e79 (0x8045 bytes) 000489 1787787489.780600 mapread 0x017fab thru 0x019839 (0x188f bytes) 000490 1787787489.781540 fallocate 0x008de3 thru 0x0173cd (0xe5eb bytes) 000491 1787787489.786460 mapread 0x00259d thru 0x0051aa (0x2c0e bytes) 000492 1787787489.800313 punch from 0x018121 to 0x01dbc2 (0x5aa2 bytes) 000493 1787787489.807864 punch from 0x005abe to 0x008999 (0x2edc bytes) 000494 1787787489.818056 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c311): Mirror 4: 0x1c609ab3 (stale) Mirror 5: 0x1c609ab3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c311): Mirror 4: 0x1c609ab3 (stale) Mirror 5: 0x1c609ab3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c311: skipped 000495 1787787490.135453 punch from 0x0207cd to 0x020959 (0x18d bytes) 000496 1787787490.141436 mapwrite 0x02f815 thru 0x0318fd (0x20e9 bytes) 000497 1787787490.184076 mapwrite 0x03f9aa thru 0x03ffff (0x656 bytes) 000500 1787787490.229520 trunc from 0x012574 to 0x0156a0 (0x312d bytes) 000501 1787787490.234326 mapwrite 0x026a85 thru 0x036648 (0xfbc4 bytes) 000502 1787787490.277991 trunc from 0x0055e7 to 0x012b8b (0xd5a5 bytes) 000503 1787787490.282836 fallocate 0x0313d0 thru 0x03e5dc (0xd20d bytes) 000504 1787787490.288347 trunc from 0x0098ab to 0x00ab19 (0x126f bytes) 000505 1787787490.292762 read 0x02c673 thru 0x02e09f (0x1a2d bytes) 000506 1787787490.294036 trunc from 0x040000 to 0x01c5d6 (0x23a29 bytes) 000507 1787787490.315406 fallocate 0x03c7f5 thru 0x03ffff (0x380b bytes) 000508 1787787490.325983 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c5d7): Mirror 5: 0x1aef84f9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c5d7): Mirror 5: 0x1aef84f9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c5d7: skipped 000509 1787787490.622424 read 0x00e822 thru 0x01180f (0x2fee bytes) 000510 1787787490.628875 mapread 0x008b90 thru 0x00da22 (0x4e93 bytes) 000511 1787787490.647907 write 0x0306a8 thru 0x03e43a (0xdd93 bytes) 000512 1787787490.655074 mapwrite 0x03d8df thru 0x03ffff (0x2721 bytes) 000513 1787787490.695457 fallocate 0x036091 thru 0x03a125 (0x4095 bytes) 000514 1787787490.702614 fallocate 0x025670 thru 0x02f102 (0x9a93 bytes) 000515 1787787490.714905 punch from 0x03f023 to 0x03ffff (0xfdd bytes) 000516 1787787490.721313 punch from 0x03d2a2 to 0x03da3b (0x79a bytes) 000517 1787787490.726652 trunc from 0x00ecba to 0x01419d (0x54e4 bytes) 000518 1787787490.736430 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x85f221de (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa5d7faf5 Mirror 6: 0xa5d7faf5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000520 1787787491.683319 trunc from 0x040000 to 0x03b301 (0x4cfe bytes) 000521 1787787491.765308 mapread 0x0054c1 thru 0x011529 (0xc069 bytes) 000522 1787787491.797254 trunc from 0x03b302 to 0x038387 (0x2f7a bytes) 000523 1787787491.825479 fallocate 0x00bbd2 thru 0x0118ca (0x5cf9 bytes) 000524 1787787491.831445 trunc from 0x030e5b to 0x038387 (0x752d bytes) 000525 1787787491.836389 punch from 0x00574f to 0x00f2b8 (0x9b6a bytes) 000526 1787787491.841912 trunc from 0x038388 to 0x025408 (0x12f7f bytes) 000527 1787787491.862759 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x25409): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25409): Mirror 5: 0xdebf043c (stale) Mirror 6: 0x894a9ab1 (stale) CRC-32 checksum value for chunk [0, 0x25409): Mirror 1: 0x894a9ab1 Mirror 7: 0x894a9ab1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25409: skipped 000528 1787787492.634413 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x25409): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25409): Mirror 5: 0xdebf043c (stale) Mirror 6: 0x894a9ab1 (stale) CRC-32 checksum value for chunk [0, 0x25409): Mirror 1: 0x894a9ab1 Mirror 7: 0x894a9ab1 Mirror 8: 0x894a9ab1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25409: skipped 000529 1787787493.397975 trunc from 0x00fdc0 to 0x01b2a5 (0xb4e6 bytes) 000530 1787787493.440004 write 0x01d10f thru 0x01dfb8 (0xeaa bytes) 000531 1787787493.452205 trunc from 0x0205a0 to 0x022781 (0x21e2 bytes) 000532 1787787493.456434 punch from 0x01880f to 0x01dbbc (0x53ae bytes) 000533 1787787493.467290 read 0x01ffac thru 0x020796 (0x7eb bytes) 000534 1787787493.474764 mapwrite 0x00c20b thru 0x011d56 (0x5b4c bytes) 000535 1787787493.510260 trunc from 0x00a4b8 to 0x016189 (0xbcd2 bytes) 000536 1787787493.514838 read 0x000103 thru 0x006e52 (0x6d50 bytes) 000537 1787787493.524149 punch from 0x0046c0 to 0x00fcf8 (0xb639 bytes) 000538 1787787493.527980 punch from 0x013de2 to 0x01acce (0x6eed bytes) 000539 1787787493.535259 MIRROR_RESYNC 5 mirrors 000540 1787787494.497148 mapwrite 0x023de3 thru 0x02bee0 (0x80fe bytes) 000541 1787787494.594848 read 0x02014b thru 0x02bee0 (0xbd96 bytes) 000542 1787787494.595515 mapwrite 0x000460 thru 0x006288 (0x5e29 bytes) 000544 1787787494.627852 MIRROR_RESYNC 5 mirrors 000545 1787787495.789822 read 0x017d81 thru 0x021cfc (0x9f7c bytes) 000546 1787787495.809351 MIRROR_EXTEND to 6 mirrors 000547 1787787496.611108 fallocate 0x01baaa thru 0x021f8d (0x64e4 bytes) 000549 1787787496.657540 trunc from 0x002694 to 0x00714f (0x4abc bytes) 000550 1787787496.662033 mapread 0x01414f thru 0x016366 (0x2218 bytes) 000551 1787787496.684543 read 0x01901a thru 0x01aa57 (0x1a3e bytes) 000552 1787787496.685218 trunc from 0x02bee1 to 0x0029fd (0x294e3 bytes) 000553 1787787496.707739 trunc from 0x000e47 to 0x0029fd (0x1bb7 bytes) 000554 1787787496.713277 mapwrite 0x00532b thru 0x00c48a (0x7160 bytes) 000555 1787787496.774032 trunc from 0x00c48b to 0x01055a (0x40d0 bytes) 000556 1787787496.786872 fallocate 0x01174b thru 0x01b8d3 (0xa189 bytes) 000557 1787787496.793136 read 0x00352b thru 0x01055a (0xd030 bytes) 000558 1787787496.801707 punch from 0x00a6d3 to 0x01055a (0x5e88 bytes) 000559 1787787496.806723 trunc from 0x01055b to 0x02da60 (0x1d506 bytes) 000560 1787787496.823347 punch from 0x00a63a to 0x013a63 (0x942a bytes) 000561 1787787496.827947 trunc from 0x02da61 to 0x01b731 (0x1232f bytes) 000562 1787787496.842907 punch from 0x003756 to 0x005a43 (0x22ee bytes) 000564 1787787496.847987 mapwrite 0x002d16 thru 0x0046dd (0x19c8 bytes) 000565 1787787496.871736 write 0x03722d thru 0x03ffff (0x8dd3 bytes) 000566 1787787496.876514 write 0x012dd3 thru 0x0137b4 (0x9e2 bytes) 000567 1787787496.877655 read 0x027e42 thru 0x0294d0 (0x168f bytes) 000569 1787787496.890348 mapwrite 0x021e1a thru 0x02a279 (0x8460 bytes) 000570 1787787496.912192 trunc from 0x0194ac to 0x024e54 (0xb9a9 bytes) 000571 1787787496.918149 trunc from 0x001c4c to 0x00478e (0x2b43 bytes) 000572 1787787496.927489 MIRROR_RESYNC 6 mirrors 000573 1787787498.799428 mapread 0x033545 thru 0x03d97e (0xa43a bytes) 000574 1787787498.814243 write 0x028738 thru 0x0300af (0x7978 bytes) 000576 1787787498.851024 trunc from 0x03aa25 to 0x03ffff (0x55db bytes) 000577 1787787498.859886 mapwrite 0x01c767 thru 0x0270a3 (0xa93d bytes) 000578 1787787498.900021 read 0x016d34 thru 0x020a3d (0x9d0a bytes) 000579 1787787498.901970 trunc from 0x040000 to 0x0348d1 (0xb72e bytes) 000580 1787787498.939722 fallocate 0x011899 thru 0x020f80 (0xf6e8 bytes) 000581 1787787498.946143 mapread 0x0240ca thru 0x025e46 (0x1d7d bytes) 000582 1787787498.948125 mapwrite 0x004b57 thru 0x010c03 (0xc0ad bytes) 000583 1787787499.003451 trunc from 0x0348d2 to 0x016d63 (0x1db6e bytes) 000584 1787787499.031331 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16d64): Mirror 5: 0x1b2bea7e (stale) Mirror 6: 0x1b2bea7e (stale) Mirror 8: 0x1b2bea7e (stale) Mirror 9: 0x1b2bea7e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16d64): Mirror 5: 0x1b2bea7e (stale) Mirror 6: 0x1b2bea7e (stale) Mirror 8: 0x1b2bea7e (stale) Mirror 9: 0x1b2bea7e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16d64: skipped 000585 1787787499.338652 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x16d64): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16d64): Mirror 5: 0x1b2bea7e (stale) Mirror 6: 0x1b2bea7e (stale) Mirror 8: 0x1b2bea7e (stale) Mirror 9: 0x1b2bea7e (stale) CRC-32 checksum value for chunk [0, 0x16d64): Mirror 1: 0xc6ca8c62 Mirror 10: 0xc6ca8c62 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16d64: skipped 000586 1787787500.369956 write 0x00f352 thru 0x011f1c (0x2bcb bytes) 000588 1787787500.421068 fallocate 0x007c13 thru 0x010263 (0x8651 bytes) 000589 1787787500.426827 trunc from 0x016d64 to 0x039ef4 (0x23191 bytes) 000590 1787787500.447090 write 0x032cc1 thru 0x033afa (0xe3a bytes) 000591 1787787500.456993 punch from 0x03402d to 0x039ef4 (0x5ec8 bytes) 000592 1787787500.461577 trunc from 0x039ef5 to 0x033767 (0x678d bytes) 000593 1787787500.477374 punch from 0x02ae5e to 0x033767 (0x890a bytes) 000595 1787787500.495026 mapread 0x01f52f thru 0x0282f8 (0x8dca bytes) 000597 1787787500.520026 trunc from 0x033768 to 0x0396f5 (0x5f8e bytes) 000598 1787787500.545308 fallocate 0x0184ce thru 0x024295 (0xbdc8 bytes) 000599 1787787500.556634 read 0x037449 thru 0x0396f5 (0x22ad bytes) 000600 1787787500.568164 write 0x003154 thru 0x0049d7 (0x1884 bytes) 000601 1787787500.579091 write 0x033675 thru 0x03a02a (0x69b6 bytes) 000602 1787787500.586218 trunc from 0x012be7 to 0x0201ab (0xd5c5 bytes) 000603 1787787500.593101 trunc from 0x03a02b to 0x020b64 (0x194c6 bytes) 000604 1787787500.608206 mapread 0x012591 thru 0x01ffb1 (0xda21 bytes) 000605 1787787500.631332 read 0x005b04 thru 0x009e25 (0x4322 bytes) 000606 1787787500.643619 mapread 0x0114f4 thru 0x01e08b (0xcb98 bytes) 000607 1787787500.645252 fallocate 0x0359a2 thru 0x037b7c (0x21db bytes) 000608 1787787500.650966 read 0x0158cb thru 0x020724 (0xae5a bytes) 000609 1787787500.652027 mapwrite 0x035624 thru 0x03d9d3 (0x83b0 bytes) 000610 1787787500.719300 write 0x01420f thru 0x01701c (0x2e0e bytes) 000611 1787787500.722892 punch from 0x030537 to 0x03c480 (0xbf4a bytes) 000612 1787787500.732440 trunc from 0x03d9d4 to 0x0139f8 (0x29fdb bytes) 000613 1787787500.752260 trunc from 0x00c5df to 0x0139f8 (0x741a bytes) 000614 1787787500.762479 write 0x010475 thru 0x0155b2 (0x513e bytes) 000615 1787787500.775737 mapread 0x00df3b thru 0x0155b2 (0x7678 bytes) 000617 1787787500.796596 trunc from 0x008cf5 to 0x0155b2 (0xc8be bytes) 000618 1787787500.808504 trunc from 0x00a7b0 to 0x0155b2 (0xae03 bytes) 000619 1787787500.812576 trunc from 0x0155b3 to 0x00d569 (0x8049 bytes) 000620 1787787500.824740 mapwrite 0x000528 thru 0x003360 (0x2e39 bytes) 000621 1787787500.837071 mapwrite 0x03ae1c thru 0x03ffff (0x51e4 bytes) 000622 1787787500.886514 mapwrite 0x00e07a thru 0x019d86 (0xbd0d bytes) 000623 1787787500.992762 mapwrite 0x0100c8 thru 0x0186cb (0x8604 bytes) 000624 1787787501.029842 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x5e056278 (stale) Mirror 6: 0x5e056278 (stale) Mirror 8: 0x5e056278 (stale) Mirror 9: 0x5e056278 (stale) Mirror 10: 0xae9dfd85 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7433b0d2 Mirror 11: 0x7433b0d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000625 1787787502.090536 MIRROR_RESYNC 7 mirrors 000626 1787787503.788131 trunc from 0x040000 to 0x033298 (0xcd67 bytes) 000627 1787787503.865411 mapread 0x004e0f thru 0x00c36a (0x755c bytes) 000628 1787787503.899213 write 0x027314 thru 0x02d683 (0x6370 bytes) 000629 1787787503.915949 mapwrite 0x00ef12 thru 0x01cf7e (0xe06d bytes) 000630 1787787503.962505 write 0x026cfa thru 0x035b8d (0xee94 bytes) 000631 1787787503.970045 trunc from 0x0032c2 to 0x0074e3 (0x4222 bytes) 000632 1787787503.976196 read 0x034708 thru 0x035b8d (0x1486 bytes) 000633 1787787503.977721 trunc from 0x035b8e to 0x02e6b3 (0x74da bytes) 000634 1787787504.006385 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e6b4): Mirror 5: 0xf3917f0b (stale) Mirror 6: 0xf3917f0b (stale) Mirror 8: 0xf3917f0b (stale) Mirror 10: 0xf3917f0b (stale) Mirror 11: 0xf3917f0b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e6b4): Mirror 5: 0xf3917f0b (stale) Mirror 6: 0xf3917f0b (stale) Mirror 8: 0xf3917f0b (stale) Mirror 10: 0xf3917f0b (stale) Mirror 11: 0xf3917f0b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e6b4: skipped 000635 1787787504.562464 trunc from 0x02e6b4 to 0x02bb9f (0x2b14 bytes) 000636 1787787504.583823 trunc from 0x02bba0 to 0x01ff6c (0xbc33 bytes) 000637 1787787504.601682 trunc from 0x0098e1 to 0x011716 (0x7e36 bytes) 000638 1787787504.611238 fallocate 0x01e390 thru 0x0226de (0x434f bytes) 000639 1787787504.616581 trunc from 0x01c3fc to 0x0226de (0x62e3 bytes) 000640 1787787504.626808 fallocate 0x017484 thru 0x026e5e (0xf9db bytes) 000641 1787787504.638677 fallocate 0x03eb4b thru 0x03ffff (0x14b5 bytes) 000642 1787787504.661873 MIRROR_RESYNC 6 mirrors 000643 1787787506.356260 trunc from 0x014f70 to 0x01a430 (0x54c1 bytes) 000644 1787787506.395890 write 0x03c2bd thru 0x03ffff (0x3d43 bytes) 000645 1787787506.401875 mapwrite 0x006eca thru 0x008dd5 (0x1f0c bytes) 000646 1787787506.458020 mapread 0x027c33 thru 0x02f4e0 (0x78ae bytes) 000647 1787787506.474564 mapread 0x027c77 thru 0x035414 (0xd79e bytes) 000648 1787787506.488516 punch from 0x017f40 to 0x026335 (0xe3f6 bytes) 000649 1787787506.501606 read 0x011a66 thru 0x01dbe3 (0xc17e bytes) 000650 1787787506.526313 mapwrite 0x015f00 thru 0x017efb (0x1ffc bytes) 000651 1787787506.550301 fallocate 0x030880 thru 0x03b641 (0xadc2 bytes) 000652 1787787506.555942 trunc from 0x040000 to 0x03315f (0xcea0 bytes) 000653 1787787506.595759 mapwrite 0x003c8d thru 0x006995 (0x2d09 bytes) 000654 1787787506.620540 trunc from 0x024fa5 to 0x030c82 (0xbcde bytes) 000655 1787787506.637944 MIRROR_RESYNC 6 mirrors 000656 1787787507.963397 fallocate 0x020694 thru 0x02c5c6 (0xbf33 bytes) 000657 1787787508.043633 trunc from 0x00dade to 0x014e92 (0x73b5 bytes) 000658 1787787508.056601 mapwrite 0x01edba thru 0x022034 (0x327b bytes) 000660 1787787508.127729 read 0x02bbb9 thru 0x02ff1b (0x4363 bytes) 000661 1787787508.151116 trunc from 0x01e66b to 0x0204d0 (0x1e66 bytes) 000662 1787787508.170072 trunc from 0x00efc9 to 0x01b9d1 (0xca09 bytes) 000663 1787787508.181852 mapread 0x031c5d thru 0x03315f (0x1503 bytes) 000664 1787787508.210787 mapread 0x0089ea thru 0x015dfe (0xd415 bytes) 000665 1787787508.247691 fallocate 0x00f628 thru 0x0159ff (0x63d8 bytes) 000666 1787787508.268578 MIRROR_RESYNC 6 mirrors 000667 1787787509.854452 trunc from 0x01f0cc to 0x02cf75 (0xdeaa bytes) 000668 1787787509.907816 punch from 0x01c86f to 0x01f00e (0x27a0 bytes) 000669 1787787509.917047 punch from 0x02ed7f to 0x03315f (0x43e1 bytes) 000670 1787787509.923058 mapwrite 0x02307f thru 0x032b49 (0xfacb bytes) 000671 1787787509.990055 fallocate 0x031f5a thru 0x03cc0a (0xacb1 bytes) 000674 1787787509.998013 punch from 0x0307d9 to 0x03315f (0x2987 bytes) 000675 1787787510.004073 read 0x022970 thru 0x02ff63 (0xd5f4 bytes) 000676 1787787510.005375 fallocate 0x0360e9 thru 0x03ffff (0x9f17 bytes) 000679 1787787510.013610 trunc from 0x040000 to 0x026bd9 (0x19426 bytes) 000680 1787787510.045695 MIRROR_RESYNC 6 mirrors 000681 1787787511.069914 write 0x01469f thru 0x0232e7 (0xec49 bytes) 000682 1787787511.153247 mapwrite 0x02aa9b thru 0x0352ea (0xa850 bytes) 000683 1787787511.237508 trunc from 0x015ca8 to 0x0220e7 (0xc440 bytes) 000684 1787787511.245749 mapread 0x0305c8 thru 0x0352ea (0x4d23 bytes) 000685 1787787511.247481 trunc from 0x01e1a7 to 0x023ef2 (0x5d4c bytes) 000688 1787787511.253418 read 0x008c12 thru 0x01429f (0xb68e bytes) 000689 1787787511.260760 mapwrite 0x011bc0 thru 0x018a21 (0x6e62 bytes) 000690 1787787511.292553 trunc from 0x010174 to 0x01dd67 (0xdbf4 bytes) 000691 1787787511.298989 write 0x00a848 thru 0x0133f5 (0x8bae bytes) 000692 1787787511.305439 write 0x015f11 thru 0x017147 (0x1237 bytes) 000693 1787787511.312957 trunc from 0x015239 to 0x01dd9a (0x8b62 bytes) 000694 1787787511.325048 mapwrite 0x02f3aa thru 0x034277 (0x4ece bytes) 000695 1787787511.338951 mapwrite 0x018a22 thru 0x01b806 (0x2de5 bytes) 000696 1787787511.357883 write 0x004f7b thru 0x00d810 (0x8896 bytes) 000697 1787787511.362391 read 0x000368 thru 0x00fcb6 (0xf94f bytes) 000698 1787787511.371117 trunc from 0x0352eb to 0x036c3d (0x1953 bytes) 000699 1787787511.393585 MIRROR_RESYNC 6 mirrors 000700 1787787513.071054 read 0x00b2d3 thru 0x00ba25 (0x753 bytes) 000702 1787787513.077637 mapwrite 0x0186d9 thru 0x027da0 (0xf6c8 bytes) 000704 1787787513.206883 write 0x01a0fc thru 0x026c97 (0xcb9c bytes) 000705 1787787513.212418 mapread 0x0352f8 thru 0x036c3d (0x1946 bytes) 000706 1787787513.229487 write 0x01a60f thru 0x02924a (0xec3c bytes) 000707 1787787513.233091 trunc from 0x036c3e to 0x03dd32 (0x70f5 bytes) 000708 1787787513.258182 mapwrite 0x03a913 thru 0x03ffff (0x56ed bytes) 000709 1787787513.302929 write 0x00a2e0 thru 0x01785b (0xd57c bytes) 000710 1787787513.309759 mapwrite 0x0184b2 thru 0x019f63 (0x1ab2 bytes) 000712 1787787513.329222 mapwrite 0x001eb2 thru 0x007c07 (0x5d56 bytes) 000713 1787787513.364213 mapwrite 0x01e6c8 thru 0x022574 (0x3ead bytes) 000714 1787787513.387567 mapread 0x003c88 thru 0x00fe35 (0xc1ae bytes) 000715 1787787513.389456 mapread 0x02b3b4 thru 0x02b4d1 (0x11e bytes) 000716 1787787513.395981 trunc from 0x0292f4 to 0x02a0ae (0xdbb bytes) 000717 1787787513.406826 MIRROR_RESYNC 6 mirrors 000718 1787787514.530129 punch from 0x0369dc to 0x037990 (0xfb5 bytes) 000719 1787787514.568590 trunc from 0x040000 to 0x02648c (0x19b73 bytes) 000720 1787787514.597593 mapwrite 0x02196a thru 0x021c5d (0x2f4 bytes) 000721 1787787514.622997 mapwrite 0x03bacf thru 0x03ffbb (0x44ed bytes) 000722 1787787514.674928 trunc from 0x005ba7 to 0x0065eb (0xa45 bytes) 000723 1787787514.680464 read 0x0233d1 thru 0x0259b3 (0x25e3 bytes) 000724 1787787514.685964 write 0x036354 thru 0x03ffff (0x9cac bytes) 000725 1787787514.688013 read 0x00fac7 thru 0x01766a (0x7ba4 bytes) 000726 1787787514.707915 trunc from 0x012d62 to 0x01c9c0 (0x9c5f bytes) 000727 1787787514.721923 fallocate 0x03534a thru 0x03ffff (0xacb6 bytes) 000728 1787787514.737086 mapread 0x03a685 thru 0x03ffff (0x597b bytes) 000729 1787787514.738331 mapread 0x01ab65 thru 0x028c5e (0xe0fa bytes) 000730 1787787514.762833 fallocate 0x022a3b thru 0x030e0b (0xe3d1 bytes) 000731 1787787514.769337 trunc from 0x01b880 to 0x025772 (0x9ef3 bytes) 000732 1787787514.774335 trunc from 0x040000 to 0x03b5fa (0x4a05 bytes) 000733 1787787514.797757 mapwrite 0x039342 thru 0x03ea16 (0x56d5 bytes) 000734 1787787514.871627 mapread 0x01f8d0 thru 0x026374 (0x6aa5 bytes) 000736 1787787514.908053 punch from 0x0244e1 to 0x03345a (0xef7a bytes) 000737 1787787514.912236 punch from 0x031fb5 to 0x03df65 (0xbfb1 bytes) 000738 1787787514.922845 MIRROR_RESYNC 6 mirrors 000739 1787787515.994944 read 0x0392e0 thru 0x03ea16 (0x5737 bytes) 000740 1787787516.012218 trunc from 0x03ea17 to 0x0097f0 (0x35226 bytes) 000741 1787787516.057260 punch from 0x00594e to 0x0097f0 (0x3ea3 bytes) 000742 1787787516.063515 mapwrite 0x009532 thru 0x0180ba (0xeb89 bytes) 000743 1787787516.183822 trunc from 0x0180bb to 0x0295c0 (0x11506 bytes) 000744 1787787516.223128 trunc from 0x0295c1 to 0x033dc6 (0xa806 bytes) 000745 1787787516.252838 mapwrite 0x008b86 thru 0x013a13 (0xae8e bytes) 000746 1787787516.279966 read 0x02fd0c thru 0x033dc6 (0x40bb bytes) 000747 1787787516.288691 trunc from 0x018181 to 0x019f2e (0x1dae bytes) 000748 1787787516.293535 mapread 0x031a49 thru 0x033dc6 (0x237e bytes) 000750 1787787516.295004 mapread 0x01bdd1 thru 0x0262e0 (0xa510 bytes) 000751 1787787516.296633 read 0x02d584 thru 0x033dc6 (0x6843 bytes) 000752 1787787516.297616 write 0x02ab30 thru 0x02e22b (0x36fc bytes) 000753 1787787516.298940 read 0x030537 thru 0x033dc6 (0x3890 bytes) 000754 1787787516.299749 fallocate 0x03145b thru 0x0327be (0x1364 bytes) 000755 1787787516.304515 mapwrite 0x004c97 thru 0x00843e (0x37a8 bytes) 000756 1787787516.320093 write 0x037276 thru 0x03ffff (0x8d8a bytes) 000757 1787787516.325756 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x41520c37 (stale) Mirror 6: 0x41520c37 (stale) Mirror 8: 0x41520c37 (stale) Mirror 10: 0x41520c37 (stale) Mirror 11: 0x41520c37 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb8dd19e8 Mirror 12: 0xb8dd19e8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000758 1787787517.189626 fallocate 0x009c56 thru 0x00bfff (0x23aa bytes) 000759 1787787517.251573 punch from 0x037eb4 to 0x03ffff (0x814c bytes) 000760 1787787517.259620 write 0x029f24 thru 0x033058 (0x9135 bytes) 000761 1787787517.277643 read 0x02c82c thru 0x039319 (0xcaee bytes) 000762 1787787517.285958 read 0x001d3c thru 0x0069f8 (0x4cbd bytes) 000763 1787787517.300550 trunc from 0x040000 to 0x02e440 (0x11bbf bytes) 000764 1787787517.347852 mapread 0x004252 thru 0x007221 (0x2fd0 bytes) 000765 1787787517.348955 write 0x035d9c thru 0x03ef8d (0x91f2 bytes) 000766 1787787517.361939 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3ef8e): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ef8e): Mirror 5: 0xf1d5897b (stale) Mirror 6: 0xf1d5897b (stale) Mirror 8: 0xf1d5897b (stale) Mirror 10: 0xf1d5897b (stale) Mirror 11: 0xf1d5897b (stale) Mirror 12: 0x4063c227 (stale) CRC-32 checksum value for chunk [0, 0x3ef8e): Mirror 1: 0xb9c8bb2e Mirror 13: 0xb9c8bb2e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ef8e: skipped 000768 1787787518.281668 read 0x015ffe thru 0x01da20 (0x7a23 bytes) 000769 1787787518.287624 punch from 0x004c55 to 0x010760 (0xbb0c bytes) 000770 1787787518.324377 trunc from 0x03ef8e to 0x003c63 (0x3b32a bytes) 000771 1787787518.380578 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c64): Mirror 6: 0x5c5c6b9c (stale) Mirror 8: 0x5c5c6b9c (stale) Mirror 10: 0x5c5c6b9c (stale) Mirror 11: 0x5c5c6b9c (stale) Mirror 12: 0x5c5c6b9c (stale) Mirror 13: 0x5c5c6b9c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c64): Mirror 6: 0x5c5c6b9c (stale) Mirror 8: 0x5c5c6b9c (stale) Mirror 10: 0x5c5c6b9c (stale) Mirror 11: 0x5c5c6b9c (stale) Mirror 12: 0x5c5c6b9c (stale) Mirror 13: 0x5c5c6b9c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c64: skipped 000772 1787787518.774613 read 0x0032af thru 0x003c63 (0x9b5 bytes) 000774 1787787518.782920 fallocate 0x020a50 thru 0x02541f (0x49d0 bytes) 000776 1787787518.800237 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25420): Mirror 6: 0xe06ac22a (stale) Mirror 8: 0xe06ac22a (stale) Mirror 10: 0xe06ac22a (stale) Mirror 12: 0xb8527de1 (stale) Mirror 13: 0xb8527de1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25420): Mirror 6: 0xe06ac22a (stale) Mirror 8: 0xe06ac22a (stale) Mirror 10: 0xe06ac22a (stale) Mirror 12: 0xb8527de1 (stale) Mirror 13: 0xb8527de1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25420: skipped 000777 1787787519.294410 write 0x02b479 thru 0x039764 (0xe2ec bytes) 000778 1787787519.300102 mapread 0x011c78 thru 0x01f3dc (0xd765 bytes) 000779 1787787519.328976 trunc from 0x039765 to 0x03d86a (0x4106 bytes) 000780 1787787519.345169 mapread 0x0115fa thru 0x01df6d (0xc974 bytes) 000781 1787787519.351388 MIRROR_RESYNC 6 mirrors 000783 1787787520.762740 read 0x007602 thru 0x009aab (0x24aa bytes) 000784 1787787520.779073 MIRROR_RESYNC 6 mirrors 000785 1787787521.201949 write 0x03dbb4 thru 0x03ffff (0x244c bytes) 000786 1787787521.253674 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x9ca94a2c (stale) Mirror 10: 0x9ca94a2c (stale) Mirror 12: 0x9ca94a2c (stale) Mirror 13: 0x9ca94a2c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x9ca94a2c (stale) Mirror 10: 0x9ca94a2c (stale) Mirror 12: 0x9ca94a2c (stale) Mirror 13: 0x9ca94a2c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000787 1787787521.662497 read 0x0392b5 thru 0x03c6c5 (0x3411 bytes) 000788 1787787521.673127 read 0x02ba4f thru 0x030dac (0x535e bytes) 000789 1787787521.685279 punch from 0x006739 to 0x016146 (0xfa0e bytes) 000791 1787787521.695692 MIRROR_RESYNC 5 mirrors 000793 1787787522.770584 write 0x030f93 thru 0x03ed5c (0xddca bytes) 000794 1787787522.854090 fallocate 0x01eb5a thru 0x021c16 (0x30bd bytes) 000795 1787787522.863325 MIRROR_RESYNC 5 mirrors 000796 1787787523.741430 punch from 0x03953b to 0x03ffff (0x6ac5 bytes) 000797 1787787523.789890 write 0x00b91b thru 0x0164ba (0xaba0 bytes) 000798 1787787523.802972 fallocate 0x039e80 thru 0x03ffff (0x6180 bytes) 000799 1787787523.810048 read 0x00f1ab thru 0x01c352 (0xd1a8 bytes) 000800 1787787523.822258 mapread 0x00f718 thru 0x011ae4 (0x23cd bytes) 000802 1787787523.824560 mapread 0x01ddf4 thru 0x024a40 (0x6c4d bytes) 000803 1787787523.849303 punch from 0x00f028 to 0x014562 (0x553b bytes) 000804 1787787523.881022 mapwrite 0x01591b thru 0x01c2ec (0x69d2 bytes) 000805 1787787523.900903 punch from 0x00903e to 0x01532f (0xc2f2 bytes) 000806 1787787523.906019 write 0x013f62 thru 0x01d86e (0x990d bytes) 000807 1787787523.912126 trunc from 0x040000 to 0x021b24 (0x1e4db bytes) 000808 1787787523.959753 mapwrite 0x00530a thru 0x009740 (0x4437 bytes) 000809 1787787524.008315 mapread 0x014710 thru 0x01e5f8 (0x9ee9 bytes) 000810 1787787524.009271 read 0x006768 thru 0x012b77 (0xc410 bytes) 000811 1787787524.016493 punch from 0x00751f to 0x00f5dc (0x80be bytes) 000812 1787787524.024018 read 0x01ad54 thru 0x021b24 (0x6dd1 bytes) 000813 1787787524.026909 trunc from 0x014b05 to 0x01f594 (0xaa90 bytes) 000814 1787787524.039968 trunc from 0x021b25 to 0x006225 (0x1b8ff bytes) 000815 1787787524.059435 trunc from 0x006226 to 0x00bec5 (0x5ca0 bytes) 000816 1787787524.074489 write 0x026858 thru 0x0318a1 (0xb04a bytes) 000817 1787787524.077443 trunc from 0x0318a2 to 0x034f4f (0x36ae bytes) 000818 1787787524.093733 mapwrite 0x002891 thru 0x00d68c (0xadfc bytes) 000819 1787787524.117725 punch from 0x02ad06 to 0x02baea (0xde5 bytes) 000821 1787787524.122431 mapwrite 0x030fc2 thru 0x038aeb (0x7b2a bytes) 000822 1787787524.184886 write 0x03eb0b thru 0x03ffff (0x14f5 bytes) 000823 1787787524.203344 MIRROR_RESYNC 5 mirrors 000824 1787787525.509626 fallocate 0x0043ea thru 0x013aa5 (0xf6bc bytes) 000825 1787787525.565299 read 0x0109bf thru 0x0186aa (0x7cec bytes) 000826 1787787525.581810 trunc from 0x040000 to 0x02d1de (0x12e21 bytes) 000827 1787787525.602808 trunc from 0x01bf06 to 0x028085 (0xc180 bytes) 000828 1787787525.610027 write 0x0200ac thru 0x029794 (0x96e9 bytes) 000831 1787787525.635842 read 0x01e817 thru 0x01f1d7 (0x9c1 bytes) 000832 1787787525.654136 read 0x01436b thru 0x01bb00 (0x7796 bytes) 000833 1787787525.665662 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2d1df): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d1df): Mirror 8: 0xc578b676 (stale) Mirror 10: 0xc578b676 (stale) Mirror 12: 0xc578b676 (stale) Mirror 13: 0xc578b676 (stale) CRC-32 checksum value for chunk [0, 0x2d1df): Mirror 1: 0xbc7b97eb Mirror 14: 0xbc7b97eb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d1df: skipped 000834 1787787526.508565 mapwrite 0x02778a thru 0x0344b4 (0xcd2b bytes) 000835 1787787526.609113 mapread 0x0215af thru 0x02434b (0x2d9d bytes) 000836 1787787526.623511 mapwrite 0x021ec8 thru 0x02634f (0x4488 bytes) 000837 1787787526.664160 write 0x027938 thru 0x02ae0a (0x34d3 bytes) 000838 1787787526.665324 trunc from 0x017400 to 0x026cb6 (0xf8b7 bytes) 000839 1787787526.676482 fallocate 0x0212eb thru 0x02eaac (0xd7c2 bytes) 000840 1787787526.684385 read 0x02199c thru 0x02d982 (0xbfe7 bytes) 000841 1787787526.694970 read 0x00e43a thru 0x010a8c (0x2653 bytes) 000842 1787787526.706761 mapread 0x033033 thru 0x0344b4 (0x1482 bytes) 000843 1787787526.708008 write 0x00adfd thru 0x0182c4 (0xd4c8 bytes) 000844 1787787526.714637 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x344b5): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x344b5): Mirror 8: 0x617cc3e4 (stale) Mirror 10: 0x617cc3e4 (stale) Mirror 12: 0x617cc3e4 (stale) Mirror 13: 0x617cc3e4 (stale) Mirror 14: 0x449d71f (stale) CRC-32 checksum value for chunk [0, 0x344b5): Mirror 1: 0x7d3ec975 Mirror 15: 0x7d3ec975 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x344b5: skipped 000845 1787787527.623533 fallocate 0x01b0e0 thru 0x023f11 (0x8e32 bytes) 000846 1787787527.664307 write 0x037b37 thru 0x038442 (0x90c bytes) 000847 1787787527.670502 write 0x03e6c7 thru 0x03ffff (0x1939 bytes) 000848 1787787527.671609 trunc from 0x040000 to 0x02c0ab (0x13f54 bytes) 000849 1787787527.699551 punch from 0x00a1d0 to 0x011a17 (0x7848 bytes) 000850 1787787527.702924 mapwrite 0x03f9fe thru 0x03ffff (0x602 bytes) 000851 1787787527.744620 trunc from 0x00b028 to 0x00ff84 (0x4f5d bytes) 000852 1787787527.750310 read 0x01f2f4 thru 0x02db0d (0xe81a bytes) 000853 1787787527.764602 write 0x032859 thru 0x0330d8 (0x880 bytes) 000854 1787787527.776347 trunc from 0x0270f8 to 0x02cf1e (0x5e27 bytes) 000855 1787787527.784846 trunc from 0x040000 to 0x001372 (0x3ec8d bytes) 000856 1787787527.817205 mapwrite 0x027142 thru 0x0357a2 (0xe661 bytes) 000857 1787787527.881158 mapread 0x0327d6 thru 0x0357a2 (0x2fcd bytes) 000858 1787787527.882203 fallocate 0x03fe53 thru 0x03ffff (0x1ad bytes) 000859 1787787527.887999 trunc from 0x00eb20 to 0x017cb8 (0x9199 bytes) 000860 1787787527.892427 mapwrite 0x026830 thru 0x035cd1 (0xf4a2 bytes) 000862 1787787527.929446 trunc from 0x01f49c to 0x0219c4 (0x2529 bytes) 000864 1787787527.938444 trunc from 0x003252 to 0x00b5ea (0x8399 bytes) 000865 1787787527.942613 fallocate 0x029edc thru 0x02bb88 (0x1cad bytes) 000866 1787787527.947385 trunc from 0x005548 to 0x00b014 (0x5acd bytes) 000867 1787787527.955537 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35cd2): Mirror 8: 0x69035d5a (stale) Mirror 10: 0x69035d5a (stale) Mirror 12: 0x69035d5a (stale) Mirror 13: 0x69035d5a (stale) Mirror 15: 0x55eeb92b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35cd2): Mirror 8: 0x69035d5a (stale) Mirror 10: 0x69035d5a (stale) Mirror 12: 0x69035d5a (stale) Mirror 13: 0x69035d5a (stale) Mirror 15: 0x55eeb92b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35cd2: skipped 000868 1787787528.321580 trunc from 0x035cd2 to 0x03dd54 (0x8083 bytes) 000869 1787787528.341174 fallocate 0x022787 thru 0x0321ce (0xfa48 bytes) 000870 1787787528.345743 mapread 0x033760 thru 0x038ba0 (0x5441 bytes) 000871 1787787528.365431 fallocate 0x0359bb thru 0x036fe0 (0x1626 bytes) 000873 1787787528.370040 read 0x035fc9 thru 0x03dd54 (0x7d8c bytes) 000874 1787787528.370580 trunc from 0x03dd55 to 0x01f3b5 (0x1e99f bytes) 000875 1787787528.385440 mapwrite 0x02f0ac thru 0x0308bd (0x1812 bytes) 000876 1787787528.417844 fallocate 0x015767 thru 0x01d726 (0x7fc0 bytes) 000877 1787787528.422549 mapread 0x0063d1 thru 0x00ff26 (0x9b56 bytes) 000878 1787787528.434233 punch from 0x01b274 to 0x0286b6 (0xd443 bytes) 000879 1787787528.438935 write 0x00cb30 thru 0x00ef6e (0x243f bytes) 000880 1787787528.440159 trunc from 0x0308be to 0x0333c3 (0x2b06 bytes) 000881 1787787528.456180 punch from 0x00a808 to 0x010a42 (0x623b bytes) 000882 1787787528.465294 fallocate 0x028e22 thru 0x029349 (0x528 bytes) 000883 1787787528.470147 mapwrite 0x0099b6 thru 0x01409f (0xa6ea bytes) 000884 1787787528.505729 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x333c4): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x333c4): Mirror 8: 0x98685cfe (stale) Mirror 10: 0x98685cfe (stale) Mirror 12: 0x98685cfe (stale) Mirror 13: 0x98685cfe (stale) Mirror 15: 0x1af77c87 (stale) CRC-32 checksum value for chunk [0, 0x333c4): Mirror 1: 0xdc6ff68b Mirror 16: 0xdc6ff68b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x333c4: skipped 000885 1787787529.293282 trunc from 0x0333c4 to 0x01f266 (0x1415d bytes) 000886 1787787529.358726 punch from 0x013e69 to 0x01ed5a (0xaef2 bytes) 000887 1787787529.368401 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f267): Mirror 10: 0x4ff1d9f5 (stale) Mirror 12: 0x4ff1d9f5 (stale) Mirror 13: 0x4ff1d9f5 (stale) Mirror 15: 0x772b3386 (stale) Mirror 16: 0xc7388fc0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f267): Mirror 10: 0x4ff1d9f5 (stale) Mirror 12: 0x4ff1d9f5 (stale) Mirror 13: 0x4ff1d9f5 (stale) Mirror 15: 0x772b3386 (stale) Mirror 16: 0xc7388fc0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f267: skipped 000888 1787787529.707415 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1f267): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f267): Mirror 10: 0x4ff1d9f5 (stale) Mirror 12: 0x4ff1d9f5 (stale) Mirror 13: 0x4ff1d9f5 (stale) Mirror 15: 0x772b3386 (stale) Mirror 16: 0xc7388fc0 (stale) CRC-32 checksum value for chunk [0, 0x1f267): Mirror 1: 0xdd853dec Mirror 17: 0xdd853dec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f267: skipped 000889 1787787530.423245 fallocate 0x02e0ed thru 0x03cac1 (0xe9d5 bytes) 000891 1787787530.463475 fallocate 0x00653b thru 0x00daef (0x75b5 bytes) 000893 1787787530.469482 trunc from 0x0103e1 to 0x0193f7 (0x9017 bytes) 000894 1787787530.478333 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cac2): Mirror 12: 0x9ad6cf24 (stale) Mirror 13: 0x9ad6cf24 (stale) Mirror 15: 0x153b9261 (stale) Mirror 16: 0xb36e755a (stale) Mirror 17: 0x9d4409e2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cac2): Mirror 12: 0x9ad6cf24 (stale) Mirror 13: 0x9ad6cf24 (stale) Mirror 15: 0x153b9261 (stale) Mirror 16: 0xb36e755a (stale) Mirror 17: 0x9d4409e2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cac2: skipped 000895 1787787531.061257 MIRROR_RESYNC 6 mirrors 000896 1787787532.550169 mapwrite 0x02cb50 thru 0x031041 (0x44f2 bytes) 000897 1787787532.640940 MIRROR_RESYNC 6 mirrors 000899 1787787534.094418 mapread 0x0113f2 thru 0x014e10 (0x3a1f bytes) 000900 1787787534.111423 write 0x0216b5 thru 0x02a059 (0x89a5 bytes) 000901 1787787534.178329 mapwrite 0x035768 thru 0x038d11 (0x35aa bytes) 000903 1787787534.219740 fallocate 0x002e6c thru 0x008e4f (0x5fe4 bytes) 000905 1787787534.236358 punch from 0x01094f to 0x01f122 (0xe7d4 bytes) 000906 1787787534.244348 mapread 0x02e9e5 thru 0x03a0a6 (0xb6c2 bytes) 000907 1787787534.246232 write 0x03f95e thru 0x03ffff (0x6a2 bytes) 000908 1787787534.248109 write 0x03e54e thru 0x03ffff (0x1ab2 bytes) 000909 1787787534.257094 read 0x03c1ba thru 0x03ffff (0x3e46 bytes) 000910 1787787534.262229 fallocate 0x0321da thru 0x037ba5 (0x59cc bytes) 000911 1787787534.266093 punch from 0x01de79 to 0x024922 (0x6aaa bytes) 000913 1787787534.271598 punch from 0x01044c to 0x014fc7 (0x4b7c bytes) 000914 1787787534.276368 punch from 0x02de45 to 0x0368d8 (0x8a94 bytes) 000915 1787787534.282431 mapwrite 0x032c75 thru 0x039a6e (0x6dfa bytes) 000916 1787787534.306739 mapwrite 0x023118 thru 0x0236d7 (0x5c0 bytes) 000917 1787787534.324288 mapread 0x033a1f thru 0x03ffff (0xc5e1 bytes) 000918 1787787534.326342 mapread 0x015574 thru 0x0202cc (0xad59 bytes) 000919 1787787534.336410 trunc from 0x01b1d5 to 0x029395 (0xe1c1 bytes) 000920 1787787534.341861 trunc from 0x023f4e to 0x025de6 (0x1e99 bytes) 000921 1787787534.346358 trunc from 0x00cb27 to 0x00d18e (0x668 bytes) 000922 1787787534.355304 fallocate 0x019964 thru 0x024256 (0xa8f3 bytes) 000923 1787787534.365690 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x5d8ad65c (stale) Mirror 15: 0x5d8ad65c (stale) Mirror 16: 0x5d8ad65c (stale) Mirror 17: 0x5d8ad65c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x5d8ad65c (stale) Mirror 15: 0x5d8ad65c (stale) Mirror 16: 0x5d8ad65c (stale) Mirror 17: 0x5d8ad65c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000924 1787787534.788355 mapwrite 0x023039 thru 0x02fbe7 (0xcbaf bytes) 000925 1787787534.837905 MIRROR_RESYNC 5 mirrors 000926 1787787536.280538 mapwrite 0x0158da thru 0x021f91 (0xc6b8 bytes) 000927 1787787536.418590 fallocate 0x02c9a3 thru 0x03485d (0x7ebb bytes) 000928 1787787536.428936 fallocate 0x01e45e thru 0x023f05 (0x5aa8 bytes) 000929 1787787536.441878 punch from 0x026465 to 0x02ed99 (0x8935 bytes) 000930 1787787536.457604 trunc from 0x028152 to 0x03565e (0xd50d bytes) 000932 1787787536.474248 mapread 0x01269e thru 0x016daa (0x470d bytes) 000933 1787787536.475276 mapread 0x02a896 thru 0x038d3d (0xe4a8 bytes) 000934 1787787536.513567 write 0x03c2c8 thru 0x03c30a (0x43 bytes) 000936 1787787536.523693 mapread 0x0268ad thru 0x02aa73 (0x41c7 bytes) 000937 1787787536.527051 mapwrite 0x02864d thru 0x02ace1 (0x2695 bytes) 000938 1787787536.542830 fallocate 0x0399ce thru 0x03ffff (0x6632 bytes) 000940 1787787536.547409 fallocate 0x03219c thru 0x03ae47 (0x8cac bytes) 000941 1787787536.551265 fallocate 0x030b83 thru 0x03b5b8 (0xaa36 bytes) 000943 1787787536.560191 read 0x011d23 thru 0x020c20 (0xeefe bytes) 000944 1787787536.574049 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x7b69cf08 (stale) Mirror 15: 0x7b69cf08 (stale) Mirror 16: 0x7b69cf08 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x7b69cf08 (stale) Mirror 15: 0x7b69cf08 (stale) Mirror 16: 0x7b69cf08 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000945 1787787536.941114 trunc from 0x040000 to 0x021992 (0x1e66d bytes) 000946 1787787536.970439 mapwrite 0x015069 thru 0x022351 (0xd2e9 bytes) 000947 1787787537.051509 mapwrite 0x0235ef thru 0x02741f (0x3e31 bytes) 000948 1787787537.094247 punch from 0x0231ef to 0x02741f (0x4231 bytes) 000949 1787787537.102727 write 0x01d74d thru 0x022594 (0x4e48 bytes) 000950 1787787537.104254 trunc from 0x0058c6 to 0x00a8a7 (0x4fe2 bytes) 000951 1787787537.110153 mapwrite 0x01c3c2 thru 0x01eef0 (0x2b2f bytes) 000952 1787787537.127208 trunc from 0x027420 to 0x004203 (0x2321c bytes) 000953 1787787537.148620 trunc from 0x004204 to 0x008488 (0x4285 bytes) 000954 1787787537.165413 fallocate 0x025874 thru 0x02eb30 (0x92bd bytes) 000955 1787787537.186203 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2eb31): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2eb31): Mirror 13: 0x6ae4b9a2 (stale) Mirror 15: 0x6ae4b9a2 (stale) Mirror 16: 0x6ae4b9a2 (stale) CRC-32 checksum value for chunk [0, 0x2eb31): Mirror 1: 0x4305b1c Mirror 17: 0x4305b1c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2eb31: skipped 000956 1787787538.052166 punch from 0x019f07 to 0x0236c4 (0x97be bytes) 000957 1787787538.083393 fallocate 0x01a4ae thru 0x01bacb (0x161e bytes) 000958 1787787538.088731 fallocate 0x03c858 thru 0x03ffff (0x37a8 bytes) 000959 1787787538.096843 fallocate 0x012bba thru 0x017b74 (0x4fbb bytes) 000960 1787787538.103507 read 0x00dd3c thru 0x017521 (0x97e6 bytes) 000961 1787787538.127686 read 0x030583 thru 0x03a762 (0xa1e0 bytes) 000962 1787787538.142224 write 0x016748 thru 0x01b3e0 (0x4c99 bytes) 000963 1787787538.147102 mapread 0x03b20b thru 0x03ffff (0x4df5 bytes) 000965 1787787538.151882 mapwrite 0x005aa0 thru 0x00a94c (0x4ead bytes) 000966 1787787538.196968 punch from 0x026caa to 0x02bc31 (0x4f88 bytes) 000967 1787787538.206421 mapwrite 0x0119ee thru 0x020682 (0xec95 bytes) 000968 1787787538.244187 trunc from 0x040000 to 0x02dad1 (0x1252e bytes) 000969 1787787538.280942 fallocate 0x000a83 thru 0x00c776 (0xbcf4 bytes) 000970 1787787538.289053 mapread 0x019644 thru 0x01cce1 (0x369e bytes) 000972 1787787538.296115 mapwrite 0x02af80 thru 0x02d8a9 (0x292a bytes) 000973 1787787538.350059 punch from 0x028c51 to 0x02d4e0 (0x4890 bytes) 000974 1787787538.354767 mapread 0x00c417 thru 0x0185db (0xc1c5 bytes) 000975 1787787538.359380 trunc from 0x02ce87 to 0x02dad1 (0xc4b bytes) 000976 1787787538.371090 mapread 0x000c9d thru 0x00ba57 (0xadbb bytes) 000977 1787787538.372728 punch from 0x025e25 to 0x02dad1 (0x7cad bytes) 000978 1787787538.377164 trunc from 0x00e6b1 to 0x01b2a9 (0xcbf9 bytes) 000979 1787787538.383560 mapwrite 0x00755a thru 0x0111fb (0x9ca2 bytes) 000980 1787787538.425299 mapread 0x009226 thru 0x00e282 (0x505d bytes) 000981 1787787538.426463 fallocate 0x00723c thru 0x014a1b (0xd7e0 bytes) 000982 1787787538.431041 trunc from 0x02dad2 to 0x03e0b3 (0x105e2 bytes) 000983 1787787538.443044 punch from 0x03084a to 0x034cb7 (0x446e bytes) 000987 1787787538.453633 punch from 0x00a654 to 0x00cded (0x279a bytes) 000988 1787787538.459172 trunc from 0x01443e to 0x01d3f0 (0x8fb3 bytes) 000989 1787787538.463951 trunc from 0x0147db to 0x019147 (0x496d bytes) 000992 1787787538.469793 write 0x033c0a thru 0x03d380 (0x9777 bytes) 000993 1787787538.477871 write 0x0249c9 thru 0x032df4 (0xe42c bytes) 000994 1787787538.482960 write 0x033428 thru 0x03ffff (0xcbd8 bytes) 000995 1787787538.484574 punch from 0x036991 to 0x03ffff (0x966f bytes) 000996 1787787538.500401 mapread 0x0018ce thru 0x005cc4 (0x43f7 bytes) 000997 1787787538.501938 trunc from 0x034eee to 0x034fda (0xed bytes) 000998 1787787538.506397 punch from 0x00ba31 to 0x00cc6a (0x123a bytes) 000999 1787787538.510168 trunc from 0x040000 to 0x0376bd (0x8942 bytes) 001000 1787787538.525410 mapwrite 0x024196 thru 0x0287ac (0x4617 bytes) All operations completed A-OK! PASS 70a (100s) == sanity-flr test 71: check mirror extend parent fid ==== 19:39:03 (1787787543) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x53:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (9s) == sanity-flr test 72: check append on FLR file ========== 19:39:13 (1787787553) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x117:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.213801 s, 9.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x117:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176289 s, 5.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x118:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x117:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x118:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x117:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11a:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 19:39:21 (1787787561) Starting client: oleg449-client.virtnet: -o user_xattr,flock 192.168.204.149@tcp:/lustre /mnt/lustre2 Starting client: oleg449-client.virtnet: -o user_xattr,flock 192.168.204.149@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1075200 .. Extending file size to 715200 .. Extending file size to 8160544 .. Extending file size to 5891712 .. Extending file size to 6995200 .. Extending file size to 7569824 .. Extending file size to 8372960 .. Extending file size to 7712864 .. Extending file size to 8593664 .. Extending file size to 8871616 .. Extending file size to 9004096 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 62716 62718 62719 62720 62723 failed 192.168.204.149@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg449-client.virtnet /mnt/lustre2 (opts:) 192.168.204.149@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg449-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (78s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 19:40:39 (1787787639) Starting client: oleg449-client.virtnet: -o user_xattr,flock 192.168.204.149@tcp:/lustre /mnt/lustre2 Starting client: oleg449-client.virtnet: -o user_xattr,flock 192.168.204.149@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 829696 .. Extending file size to 6699936 .. failed Extending file size to 4050720 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9039872 .. Extending file size to 8172000 .. Extending file size to 8500256 .. Extending file size to 8768544 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8833248 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 64280 64281 64284 64285 64287 192.168.204.149@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg449-client.virtnet /mnt/lustre2 (opts:) 192.168.204.149@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg449-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 19:41:56 (1787787716) Starting client: oleg449-client.virtnet: -o user_xattr,flock 192.168.204.149@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2320 1268984 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2276 1269028 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18780 3588240 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24484 3582536 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43264 7170776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2320 1268984 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2276 1269028 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18780 3588240 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24484 3582536 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43264 7170776 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.111469 s, 18.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.225778 s, 9.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0xd4d16f63 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.204.149@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg449-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 19:42:16 (1787787736) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.157541 s, 13.3 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 19:42:24 (1787787744) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x62:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x135:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 19:42:32 (1787787752) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0755458 s, 27.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x260:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x25e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x25f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 19:42:41 (1787787761) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.108089 s, 19.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x261:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x262:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x260:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x261:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x262:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x260:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x261:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 19:42:50 (1787787770) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0777907 s, 27.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x136:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x136:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 19:43:23 (1787787803) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg449-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106421 s, 9.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x263:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x263:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x262:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x263:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x263:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x262:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x264:0x0] } PASS 204d (50s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 19:44:13 (1787787853) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg449-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x138:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0718428 s, 29.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x138:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x138:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x138:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x138:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (54s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 19:45:07 (1787787907) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg449-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg449-server: Pool lustre.test_204f created Adding targets to pool oleg449-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg449-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.140622 s, 14.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x139:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x139:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x139:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x139:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg449-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg449-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg449-server: Pool lustre.test_204f destroyed PASS 204f (76s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 19:46:23 (1787787983) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 19:46:32 (1787787992) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 19:46:40 (1787788000) oleg449-server: Pool lustre.test_206 created Adding targets to pool oleg449-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg449-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg449-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg449-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg449-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 19:47:07 (1787788027) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822883 s, 12.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296675 s, 35.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xe184560f Mirror 3: 0xe184560f PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 19:47:18 (1787788038) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 19:47:23 (1787788043) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 19:47:28 (1787788048) Starting client: oleg449-client.virtnet: -o user_xattr,flock 192.168.204.149@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.037696 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1930000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x15f0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x15f0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x15f0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x800000 Read this (4 bytes): BBBB 192.168.204.149@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg449-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 19:48:08 (1787788088) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0237562 s, 345 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 19:48:17 (1787788097) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374598 s, 28.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x81 lmm_fid: [0x200000405:0x81:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x81 lmm_fid: [0x200000405:0x81:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x143:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611265 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 19:48:19.000000000 -0400 Modify: 2026-08-26 19:48:19.000000000 -0400 Change: 2026-08-26 19:48:20.000000000 -0400 Birth: 2026-08-26 19:48:19.000000000 -0400 PASS 210a (11s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 19:48:27 (1787788107) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509387 s, 20.6 MB/s oleg449-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg449-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg449-server: O/280000401/d4/144: File not found by ext2_lookup PASS 210b (23s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 19:48:50 (1787788130) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.168892 s, 243 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0198406 s, 206 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 26 19:48 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x86:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 26 19:48 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 19:48:58 (1787788138) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 19:49:06 (1787788146) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1759 sec =========== 19:49:16 (1787788156) === sanity-flr: start cleanup 19:49:19 (1787788159) === === sanity-flr: finish cleanup 19:49:29 (1787788169) ===