-----============= acceptance-small: sanity-flr ============----- Sun Sep 6 16:22:20 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg104-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 16:22:40 (1788726160) === oleg104-client.virtnet: executing check_config_client /mnt/lustre oleg104-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg104-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff970891e6f800.idle_timeout=debug osc.lustre-OST0001-osc-ffff970891e6f800.idle_timeout=debug disable quota as required oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 16:23:15 (1788726195) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8230] == sanity-flr test 0a: lfs mirror create with -N option == 16:23:18 (1788726198) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 16:23:36 (1788726216) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 16:23:39 (1788726219) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 16:23:42 (1788726222) PASS 0d (21s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 16:24:03 (1788726243) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 16:24:06 (1788726246) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 16:24:09 (1788726249) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff970891e6f800.stats=clear osc.lustre-OST0001-osc-ffff970891e6f800.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 16:24:15 (1788726255) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 16:24:26 (1788726266) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 16:24:34 (1788726274) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 16:24:37 (1788726277) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 16:24:45 (1788726285) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 16:24:56 (1788726296) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 16:25:02 (1788726302) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 16:25:10 (1788726310) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 16:25:19 (1788726319) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 16:25:29 (1788726329) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 16:25:36 (1788726336) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 1.11596 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810972 s, 12.9 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 16:25:47 (1788726347) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.500483 s, 25.1 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 16:25:55 (1788726355) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 16:26:05 (1788726365) Stopping /mnt/lustre-ost1 (opts:) on oleg104-server oleg104-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff970891e6f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff970891e6f800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg104-server: oleg104-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg104-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg104-server oleg104-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff970891e6f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff970891e6f800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg104-server: oleg104-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg104-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (78s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 16:27:24 (1788726444) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.410204 s, 15.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg104-server oleg104-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff970891e6f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff970891e6f800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg104-server: oleg104-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg104-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (49s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 16:28:13 (1788726493) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg104-server oleg104-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff970891e6f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff970891e6f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg104-server: oleg104-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg104-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg104-server oleg104-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff970891e6f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff970891e6f800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg104-server: oleg104-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg104-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (81s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 16:29:34 (1788726574) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.27045 s, 23.1 MB/s -rw-r--r-- 1 root root 52428800 Sep 6 16:29 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.960s user 0m0.000s sys 0m0.198s Stopping /mnt/lustre-ost1 (opts:) on oleg104-server oleg104-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff970891e6f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff970891e6f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.043s user 0m0.000s sys 0m0.203s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg104-server: oleg104-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg104-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg104-server oleg104-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff970891e6f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff970891e6f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.206s user 0m0.001s sys 0m0.198s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg104-server: oleg104-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg104-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg104-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff970891e6f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff970891e6f800.ost_server_uuid in FULL state after 0 sec PASS 33b (91s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 16:31:05 (1788726665) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 16:31:09 (1788726669) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 16:31:13 (1788726673) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 16:31:17 (1788726677) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 16:31:26 (1788726686) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.539531 s, 23.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (16s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 16:31:42 (1788726702) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310447 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287506 s, 36.5 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 16:31:50 (1788726710) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 16:31:59 (1788726719) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00198429 s, 516 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00222957 s, 459 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00163183 s, 628 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000966873 s, 1.1 MB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 16:32:10 (1788726730) create test files with size 15741859 .. 96a9316ceb569142cf89d27600f3730e - 928a2820c8a727316c701f8eb88c67b9 - 498716ccdaefbb76bd3a5242753f162a - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg104-client.virtnet /mnt/lustre (opts:) Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre PASS 37 (28s) == sanity-flr test 38: resync ============================ 16:32:38 (1788726758) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 16:32:52 (1788726772) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 16:33:00 (1788726780) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139495 s, 7.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290378 s, 36.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 16:33:10 (1788726790) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0485116 s, 43.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.405188 s, 10.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.149003 s, 14.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 16:33:28 (1788726808) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (5s) == sanity-flr test 43a: mirror pick on write ============= 16:33:33 (1788726813) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 16:33:38 (1788726818) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628603 s, 16.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 16:33:46 (1788726826) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.196513 s, 16.0 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 16:33:58 (1788726838) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (12s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 16:34:12 (1788726852) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.504967 s, 20.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg104-client.virtnet /mnt/lustre (opts:) Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (19s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 16:34:31 (1788726871) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff97089220f800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.270818 s, 38.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg104-client.virtnet /mnt/lustre (opts:) Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff97088778d000.xattr_cache=1 PASS 44e (18s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 16:34:49 (1788726889) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (47s) == sanity-flr test 46: Verify setstripe --copy option ==== 16:35:37 (1788726937) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 16:36:18 (1788726978) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 16:36:22 (1788726982) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.275365 s, 11.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.215153 s, 14.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xb632ba95 (nosync,stale) Mirror 2: 0xb632ba95 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 92adff8f7539ee2dcc7c95f07ab695d5 - mirror 1 checksum: fd3c4e9b8b6af09cc7bf9c358624e6d7 - mirror 2 checksum: 92adff8f7539ee2dcc7c95f07ab695d5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-06T16:36:25-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-06T16:36:25-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 92adff8f7539ee2dcc7c95f07ab695d5 - mirror 1 checksum: fd3c4e9b8b6af09cc7bf9c358624e6d7 - mirror 2 checksum: fd3c4e9b8b6af09cc7bf9c358624e6d7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-06T16:36:25-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-06T16:36:25-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } PASS 48 (13s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 16:36:36 (1788726996) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388894 s, 27.0 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (11s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 16:36:46 (1788727006) Failing mds1 on oleg104-server Stopping /mnt/lustre-mds1 (opts:) on oleg104-server 16:37:00 (1788727020) shut down facet: mds1 facet_host: oleg104-server facet_failover_host: oleg104-server Failover mds1 to oleg104-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0000 16:37:33 (1788727053) targets are mounted 16:37:33 (1788727053) facet_failover done oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045987 s, 22.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } PASS 50A (69s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 16:37:55 (1788727075) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0528834 s, 387 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0228208 s, 897 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (12s) == sanity-flr test 50b: mirror rsync handles sparseness == 16:38:08 (1788727088) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.061057 s, 335 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00922722 s, 2.2 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (12s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 16:38:20 (1788727100) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0109206 s, 1.5 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 16:38:28 (1788727108) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0596991 s, 343 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 16:38:37 (1788727117) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00472361 s, 867 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0397295 s, 103 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 16:38:45 (1788727125) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 16:39:08 (1788727148) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (32s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 16:39:41 (1788727181) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (27s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 16:40:08 (1788727208) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 16:40:36 (1788727236) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 16:40:45 (1788727245) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57577 mirror split pid 57578 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7845: 57577 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7019: 57578 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7019: 57578 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57578 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 16:41:54 (1788727314) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 7316 skipping zero size read skipping zero size read skipping zero length zero range skipping zero size read skipping zero size read 000007 1788727316.733242 mapwrite 0x00da5b thru 0x01d51b (0xfac1 bytes) 000009 1788727316.826458 read 0x011791 thru 0x01d51b (0xbd8b bytes) 000010 1788727316.828199 write 0x01ad9a thru 0x0263c1 (0xb628 bytes) 000011 1788727316.841454 read 0x022b62 thru 0x0263c1 (0x3860 bytes) 000012 1788727316.844263 punch from 0x005c21 to 0x00c7ab (0x6b8b bytes) 000013 1788727316.861107 MIRROR_EXTEND to 2 mirrors 000014 1788727317.745164 read 0x01f370 thru 0x0263c1 (0x7052 bytes) 000015 1788727317.753205 read 0x024255 thru 0x0263c1 (0x216d bytes) 000016 1788727317.753797 punch from 0x019902 to 0x01fb7b (0x627a bytes) truncating to largest ever: 0x16ba5 000017 1788727317.793294 trunc from 0x0263c2 to 0x016ba4 (0xf81d bytes) 000018 1788727317.818090 write 0x03033d thru 0x03a4c4 (0xa188 bytes) 000019 1788727317.819558 mapwrite 0x038ddd thru 0x03ffff (0x7223 bytes) truncating to largest ever: 0x1f4b2 000020 1788727317.907777 trunc from 0x040000 to 0x01f4b1 (0x20b4e bytes) 000021 1788727317.934608 write 0x00ad58 thru 0x01259e (0x7847 bytes) 000022 1788727317.937584 trunc from 0x008826 to 0x0151eb (0xc9c6 bytes) 000023 1788727317.954794 read 0x01a107 thru 0x01f2d5 (0x51cf bytes) 000024 1788727317.961602 MIRROR_RESYNC 2 mirrors 000025 1788727318.871091 mapwrite 0x008de8 thru 0x00baac (0x2cc5 bytes) 000026 1788727318.966703 read 0x0158ef thru 0x015fc2 (0x6d4 bytes) 000027 1788727318.977754 mapwrite 0x02b001 thru 0x02ca64 (0x1a64 bytes) 000028 1788727319.087076 MIRROR_RESYNC 2 mirrors 000030 1788727320.071839 mapread 0x00ee98 thru 0x012377 (0x34e0 bytes) fallocating to largest ever: 0x2f8f4 000031 1788727320.092824 fallocate 0x02a7f2 thru 0x02f8f3 (0x5102 bytes) truncating to largest ever: 0x3fb42 000032 1788727320.136364 trunc from 0x02f8f4 to 0x03fb41 (0x1024e bytes) 000033 1788727320.159560 read 0x00ac00 thru 0x00c664 (0x1a65 bytes) 000035 1788727320.168669 write 0x02e07e thru 0x02e8f3 (0x876 bytes) 000036 1788727320.175155 mapwrite 0x03c010 thru 0x03ffff (0x3ff0 bytes) 000038 1788727320.240288 trunc from 0x00d354 to 0x0150be (0x7d6b bytes) 000039 1788727320.246462 trunc from 0x040000 to 0x02da1f (0x125e0 bytes) 000040 1788727320.266810 mapwrite 0x000624 thru 0x00cf74 (0xc951 bytes) 000041 1788727320.306764 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) 000042 1788727320.536292 fallocate 0x02b866 thru 0x03b1c8 (0xf963 bytes) 000043 1788727320.545276 mapwrite 0x01bf12 thru 0x02b2ca (0xf3b9 bytes) 000044 1788727320.624761 mapread 0x00f81e thru 0x011d13 (0x24f6 bytes) 000045 1788727320.647452 trunc from 0x02da20 to 0x00a268 (0x237b7 bytes) 000046 1788727320.678872 read 0x00430e thru 0x00a268 (0x5f5b bytes) 000047 1788727320.685383 trunc from 0x00a269 to 0x037878 (0x2d610 bytes) 000048 1788727320.713551 mapread 0x02aabc thru 0x03626c (0xb7b1 bytes) 000049 1788727320.749830 read 0x00ed45 thru 0x00f5e6 (0x8a2 bytes) 000051 1788727320.769407 write 0x017d51 thru 0x01886c (0xb1c bytes) 000052 1788727320.770595 read 0x01038a thru 0x014f60 (0x4bd7 bytes) 000053 1788727320.773181 mapread 0x02cdf4 thru 0x0351e8 (0x83f5 bytes) 000054 1788727320.775874 trunc from 0x02c1b7 to 0x0336d6 (0x7520 bytes) 000057 1788727320.787622 trunc from 0x037879 to 0x0293da (0xe49e bytes) 000058 1788727320.823111 read 0x01e47c thru 0x0293da (0xaf5f bytes) 000060 1788727320.825299 mapread 0x000618 thru 0x00ce4e (0xc837 bytes) 000061 1788727320.826801 read 0x0009da thru 0x00f55b (0xeb82 bytes) 000062 1788727320.828094 read 0x006343 thru 0x012c47 (0xc905 bytes) 000063 1788727320.829846 mapread 0x003148 thru 0x00f43e (0xc2f7 bytes) 000065 1788727320.833472 fallocate 0x01b754 thru 0x01f6de (0x3f8b bytes) 000066 1788727320.843620 read 0x003863 thru 0x0090fa (0x5898 bytes) 000067 1788727320.844561 trunc from 0x0293db to 0x0368cb (0xd4f1 bytes) 000068 1788727320.890468 mapread 0x005c72 thru 0x007a1f (0x1dae bytes) 000069 1788727320.894171 fallocate 0x00a3a7 thru 0x014723 (0xa37d bytes) 000071 1788727320.903248 mapread 0x00c0d1 thru 0x014c64 (0x8b94 bytes) 000072 1788727320.905585 trunc from 0x016835 to 0x01e768 (0x7f34 bytes) 000073 1788727320.940782 punch from 0x022ea3 to 0x02e8ba (0xba18 bytes) 000074 1788727320.956473 mapwrite 0x01550b thru 0x016b04 (0x15fa bytes) 000075 1788727321.007666 trunc from 0x0368cc to 0x02f6f6 (0x71d5 bytes) 000076 1788727321.063625 punch from 0x017402 to 0x025df8 (0xe9f7 bytes) 000077 1788727321.071227 punch from 0x0141e6 to 0x0212f0 (0xd10b bytes) 000078 1788727321.088223 trunc from 0x02f6f7 to 0x02b805 (0x3ef1 bytes) 000080 1788727321.119460 write 0x0229e6 thru 0x02ff1b (0xd536 bytes) 000081 1788727321.141797 read 0x00979f thru 0x00af0a (0x176c bytes) 000082 1788727321.143236 trunc from 0x02ff1c to 0x020d8e (0xf18d bytes) 000083 1788727321.171784 mapwrite 0x009225 thru 0x00d77b (0x4557 bytes) 000085 1788727321.224658 trunc from 0x020d8f to 0x00f21f (0x11b6f bytes) 000086 1788727321.248297 punch from 0x008ade to 0x00b233 (0x2756 bytes) 000087 1788727321.272715 MIRROR_EXTEND to 2 mirrors 000088 1788727322.300270 trunc from 0x00f220 to 0x0174c1 (0x82a2 bytes) 000089 1788727322.383645 mapwrite 0x0103cd thru 0x01dbda (0xd80e bytes) 000090 1788727322.472447 mapread 0x0156c5 thru 0x01dbda (0x8516 bytes) 000091 1788727322.473460 read 0x00077f thru 0x010736 (0xffb8 bytes) 000092 1788727322.486386 mapwrite 0x03cdaa thru 0x03ffff (0x3256 bytes) 000094 1788727322.533727 write 0x03ac8c thru 0x03ffff (0x5374 bytes) 000096 1788727322.536713 mapwrite 0x02b891 thru 0x02f519 (0x3c89 bytes) 000097 1788727322.591845 punch from 0x0382f4 to 0x0389d8 (0x6e5 bytes) 000098 1788727322.599573 trunc from 0x02f288 to 0x037ef4 (0x8c6d bytes) 000099 1788727322.608699 mapwrite 0x02c439 thru 0x035de0 (0x99a8 bytes) 000100 1788727322.653515 trunc from 0x040000 to 0x03cc10 (0x33ef bytes) 000102 1788727322.675773 write 0x036341 thru 0x03ffff (0x9cbf bytes) 000103 1788727322.678767 read 0x007d7a thru 0x0165ca (0xe851 bytes) 000105 1788727322.680161 trunc from 0x035007 to 0x03a386 (0x5380 bytes) 000106 1788727322.697626 read 0x01359b thru 0x01fc56 (0xc6bc bytes) 000107 1788727322.708404 read 0x011701 thru 0x01e7c0 (0xd0c0 bytes) 000108 1788727322.709332 trunc from 0x013118 to 0x01af90 (0x7e79 bytes) 000109 1788727322.715189 write 0x02e842 thru 0x03a11d (0xb8dc bytes) 000110 1788727322.718487 mapread 0x02ab3b thru 0x033eac (0x9372 bytes) 000111 1788727322.719713 fallocate 0x029d56 thru 0x03250b (0x87b6 bytes) 000112 1788727322.724802 trunc from 0x02c7c5 to 0x03b46e (0xecaa bytes) 000113 1788727322.740822 trunc from 0x032247 to 0x03e150 (0xbf0a bytes) 000114 1788727322.746796 trunc from 0x040000 to 0x028f86 (0x17079 bytes) 000115 1788727322.784520 trunc from 0x00b17e to 0x01ad2e (0xfbb1 bytes) 000116 1788727322.794320 trunc from 0x028f87 to 0x00cd38 (0x1c24e bytes) 000118 1788727322.837813 mapread 0x0092bc thru 0x00986b (0x5b0 bytes) 000119 1788727322.838955 trunc from 0x004236 to 0x00c4c2 (0x828d bytes) 000120 1788727322.846659 punch from 0x005c1a to 0x009065 (0x344c bytes) 000121 1788727322.871480 MIRROR_RESYNC 2 mirrors 000122 1788727323.728797 mapread 0x000275 thru 0x008d6c (0x8af8 bytes) 000123 1788727323.777554 mapwrite 0x031b9e thru 0x037d49 (0x61ac bytes) 000124 1788727323.934019 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37d4a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37d4a): Mirror 2: 0xbd3cb1eb (stale) CRC-32 checksum value for chunk [0, 0x37d4a): Mirror 1: 0xff65cc37 Mirror 3: 0xff65cc37 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37d4a: skipped 000125 1788727325.134766 fallocate 0x00ed21 thru 0x01ed0c (0xffec bytes) 000126 1788727325.188402 write 0x026e03 thru 0x032503 (0xb701 bytes) 000127 1788727325.207815 fallocate 0x00d6e4 thru 0x01cbce (0xf4eb bytes) 000128 1788727325.215753 read 0x018e69 thru 0x024593 (0xb72b bytes) 000129 1788727325.236332 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37d4a): Mirror 2: 0xbd3cb1eb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37d4a): Mirror 2: 0xbd3cb1eb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37d4a: skipped 000130 1788727325.571737 trunc from 0x01e56c to 0x028acc (0xa561 bytes) 000131 1788727325.580529 read 0x029bfc thru 0x02d4ab (0x38b0 bytes) 000132 1788727325.607389 trunc from 0x018ede to 0x026d71 (0xde94 bytes) 000133 1788727325.614809 write 0x011510 thru 0x0186f8 (0x71e9 bytes) 000135 1788727325.667179 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) 000136 1788727325.977612 mapread 0x016cf9 thru 0x026582 (0xf88a bytes) 000137 1788727326.028334 mapwrite 0x0100ec thru 0x019cd5 (0x9bea bytes) 000138 1788727326.074101 mapwrite 0x028fbf thru 0x038fbe (0x10000 bytes) 000139 1788727326.134682 mapread 0x0009e3 thru 0x008829 (0x7e47 bytes) 000140 1788727326.143463 mapwrite 0x03f923 thru 0x03ffff (0x6dd bytes) 000142 1788727326.183036 mapread 0x00038c thru 0x007788 (0x73fd bytes) 000144 1788727326.185806 trunc from 0x040000 to 0x0340ad (0xbf52 bytes) 000145 1788727326.204044 trunc from 0x02d52b to 0x0340ad (0x6b83 bytes) 000146 1788727326.210824 punch from 0x02100a to 0x026554 (0x554b bytes) 000148 1788727326.222410 mapwrite 0x03b909 thru 0x03d54a (0x1c42 bytes) 000150 1788727326.318231 trunc from 0x02b711 to 0x0355a4 (0x9e94 bytes) 000152 1788727326.339596 read 0x017804 thru 0x01b450 (0x3c4d bytes) 000154 1788727326.340667 write 0x0195da thru 0x0248c8 (0xb2ef bytes) 000156 1788727326.379740 mapread 0x011b03 thru 0x019c71 (0x816f bytes) 000157 1788727326.380560 trunc from 0x03d54b to 0x00b16c (0x323de bytes) 000158 1788727326.421223 write 0x01fadd thru 0x0283a2 (0x88c6 bytes) 000159 1788727326.422849 punch from 0x005d3f to 0x006ae3 (0xda5 bytes) 000160 1788727326.447477 trunc from 0x0283a3 to 0x001a07 (0x2699b bytes) 000162 1788727326.505205 punch from 0x0013b0 to 0x001a07 (0x658 bytes) 000163 1788727326.512439 fallocate 0x019de5 thru 0x026b9b (0xcdb7 bytes) 000164 1788727326.535769 fallocate 0x0047bd thru 0x004fc8 (0x80c bytes) 000165 1788727326.558685 mapwrite 0x02de05 thru 0x030982 (0x2b7e bytes) 000166 1788727326.668069 MIRROR_EXTEND to 2 mirrors 000167 1788727327.539589 trunc from 0x00d75a to 0x01b2da (0xdb81 bytes) 000168 1788727327.641468 mapwrite 0x029316 thru 0x031049 (0x7d34 bytes) 000169 1788727327.709003 mapwrite 0x0093f7 thru 0x00c598 (0x31a2 bytes) 000170 1788727327.735326 write 0x022930 thru 0x025c94 (0x3365 bytes) 000171 1788727327.743631 fallocate 0x008855 thru 0x013873 (0xb01f bytes) 000172 1788727327.750067 trunc from 0x02b61d to 0x031049 (0x5a2d bytes) 000173 1788727327.763206 MIRROR_RESYNC 2 mirrors 000174 1788727328.841952 MIRROR_EXTEND to 3 mirrors 000175 1788727329.895072 mapread 0x0188dd thru 0x01c180 (0x38a4 bytes) 000176 1788727329.958629 fallocate 0x01d66c thru 0x025dcc (0x8761 bytes) 000177 1788727330.012440 mapread 0x00b36d thru 0x00deaf (0x2b43 bytes) 000178 1788727330.034426 mapread 0x003766 thru 0x009816 (0x60b1 bytes) 000179 1788727330.044452 trunc from 0x02902c to 0x02a3c8 (0x139d bytes) 000180 1788727330.051252 trunc from 0x03104a to 0x02272f (0xe91a bytes) 000181 1788727330.093438 read 0x01e226 thru 0x02272f (0x450a bytes) 000182 1788727330.103436 trunc from 0x01b591 to 0x02272f (0x719f bytes) 000183 1788727330.110376 trunc from 0x004a60 to 0x0138ff (0xeea0 bytes) 000184 1788727330.116509 trunc from 0x00e338 to 0x01a2c6 (0xbf8f bytes) 000186 1788727330.124751 fallocate 0x022743 thru 0x031a9a (0xf358 bytes) 000187 1788727330.130673 punch from 0x01062b to 0x0137b1 (0x3187 bytes) 000188 1788727330.135728 trunc from 0x031a9b to 0x02c54b (0x554f bytes) 000189 1788727330.160349 mapread 0x02a151 thru 0x02c54b (0x23fb bytes) 000190 1788727330.174026 trunc from 0x02c54c to 0x03e273 (0x11d28 bytes) 000191 1788727330.198365 mapread 0x003d53 thru 0x00bb1d (0x7dcb bytes) 000193 1788727330.213651 mapwrite 0x029986 thru 0x037c49 (0xe2c4 bytes) 000194 1788727330.260488 mapwrite 0x02e7d8 thru 0x0344b1 (0x5cda bytes) 000196 1788727330.285870 fallocate 0x003249 thru 0x007f8b (0x4d43 bytes) 000197 1788727330.291306 write 0x00b227 thru 0x019596 (0xe370 bytes) 000198 1788727330.298298 read 0x016638 thru 0x017767 (0x1130 bytes) 000199 1788727330.299052 mapread 0x01ee02 thru 0x020d57 (0x1f56 bytes) 000200 1788727330.312999 trunc from 0x00bf41 to 0x00e7e1 (0x28a1 bytes) 000201 1788727330.350139 read 0x00ffda thru 0x0178c1 (0x78e8 bytes) 000202 1788727330.350779 punch from 0x03bbb8 to 0x03e273 (0x26bc bytes) 000203 1788727330.364456 trunc from 0x03e274 to 0x0131d8 (0x2b09b bytes) 000204 1788727330.388878 write 0x02677f thru 0x03516a (0xe9ec bytes) 000205 1788727330.391111 write 0x005c7f thru 0x006bb0 (0xf32 bytes) fallocating to largest ever: 0x40000 000206 1788727330.394089 fallocate 0x033c80 thru 0x03ffff (0xc380 bytes) 000207 1788727330.404342 fallocate 0x039440 thru 0x03ffff (0x6bc0 bytes) 000208 1788727330.409471 fallocate 0x0114b1 thru 0x014944 (0x3494 bytes) 000209 1788727330.415716 trunc from 0x00f59c to 0x00fe76 (0x8db bytes) 000210 1788727330.426353 trunc from 0x03516b to 0x003bb5 (0x315b5 bytes) 000211 1788727330.444749 write 0x030f6c thru 0x035d59 (0x4dee bytes) 000212 1788727330.445699 fallocate 0x03f03b thru 0x03ffff (0xfc5 bytes) 000213 1788727330.454897 read 0x0011e8 thru 0x00541d (0x4236 bytes) 000214 1788727330.475461 MIRROR_RESYNC 3 mirrors 000215 1788727331.678796 punch from 0x021e1a to 0x02d8c5 (0xbaac bytes) 000216 1788727331.774234 write 0x01ac04 thru 0x02a11e (0xf51b bytes) 000217 1788727331.796413 trunc from 0x040000 to 0x03bcba (0x4345 bytes) 000218 1788727331.836405 write 0x02ee2b thru 0x030881 (0x1a57 bytes) 000219 1788727331.852040 read 0x030713 thru 0x0357e9 (0x50d7 bytes) 000220 1788727331.865863 fallocate 0x031f33 thru 0x03ffff (0xe0cd bytes) 000221 1788727331.874382 trunc from 0x00a5fb to 0x0165fa (0xc000 bytes) 000222 1788727331.879124 write 0x03071f thru 0x03cb9c (0xc47e bytes) 000223 1788727331.888082 MIRROR_RESYNC 3 mirrors 000225 1788727332.759191 read 0x031cd0 thru 0x03ba12 (0x9d43 bytes) 000226 1788727332.768530 write 0x032a5a thru 0x038004 (0x55ab bytes) 000228 1788727332.811117 write 0x002f94 thru 0x00b41b (0x8488 bytes) 000230 1788727332.824234 punch from 0x02dfc1 to 0x0339bb (0x59fb bytes) 000231 1788727332.863787 trunc from 0x0381a8 to 0x03cb9c (0x49f5 bytes) 000232 1788727332.870934 trunc from 0x034ca9 to 0x0399e1 (0x4d39 bytes) 000233 1788727332.880049 fallocate 0x025df5 thru 0x02bf74 (0x6180 bytes) 000234 1788727332.888396 mapwrite 0x007168 thru 0x009573 (0x240c bytes) 000235 1788727332.914455 punch from 0x01b792 to 0x01bbaf (0x41e bytes) 000236 1788727332.920385 trunc from 0x03cb9d to 0x00372b (0x39471 bytes) 000237 1788727332.949977 mapread 0x000bb3 thru 0x00372b (0x2b79 bytes) 000238 1788727332.968022 read 0x0031ea thru 0x00372b (0x542 bytes) 000239 1788727332.969215 trunc from 0x001345 to 0x002765 (0x1421 bytes) 000240 1788727332.979677 write 0x011c4a thru 0x01ce63 (0xb21a bytes) 000241 1788727332.990090 write 0x01c084 thru 0x024d97 (0x8d14 bytes) 000242 1788727333.002175 trunc from 0x024d98 to 0x01c7cb (0x85cc bytes) 000243 1788727333.050595 read 0x0130af thru 0x013524 (0x476 bytes) 000244 1788727333.051519 fallocate 0x027ced thru 0x0373fb (0xf70f bytes) 000246 1788727333.063345 trunc from 0x01da83 to 0x029f5b (0xc4d9 bytes) 000248 1788727333.070552 trunc from 0x00ce89 to 0x013620 (0x6798 bytes) 000250 1788727333.086422 punch from 0x02e3f9 to 0x0373fb (0x9003 bytes) 000251 1788727333.091607 read 0x027fe8 thru 0x0307bf (0x87d8 bytes) 000252 1788727333.120341 write 0x00f777 thru 0x01abf2 (0xb47c bytes) 000253 1788727333.121805 mapwrite 0x03dfe4 thru 0x03f35f (0x137c bytes) 000254 1788727333.239966 mapread 0x03a786 thru 0x03f35f (0x4bda bytes) 000255 1788727333.241002 trunc from 0x013233 to 0x014e70 (0x1c3e bytes) 000256 1788727333.253936 fallocate 0x002d53 thru 0x006801 (0x3aaf bytes) 000257 1788727333.268622 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f360): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f360): Mirror 2: 0x5ad11b1d (stale) Mirror 3: 0x5ad11b1d (stale) CRC-32 checksum value for chunk [0, 0x3f360): Mirror 1: 0x6fbe4380 Mirror 4: 0x6fbe4380 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f360: skipped 000258 1788727334.143583 write 0x037639 thru 0x03a40c (0x2dd4 bytes) 000259 1788727334.174962 write 0x035359 thru 0x03ffff (0xaca7 bytes) 000260 1788727334.179543 punch from 0x02ca20 to 0x02f830 (0x2e11 bytes) 000261 1788727334.183056 trunc from 0x033839 to 0x03ffff (0xc7c7 bytes) 000263 1788727334.192562 trunc from 0x02a98c to 0x038472 (0xdae7 bytes) 000265 1788727334.198479 mapwrite 0x01593e thru 0x01ae5e (0x5521 bytes) 000266 1788727334.250677 trunc from 0x03917c to 0x03ffff (0x6e84 bytes) 000268 1788727334.260556 trunc from 0x040000 to 0x005276 (0x3ad89 bytes) 000269 1788727334.321502 mapread 0x0011fd thru 0x005276 (0x407a bytes) 000270 1788727334.339899 fallocate 0x02ba09 thru 0x03a072 (0xe66a bytes) 000271 1788727334.345837 mapread 0x0018a1 thru 0x005276 (0x39d6 bytes) 000272 1788727334.347325 mapwrite 0x00fdb5 thru 0x010fab (0x11f7 bytes) 000273 1788727334.389766 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10fac): Mirror 2: 0xc9d0dbf1 (stale) Mirror 3: 0xc9d0dbf1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10fac): Mirror 2: 0xc9d0dbf1 (stale) Mirror 3: 0xc9d0dbf1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10fac: skipped 000274 1788727334.994681 trunc from 0x010fac to 0x00f09f (0x1f0c bytes) 000275 1788727335.076065 write 0x039990 thru 0x03ffff (0x6670 bytes) 000277 1788727335.088998 trunc from 0x040000 to 0x0348c7 (0xb738 bytes) 000279 1788727335.147031 mapwrite 0x018f3c thru 0x01cb8e (0x3c53 bytes) 000280 1788727335.229327 mapread 0x004dbe thru 0x0053bf (0x602 bytes) 000281 1788727335.279125 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x348c8): Mirror 2: 0x5c619953 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x348c8): Mirror 2: 0x5c619953 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x348c8: skipped 000282 1788727335.859890 mapread 0x00cc11 thru 0x0132a8 (0x6698 bytes) 000283 1788727335.935576 mapread 0x02df36 thru 0x0343a7 (0x6472 bytes) 000284 1788727335.955071 read 0x01582a thru 0x017a4a (0x2221 bytes) 000285 1788727335.964021 trunc from 0x00f451 to 0x01075f (0x130f bytes) 000286 1788727335.975498 fallocate 0x0047e4 thru 0x00a035 (0x5852 bytes) 000287 1788727336.003798 write 0x013fea thru 0x0184b7 (0x44ce bytes) 000288 1788727336.029563 MIRROR_RESYNC 2 mirrors 000289 1788727337.437091 mapwrite 0x038f19 thru 0x03ffff (0x70e7 bytes) 000290 1788727337.636761 trunc from 0x040000 to 0x028406 (0x17bf9 bytes) 000292 1788727337.660865 write 0x019f21 thru 0x01b8dd (0x19bd bytes) 000293 1788727337.674103 mapwrite 0x03140f thru 0x036681 (0x5273 bytes) 000294 1788727337.808424 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36682): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36682): Mirror 2: 0x5dfc3c29 (stale) CRC-32 checksum value for chunk [0, 0x36682): Mirror 1: 0xd8589950 Mirror 3: 0xd8589950 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36682: skipped 000295 1788727339.259283 read 0x01155f thru 0x012065 (0xb07 bytes) 000296 1788727339.270357 write 0x01eb03 thru 0x0290a0 (0xa59e bytes) 000297 1788727339.299320 fallocate 0x00213c thru 0x005ca8 (0x3b6d bytes) 000298 1788727339.303931 trunc from 0x036682 to 0x01a3f0 (0x1c291 bytes) 000299 1788727339.321443 read 0x010f94 thru 0x01a3f0 (0x945d bytes) 000300 1788727339.336348 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1a3f1): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a3f1): Mirror 2: 0x6de06761 (stale) Mirror 3: 0xc6e4133a (stale) CRC-32 checksum value for chunk [0, 0x1a3f1): Mirror 1: 0xc6e4133a Mirror 4: 0xc6e4133a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a3f1: skipped 000301 1788727340.099846 punch from 0x004961 to 0x007100 (0x27a0 bytes) 000302 1788727340.210423 read 0x0073c5 thru 0x0080f0 (0xd2c bytes) 000303 1788727340.216798 mapread 0x012e59 thru 0x014fb2 (0x215a bytes) 000304 1788727340.240952 mapread 0x0012da thru 0x009a01 (0x8728 bytes) 000305 1788727340.252809 fallocate 0x033be4 thru 0x03c2e2 (0x86ff bytes) 000306 1788727340.262508 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1a3f1): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a3f1): Mirror 2: 0x6de06761 (stale) Mirror 3: 0xc6e4133a (stale) Mirror 4: 0xc6e4133a (stale) CRC-32 checksum value for chunk [0, 0x1a3f1): Mirror 1: 0xc6e4133a Mirror 5: 0xc6e4133a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a3f1: skipped 000307 1788727341.457429 trunc from 0x01a3f1 to 0x01c44d (0x205d bytes) 000308 1788727341.516398 trunc from 0x01c44e to 0x00db16 (0xe937 bytes) 000309 1788727341.540201 punch from 0x003506 to 0x007dd5 (0x48d0 bytes) 000310 1788727341.551465 MIRROR_RESYNC 5 mirrors 000311 1788727342.597869 fallocate 0x02fc19 thru 0x0319ad (0x1d95 bytes) 000312 1788727342.644285 fallocate 0x005640 thru 0x00c504 (0x6ec5 bytes) 000313 1788727342.650060 write 0x03cf93 thru 0x03e20e (0x127c bytes) 000314 1788727342.652413 mapread 0x02f47c thru 0x03b875 (0xc3fa bytes) 000315 1788727342.687341 fallocate 0x02d895 thru 0x034cb6 (0x7422 bytes) 000316 1788727342.694074 fallocate 0x0287b4 thru 0x0290c0 (0x90d bytes) 000317 1788727342.707173 MIRROR_RESYNC 5 mirrors 000318 1788727343.600591 trunc from 0x03e20f to 0x02a8a4 (0x1396a bytes) 000320 1788727343.647756 trunc from 0x02a8a5 to 0x027866 (0x303e bytes) 000321 1788727343.666202 mapread 0x014050 thru 0x01766e (0x361f bytes) 000322 1788727343.683722 write 0x008b24 thru 0x0103b1 (0x788e bytes) 000323 1788727343.689309 read 0x00729e thru 0x016eb0 (0xfc13 bytes) 000324 1788727343.699654 trunc from 0x023fd3 to 0x027866 (0x3894 bytes) 000325 1788727343.704788 fallocate 0x0018a9 thru 0x00b243 (0x999b bytes) 000326 1788727343.710865 trunc from 0x027867 to 0x02c3ac (0x4b46 bytes) 000327 1788727343.734516 mapread 0x003839 thru 0x00f2c1 (0xba89 bytes) 000328 1788727343.746355 trunc from 0x02c3ad to 0x0105e4 (0x1bdc8 bytes) 000329 1788727343.778469 fallocate 0x039b24 thru 0x03ffff (0x64dc bytes) 000330 1788727343.786792 read 0x01e00a thru 0x02afa3 (0xcf9a bytes) 000331 1788727343.902777 read 0x028acf thru 0x032d31 (0xa263 bytes) 000332 1788727343.904884 mapwrite 0x00849a thru 0x013084 (0xabeb bytes) 000333 1788727343.942297 trunc from 0x040000 to 0x026ede (0x19121 bytes) 000334 1788727343.972087 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x26edf): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26edf): Mirror 2: 0x9b0b4331 (stale) Mirror 3: 0x9b0b4331 (stale) Mirror 4: 0x9b0b4331 (stale) Mirror 5: 0x9b0b4331 (stale) CRC-32 checksum value for chunk [0, 0x26edf): Mirror 1: 0x3c4f9fbf Mirror 6: 0x3c4f9fbf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26edf: skipped 000335 1788727345.347874 trunc from 0x007f1d to 0x01227e (0xa362 bytes) 000336 1788727345.531369 punch from 0x00679a to 0x00ef0b (0x8772 bytes) 000337 1788727345.558038 MIRROR_RESYNC 6 mirrors 000338 1788727347.034593 write 0x0337df thru 0x0399ef (0x6211 bytes) 000339 1788727347.073113 punch from 0x030060 to 0x033990 (0x3931 bytes) 000340 1788727347.103182 read 0x02ed30 thru 0x03382c (0x4afd bytes) 000341 1788727347.112652 trunc from 0x0399f0 to 0x008a1b (0x30fd4 bytes) 000342 1788727347.142049 mapwrite 0x036d56 thru 0x03f21a (0x84c5 bytes) 000343 1788727347.221728 read 0x037b2a thru 0x03ece3 (0x71ba bytes) 000344 1788727347.222454 mapread 0x02d016 thru 0x03ab8e (0xdb79 bytes) 000345 1788727347.240796 mapwrite 0x02eb02 thru 0x03be8b (0xd38a bytes) 000346 1788727347.265817 write 0x01026a thru 0x01fd47 (0xfade bytes) 000347 1788727347.267163 write 0x018b22 thru 0x021422 (0x8901 bytes) 000348 1788727347.271359 trunc from 0x00f708 to 0x015924 (0x621d bytes) 000350 1788727347.290773 mapwrite 0x026276 thru 0x02949f (0x322a bytes) 000351 1788727347.305311 read 0x003bd2 thru 0x00c041 (0x8470 bytes) 000353 1788727347.306674 mapwrite 0x00ce10 thru 0x01786f (0xaa60 bytes) 000355 1788727347.352147 write 0x033cc8 thru 0x034e5f (0x1198 bytes) 000356 1788727347.352780 trunc from 0x03f21b to 0x00cc6d (0x325ad bytes) 000357 1788727347.390731 mapread 0x00a81f thru 0x00cc6d (0x244f bytes) 000358 1788727347.391676 mapwrite 0x01e0ea thru 0x01f2ef (0x1206 bytes) 000359 1788727347.444941 trunc from 0x016fb3 to 0x01f2ef (0x833d bytes) 000360 1788727347.450528 read 0x00fadd thru 0x0199d4 (0x9ef8 bytes) 000361 1788727347.459541 fallocate 0x01fe36 thru 0x022c2d (0x2df8 bytes) 000362 1788727347.464599 write 0x01f14e thru 0x025f8c (0x6e3f bytes) 000363 1788727347.466586 mapwrite 0x038bfb thru 0x03c582 (0x3988 bytes) 000364 1788727347.526038 fallocate 0x02fc26 thru 0x03a572 (0xa94d bytes) 000365 1788727347.541270 fallocate 0x036044 thru 0x03ffff (0x9fbc bytes) 000366 1788727347.589243 mapread 0x02ceb8 thru 0x032301 (0x544a bytes) 000367 1788727347.590358 punch from 0x02cdee to 0x03c302 (0xf515 bytes) 000368 1788727347.608161 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x57284f75 (stale) Mirror 3: 0x57284f75 (stale) Mirror 4: 0x57284f75 (stale) Mirror 6: 0x57284f75 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x57284f75 (stale) Mirror 3: 0x57284f75 (stale) Mirror 4: 0x57284f75 (stale) Mirror 6: 0x57284f75 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000369 1788727348.403689 read 0x0223f8 thru 0x02a9f4 (0x85fd bytes) 000370 1788727348.451485 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x57284f75 (stale) Mirror 4: 0x57284f75 (stale) Mirror 6: 0x57284f75 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x57284f75 (stale) Mirror 4: 0x57284f75 (stale) Mirror 6: 0x57284f75 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000371 1788727348.945124 mapwrite 0x02d9f2 thru 0x039640 (0xbc4f bytes) 000372 1788727349.102997 trunc from 0x040000 to 0x037760 (0x889f bytes) 000373 1788727349.159890 write 0x01253f thru 0x013623 (0x10e5 bytes) 000374 1788727349.183418 punch from 0x02ad9b to 0x031b57 (0x6dbd bytes) 000375 1788727349.200718 trunc from 0x037761 to 0x016875 (0x20eeb bytes) 000376 1788727349.257820 trunc from 0x015dc6 to 0x016875 (0xab0 bytes) 000377 1788727349.285564 trunc from 0x00d1d5 to 0x016875 (0x96a1 bytes) 000378 1788727349.326010 mapread 0x01406b thru 0x016875 (0x280b bytes) 000379 1788727349.337364 punch from 0x0126c4 to 0x016875 (0x41b2 bytes) 000380 1788727349.360725 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16876): Mirror 3: 0xc605c2b (stale) Mirror 4: 0xc605c2b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16876): Mirror 3: 0xc605c2b (stale) Mirror 4: 0xc605c2b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16876: skipped 000381 1788727349.953831 punch from 0x00c692 to 0x013396 (0x6d05 bytes) 000382 1788727349.961193 punch from 0x01338d to 0x016875 (0x34e9 bytes) 000383 1788727349.966497 mapwrite 0x018e8a thru 0x020cef (0x7e66 bytes) 000384 1788727350.005879 trunc from 0x020cf0 to 0x0217c9 (0xada bytes) 000385 1788727350.025406 trunc from 0x01381f to 0x016dd4 (0x35b6 bytes) 000386 1788727350.030580 trunc from 0x0217ca to 0x018624 (0x91a5 bytes) 000387 1788727350.053899 trunc from 0x00ccea to 0x012223 (0x553a bytes) 000388 1788727350.065113 trunc from 0x018625 to 0x031fe7 (0x199c3 bytes) 000389 1788727350.083803 mapwrite 0x01f10f thru 0x01f456 (0x348 bytes) 000390 1788727350.105147 trunc from 0x031fe8 to 0x03caa7 (0xaac0 bytes) 000391 1788727350.128289 mapread 0x01f608 thru 0x026f01 (0x78fa bytes) 000392 1788727350.132092 mapread 0x03ba02 thru 0x03caa7 (0x10a6 bytes) 000393 1788727350.147033 read 0x03bf7b thru 0x03caa7 (0xb2d bytes) 000394 1788727350.148125 mapread 0x0284c3 thru 0x033801 (0xb33f bytes) 000395 1788727350.175880 trunc from 0x01f95a to 0x028fc9 (0x9670 bytes) 000396 1788727350.189433 trunc from 0x03caa8 to 0x013143 (0x29964 bytes) 000397 1788727350.215204 punch from 0x000edd to 0x00a8f9 (0x9a1d bytes) 000398 1788727350.237027 mapread 0x006a21 thru 0x00aba2 (0x4182 bytes) 000400 1788727350.252440 trunc from 0x00da49 to 0x013143 (0x56fb bytes) 000401 1788727350.256787 read 0x008828 thru 0x013143 (0xa91c bytes) 000403 1788727350.266191 write 0x000e9b thru 0x003d39 (0x2e9f bytes) 000404 1788727350.268173 punch from 0x0008ea to 0x00172b (0xe42 bytes) 000405 1788727350.289428 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x13144): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13144): Mirror 3: 0x843f3074 (stale) Mirror 4: 0x843f3074 (stale) CRC-32 checksum value for chunk [0, 0x13144): Mirror 1: 0xe4ce9edb Mirror 5: 0xe4ce9edb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13144: skipped 000406 1788727351.074847 MIRROR_RESYNC 4 mirrors 000407 1788727351.760796 mapread 0x00b36e thru 0x013143 (0x7dd6 bytes) 000408 1788727351.778854 punch from 0x006645 to 0x013143 (0xcaff bytes) 000409 1788727351.825048 MIRROR_RESYNC 4 mirrors 000410 1788727352.930669 fallocate 0x03b28d thru 0x03ffff (0x4d73 bytes) 000411 1788727353.036754 write 0x014866 thru 0x015d33 (0x14ce bytes) 000412 1788727353.065241 mapread 0x000f42 thru 0x0099ee (0x8aad bytes) 000413 1788727353.118173 fallocate 0x03b3c0 thru 0x03ffff (0x4c40 bytes) 000414 1788727353.131009 trunc from 0x004535 to 0x0047fb (0x2c7 bytes) 000415 1788727353.146506 read 0x02735c thru 0x0314a9 (0xa14e bytes) 000416 1788727353.165106 punch from 0x0121e7 to 0x01ffc7 (0xdde1 bytes) 000417 1788727353.196744 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x84e29667 (stale) Mirror 4: 0x84e29667 (stale) Mirror 5: 0x84e29667 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x84e29667 Mirror 6: 0x84e29667 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000418 1788727354.570976 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x84e29667 (stale) Mirror 4: 0x84e29667 (stale) Mirror 5: 0x84e29667 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x84e29667 Mirror 6: 0x84e29667 Mirror 7: 0x84e29667 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000419 1788727355.640810 trunc from 0x020cd1 to 0x02a78a (0x9aba bytes) 000420 1788727355.718727 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x84e29667 (stale) Mirror 4: 0x84e29667 (stale) Mirror 5: 0x84e29667 (stale) Mirror 6: 0x84e29667 (stale) Mirror 7: 0x84e29667 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x84e29667 Mirror 8: 0x84e29667 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000421 1788727356.533757 trunc from 0x03212b to 0x03f6bb (0xd591 bytes) 000422 1788727356.572792 write 0x0129a6 thru 0x017ec8 (0x5523 bytes) 000423 1788727356.583513 mapread 0x00fd61 thru 0x017218 (0x74b8 bytes) 000424 1788727356.600477 read 0x008dba thru 0x00e45d (0x56a4 bytes) 000425 1788727356.607056 read 0x031f71 thru 0x0357e1 (0x3871 bytes) 000426 1788727356.616237 mapwrite 0x039c37 thru 0x03ecf1 (0x50bb bytes) 000428 1788727356.631232 read 0x0362bf thru 0x03ffff (0x9d41 bytes) 000429 1788727356.631668 trunc from 0x040000 to 0x010193 (0x2fe6c bytes) 000430 1788727356.653354 mapwrite 0x01a765 thru 0x021926 (0x71c2 bytes) 000431 1788727356.686277 MIRROR_RESYNC 7 mirrors 000432 1788727357.519687 punch from 0x01e5e0 to 0x021926 (0x3347 bytes) 000433 1788727357.547837 fallocate 0x0078fe thru 0x00b8e2 (0x3fe5 bytes) 000434 1788727357.554557 mapwrite 0x030a71 thru 0x031208 (0x798 bytes) 000435 1788727357.612586 mapread 0x0193aa thru 0x026572 (0xd1c9 bytes) 000436 1788727357.631166 read 0x02d8e2 thru 0x031208 (0x3927 bytes) 000437 1788727357.636703 trunc from 0x031209 to 0x037daa (0x6ba2 bytes) 000438 1788727357.659589 read 0x005c6c thru 0x011df9 (0xc18e bytes) 000439 1788727357.674386 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x37dab): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37dab): Mirror 3: 0x18ace86b (stale) Mirror 4: 0x18ace86b (stale) Mirror 5: 0x18ace86b (stale) Mirror 6: 0x18ace86b (stale) Mirror 7: 0x18ace86b (stale) Mirror 8: 0x18ace86b (stale) CRC-32 checksum value for chunk [0, 0x37dab): Mirror 1: 0x41170289 Mirror 9: 0x41170289 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37dab: skipped 000440 1788727359.545874 write 0x01d33c thru 0x0277b3 (0xa478 bytes) 000441 1788727359.604621 read 0x00c7e5 thru 0x00ef42 (0x275e bytes) 000442 1788727359.611465 mapread 0x00c50b thru 0x010553 (0x4049 bytes) 000443 1788727359.621242 trunc from 0x0227a9 to 0x02b865 (0x90bd bytes) 000444 1788727359.633069 mapread 0x0352ad thru 0x037daa (0x2afe bytes) 000445 1788727359.644605 read 0x006499 thru 0x0135cd (0xd135 bytes) 000446 1788727359.668747 write 0x02c41c thru 0x0337f7 (0x73dc bytes) 000447 1788727359.670286 punch from 0x00e2d9 to 0x0161f3 (0x7f1b bytes) 000448 1788727359.674611 trunc from 0x037dab to 0x00b191 (0x2cc19 bytes) 000449 1788727359.694869 mapwrite 0x00dc77 thru 0x0124a6 (0x4830 bytes) 000450 1788727359.758591 trunc from 0x0124a7 to 0x03c24b (0x29da5 bytes) 000451 1788727359.774858 trunc from 0x01b841 to 0x0218b9 (0x6079 bytes) 000452 1788727359.780482 fallocate 0x02652a thru 0x027d04 (0x17db bytes) 000454 1788727359.787661 trunc from 0x0391d7 to 0x03c24b (0x3075 bytes) 000455 1788727359.794077 mapread 0x0226ae thru 0x02b16f (0x8ac2 bytes) 000456 1788727359.852877 mapread 0x037801 thru 0x03c24b (0x4a4b bytes) 000457 1788727359.861000 mapread 0x0359d3 thru 0x03c24b (0x6879 bytes) 000459 1788727359.876492 fallocate 0x01b5fb thru 0x01caeb (0x14f1 bytes) 000460 1788727359.882150 punch from 0x035574 to 0x03c24b (0x6cd8 bytes) 000461 1788727359.896133 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c24c): Mirror 3: 0x5543412 (stale) Mirror 5: 0x5543412 (stale) Mirror 6: 0x5543412 (stale) Mirror 7: 0x5543412 (stale) Mirror 8: 0x5543412 (stale) Mirror 9: 0xf0d2b079 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c24c): Mirror 3: 0x5543412 (stale) Mirror 5: 0x5543412 (stale) Mirror 6: 0x5543412 (stale) Mirror 7: 0x5543412 (stale) Mirror 8: 0x5543412 (stale) Mirror 9: 0xf0d2b079 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c24c: skipped 000462 1788727360.358744 punch from 0x037c9c to 0x0389f2 (0xd57 bytes) 000463 1788727360.364535 write 0x03fce3 thru 0x03ffff (0x31d bytes) 000465 1788727360.372754 mapwrite 0x010711 thru 0x019b5d (0x944d bytes) 000466 1788727360.424757 punch from 0x014663 to 0x022f00 (0xe89e bytes) 000468 1788727360.438177 punch from 0x004369 to 0x0137ec (0xf484 bytes) 000469 1788727360.445035 read 0x02c82c thru 0x032375 (0x5b4a bytes) 000470 1788727360.464809 fallocate 0x029a4e thru 0x02db1e (0x40d1 bytes) 000471 1788727360.485429 mapread 0x03ab3c thru 0x03d95a (0x2e1f bytes) 000472 1788727360.501525 trunc from 0x0014ff to 0x00c4a4 (0xafa6 bytes) 000473 1788727360.509101 punch from 0x012435 to 0x016bb8 (0x4784 bytes) 000474 1788727360.515306 punch from 0x037166 to 0x03953a (0x23d5 bytes) 000475 1788727360.523509 read 0x0230d6 thru 0x025374 (0x229f bytes) 000476 1788727360.543725 trunc from 0x040000 to 0x02863d (0x179c2 bytes) 000477 1788727360.565579 mapread 0x00672d thru 0x011282 (0xab56 bytes) 000478 1788727360.566634 read 0x00bae3 thru 0x010d36 (0x5254 bytes) 000479 1788727360.567266 trunc from 0x02863e to 0x037eba (0xf87d bytes) 000480 1788727360.607988 trunc from 0x028111 to 0x037713 (0xf603 bytes) 000481 1788727360.615436 trunc from 0x037ebb to 0x02ae4e (0xd06c bytes) 000482 1788727360.640121 read 0x00191b thru 0x003b14 (0x21fa bytes) 000483 1788727360.643374 punch from 0x01f5e6 to 0x02220e (0x2c29 bytes) 000485 1788727360.649891 mapread 0x002913 thru 0x00e27e (0xb96c bytes) 000486 1788727360.650882 read 0x0049f2 thru 0x010b03 (0xc112 bytes) 000487 1788727360.651417 mapread 0x001838 thru 0x006565 (0x4d2e bytes) 000488 1788727360.654159 trunc from 0x02ae4f to 0x00930d (0x21b41 bytes) 000489 1788727360.675241 trunc from 0x00930e to 0x017a55 (0xe748 bytes) 000491 1788727360.718830 trunc from 0x017a56 to 0x02bdc1 (0x1436c bytes) 000492 1788727360.748640 fallocate 0x0045ac thru 0x004daa (0x7ff bytes) 000493 1788727360.753125 punch from 0x023706 to 0x02bdc1 (0x86bc bytes) 000494 1788727360.762011 mapread 0x0246b8 thru 0x02bdc1 (0x770a bytes) 000495 1788727360.778507 punch from 0x0104be to 0x010764 (0x2a7 bytes) 000496 1788727360.786151 punch from 0x01ee07 to 0x025059 (0x6253 bytes) 000498 1788727360.793578 trunc from 0x01ddb2 to 0x0251e9 (0x7438 bytes) 000499 1788727360.808807 trunc from 0x028f8e to 0x02bdc1 (0x2e34 bytes) 000500 1788727360.818969 trunc from 0x011055 to 0x01ef0d (0xdeb9 bytes) 000501 1788727360.826778 write 0x039e0d thru 0x03caca (0x2cbe bytes) 000502 1788727360.827650 trunc from 0x03cacb to 0x037a96 (0x5034 bytes) 000503 1788727360.839899 fallocate 0x02db90 thru 0x03d8e3 (0xfd54 bytes) 000504 1788727360.847704 MIRROR_RESYNC 7 mirrors 000505 1788727361.952337 punch from 0x006176 to 0x00bf06 (0x5d91 bytes) 000507 1788727361.998393 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x37a97): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a97): Mirror 3: 0x12dc6b2b (stale) Mirror 5: 0x12dc6b2b (stale) Mirror 6: 0x12dc6b2b (stale) Mirror 7: 0x12dc6b2b (stale) Mirror 8: 0x12dc6b2b (stale) Mirror 9: 0x12dc6b2b (stale) CRC-32 checksum value for chunk [0, 0x37a97): Mirror 1: 0x12dc6b2b Mirror 10: 0x12dc6b2b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a97: skipped 000509 1788727363.071841 mapread 0x03317e thru 0x037a96 (0x4919 bytes) 000510 1788727363.120327 mapwrite 0x0368ce thru 0x03974c (0x2e7f bytes) 000511 1788727363.219836 write 0x019fae thru 0x025ea5 (0xbef8 bytes) 000512 1788727363.230688 punch from 0x011545 to 0x01e348 (0xce04 bytes) 000513 1788727363.258431 MIRROR_RESYNC 8 mirrors 000514 1788727365.246885 punch from 0x0142c7 to 0x02168d (0xd3c7 bytes) 000516 1788727365.311059 trunc from 0x03974d to 0x0369e6 (0x2d66 bytes) 000517 1788727365.344791 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x369e7): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x369e7): Mirror 3: 0x91b1a295 (stale) Mirror 5: 0x91b1a295 (stale) Mirror 6: 0x91b1a295 (stale) Mirror 7: 0x91b1a295 (stale) Mirror 8: 0x91b1a295 (stale) Mirror 9: 0x91b1a295 (stale) Mirror 10: 0x91b1a295 (stale) CRC-32 checksum value for chunk [0, 0x369e7): Mirror 1: 0x7261c367 Mirror 11: 0x7261c367 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x369e7: skipped 000518 1788727366.345237 read 0x0269ca thru 0x02f63f (0x8c76 bytes) 000519 1788727366.375985 mapwrite 0x028e02 thru 0x034095 (0xb294 bytes) 000520 1788727366.523403 mapwrite 0x00133f thru 0x010b79 (0xf83b bytes) 000521 1788727366.636136 punch from 0x003fa1 to 0x01171f (0xd77f bytes) 000522 1788727366.651164 write 0x01681d thru 0x022a74 (0xc258 bytes) 000523 1788727366.663242 read 0x00c8e7 thru 0x01172f (0x4e49 bytes) 000524 1788727366.678855 read 0x034fd3 thru 0x035303 (0x331 bytes) 000525 1788727366.680360 trunc from 0x02610d to 0x02c5cc (0x64c0 bytes) 000526 1788727366.685547 trunc from 0x0369e7 to 0x00b66f (0x2b377 bytes) 000527 1788727366.711152 fallocate 0x01715b thru 0x0207ba (0x9660 bytes) 000528 1788727366.717733 trunc from 0x00b25b to 0x00b66f (0x415 bytes) 000529 1788727366.733373 MIRROR_RESYNC 9 mirrors 000530 1788727367.997457 mapread 0x009578 thru 0x00aa9c (0x1525 bytes) 000531 1788727368.014508 mapread 0x0089ff thru 0x00b66f (0x2c71 bytes) 000532 1788727368.023566 trunc from 0x000859 to 0x0063ca (0x5b72 bytes) 000533 1788727368.060414 read 0x0091ea thru 0x00b66f (0x2486 bytes) 000535 1788727368.069974 mapwrite 0x005b9a thru 0x010404 (0xa86b bytes) 000536 1788727368.119353 fallocate 0x0314d4 thru 0x033c3c (0x2769 bytes) 000537 1788727368.124767 trunc from 0x033c3d to 0x0367c6 (0x2b8a bytes) 000538 1788727368.138718 trunc from 0x0367c7 to 0x01ee1e (0x179a8 bytes) 000539 1788727368.158464 punch from 0x0138e6 to 0x01ee1e (0xb539 bytes) 000540 1788727368.164727 trunc from 0x01ee1f to 0x01dcc8 (0x1156 bytes) 000541 1788727368.180788 write 0x0269d4 thru 0x02838b (0x19b8 bytes) 000543 1788727368.184471 write 0x027104 thru 0x033f45 (0xce42 bytes) 000544 1788727368.190394 mapwrite 0x02f110 thru 0x03cc25 (0xdb16 bytes) 000545 1788727368.240612 write 0x028eb8 thru 0x0323d3 (0x951c bytes) 000546 1788727368.242153 read 0x018fa9 thru 0x02088a (0x78e2 bytes) 000547 1788727368.270802 mapwrite 0x032963 thru 0x0365bc (0x3c5a bytes) 000548 1788727368.313594 mapread 0x0324e2 thru 0x035c1b (0x373a bytes) 000549 1788727368.315352 trunc from 0x038613 to 0x03cc25 (0x4613 bytes) 000550 1788727368.331438 fallocate 0x00346c thru 0x009be4 (0x6779 bytes) 000551 1788727368.338394 trunc from 0x019237 to 0x01ab9e (0x1968 bytes) 000552 1788727368.360477 read 0x00eaeb thru 0x01505d (0x6573 bytes) 000553 1788727368.363935 trunc from 0x03cc26 to 0x00e53d (0x2e6e8 bytes) 000554 1788727368.403685 trunc from 0x00e53e to 0x02febe (0x21981 bytes) 000555 1788727368.423614 mapread 0x026cd6 thru 0x02febe (0x91e9 bytes) 000556 1788727368.446076 MIRROR_RESYNC 9 mirrors 000557 1788727370.359382 mapwrite 0x03684e thru 0x03ffff (0x97b2 bytes) 000558 1788727370.516276 fallocate 0x01cd99 thru 0x02a9cb (0xdc33 bytes) 000559 1788727370.533484 trunc from 0x02dedc to 0x034982 (0x6aa7 bytes) 000560 1788727370.543804 trunc from 0x018849 to 0x028786 (0xff3e bytes) 000561 1788727370.574762 mapread 0x03a626 thru 0x03ffff (0x59da bytes) 000562 1788727370.580571 read 0x021f0e thru 0x025057 (0x314a bytes) 000563 1788727370.599138 read 0x00128c thru 0x00bb03 (0xa878 bytes) 000564 1788727370.621812 punch from 0x02c29d to 0x0395fa (0xd35e bytes) 000565 1788727370.639741 trunc from 0x02ea8c to 0x0320af (0x3624 bytes) 000566 1788727370.662206 read 0x004f81 thru 0x009d9e (0x4e1e bytes) 000567 1788727370.663038 write 0x03085d thru 0x032afa (0x229e bytes) 000568 1788727370.692808 write 0x0082be thru 0x008e43 (0xb86 bytes) 000569 1788727370.706367 write 0x022eb8 thru 0x02fb00 (0xcc49 bytes) 000570 1788727370.714080 trunc from 0x01c65e to 0x01ddb3 (0x1756 bytes) 000571 1788727370.743347 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x909965e0 (stale) Mirror 5: 0x909965e0 (stale) Mirror 6: 0x909965e0 (stale) Mirror 7: 0x909965e0 (stale) Mirror 8: 0x909965e0 (stale) Mirror 9: 0x909965e0 (stale) Mirror 10: 0x909965e0 (stale) Mirror 11: 0x909965e0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x40dfcfc7 Mirror 12: 0x40dfcfc7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000572 1788727372.048063 mapwrite 0x03241d thru 0x037809 (0x53ed bytes) 000573 1788727372.151831 MIRROR_RESYNC 10 mirrors 000574 1788727374.744127 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) 000577 1788727375.427158 punch from 0x00fc0d to 0x0171cb (0x75bf bytes) 000578 1788727375.508625 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf16c7ca5 (stale) Mirror 5: 0xf16c7ca5 (stale) Mirror 7: 0xf16c7ca5 (stale) Mirror 8: 0xf16c7ca5 (stale) Mirror 10: 0xf16c7ca5 (stale) Mirror 11: 0xf16c7ca5 (stale) Mirror 12: 0xf16c7ca5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf16c7ca5 (stale) Mirror 5: 0xf16c7ca5 (stale) Mirror 7: 0xf16c7ca5 (stale) Mirror 8: 0xf16c7ca5 (stale) Mirror 10: 0xf16c7ca5 (stale) Mirror 11: 0xf16c7ca5 (stale) Mirror 12: 0xf16c7ca5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000579 1788727376.149241 mapread 0x003086 thru 0x00912f (0x60aa bytes) 000580 1788727376.182384 mapread 0x038ba0 thru 0x03dce8 (0x5149 bytes) 000581 1788727376.205132 mapwrite 0x02a4bb thru 0x030d97 (0x68dd bytes) 000582 1788727376.272413 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf16c7ca5 (stale) Mirror 5: 0xf16c7ca5 (stale) Mirror 7: 0xf16c7ca5 (stale) Mirror 8: 0xf16c7ca5 (stale) Mirror 10: 0xf16c7ca5 (stale) Mirror 11: 0xf16c7ca5 (stale) Mirror 12: 0xf16c7ca5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4c3e4dbc Mirror 13: 0x4c3e4dbc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000583 1788727377.417672 read 0x035d28 thru 0x03f858 (0x9b31 bytes) 000584 1788727377.461326 MIRROR_RESYNC 9 mirrors 000585 1788727379.468969 punch from 0x00f873 to 0x0142ab (0x4a39 bytes) 000586 1788727379.517245 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4c3e4dbc (stale) Mirror 5: 0x4c3e4dbc (stale) Mirror 7: 0x4c3e4dbc (stale) Mirror 8: 0x4c3e4dbc (stale) Mirror 10: 0x4c3e4dbc (stale) Mirror 11: 0x4c3e4dbc (stale) Mirror 12: 0x4c3e4dbc (stale) Mirror 13: 0x4c3e4dbc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4c3e4dbc Mirror 14: 0x4c3e4dbc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000587 1788727380.945501 write 0x01e743 thru 0x026ccc (0x858a bytes) 000588 1788727381.060506 punch from 0x009769 to 0x00dc6f (0x4507 bytes) 000590 1788727381.079572 read 0x029117 thru 0x032871 (0x975b bytes) 000591 1788727381.100015 trunc from 0x040000 to 0x019477 (0x26b88 bytes) 000592 1788727381.156575 trunc from 0x016f78 to 0x019477 (0x2500 bytes) 000593 1788727381.162880 fallocate 0x0128e2 thru 0x01cecc (0xa5eb bytes) 000594 1788727381.167989 punch from 0x00903e to 0x017397 (0xe35a bytes) 000595 1788727381.175177 read 0x00e674 thru 0x019477 (0xae04 bytes) 000596 1788727381.186812 trunc from 0x019478 to 0x001f7e (0x174f9 bytes) 000597 1788727381.223258 write 0x00c929 thru 0x0125aa (0x5c82 bytes) 000598 1788727381.225235 trunc from 0x0125ab to 0x03b502 (0x28f58 bytes) 000599 1788727381.265664 write 0x028099 thru 0x0371c8 (0xf130 bytes) 000600 1788727381.284798 punch from 0x02a7aa to 0x037851 (0xd0a8 bytes) 000602 1788727381.309712 write 0x00dbf4 thru 0x0145cb (0x69d8 bytes) 000603 1788727381.316791 mapread 0x02ab2c thru 0x02b9a9 (0xe7e bytes) 000604 1788727381.330780 fallocate 0x005af1 thru 0x00bf0a (0x641a bytes) 000605 1788727381.337786 read 0x027c0f thru 0x02c708 (0x4afa bytes) 000606 1788727381.354297 punch from 0x00e649 to 0x01aa36 (0xc3ee bytes) 000607 1788727381.367606 mapwrite 0x02cfe6 thru 0x02d49b (0x4b6 bytes) 000608 1788727381.379343 mapread 0x0133cb thru 0x01ec96 (0xb8cc bytes) 000609 1788727381.430701 mapread 0x007287 thru 0x00b824 (0x459e bytes) 000611 1788727381.434670 trunc from 0x03b503 to 0x0033de (0x38124 bytes) 000612 1788727381.458896 trunc from 0x0033df to 0x02ceee (0x29b10 bytes) 000614 1788727381.478882 mapwrite 0x02315a thru 0x02fa22 (0xc8c9 bytes) 000615 1788727381.611709 mapread 0x01cce8 thru 0x027654 (0xa96d bytes) 000616 1788727381.652540 trunc from 0x02b69d to 0x02fa22 (0x4386 bytes) 000617 1788727381.657789 read 0x0049fa thru 0x00f013 (0xa61a bytes) 000619 1788727381.668497 trunc from 0x02fa23 to 0x03821c (0x87fa bytes) 000620 1788727381.691782 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3821d): Mirror 5: 0xef28412b (stale) Mirror 7: 0xef28412b (stale) Mirror 8: 0xef28412b (stale) Mirror 10: 0xef28412b (stale) Mirror 11: 0xef28412b (stale) Mirror 12: 0xef28412b (stale) Mirror 13: 0xef28412b (stale) Mirror 14: 0xef28412b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3821d): Mirror 5: 0xef28412b (stale) Mirror 7: 0xef28412b (stale) Mirror 8: 0xef28412b (stale) Mirror 10: 0xef28412b (stale) Mirror 11: 0xef28412b (stale) Mirror 12: 0xef28412b (stale) Mirror 13: 0xef28412b (stale) Mirror 14: 0xef28412b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3821d: skipped 000621 1788727382.406462 write 0x0097ac thru 0x00c731 (0x2f86 bytes) 000622 1788727382.438603 mapread 0x02a401 thru 0x02f92c (0x552c bytes) 000623 1788727382.490271 read 0x02924e thru 0x037856 (0xe609 bytes) 000624 1788727382.499562 mapwrite 0x028f71 thru 0x036a45 (0xdad5 bytes) 000625 1788727382.537530 mapwrite 0x019ec0 thru 0x029a70 (0xfbb1 bytes) 000626 1788727382.650743 MIRROR_RESYNC 9 mirrors 000627 1788727385.559210 fallocate 0x036cb0 thru 0x03fc9e (0x8fef bytes) 000628 1788727385.693008 mapread 0x03db33 thru 0x03fc9e (0x216c bytes) 000629 1788727385.732853 read 0x034c82 thru 0x037e89 (0x3208 bytes) 000630 1788727385.748584 mapread 0x004965 thru 0x00da57 (0x90f3 bytes) 000631 1788727385.760736 mapread 0x01b370 thru 0x0242c8 (0x8f59 bytes) 000632 1788727385.788513 mapread 0x00ab25 thru 0x01232b (0x7807 bytes) 000633 1788727385.803084 fallocate 0x01cb21 thru 0x029829 (0xcd09 bytes) 000634 1788727385.814631 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fc9f): Mirror 5: 0x23b60461 (stale) Mirror 7: 0x23b60461 (stale) Mirror 8: 0x23b60461 (stale) Mirror 10: 0x23b60461 (stale) Mirror 12: 0x23b60461 (stale) Mirror 13: 0x23b60461 (stale) Mirror 14: 0x23b60461 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fc9f): Mirror 5: 0x23b60461 (stale) Mirror 7: 0x23b60461 (stale) Mirror 8: 0x23b60461 (stale) Mirror 10: 0x23b60461 (stale) Mirror 12: 0x23b60461 (stale) Mirror 13: 0x23b60461 (stale) Mirror 14: 0x23b60461 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fc9f: skipped 000635 1788727386.644259 fallocate 0x015636 thru 0x01affa (0x59c5 bytes) 000636 1788727386.651768 mapread 0x026dc5 thru 0x026eee (0x12a bytes) 000637 1788727386.659475 mapwrite 0x002fdd thru 0x00bbd4 (0x8bf8 bytes) 000638 1788727386.696427 write 0x0358d5 thru 0x03ffff (0xa72b bytes) 000639 1788727386.709980 mapread 0x0257be thru 0x0318fb (0xc13e bytes) 000640 1788727386.739037 trunc from 0x038241 to 0x03d574 (0x5334 bytes) 000642 1788727386.756266 punch from 0x001785 to 0x006b2d (0x53a9 bytes) 000643 1788727386.763056 read 0x004ccb thru 0x010481 (0xb7b7 bytes) 000644 1788727386.770695 fallocate 0x01c987 thru 0x01e683 (0x1cfd bytes) 000645 1788727386.777771 trunc from 0x040000 to 0x00d896 (0x32769 bytes) 000646 1788727386.798611 trunc from 0x00014d to 0x0002cc (0x180 bytes) 000647 1788727386.805386 trunc from 0x00d897 to 0x03ac5d (0x2d3c7 bytes) 000648 1788727386.824499 trunc from 0x02cebd to 0x03673b (0x987f bytes) 000649 1788727386.830351 trunc from 0x002601 to 0x005b06 (0x3506 bytes) 000650 1788727386.836614 mapwrite 0x00687a thru 0x006d2b (0x4b2 bytes) 000651 1788727386.851347 read 0x0362da thru 0x038fbd (0x2ce4 bytes) 000652 1788727386.863044 punch from 0x026e38 to 0x02c34c (0x5515 bytes) 000653 1788727386.874408 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3ac5e): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ac5e): Mirror 5: 0x26f0b866 (stale) Mirror 7: 0x26f0b866 (stale) Mirror 8: 0x26f0b866 (stale) Mirror 10: 0x26f0b866 (stale) Mirror 12: 0x26f0b866 (stale) Mirror 13: 0x26f0b866 (stale) Mirror 14: 0x26f0b866 (stale) CRC-32 checksum value for chunk [0, 0x3ac5e): Mirror 1: 0x4dd0699e Mirror 15: 0x4dd0699e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ac5e: skipped 000654 1788727388.000535 mapwrite 0x00d3be thru 0x019c9d (0xc8e0 bytes) 000655 1788727388.144490 mapwrite 0x0159f6 thru 0x0165e3 (0xbee bytes) 000656 1788727388.166753 write 0x0322e1 thru 0x039590 (0x72b0 bytes) 000657 1788727388.190494 mapwrite 0x00fa0b thru 0x010b1c (0x1112 bytes) 000658 1788727388.216544 trunc from 0x03ac5e to 0x02a398 (0x108c5 bytes) 000659 1788727388.238634 read 0x00640a thru 0x013c8c (0xd883 bytes) 000660 1788727388.245434 mapread 0x01ac59 thru 0x027d54 (0xd0fc bytes) 000661 1788727388.256676 punch from 0x00fe8e to 0x01c50c (0xc67f bytes) 000662 1788727388.263066 mapwrite 0x02f968 thru 0x03523e (0x58d7 bytes) 000663 1788727388.317430 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3523f): Mirror 5: 0x885c9b78 (stale) Mirror 7: 0x885c9b78 (stale) Mirror 10: 0x885c9b78 (stale) Mirror 12: 0x885c9b78 (stale) Mirror 13: 0x885c9b78 (stale) Mirror 14: 0x885c9b78 (stale) Mirror 15: 0xa5e13956 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3523f): Mirror 5: 0x885c9b78 (stale) Mirror 7: 0x885c9b78 (stale) Mirror 10: 0x885c9b78 (stale) Mirror 12: 0x885c9b78 (stale) Mirror 13: 0x885c9b78 (stale) Mirror 14: 0x885c9b78 (stale) Mirror 15: 0xa5e13956 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3523f: skipped 000664 1788727389.030386 punch from 0x0227ee to 0x02718a (0x499d bytes) 000665 1788727389.043267 MIRROR_RESYNC 8 mirrors 000666 1788727391.087099 punch from 0x0232ad to 0x031fe9 (0xed3d bytes) 000667 1788727391.127629 mapwrite 0x0094d6 thru 0x0191fd (0xfd28 bytes) 000668 1788727391.202043 fallocate 0x0072ae thru 0x0154c5 (0xe218 bytes) 000669 1788727391.211105 mapwrite 0x026ab2 thru 0x02b4b3 (0x4a02 bytes) 000671 1788727391.250515 trunc from 0x03523f to 0x02effc (0x6242 bytes) 000674 1788727391.279553 fallocate 0x037245 thru 0x03ffff (0x8dbb bytes) 000675 1788727391.285489 mapwrite 0x012ffb thru 0x019bc6 (0x6bcc bytes) 000676 1788727391.309666 fallocate 0x00c9f8 thru 0x01163e (0x4c47 bytes) 000677 1788727391.316732 write 0x0153e7 thru 0x01989e (0x44b8 bytes) 000678 1788727391.324559 MIRROR_RESYNC 8 mirrors 000679 1788727393.619036 mapwrite 0x0357ee thru 0x03dc13 (0x8426 bytes) 000681 1788727393.755191 trunc from 0x0368e7 to 0x037269 (0x983 bytes) 000682 1788727393.773003 read 0x00a90a thru 0x011381 (0x6a78 bytes) 000683 1788727393.790518 mapread 0x007eb4 thru 0x00c9d0 (0x4b1d bytes) 000684 1788727393.803022 trunc from 0x040000 to 0x0336f1 (0xc90e bytes) 000685 1788727393.860248 mapwrite 0x005103 thru 0x01258d (0xd48b bytes) 000686 1788727393.944157 fallocate 0x03ce67 thru 0x03ffff (0x3199 bytes) 000687 1788727393.957134 fallocate 0x006f25 thru 0x00c70b (0x57e7 bytes) 000688 1788727393.964208 write 0x005fa7 thru 0x01421f (0xe279 bytes) 000689 1788727393.970048 write 0x0014ea thru 0x00dc1c (0xc733 bytes) 000690 1788727393.975071 mapwrite 0x0206ca thru 0x02d470 (0xcda7 bytes) 000692 1788727394.106817 read 0x00924b thru 0x019223 (0xffd9 bytes) 000693 1788727394.110292 punch from 0x00e05f to 0x01ca56 (0xe9f8 bytes) 000694 1788727394.131799 fallocate 0x0042dc thru 0x00b864 (0x7589 bytes) 000695 1788727394.152203 write 0x01521a thru 0x0212e6 (0xc0cd bytes) 000697 1788727394.168996 trunc from 0x006bf1 to 0x00e863 (0x7c73 bytes) 000698 1788727394.176981 punch from 0x03205e to 0x0336f1 (0x1694 bytes) 000699 1788727394.186543 fallocate 0x017351 thru 0x0217a2 (0xa452 bytes) 000700 1788727394.200189 trunc from 0x019411 to 0x01e8fe (0x54ee bytes) 000701 1788727394.235110 write 0x023eca thru 0x0313d4 (0xd50b bytes) 000702 1788727394.239335 punch from 0x004277 to 0x012f09 (0xec93 bytes) 000703 1788727394.259451 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x336f2): Mirror 5: 0xa009458c (stale) Mirror 7: 0xa009458c (stale) Mirror 10: 0xa009458c (stale) Mirror 12: 0xa009458c (stale) Mirror 13: 0xa009458c (stale) Mirror 15: 0xa009458c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x336f2): Mirror 5: 0xa009458c (stale) Mirror 7: 0xa009458c (stale) Mirror 10: 0xa009458c (stale) Mirror 12: 0xa009458c (stale) Mirror 13: 0xa009458c (stale) Mirror 15: 0xa009458c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x336f2: skipped 000704 1788727395.122380 write 0x016c82 thru 0x02385a (0xcbd9 bytes) 000705 1788727395.142874 read 0x01cae8 thru 0x025834 (0x8d4d bytes) 000707 1788727395.151786 mapwrite 0x003d60 thru 0x011f75 (0xe216 bytes) 000708 1788727395.228501 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x336f2): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x336f2): Mirror 5: 0xa009458c (stale) Mirror 7: 0xa009458c (stale) Mirror 10: 0xa009458c (stale) Mirror 12: 0xa009458c (stale) Mirror 13: 0xa009458c (stale) Mirror 15: 0xa009458c (stale) CRC-32 checksum value for chunk [0, 0x336f2): Mirror 1: 0x6b73cd48 Mirror 16: 0x6b73cd48 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x336f2: skipped 000709 1788727396.596107 mapread 0x029535 thru 0x0336f1 (0xa1bd bytes) 000710 1788727396.663248 fallocate 0x03545b thru 0x039abc (0x4662 bytes) 000711 1788727396.741878 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39abd): Mirror 5: 0x578f2e44 (stale) Mirror 7: 0x578f2e44 (stale) Mirror 10: 0x578f2e44 (stale) Mirror 12: 0x578f2e44 (stale) Mirror 13: 0x578f2e44 (stale) Mirror 16: 0x7a1e432d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39abd): Mirror 5: 0x578f2e44 (stale) Mirror 7: 0x578f2e44 (stale) Mirror 10: 0x578f2e44 (stale) Mirror 12: 0x578f2e44 (stale) Mirror 13: 0x578f2e44 (stale) Mirror 16: 0x7a1e432d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39abd: skipped 000712 1788727397.393375 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39abd): Mirror 5: 0x578f2e44 (stale) Mirror 7: 0x578f2e44 (stale) Mirror 10: 0x578f2e44 (stale) Mirror 12: 0x578f2e44 (stale) Mirror 16: 0x7a1e432d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39abd): Mirror 5: 0x578f2e44 (stale) Mirror 7: 0x578f2e44 (stale) Mirror 10: 0x578f2e44 (stale) Mirror 12: 0x578f2e44 (stale) Mirror 16: 0x7a1e432d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39abd: skipped 000713 1788727397.816852 punch from 0x016c7a to 0x0207ba (0x9b41 bytes) 000714 1788727397.823849 mapwrite 0x014f74 thru 0x01be84 (0x6f11 bytes) 000715 1788727397.872407 read 0x016e2e thru 0x01b6ac (0x487f bytes) 000717 1788727397.875102 write 0x013f0a thru 0x0222c1 (0xe3b8 bytes) 000718 1788727397.890384 mapread 0x018242 thru 0x01b622 (0x33e1 bytes) 000720 1788727397.892603 punch from 0x008a3f to 0x00b154 (0x2716 bytes) 000721 1788727397.899820 mapread 0x0296b0 thru 0x0342d9 (0xac2a bytes) 000722 1788727397.932798 trunc from 0x037bf5 to 0x0399a0 (0x1dac bytes) 000723 1788727397.941755 punch from 0x03973d to 0x039abc (0x380 bytes) 000724 1788727397.951778 read 0x025617 thru 0x02af27 (0x5911 bytes) 000725 1788727397.952757 fallocate 0x006e33 thru 0x01519c (0xe36a bytes) 000726 1788727397.967350 trunc from 0x005779 to 0x007936 (0x21be bytes) 000727 1788727397.974867 punch from 0x015683 to 0x01e921 (0x929f bytes) 000728 1788727397.999947 trunc from 0x01734e to 0x0198c5 (0x2578 bytes) 000729 1788727398.012957 fallocate 0x0161e8 thru 0x017e5d (0x1c76 bytes) 000730 1788727398.021059 mapread 0x020f41 thru 0x026c10 (0x5cd0 bytes) 000731 1788727398.024738 read 0x01272e thru 0x01303f (0x912 bytes) 000732 1788727398.026695 write 0x0060c1 thru 0x006560 (0x4a0 bytes) 000733 1788727398.043889 mapread 0x0252d1 thru 0x02a964 (0x5694 bytes) 000734 1788727398.046527 read 0x02d719 thru 0x037692 (0x9f7a bytes) 000735 1788727398.048498 trunc from 0x0286c5 to 0x038660 (0xff9c bytes) 000737 1788727398.056596 mapread 0x021598 thru 0x0301ff (0xec68 bytes) 000738 1788727398.072052 write 0x019bc7 thru 0x01ad3f (0x1179 bytes) 000739 1788727398.073143 trunc from 0x039abd to 0x0284ce (0x115ee bytes) 000741 1788727398.121682 punch from 0x0095dc to 0x016768 (0xd18d bytes) 000742 1788727398.128839 write 0x004d71 thru 0x00efc2 (0xa252 bytes) 000743 1788727398.137035 write 0x03dd7c thru 0x03ffff (0x2284 bytes) 000745 1788727398.143991 mapread 0x017076 thru 0x01a9a7 (0x3932 bytes) 000746 1788727398.144984 write 0x018e86 thru 0x024e51 (0xbfcc bytes) 000747 1788727398.159878 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xbce1bb7e (stale) Mirror 7: 0xbce1bb7e (stale) Mirror 10: 0xbce1bb7e (stale) Mirror 12: 0xbce1bb7e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xbce1bb7e (stale) Mirror 7: 0xbce1bb7e (stale) Mirror 10: 0xbce1bb7e (stale) Mirror 12: 0xbce1bb7e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000748 1788727398.808546 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xbce1bb7e (stale) Mirror 10: 0xbce1bb7e (stale) Mirror 12: 0xbce1bb7e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xbce1bb7e (stale) Mirror 10: 0xbce1bb7e (stale) Mirror 12: 0xbce1bb7e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000749 1788727399.232128 write 0x030cd2 thru 0x037ea0 (0x71cf bytes) 000750 1788727399.258326 punch from 0x03c65a to 0x03f045 (0x29ec bytes) 000752 1788727399.269682 write 0x027733 thru 0x033f53 (0xc821 bytes) 000753 1788727399.285339 mapwrite 0x0114db thru 0x01f33f (0xde65 bytes) 000754 1788727399.346731 read 0x032339 thru 0x036fd1 (0x4c99 bytes) 000755 1788727399.347367 trunc from 0x023eb5 to 0x032b5d (0xeca9 bytes) 000756 1788727399.355079 mapwrite 0x0319b3 thru 0x0341d9 (0x2827 bytes) 000757 1788727399.427798 read 0x021415 thru 0x02f8be (0xe4aa bytes) 000758 1788727399.456162 MIRROR_RESYNC 4 mirrors 000759 1788727401.448149 MIRROR_EXTEND to 5 mirrors 000761 1788727402.673563 punch from 0x00f4a5 to 0x0121d5 (0x2d31 bytes) 000762 1788727402.732739 punch from 0x03a309 to 0x03ffff (0x5cf7 bytes) 000764 1788727402.739293 trunc from 0x009ab0 to 0x0186a8 (0xebf9 bytes) 000765 1788727402.745624 trunc from 0x031a05 to 0x03dcc0 (0xc2bc bytes) 000766 1788727402.751991 mapread 0x010009 thru 0x01ad0f (0xad07 bytes) 000767 1788727402.777478 fallocate 0x02bff1 thru 0x02c124 (0x134 bytes) 000769 1788727402.783010 trunc from 0x040000 to 0x02ea0c (0x115f3 bytes) 000770 1788727402.805130 fallocate 0x03eb3c thru 0x03ffff (0x14c4 bytes) 000771 1788727402.815577 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x735a5551 (stale) Mirror 10: 0x735a5551 (stale) Mirror 12: 0x735a5551 (stale) Mirror 13: 0x735a5551 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x21229c95 Mirror 14: 0x21229c95 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000772 1788727403.753291 trunc from 0x040000 to 0x0295e7 (0x16a18 bytes) 000773 1788727403.803769 write 0x02e587 thru 0x02ebda (0x654 bytes) 000774 1788727403.806297 read 0x01c653 thru 0x0233c8 (0x6d76 bytes) 000776 1788727403.838782 read 0x005749 thru 0x011941 (0xc1f9 bytes) 000777 1788727403.880167 read 0x0228a5 thru 0x028777 (0x5ed3 bytes) 000779 1788727403.883107 trunc from 0x02ebdb to 0x03f2af (0x106d5 bytes) 000780 1788727403.931383 mapread 0x017ecc thru 0x0246bf (0xc7f4 bytes) 000781 1788727403.932443 trunc from 0x02df91 to 0x031df3 (0x3e63 bytes) 000782 1788727403.946100 write 0x02cab6 thru 0x0392a5 (0xc7f0 bytes) 000784 1788727403.971097 fallocate 0x005749 thru 0x00dab4 (0x836c bytes) 000786 1788727403.984479 mapread 0x0179bb thru 0x01d9ba (0x6000 bytes) 000787 1788727403.985496 mapread 0x02800d thru 0x02d3a0 (0x5394 bytes) 000788 1788727403.987339 trunc from 0x03f2b0 to 0x011281 (0x2e02e bytes) 000789 1788727404.013051 trunc from 0x011282 to 0x001024 (0x1025d bytes) 000790 1788727404.038997 write 0x03e42d thru 0x03ffff (0x1bd3 bytes) 000791 1788727404.040298 trunc from 0x040000 to 0x02c3f9 (0x13c06 bytes) 000792 1788727404.097422 mapwrite 0x027e1f thru 0x02dce7 (0x5ec9 bytes) 000794 1788727404.202382 mapwrite 0x0094fd thru 0x015f35 (0xca39 bytes) 000796 1788727404.339245 mapread 0x018db6 thru 0x02689a (0xdae5 bytes) 000797 1788727404.469827 punch from 0x009105 to 0x0101a8 (0x70a4 bytes) 000798 1788727404.486808 MIRROR_RESYNC 6 mirrors 000799 1788727406.254243 fallocate 0x0107a7 thru 0x013864 (0x30be bytes) 000800 1788727406.282771 read 0x017266 thru 0x02485b (0xd5f6 bytes) 000801 1788727406.294015 punch from 0x025164 to 0x02dce7 (0x8b84 bytes) 000802 1788727406.299902 mapread 0x01f776 thru 0x02d2f0 (0xdb7b bytes) 000803 1788727406.315759 punch from 0x00bcf5 to 0x018057 (0xc363 bytes) 000804 1788727406.320359 mapread 0x01b4b3 thru 0x01f1b3 (0x3d01 bytes) 000805 1788727406.325583 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2dce8): Mirror 7: 0xec17e71e (stale) Mirror 10: 0xec17e71e (stale) Mirror 13: 0xec17e71e (stale) Mirror 14: 0xec17e71e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dce8): Mirror 7: 0xec17e71e (stale) Mirror 10: 0xec17e71e (stale) Mirror 13: 0xec17e71e (stale) Mirror 14: 0xec17e71e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dce8: skipped 000807 1788727406.912678 mapread 0x029606 thru 0x02dce7 (0x46e2 bytes) 000808 1788727406.934379 fallocate 0x004753 thru 0x00f8c6 (0xb174 bytes) 000809 1788727406.940425 read 0x006f8d thru 0x010295 (0x9309 bytes) 000810 1788727406.951147 trunc from 0x00de58 to 0x0173c1 (0x956a bytes) 000812 1788727406.958413 mapwrite 0x02a8bc thru 0x032e04 (0x8549 bytes) 000814 1788727407.070317 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32e05): Mirror 10: 0xd46e2bed (stale) Mirror 13: 0xd46e2bed (stale) Mirror 14: 0xd46e2bed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32e05): Mirror 10: 0xd46e2bed (stale) Mirror 13: 0xd46e2bed (stale) Mirror 14: 0xd46e2bed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32e05: skipped 000815 1788727407.682668 fallocate 0x03605f thru 0x03ffff (0x9fa1 bytes) 000816 1788727407.698268 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x994f0caf (stale) Mirror 13: 0x994f0caf (stale) Mirror 14: 0x994f0caf (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xef18bd9d Mirror 15: 0xef18bd9d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000817 1788727408.686187 read 0x006a26 thru 0x00cb55 (0x6130 bytes) 000818 1788727408.696941 fallocate 0x000065 thru 0x00a5af (0xa54b bytes) 000819 1788727408.746769 trunc from 0x040000 to 0x01819e (0x27e61 bytes) 000820 1788727408.778144 trunc from 0x00545c to 0x00cf8f (0x7b34 bytes) 000821 1788727408.783955 trunc from 0x003391 to 0x008323 (0x4f93 bytes) 000822 1788727408.788645 fallocate 0x029b94 thru 0x02f7ed (0x5c5a bytes) 000824 1788727408.794304 trunc from 0x00f2d9 to 0x01281b (0x3543 bytes) 000825 1788727408.798772 trunc from 0x00e38e to 0x01cd55 (0xe9c8 bytes) 000826 1788727408.808014 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f7ee): Mirror 10: 0xdf46c9c2 (stale) Mirror 13: 0xdf46c9c2 (stale) Mirror 15: 0x5b54beb7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f7ee): Mirror 10: 0xdf46c9c2 (stale) Mirror 13: 0xdf46c9c2 (stale) Mirror 15: 0x5b54beb7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f7ee: skipped 000827 1788727409.165474 fallocate 0x016588 thru 0x01f87b (0x92f4 bytes) 000828 1788727409.172382 mapread 0x0283ad thru 0x0286f0 (0x344 bytes) 000829 1788727409.178609 mapread 0x01a6fb thru 0x028eb5 (0xe7bb bytes) 000830 1788727409.203248 mapwrite 0x03e8cc thru 0x03ffff (0x1734 bytes) 000831 1788727409.249668 write 0x03bc01 thru 0x03ffff (0x43ff bytes) 000832 1788727409.251809 trunc from 0x01ad8c to 0x028459 (0xd6ce bytes) 000834 1788727409.258944 trunc from 0x03a767 to 0x03ffff (0x5899 bytes) 000835 1788727409.273703 trunc from 0x040000 to 0x021239 (0x1edc6 bytes) 000836 1788727409.294211 read 0x0168cd thru 0x021239 (0xa96d bytes) 000837 1788727409.303992 mapread 0x0204c2 thru 0x021239 (0xd78 bytes) 000839 1788727409.306336 trunc from 0x007860 to 0x00b979 (0x411a bytes) 000840 1788727409.311812 fallocate 0x025ea8 thru 0x02e05f (0x81b8 bytes) 000841 1788727409.316941 trunc from 0x01f65e to 0x021239 (0x1bdc bytes) 000842 1788727409.321758 trunc from 0x02123a to 0x030c42 (0xfa09 bytes) 000843 1788727409.337571 mapread 0x00b457 thru 0x012bfa (0x77a4 bytes) 000845 1788727409.348088 trunc from 0x005f3a to 0x0085fa (0x26c1 bytes) 000846 1788727409.353382 punch from 0x02af8f to 0x030c42 (0x5cb4 bytes) 000847 1788727409.359671 write 0x036fd9 thru 0x03e855 (0x787d bytes) 000848 1788727409.361370 write 0x00b8b5 thru 0x011571 (0x5cbd bytes) 000849 1788727409.363195 write 0x01d231 thru 0x022745 (0x5515 bytes) 000850 1788727409.368112 read 0x032d51 thru 0x03ca53 (0x9d03 bytes) 000851 1788727409.381071 trunc from 0x005b76 to 0x00da07 (0x7e92 bytes) 000852 1788727409.403230 mapwrite 0x039ae2 thru 0x03ffff (0x651e bytes) 000853 1788727409.543012 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x994f0caf (stale) Mirror 15: 0xef18bd9d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x994f0caf (stale) Mirror 15: 0xef18bd9d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000854 1788727409.969683 punch from 0x007bc9 to 0x0146ca (0xcb02 bytes) 000855 1788727409.981039 fallocate 0x00acce thru 0x014b2f (0x9e62 bytes) 000856 1788727410.000008 MIRROR_RESYNC 3 mirrors 000857 1788727411.070769 write 0x02f158 thru 0x03998c (0xa835 bytes) 000858 1788727411.100503 trunc from 0x040000 to 0x0178a5 (0x2875a bytes) 000859 1788727411.130722 punch from 0x006eee to 0x013e79 (0xcf8c bytes) 000860 1788727411.140581 trunc from 0x00c18f to 0x00c3cf (0x241 bytes) 000861 1788727411.144360 fallocate 0x00e99a thru 0x01cb67 (0xe1ce bytes) 000862 1788727411.150887 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x178a6): Mirror 10: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x178a6): Mirror 10: 0xfe713edf (stale) Mirror 15: 0xfe713edf (stale) CRC-32 checksum value for chunk [0, 0x178a6): Mirror 1: 0xfe713edf Mirror 16: 0xfe713edf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x178a6: skipped 000864 1788727412.143886 mapread 0x0116fc thru 0x0169ef (0x52f4 bytes) 000865 1788727412.172090 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x178a6): Mirror 10: 0xfe713edf (stale) Mirror 15: 0xfe713edf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x178a6): Mirror 10: 0xfe713edf (stale) Mirror 15: 0xfe713edf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x178a6: skipped 000866 1788727412.480455 trunc from 0x002b06 to 0x00ebf6 (0xc0f1 bytes) 000867 1788727412.500277 trunc from 0x0178a6 to 0x015c58 (0x1c4d bytes) 000868 1788727412.541346 MIRROR_RESYNC 3 mirrors 000869 1788727413.525445 mapwrite 0x01fb5f thru 0x021edb (0x237d bytes) 000870 1788727413.600053 fallocate 0x0272f1 thru 0x02b21b (0x3f2b bytes) 000871 1788727413.604720 fallocate 0x02311e thru 0x0301af (0xd092 bytes) 000872 1788727413.609300 fallocate 0x01d669 thru 0x02c90f (0xf2a7 bytes) 000873 1788727413.614162 fallocate 0x03164f thru 0x03a478 (0x8e2a bytes) 000874 1788727413.624489 MIRROR_RESYNC 3 mirrors 000876 1788727414.669708 trunc from 0x00ba06 to 0x0179e0 (0xbfdb bytes) 000877 1788727414.705425 trunc from 0x02c910 to 0x03fbc1 (0x132b2 bytes) 000878 1788727414.738596 fallocate 0x033a62 thru 0x03ffff (0xc59e bytes) 000879 1788727414.745101 trunc from 0x03d665 to 0x03fbc1 (0x255d bytes) 000880 1788727414.751158 fallocate 0x031b1e thru 0x03cf5a (0xb43d bytes) 000881 1788727414.761128 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fbc2): Mirror 15: 0xdd2cae1d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fbc2): Mirror 15: 0xdd2cae1d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fbc2: skipped 000883 1788727415.119134 mapread 0x01eb02 thru 0x025c3d (0x713c bytes) 000884 1788727415.141876 punch from 0x001da5 to 0x008bee (0x6e4a bytes) 000885 1788727415.149949 punch from 0x01a0aa to 0x01fc06 (0x5b5d bytes) 000886 1788727415.156549 fallocate 0x0244d2 thru 0x028706 (0x4235 bytes) 000887 1788727415.161810 fallocate 0x01ea0d thru 0x02e80b (0xfdff bytes) 000888 1788727415.167658 punch from 0x00f918 to 0x01ac63 (0xb34c bytes) 000889 1788727415.173709 mapwrite 0x001551 thru 0x00275f (0x120f bytes) 000890 1788727415.200531 write 0x024fa2 thru 0x029b81 (0x4be0 bytes) 000891 1788727415.218574 write 0x03ffe2 thru 0x03ffff (0x1e bytes) 000892 1788727415.223884 read 0x03b253 thru 0x03ffff (0x4dad bytes) 000893 1788727415.229415 mapread 0x00ee4f thru 0x00ffbe (0x1170 bytes) 000894 1788727415.230267 mapwrite 0x00ed01 thru 0x01a25c (0xb55c bytes) 000895 1788727415.273276 mapread 0x03ce00 thru 0x03d0a7 (0x2a8 bytes) 000896 1788727415.274264 mapread 0x0257ea thru 0x02aecd (0x56e4 bytes) 000897 1788727415.303011 punch from 0x030f68 to 0x034773 (0x380c bytes) 000898 1788727415.318193 mapwrite 0x00d288 thru 0x014fef (0x7d68 bytes) 000899 1788727415.365968 fallocate 0x031fa4 thru 0x03343b (0x1498 bytes) 000900 1788727415.386079 fallocate 0x038d33 thru 0x03ffff (0x72cd bytes) 000901 1788727415.394568 fallocate 0x0135d4 thru 0x01ebfb (0xb628 bytes) 000902 1788727415.406077 fallocate 0x02c569 thru 0x0301fc (0x3c94 bytes) 000903 1788727415.417243 punch from 0x0012a4 to 0x00ec99 (0xd9f6 bytes) 000904 1788727415.431117 trunc from 0x040000 to 0x01ef67 (0x21098 bytes) 000905 1788727415.499832 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1ef68): Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ef68): Mirror 15: 0x50e851d7 (stale) CRC-32 checksum value for chunk [0, 0x1ef68): Mirror 1: 0x46ab9217 Mirror 16: 0x46ab9217 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ef68: skipped 000906 1788727416.239005 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1ef68): Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ef68): Mirror 15: 0x50e851d7 (stale) CRC-32 checksum value for chunk [0, 0x1ef68): Mirror 1: 0x46ab9217 Mirror 16: 0x46ab9217 Mirror 17: 0x46ab9217 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ef68: skipped 000908 1788727417.039776 mapwrite 0x027a1e thru 0x02bfd3 (0x45b6 bytes) 000909 1788727417.180494 trunc from 0x02bfd4 to 0x001569 (0x2aa6a bytes) 000911 1788727417.208349 trunc from 0x000f90 to 0x001569 (0x5da bytes) 000912 1788727417.218585 mapwrite 0x03660b thru 0x037425 (0xe1b bytes) 000913 1788727417.258153 mapread 0x014af2 thru 0x021db1 (0xd2c0 bytes) 000914 1788727417.286084 punch from 0x034efe to 0x037425 (0x2528 bytes) 000915 1788727417.296483 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x159:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37426): Mirror 16: 0x9de97713 (stale) Mirror 17: 0x9de97713 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37426): Mirror 16: 0x9de97713 (stale) Mirror 17: 0x9de97713 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37426: skipped 000916 1788727417.597879 mapwrite 0x00b55a thru 0x015861 (0xa308 bytes) 000917 1788727417.644865 fallocate 0x0201c6 thru 0x02d240 (0xd07b bytes) 000918 1788727417.650646 mapread 0x030d20 thru 0x037425 (0x6706 bytes) 000920 1788727417.662467 read 0x024807 thru 0x025b5b (0x1355 bytes) 000921 1788727417.667632 mapwrite 0x029417 thru 0x02d7cf (0x43b9 bytes) 000922 1788727417.685806 read 0x0225b4 thru 0x02733b (0x4d88 bytes) 000923 1788727417.697469 trunc from 0x037426 to 0x02eb16 (0x890f bytes) 000924 1788727417.719118 trunc from 0x004b1b to 0x0115ad (0xca93 bytes) 000925 1788727417.725787 trunc from 0x026446 to 0x027615 (0x11d0 bytes) 000926 1788727417.732108 mapread 0x024f6a thru 0x02eb16 (0x9bad bytes) 000927 1788727417.745142 mapread 0x01a377 thru 0x026fdf (0xcc69 bytes) 000928 1788727417.746349 trunc from 0x02eb17 to 0x029b85 (0x4f91 bytes) 000929 1788727417.768904 mapread 0x028089 thru 0x029b85 (0x1afd bytes) 000930 1788727417.769872 mapwrite 0x001d73 thru 0x005c50 (0x3ede bytes) 000931 1788727417.799320 fallocate 0x035a03 thru 0x03b4dc (0x5ada bytes) 000932 1788727417.805507 mapread 0x00e29c thru 0x015443 (0x71a8 bytes) 000933 1788727417.829107 fallocate 0x01c230 thru 0x02bd8e (0xfb5f bytes) 000935 1788727417.836524 write 0x0082ca thru 0x00da67 (0x579e bytes) 000936 1788727417.847107 trunc from 0x03b4dd to 0x03bd82 (0x8a6 bytes) 000937 1788727417.869582 fallocate 0x03ba3d thru 0x03ffff (0x45c3 bytes) 000938 1788727417.879779 MIRROR_RESYNC 3 mirrors 000939 1788727418.871519 write 0x031888 thru 0x036066 (0x47df bytes) 000940 1788727418.924846 trunc from 0x03bd83 to 0x02fb8b (0xc1f7 bytes) 000941 1788727418.950122 write 0x010cf9 thru 0x01a6ae (0x99b6 bytes) 000942 1788727418.963518 trunc from 0x02fb8c to 0x00ce0b (0x22d80 bytes) 000944 1788727418.985168 trunc from 0x00ce0c to 0x037ec7 (0x2b0bc bytes) 000945 1788727419.004708 trunc from 0x037ec8 to 0x00c8b4 (0x2b613 bytes) 000946 1788727419.021144 write 0x03fad3 thru 0x03ffff (0x52d bytes) 000947 1788727419.023140 fallocate 0x000eff thru 0x006be8 (0x5cea bytes) 000948 1788727419.029248 mapread 0x015b87 thru 0x017e9d (0x2317 bytes) 000949 1788727419.047097 mapread 0x004579 thru 0x011412 (0xce9a bytes) 000950 1788727419.063479 fallocate 0x016631 thru 0x01fc89 (0x9659 bytes) 000951 1788727419.069138 write 0x005b5c thru 0x007da5 (0x224a bytes) 000952 1788727419.070497 trunc from 0x01132b to 0x01dada (0xc7b0 bytes) 000953 1788727419.076576 trunc from 0x040000 to 0x01f6c8 (0x20937 bytes) 000954 1788727419.092632 punch from 0x008fbe to 0x00e746 (0x5789 bytes) 000955 1788727419.100182 trunc from 0x01f6c9 to 0x02e9c1 (0xf2f9 bytes) 000956 1788727419.113999 trunc from 0x01a288 to 0x02374b (0x94c4 bytes) 000957 1788727419.120809 trunc from 0x02e9c2 to 0x03fc19 (0x11258 bytes) 000958 1788727419.152460 mapread 0x02e43f thru 0x0372dc (0x8e9e bytes) 000960 1788727419.181430 mapread 0x004713 thru 0x01071b (0xc009 bytes) 000961 1788727419.192486 fallocate 0x007ac5 thru 0x011327 (0x9863 bytes) 000962 1788727419.198193 write 0x03e4b7 thru 0x03f132 (0xc7c bytes) 000963 1788727419.199962 trunc from 0x039a0d to 0x03fc19 (0x620d bytes) 000964 1788727419.218871 read 0x000c21 thru 0x010732 (0xfb12 bytes) 000965 1788727419.224791 mapread 0x009661 thru 0x013448 (0x9de8 bytes) 000966 1788727419.237504 write 0x006ab8 thru 0x00d1b1 (0x66fa bytes) 000967 1788727419.239470 mapwrite 0x03d05f thru 0x03ffff (0x2fa1 bytes) 000968 1788727419.295068 read 0x00c294 thru 0x0178af (0xb61c bytes) 000969 1788727419.296866 write 0x017770 thru 0x01cec8 (0x5759 bytes) 000970 1788727419.307874 mapread 0x02556d thru 0x02c201 (0x6c95 bytes) 000971 1788727419.318632 punch from 0x028d20 to 0x03782f (0xeb10 bytes) 000972 1788727419.323641 trunc from 0x035177 to 0x03f1be (0xa048 bytes) 000973 1788727419.332580 fallocate 0x029709 thru 0x036145 (0xca3d bytes) 000974 1788727419.347037 mapwrite 0x013b5a thru 0x02104a (0xd4f1 bytes) 000975 1788727419.386683 mapwrite 0x02a6ac thru 0x038c61 (0xe5b6 bytes) 000976 1788727419.484294 trunc from 0x040000 to 0x03f24f (0xdb0 bytes) 000978 1788727419.509802 mapread 0x00387e thru 0x00514d (0x18d0 bytes) 000980 1788727419.512798 trunc from 0x03138d to 0x03f24f (0xdec3 bytes) 000981 1788727419.520101 mapwrite 0x0215ee thru 0x02d771 (0xc184 bytes) 000983 1788727419.552627 fallocate 0x02e78c thru 0x03c8b5 (0xe12a bytes) 000984 1788727419.563624 fallocate 0x023dc0 thru 0x02d0de (0x931f bytes) 000985 1788727419.574107 write 0x01ccaf thru 0x024f11 (0x8263 bytes) 000986 1788727419.578153 trunc from 0x035ceb to 0x03b2c6 (0x55dc bytes) 000987 1788727419.589349 mapread 0x0128dc thru 0x0157d3 (0x2ef8 bytes) 000989 1788727419.590694 trunc from 0x01e058 to 0x0263cb (0x8374 bytes) 000990 1788727419.613131 write 0x021b5d thru 0x030bf0 (0xf094 bytes) 000991 1788727419.622097 punch from 0x01239d to 0x0189cc (0x6630 bytes) 000993 1788727419.636724 read 0x020330 thru 0x0254a6 (0x5177 bytes) 000994 1788727419.648508 trunc from 0x025cb3 to 0x030bf1 (0xaf3f bytes) 000995 1788727419.684972 trunc from 0x03f250 to 0x013d65 (0x2b4ea bytes) 000996 1788727419.728462 trunc from 0x011a19 to 0x013d65 (0x234d bytes) 000997 1788727419.736439 mapread 0x0085c8 thru 0x013d65 (0xb79e bytes) 000998 1788727419.753150 fallocate 0x00d039 thru 0x00fdcd (0x2d95 bytes) 001000 1788727419.758109 trunc from 0x013d66 to 0x03e96f (0x2ac0a bytes) All operations completed A-OK! PASS 70a (112s) == sanity-flr test 71: check mirror extend parent fid ==== 16:43:46 (1788727426) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x55:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 16:43:57 (1788727437) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.32677 s, 6.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112455 s, 9.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x117:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 16:44:07 (1788727447) Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre2 Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5138496 .. Extending file size to 6018624 .. Extending file size to 8527808 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7322464 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8420096 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 61237 61238 61240 61243 61244 failed 192.168.201.104@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg104-client.virtnet /mnt/lustre2 (opts:) 192.168.201.104@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg104-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 16:45:29 (1788727529) Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre2 Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5786336 .. Extending file size to 8375808 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7888832 .. Extending file size to 8177216 .. Extending file size to 8215904 .. Extending file size to 8612800 .. Extending file size to 8470016 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8789120 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8931008 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9387936 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 62587 62588 62590 62593 62595 done 192.168.201.104@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg104-client.virtnet /mnt/lustre2 (opts:) 192.168.201.104@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg104-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (86s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 16:46:54 (1788727614) Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2276 1269028 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2188 1269116 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25192 3581828 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 19040 3587980 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 44232 7169808 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2276 1269028 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2188 1269116 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25192 3581828 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 19040 3587980 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 44232 7169808 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.112237 s, 18.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.229166 s, 9.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xac20e14 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.104@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg104-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 16:47:14 (1788727634) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.117661 s, 17.8 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 16:47:21 (1788727641) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x64:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 16:47:29 (1788727649) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0797398 s, 26.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1eb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ec:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 16:47:36 (1788727656) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0933264 s, 22.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 16:47:43 (1788727663) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100722 s, 20.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 16:48:17 (1788727697) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg104-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971963 s, 10.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ec:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ee:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ed:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ec:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ee:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ed:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ed:0x0] } PASS 204d (54s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 16:49:11 (1788727751) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg104-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.128109 s, 16.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (50s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 16:50:01 (1788727801) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg104-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg104-server: Pool lustre.test_204f created Adding targets to pool oleg104-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg104-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.149574 s, 14.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x135:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x135:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg104-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg104-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg104-server: Pool lustre.test_204f destroyed PASS 204f (74s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 16:51:15 (1788727875) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 16:51:23 (1788727883) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 16:51:30 (1788727890) oleg104-server: Pool lustre.test_206 created Adding targets to pool oleg104-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg104-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg104-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg104-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg104-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 16:51:57 (1788727917) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896482 s, 11.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444163 s, 23.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x5369ed23 Mirror 3: 0x5369ed23 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 16:52:08 (1788727928) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 16:52:11 (1788727931) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 16:52:15 (1788727935) Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.059664 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1ad0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x6e0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x6e0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x6e0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x17e0000 Read this (4 bytes): BBBB 192.168.201.104@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg104-client.virtnet /mnt/lustre2 (opts:) PASS 209a (35s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 16:52:50 (1788727970) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00665448 s, 1.2 MB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 16:52:57 (1788727977) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587994 s, 17.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x85 lmm_fid: [0x200000405:0x85:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x85 lmm_fid: [0x200000405:0x85:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13f:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611269 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 16:52:59.000000000 -0400 Modify: 2026-09-06 16:52:59.000000000 -0400 Change: 2026-09-06 16:53:00.000000000 -0400 Birth: 2026-09-06 16:52:58.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 16:53:05 (1788727985) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494489 s, 21.2 MB/s oleg104-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg104-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg104-server: O/280000401/d30/13e: File not found by ext2_lookup PASS 210b (22s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 16:53:27 (1788728007) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.167855 s, 244 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.029171 s, 140 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 6 16:53 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x8a:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 6 16:53 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 16:53:34 (1788728014) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 16:53:42 (1788728022) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1888 sec =========== 16:53:50 (1788728030) === sanity-flr: start cleanup 16:53:51 (1788728031) === === sanity-flr: finish cleanup 16:53:58 (1788728038) ===