-----============= acceptance-small: sanity-flr ============----- Sat Sep 5 21:20:47 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg441-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 21:21:06 (1788657666) === oleg441-client.virtnet: executing check_config_client /mnt/lustre oleg441-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg441-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9333180a6000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9333180a6000.idle_timeout=debug disable quota as required oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 21:21:40 (1788657700) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7647] == sanity-flr test 0a: lfs mirror create with -N option == 21:21:42 (1788657702) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 21:21:57 (1788657717) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 21:22:00 (1788657720) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 21:22:03 (1788657723) PASS 0d (21s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 21:22:24 (1788657744) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 21:22:27 (1788657747) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 21:22:30 (1788657750) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9333180a6000.stats=clear osc.lustre-OST0001-osc-ffff9333180a6000.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 21:22:36 (1788657756) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 21:22:47 (1788657767) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 21:22:55 (1788657775) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 21:22:58 (1788657778) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 21:23:06 (1788657786) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 21:23:16 (1788657796) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 21:23:23 (1788657803) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 21:23:26 (1788657806) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 21:23:33 (1788657813) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 21:23:42 (1788657822) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 21:23:49 (1788657829) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 1.34915 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114384 s, 9.2 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 21:23:59 (1788657839) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.175109 s, 47.9 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 21:24:06 (1788657846) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 21:24:15 (1788657855) Stopping /mnt/lustre-ost1 (opts:) on oleg441-server oleg441-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9333180a6000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9333180a6000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg441-server: oleg441-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg441-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg441-server oleg441-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9333180a6000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9333180a6000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg441-server: oleg441-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg441-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (67s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 21:25:22 (1788657922) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.403699 s, 18.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg441-server oleg441-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9333180a6000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9333180a6000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg441-server: oleg441-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg441-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 21:26:04 (1788657964) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg441-server oleg441-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9333180a6000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9333180a6000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg441-server: oleg441-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg441-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg441-server oleg441-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9333180a6000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9333180a6000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg441-server: oleg441-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg441-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (91s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 21:27:36 (1788658056) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.57118 s, 9.4 MB/s -rw-r--r-- 1 root root 52428800 Sep 5 21:27 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.524s user 0m0.002s sys 0m0.203s Stopping /mnt/lustre-ost1 (opts:) on oleg441-server oleg441-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9333180a6000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9333180a6000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.085s user 0m0.001s sys 0m0.194s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg441-server: oleg441-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg441-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg441-server oleg441-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9333180a6000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9333180a6000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.121s user 0m0.001s sys 0m0.177s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg441-server: oleg441-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg441-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg441-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9333180a6000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9333180a6000.ost_server_uuid in FULL state after 0 sec PASS 33b (93s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 21:29:08 (1788658148) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 21:29:12 (1788658152) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 21:29:14 (1788658154) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 21:29:17 (1788658157) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 21:29:24 (1788658164) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.597728 s, 21.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 21:29:36 (1788658176) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252135 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306621 s, 34.2 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 21:29:44 (1788658184) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 21:29:52 (1788658192) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00294597 s, 348 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00286221 s, 358 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00078281 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000927034 s, 1.1 MB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 21:30:03 (1788658203) create test files with size 15751671 .. cd96fd6be61507040d6fa4bcdd67ece2 - bd63f179c18680b82e84f63165d21e16 - 52168ed8d7cebb9e797900e3941eb4b7 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg441-client.virtnet /mnt/lustre (opts:) Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 21:30:28 (1788658228) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 21:30:41 (1788658241) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 21:30:48 (1788658248) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906477 s, 11.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360296 s, 29.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 21:30:56 (1788658256) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.138928 s, 15.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.33415 s, 12.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.228141 s, 9.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 21:31:14 (1788658274) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 21:31:18 (1788658278) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 21:31:21 (1788658281) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307556 s, 34.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 21:31:28 (1788658288) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 21:31:32 (1788658292) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 21:31:43 (1788658303) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.263713 s, 39.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg441-client.virtnet /mnt/lustre (opts:) Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 21:31:54 (1788658314) llite.lustre-ffff933320367000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.318301 s, 32.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg441-client.virtnet /mnt/lustre (opts:) Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff933318086000.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 21:32:11 (1788658331) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (41s) == sanity-flr test 46: Verify setstripe --copy option ==== 21:32:51 (1788658371) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (35s) == sanity-flr test 47: Verify mirror obj alloc =========== 21:33:26 (1788658406) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 21:33:29 (1788658409) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.254004 s, 12.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.296758 s, 10.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x6f3fee47 (nosync,stale) Mirror 2: 0x6f3fee47 (nosync,stale) ** verify mirror 2 doesn't change original checksum: aa2e5da58389b6e789a45dd8107a1a76 - mirror 1 checksum: b08b266a038f85eba6e220e89770bcf5 - mirror 2 checksum: aa2e5da58389b6e789a45dd8107a1a76 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-05T21:33:32-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-05T21:33:32-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: aa2e5da58389b6e789a45dd8107a1a76 - mirror 1 checksum: b08b266a038f85eba6e220e89770bcf5 - mirror 2 checksum: b08b266a038f85eba6e220e89770bcf5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-05T21:33:32-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-05T21:33:32-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } PASS 48 (13s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 21:33:43 (1788658423) Failing mds1 on oleg441-server Stopping /mnt/lustre-mds1 (opts:) on oleg441-server 21:33:55 (1788658435) shut down facet: mds1 facet_host: oleg441-server facet_failover_host: oleg441-server Failover mds1 to oleg441-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 21:34:28 (1788658468) targets are mounted 21:34:28 (1788658468) facet_failover done oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639819 s, 16.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } PASS 50A (60s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 21:34:43 (1788658483) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 21:34:46 (1788658486) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 21:34:49 (1788658489) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00629077 s, 2.6 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 21:34:57 (1788658497) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 21:35:01 (1788658501) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00648309 s, 632 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0670883 s, 61.1 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 21:35:09 (1788658509) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 21:35:33 (1788658533) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 21:36:01 (1788658561) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 21:36:24 (1788658584) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 21:36:47 (1788658607) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 21:36:53 (1788658613) mirror create pid 54342 mirror split pid 54344 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54342 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54344 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 21:38:00 (1788658680) Seed set to 8682 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000004 1788658682.523515 MIRROR_EXTEND to 2 mirrors skipping zero length punch hole 000007 1788658682.915853 MIRROR_EXTEND to 3 mirrors skipping zero size read 000011 1788658683.303105 mapwrite 0x026252 thru 0x028f32 (0x2ce1 bytes) 000012 1788658683.524161 mapread 0x0249df thru 0x028f32 (0x4554 bytes) 000013 1788658683.525561 punch from 0x023964 to 0x024f4d (0x15ea bytes) 000014 1788658683.531234 mapwrite 0x023823 thru 0x02b738 (0x7f16 bytes) 000015 1788658683.646104 mapwrite 0x021293 thru 0x02e7e7 (0xd555 bytes) 000017 1788658683.762133 read 0x00f37b thru 0x019125 (0x9dab bytes) 000018 1788658683.772074 mapread 0x002fd0 thru 0x011936 (0xe967 bytes) 000020 1788658683.802330 MIRROR_RESYNC 3 mirrors 000021 1788658684.878832 mapread 0x016bea thru 0x01d9ce (0x6de5 bytes) 000022 1788658684.904788 mapwrite 0x0134a5 thru 0x021d26 (0xe882 bytes) 000023 1788658685.101831 mapread 0x0013cd thru 0x00267b (0x12af bytes) 000024 1788658685.131870 mapread 0x001f5a thru 0x009ccb (0x7d72 bytes) 000027 1788658685.140817 mapread 0x020473 thru 0x02e7e7 (0xe375 bytes) 000028 1788658685.157967 mapwrite 0x03831c thru 0x03ffff (0x7ce4 bytes) 000029 1788658685.273204 read 0x028ae0 thru 0x031930 (0x8e51 bytes) 000030 1788658685.307483 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7312d52b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7312d52b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000031 1788658685.722414 mapwrite 0x00ac2d thru 0x0163d6 (0xb7aa bytes) 000033 1788658685.817489 read 0x002637 thru 0x009738 (0x7102 bytes) truncating to largest ever: 0x2d8fa 000036 1788658685.825261 trunc from 0x040000 to 0x02d8f9 (0x12706 bytes) 000038 1788658685.847649 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) 000041 1788658686.045057 punch from 0x0097ef to 0x013b13 (0xa325 bytes) 000042 1788658686.059615 mapread 0x02b7cd thru 0x02d8f9 (0x212d bytes) 000045 1788658686.086723 read 0x00f1ec thru 0x0111c6 (0x1fdb bytes) 000046 1788658686.098795 mapread 0x01d066 thru 0x0299ca (0xc965 bytes) 000047 1788658686.118189 punch from 0x02193e to 0x024c04 (0x32c7 bytes) 000050 1788658686.141294 punch from 0x020fcd to 0x02694e (0x5982 bytes) 000051 1788658686.156076 mapwrite 0x0204f7 thru 0x026cd4 (0x67de bytes) 000052 1788658686.234955 write 0x00dee7 thru 0x018b4d (0xac67 bytes) 000053 1788658686.252553 write 0x017d34 thru 0x01a222 (0x24ef bytes) 000054 1788658686.258604 write 0x00cf44 thru 0x01b7f7 (0xe8b4 bytes) 000055 1788658686.292884 MIRROR_EXTEND to 2 mirrors 000056 1788658687.124792 punch from 0x01d4c5 to 0x01db1a (0x656 bytes) 000057 1788658687.245673 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) 000058 1788658687.396772 mapread 0x002c7c thru 0x00940d (0x6792 bytes) 000059 1788658687.418529 punch from 0x0219f5 to 0x02d8f9 (0xbf05 bytes) 000060 1788658687.427475 punch from 0x0157e5 to 0x01a8d7 (0x50f3 bytes) 000062 1788658687.436931 write 0x02dd4a thru 0x03975f (0xba16 bytes) 000064 1788658687.454573 punch from 0x015336 to 0x022629 (0xd2f4 bytes) 000065 1788658687.476801 punch from 0x008187 to 0x013753 (0xb5cd bytes) 000067 1788658687.495444 MIRROR_EXTEND to 2 mirrors 000068 1788658688.617503 write 0x035c0a thru 0x03ffff (0xa3f6 bytes) 000069 1788658688.707527 mapread 0x012bfe thru 0x021fb5 (0xf3b8 bytes) 000072 1788658688.780111 mapwrite 0x014580 thru 0x0195c9 (0x504a bytes) 000073 1788658688.905881 mapread 0x012146 thru 0x01e75e (0xc619 bytes) 000074 1788658688.906895 mapread 0x010a0b thru 0x017b68 (0x715e bytes) 000075 1788658688.907810 mapwrite 0x02056f thru 0x0236c9 (0x315b bytes) 000076 1788658688.982415 punch from 0x010bd0 to 0x020878 (0xfca9 bytes) 000077 1788658688.997018 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) 000078 1788658689.219096 write 0x028404 thru 0x030914 (0x8511 bytes) 000079 1788658689.237654 trunc from 0x040000 to 0x01e2e3 (0x21d1c bytes) 000080 1788658689.259043 read 0x002e37 thru 0x005e81 (0x304b bytes) 000081 1788658689.266023 punch from 0x00ed15 to 0x017a85 (0x8d71 bytes) 000084 1788658689.275451 mapwrite 0x016afd thru 0x01fe13 (0x9317 bytes) 000088 1788658689.385875 mapwrite 0x038b25 thru 0x03d3ff (0x48db bytes) 000090 1788658689.484854 MIRROR_EXTEND to 2 mirrors 000092 1788658690.486324 mapwrite 0x00dcf1 thru 0x018d30 (0xb040 bytes) 000093 1788658690.724955 MIRROR_RESYNC 2 mirrors 000094 1788658691.811170 mapwrite 0x0049a9 thru 0x012d8a (0xe3e2 bytes) 000096 1788658692.106721 trunc from 0x03d400 to 0x0218be (0x1bb41 bytes) 000097 1788658692.153531 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x218bf): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x218bf): Mirror 2: 0x819c3400 (stale) CRC-32 checksum value for chunk [0, 0x218bf): Mirror 1: 0xdd7f1602 Mirror 3: 0xdd7f1602 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x218bf: skipped 000099 1788658693.552434 trunc from 0x0218bf to 0x0263fe (0x4b40 bytes) 000100 1788658693.649609 read 0x026191 thru 0x02621e (0x8e bytes) 000101 1788658693.671975 write 0x035f1c thru 0x03ffff (0xa0e4 bytes) 000102 1788658693.678956 punch from 0x03bb07 to 0x03ffff (0x44f9 bytes) 000104 1788658693.698255 trunc from 0x040000 to 0x02d642 (0x129bd bytes) 000108 1788658693.736928 read 0x00bc4c thru 0x012537 (0x68ec bytes) 000110 1788658693.752607 read 0x001785 thru 0x009d91 (0x860d bytes) 000112 1788658693.753832 read 0x019663 thru 0x020a16 (0x73b4 bytes) 000114 1788658693.757182 mapread 0x02b9a8 thru 0x02cbcc (0x1225 bytes) 000115 1788658693.765576 MIRROR_RESYNC 3 mirrors 000116 1788658694.917565 punch from 0x009761 to 0x010ac2 (0x7362 bytes) 000117 1788658694.990239 trunc from 0x02d643 to 0x00ff0b (0x1d737 bytes) 000118 1788658695.032865 trunc from 0x00ff0c to 0x012f85 (0x307a bytes) 000119 1788658695.062942 write 0x01e6d2 thru 0x02afc7 (0xc8f6 bytes) 000123 1788658695.068798 mapwrite 0x031831 thru 0x034772 (0x2f42 bytes) 000124 1788658695.174824 punch from 0x0068a6 to 0x00bcbb (0x5416 bytes) truncating to largest ever: 0x3ec3a 000126 1788658695.187336 trunc from 0x034773 to 0x03ec39 (0xa4c7 bytes) 000127 1788658695.205947 punch from 0x0281b2 to 0x031cda (0x9b29 bytes) 000128 1788658695.214217 punch from 0x0356ca to 0x03b123 (0x5a5a bytes) 000130 1788658695.221800 read 0x000cbb thru 0x0060a3 (0x53e9 bytes) 000131 1788658695.234841 trunc from 0x03ec3a to 0x03c712 (0x2527 bytes) 000134 1788658695.255818 mapwrite 0x001104 thru 0x00bf47 (0xae44 bytes) 000136 1788658695.392173 MIRROR_RESYNC 3 mirrors 000137 1788658696.805184 write 0x03887d thru 0x03ffff (0x7783 bytes) 000138 1788658696.864792 read 0x0131f1 thru 0x014bd9 (0x19e9 bytes) 000139 1788658696.873857 mapwrite 0x03cb1d thru 0x03ffff (0x34e3 bytes) 000140 1788658696.937559 trunc from 0x040000 to 0x035ce2 (0xa31d bytes) 000142 1788658696.971803 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35ce3): Mirror 3: 0x9a26db12 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35ce3): Mirror 3: 0x9a26db12 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35ce3: skipped 000143 1788658697.268682 mapwrite 0x034d42 thru 0x03f9fe (0xacbd bytes) 000144 1788658697.361922 mapread 0x01ed3d thru 0x0224e6 (0x37aa bytes) 000145 1788658697.372591 read 0x026a6e thru 0x02d14f (0x66e2 bytes) 000147 1788658697.385364 read 0x00bd24 thru 0x00ea47 (0x2d24 bytes) 000148 1788658697.396636 write 0x02c0b4 thru 0x0342d5 (0x8222 bytes) 000149 1788658697.398718 trunc from 0x03f9ff to 0x01175f (0x2e29f bytes) 000151 1788658697.426675 trunc from 0x011760 to 0x03c2cb (0x2ab6c bytes) 000152 1788658697.449503 MIRROR_RESYNC 2 mirrors 000153 1788658698.367703 trunc from 0x03c2cc to 0x02ed65 (0xd566 bytes) 000154 1788658698.513835 read 0x00a472 thru 0x00f7d5 (0x5364 bytes) 000155 1788658698.524315 mapread 0x0152fd thru 0x01b786 (0x648a bytes) 000156 1788658698.549716 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2ed66): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ed66): Mirror 3: 0xf66cdad7 (stale) CRC-32 checksum value for chunk [0, 0x2ed66): Mirror 1: 0xf66cdad7 Mirror 4: 0xf66cdad7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ed66: skipped 000158 1788658699.858447 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ed66): Mirror 3: 0xf66cdad7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ed66): Mirror 3: 0xf66cdad7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ed66: skipped 000160 1788658700.218596 mapread 0x013833 thru 0x01547b (0x1c49 bytes) 000161 1788658700.249795 punch from 0x027430 to 0x02ed65 (0x7936 bytes) 000162 1788658700.263725 punch from 0x024430 to 0x02ed65 (0xa936 bytes) 000163 1788658700.269806 trunc from 0x02ed66 to 0x03e53a (0xf7d5 bytes) 000164 1788658700.291709 write 0x019542 thru 0x01988a (0x349 bytes) 000165 1788658700.300890 read 0x01a4b1 thru 0x025cdb (0xb82b bytes) 000166 1788658700.332419 read 0x007fab thru 0x01151b (0x9571 bytes) 000169 1788658700.352175 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e53b): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e53b): Mirror 3: 0xdc48d610 (stale) CRC-32 checksum value for chunk [0, 0x3e53b): Mirror 1: 0xb53e2ecc Mirror 4: 0xb53e2ecc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e53b: skipped 000172 1788658701.468645 punch from 0x019ee4 to 0x01d743 (0x3860 bytes) 000173 1788658701.564187 mapwrite 0x0241ce thru 0x02560a (0x143d bytes) 000176 1788658701.624145 mapread 0x02b540 thru 0x03a79b (0xf25c bytes) 000177 1788658701.641898 mapwrite 0x0117a2 thru 0x012735 (0xf94 bytes) 000178 1788658701.712474 read 0x001331 thru 0x0053f6 (0x40c6 bytes) 000179 1788658701.745209 punch from 0x01f31a to 0x02dae2 (0xe7c9 bytes) 000180 1788658701.758949 read 0x00139b thru 0x0022db (0xf41 bytes) 000182 1788658701.772666 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e53b): Mirror 3: 0xdc48d610 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e53b): Mirror 3: 0xdc48d610 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e53b: skipped 000183 1788658702.313838 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e53b): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e53b): Mirror 3: 0xdc48d610 (stale) CRC-32 checksum value for chunk [0, 0x3e53b): Mirror 1: 0x58a1f767 Mirror 4: 0x58a1f767 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e53b: skipped 000185 1788658703.592817 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e53b): Mirror 3: 0xdc48d610 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e53b): Mirror 3: 0xdc48d610 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e53b: skipped 000187 1788658703.890764 write 0x002fd7 thru 0x006c1d (0x3c47 bytes) 000188 1788658703.919954 mapread 0x03a2b9 thru 0x03e53a (0x4282 bytes) 000190 1788658703.948144 trunc from 0x03e53b to 0x006dbb (0x3777f bytes) 000191 1788658703.970555 mapread 0x004fa0 thru 0x006dbb (0x1e1c bytes) 000193 1788658703.977344 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) 000194 1788658704.179544 mapread 0x001b28 thru 0x004a0b (0x2ee4 bytes) 000199 1788658704.218419 read 0x003280 thru 0x006dbb (0x3b3c bytes) 000201 1788658704.219667 mapread 0x003105 thru 0x006dbb (0x3cb7 bytes) 000202 1788658704.221675 read 0x002b04 thru 0x006dbb (0x42b8 bytes) 000203 1788658704.222528 read 0x000249 thru 0x006dbb (0x6b73 bytes) 000204 1788658704.234661 read 0x006b9f thru 0x006dbb (0x21d bytes) 000205 1788658704.237630 mapread 0x002d37 thru 0x006dbb (0x4085 bytes) 000207 1788658704.241952 punch from 0x002736 to 0x0064bb (0x3d86 bytes) 000208 1788658704.256389 trunc from 0x006dbc to 0x012106 (0xb34b bytes) 000210 1788658704.290974 read 0x007ba3 thru 0x012106 (0xa564 bytes) 000212 1788658704.302141 mapread 0x00b11e thru 0x00ece7 (0x3bca bytes) 000213 1788658704.302996 mapread 0x00671b thru 0x011ef4 (0xb7da bytes) 000217 1788658704.306741 write 0x03cf1b thru 0x03f437 (0x251d bytes) 000218 1788658704.309453 mapread 0x01b152 thru 0x028f12 (0xddc1 bytes) 000219 1788658704.324057 write 0x018a54 thru 0x01f025 (0x65d2 bytes) 000222 1788658704.327650 trunc from 0x03f438 to 0x01f223 (0x20214 bytes) 000223 1788658704.361245 punch from 0x0106a5 to 0x017760 (0x70bc bytes) 000224 1788658704.379424 read 0x010243 thru 0x012f3c (0x2cfa bytes) 000225 1788658704.388506 write 0x0236fe thru 0x02c28f (0x8b92 bytes) 000230 1788658704.394580 trunc from 0x02c290 to 0x005d7e (0x26511 bytes) 000231 1788658704.419083 mapread 0x000f27 thru 0x005d7e (0x4e58 bytes) 000232 1788658704.420251 mapwrite 0x01f8a4 thru 0x02b656 (0xbdb3 bytes) 000233 1788658704.501078 write 0x0036f6 thru 0x010fdd (0xd8e8 bytes) 000235 1788658704.515464 punch from 0x011b3a to 0x01e04d (0xc514 bytes) 000239 1788658704.541654 write 0x030048 thru 0x036497 (0x6450 bytes) 000240 1788658704.547248 trunc from 0x036498 to 0x005cf6 (0x307a1 bytes) 000242 1788658704.574547 read 0x005597 thru 0x005cf6 (0x760 bytes) 000243 1788658704.575421 mapwrite 0x03b26f thru 0x03ffff (0x4d91 bytes) 000244 1788658704.709955 punch from 0x009b86 to 0x00d16d (0x35e8 bytes) 000246 1788658704.717880 mapwrite 0x0053c4 thru 0x0120c9 (0xcd06 bytes) 000250 1788658704.820054 read 0x0325e4 thru 0x033d20 (0x173d bytes) 000251 1788658704.822445 mapread 0x03d6db thru 0x03ffff (0x2925 bytes) 000254 1788658704.826973 trunc from 0x040000 to 0x005c94 (0x3a36b bytes) 000255 1788658704.842690 write 0x035a87 thru 0x037835 (0x1daf bytes) 000257 1788658704.844961 write 0x03e889 thru 0x03ffff (0x1777 bytes) 000259 1788658704.847481 mapread 0x02e26b thru 0x031dc3 (0x3b59 bytes) 000260 1788658704.861284 write 0x010919 thru 0x013353 (0x2a3b bytes) 000262 1788658704.865367 trunc from 0x040000 to 0x013f34 (0x2c0cb bytes) 000264 1788658704.890769 read 0x001c39 thru 0x00d81f (0xbbe7 bytes) 000265 1788658704.891611 write 0x0248fc thru 0x02d5c1 (0x8cc6 bytes) 000266 1788658704.895661 trunc from 0x02d5c2 to 0x02fa00 (0x243f bytes) 000267 1788658704.918504 mapread 0x006ac9 thru 0x006e28 (0x360 bytes) 000268 1788658704.919482 punch from 0x01ef59 to 0x02c20c (0xd2b4 bytes) 000270 1788658704.932748 write 0x01f8eb thru 0x029840 (0x9f56 bytes) 000271 1788658704.946930 read 0x01081f thru 0x0194a5 (0x8c87 bytes) 000273 1788658704.948652 mapwrite 0x031e5e thru 0x03dae3 (0xbc86 bytes) 000274 1788658705.042788 mapread 0x012d85 thru 0x01b2b3 (0x852f bytes) 000275 1788658705.043805 mapread 0x02eff9 thru 0x03c8c0 (0xd8c8 bytes) 000276 1788658705.044689 mapread 0x00896b thru 0x00a1f7 (0x188d bytes) 000278 1788658705.047704 read 0x01e030 thru 0x02981c (0xb7ed bytes) 000280 1788658705.049726 read 0x03ab7b thru 0x03dae3 (0x2f69 bytes) 000284 1788658705.053116 read 0x01d26a thru 0x02a68a (0xd421 bytes) 000285 1788658705.053990 read 0x037ed4 thru 0x03dae3 (0x5c10 bytes) 000286 1788658705.054910 mapread 0x00015d thru 0x0045e3 (0x4487 bytes) 000287 1788658705.056425 punch from 0x014a60 to 0x017154 (0x26f5 bytes) 000288 1788658705.061075 write 0x02de83 thru 0x03b486 (0xd604 bytes) 000289 1788658705.062722 write 0x01950e thru 0x01b1d8 (0x1ccb bytes) 000290 1788658705.064522 punch from 0x014eaa to 0x01ce57 (0x7fae bytes) 000292 1788658705.087861 trunc from 0x03dae4 to 0x026e05 (0x16cde bytes) 000293 1788658705.129466 write 0x00dc6b thru 0x017a4f (0x9de5 bytes) 000294 1788658705.141960 write 0x032264 thru 0x03760b (0x53a8 bytes) 000295 1788658705.143091 punch from 0x020b25 to 0x0264ce (0x59aa bytes) 000297 1788658705.163834 trunc from 0x03760c to 0x003f24 (0x336e7 bytes) 000298 1788658705.195348 mapwrite 0x003e53 thru 0x010693 (0xc841 bytes) 000300 1788658705.383007 trunc from 0x010694 to 0x0034b3 (0xd1e0 bytes) 000301 1788658705.408676 write 0x02f5e0 thru 0x035f99 (0x69ba bytes) 000302 1788658705.410099 write 0x01f0b3 thru 0x020c18 (0x1b66 bytes) 000303 1788658705.426684 read 0x00af32 thru 0x00e07b (0x314a bytes) 000308 1788658705.438116 write 0x00dce0 thru 0x00e670 (0x991 bytes) 000309 1788658705.440952 read 0x034547 thru 0x035f99 (0x1a53 bytes) 000310 1788658705.443351 punch from 0x002e95 to 0x00f2f6 (0xc462 bytes) 000311 1788658705.461672 mapread 0x021297 thru 0x023d9b (0x2b05 bytes) 000312 1788658705.462641 write 0x0373e1 thru 0x03fd7e (0x899e bytes) 000313 1788658705.474309 MIRROR_EXTEND to 2 mirrors 000315 1788658706.651186 punch from 0x013135 to 0x015896 (0x2762 bytes) 000316 1788658706.735180 read 0x0283d1 thru 0x02d520 (0x5150 bytes) 000318 1788658706.746179 trunc from 0x03fd7f to 0x03a5a4 (0x57da bytes) 000319 1788658706.766665 read 0x012a75 thru 0x01694c (0x3ed8 bytes) 000320 1788658706.792545 read 0x02204e thru 0x0307d6 (0xe789 bytes) 000321 1788658706.793653 punch from 0x02f96a to 0x03a5a4 (0xac3b bytes) 000322 1788658706.802163 read 0x038355 thru 0x039844 (0x14f0 bytes) 000324 1788658706.810920 mapread 0x01c81f thru 0x025375 (0x8b57 bytes) 000327 1788658706.813981 mapread 0x0151a7 thru 0x01ded0 (0x8d2a bytes) 000328 1788658706.815085 trunc from 0x03a5a5 to 0x0069ec (0x33bb8 bytes) 000330 1788658706.838876 mapwrite 0x036c92 thru 0x0394de (0x284d bytes) 000333 1788658706.947327 punch from 0x00ff4e to 0x01b324 (0xb3d7 bytes) 000336 1788658706.954089 punch from 0x03778b to 0x0394de (0x1d54 bytes) 000337 1788658706.967497 mapread 0x019af2 thru 0x023b32 (0xa041 bytes) 000338 1788658707.052595 mapread 0x0188a7 thru 0x0215de (0x8d38 bytes) 000339 1788658707.064088 trunc from 0x0394df to 0x02b061 (0xe47d bytes) 000340 1788658707.082199 write 0x019e0f thru 0x02179a (0x798c bytes) 000341 1788658707.083421 write 0x027813 thru 0x030d6b (0x9559 bytes) 000342 1788658707.095460 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) 000344 1788658707.290356 punch from 0x01f68d to 0x02c71e (0xd092 bytes) 000348 1788658707.304725 mapwrite 0x029805 thru 0x034ad5 (0xb2d1 bytes) 000349 1788658707.518645 mapread 0x031f23 thru 0x034ad5 (0x2bb3 bytes) 000350 1788658707.520591 write 0x038c7c thru 0x03ffff (0x7384 bytes) 000351 1788658707.521969 punch from 0x01581d to 0x01aac9 (0x52ad bytes) 000353 1788658707.558857 MIRROR_EXTEND to 2 mirrors 000355 1788658708.632985 trunc from 0x040000 to 0x02353d (0x1cac2 bytes) 000356 1788658708.765982 mapread 0x00802a thru 0x00927d (0x1254 bytes) 000359 1788658708.783147 punch from 0x01f639 to 0x02353d (0x3f05 bytes) 000362 1788658708.790555 mapwrite 0x000451 thru 0x00eb1d (0xe6cd bytes) 000363 1788658708.885670 read 0x01b13e thru 0x02353d (0x8400 bytes) 000365 1788658708.894555 punch from 0x0095cc to 0x00faec (0x6521 bytes) 000367 1788658708.900888 write 0x00eca6 thru 0x012d90 (0x40eb bytes) 000369 1788658708.930238 mapread 0x01101b thru 0x013aaf (0x2a95 bytes) 000376 1788658708.959630 MIRROR_RESYNC 2 mirrors 000377 1788658709.928762 punch from 0x01c3f6 to 0x02353d (0x7148 bytes) 000378 1788658710.020023 read 0x01b07d thru 0x023014 (0x7f98 bytes) 000379 1788658710.028305 trunc from 0x02353e to 0x024a41 (0x1504 bytes) 000380 1788658710.055351 mapread 0x012203 thru 0x013f6a (0x1d68 bytes) 000382 1788658710.071401 read 0x00790d thru 0x008404 (0xaf8 bytes) 000384 1788658710.082194 punch from 0x006f57 to 0x016e74 (0xff1e bytes) 000385 1788658710.087466 write 0x01761c thru 0x0268ab (0xf290 bytes) 000386 1788658710.089356 punch from 0x011bfd to 0x0198bf (0x7cc3 bytes) 000387 1788658710.104169 trunc from 0x0268ac to 0x023247 (0x3664 bytes) 000388 1788658710.124540 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x23248): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23248): Mirror 2: 0x66f8f584 (stale) CRC-32 checksum value for chunk [0, 0x23248): Mirror 1: 0x85280f73 Mirror 3: 0x85280f73 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23248: skipped 000389 1788658711.349530 punch from 0x01d020 to 0x021e37 (0x4e18 bytes) 000390 1788658711.453135 read 0x017b58 thru 0x023247 (0xb6f0 bytes) 000391 1788658711.464333 punch from 0x01a397 to 0x0207e7 (0x6451 bytes) 000392 1788658711.474314 mapwrite 0x013dea thru 0x01e661 (0xa878 bytes) 000395 1788658711.592810 write 0x025974 thru 0x026b3c (0x11c9 bytes) 000396 1788658711.596455 read 0x00fd16 thru 0x0146b4 (0x499f bytes) 000397 1788658711.605963 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x26b3d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26b3d): Mirror 2: 0x39c549f (stale) Mirror 3: 0x5d066c81 (stale) CRC-32 checksum value for chunk [0, 0x26b3d): Mirror 1: 0x7176e244 Mirror 4: 0x7176e244 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26b3d: skipped 000398 1788658713.134029 punch from 0x0146c7 to 0x01bece (0x7808 bytes) 000399 1788658713.208397 read 0x01e48c thru 0x026b3c (0x86b1 bytes) 000400 1788658713.216746 mapwrite 0x0040a0 thru 0x009478 (0x53d9 bytes) 000402 1788658713.285810 read 0x025b7a thru 0x026b3c (0xfc3 bytes) 000406 1788658713.289086 write 0x009418 thru 0x009dbd (0x9a6 bytes) 000407 1788658713.295493 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26b3d): Mirror 3: 0x5d066c81 (stale) Mirror 4: 0x7176e244 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26b3d): Mirror 3: 0x5d066c81 (stale) Mirror 4: 0x7176e244 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26b3d: skipped 000408 1788658713.752323 mapread 0x01adee thru 0x02365d (0x8870 bytes) 000409 1788658713.772067 mapread 0x012ec2 thru 0x01a231 (0x7370 bytes) 000410 1788658713.781046 read 0x018007 thru 0x0206f6 (0x86f0 bytes) 000411 1788658713.781719 punch from 0x0215e6 to 0x026b3c (0x5557 bytes) 000412 1788658713.790244 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x26b3d): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26b3d): Mirror 3: 0x5d066c81 (stale) Mirror 4: 0x7176e244 (stale) CRC-32 checksum value for chunk [0, 0x26b3d): Mirror 1: 0x2e75c46b Mirror 5: 0x2e75c46b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26b3d: skipped 000413 1788658715.313751 mapwrite 0x030914 thru 0x036cea (0x63d7 bytes) 000414 1788658715.626415 trunc from 0x036ceb to 0x03df42 (0x7258 bytes) 000415 1788658715.672107 trunc from 0x03df43 to 0x0169f7 (0x2754b bytes) 000416 1788658715.715029 mapread 0x0142cd thru 0x0169f7 (0x272b bytes) 000417 1788658715.732648 read 0x00f5f6 thru 0x0169f7 (0x7402 bytes) 000418 1788658715.736668 write 0x03aa6d thru 0x03b514 (0xaa8 bytes) 000419 1788658715.744110 write 0x000854 thru 0x00ebe1 (0xe38e bytes) 000420 1788658715.756140 trunc from 0x03b515 to 0x03daa7 (0x2593 bytes) 000422 1788658715.788822 punch from 0x034001 to 0x03a6ba (0x66ba bytes) 000423 1788658715.834582 write 0x002b8c thru 0x00a0fc (0x7571 bytes) 000424 1788658715.839008 write 0x02594b thru 0x0328e4 (0xcf9a bytes) 000425 1788658715.865151 punch from 0x03cd39 to 0x03daa7 (0xd6f bytes) 000426 1788658715.905777 punch from 0x00e345 to 0x0110af (0x2d6b bytes) 000428 1788658715.920615 punch from 0x01d566 to 0x01daf7 (0x592 bytes) 000430 1788658715.951261 mapwrite 0x02e8b6 thru 0x0339d6 (0x5121 bytes) 000431 1788658716.149913 punch from 0x032be8 to 0x035a49 (0x2e62 bytes) 000432 1788658716.166157 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3daa8): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3daa8): Mirror 3: 0xd61e0bdd (stale) Mirror 4: 0xcf3bf632 (stale) Mirror 5: 0xaa950e3a (stale) CRC-32 checksum value for chunk [0, 0x3daa8): Mirror 1: 0xa9aaccb9 Mirror 6: 0xa9aaccb9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3daa8: skipped 000433 1788658717.911044 write 0x00dfa4 thru 0x014140 (0x619d bytes) 000434 1788658718.055402 punch from 0x01f996 to 0x025bbc (0x6227 bytes) 000435 1788658718.097133 read 0x031109 thru 0x038036 (0x6f2e bytes) 000437 1788658718.128208 mapread 0x01db8a thru 0x02ab7e (0xcff5 bytes) 000438 1788658718.202065 mapwrite 0x02a2a0 thru 0x02ca33 (0x2794 bytes) 000439 1788658718.269870 mapread 0x007bc4 thru 0x012d5a (0xb197 bytes) 000440 1788658718.285334 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3daa8): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3daa8): Mirror 3: 0xd61e0bdd (stale) Mirror 4: 0xcf3bf632 (stale) Mirror 5: 0xaa950e3a (stale) Mirror 6: 0xa9aaccb9 (stale) CRC-32 checksum value for chunk [0, 0x3daa8): Mirror 1: 0x304ef43c Mirror 7: 0x304ef43c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3daa8: skipped 000442 1788658720.067349 mapwrite 0x0131e9 thru 0x016ce7 (0x3aff bytes) 000443 1788658720.216782 mapwrite 0x018dfe thru 0x023d1b (0xaf1e bytes) 000444 1788658720.280162 trunc from 0x03daa8 to 0x00b162 (0x32945 bytes) 000445 1788658720.315756 punch from 0x000b57 to 0x002a89 (0x1f33 bytes) 000447 1788658720.323992 mapread 0x007e1a thru 0x00b162 (0x3349 bytes) 000450 1788658720.337148 write 0x033511 thru 0x03a673 (0x7163 bytes) 000452 1788658720.340891 punch from 0x001874 to 0x00277d (0xf0a bytes) 000453 1788658720.367224 mapread 0x0267eb thru 0x02b0a0 (0x48b6 bytes) 000454 1788658720.379128 mapread 0x0242b4 thru 0x02fffd (0xbd4a bytes) 000455 1788658720.395458 trunc from 0x03a674 to 0x0154ae (0x251c5 bytes) 000456 1788658720.427987 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x154af): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x154af): Mirror 3: 0x96ad4e42 (stale) Mirror 4: 0x3b4e1664 (stale) Mirror 5: 0xff7d339c (stale) Mirror 6: 0x31875738 (stale) Mirror 7: 0x38693221 (stale) CRC-32 checksum value for chunk [0, 0x154af): Mirror 1: 0x186478b Mirror 8: 0x186478b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x154af: skipped 000457 1788658721.544041 trunc from 0x0154af to 0x02d31a (0x17e6c bytes) 000460 1788658721.629554 mapread 0x00fa8d thru 0x01b8ca (0xbe3e bytes) 000461 1788658721.656784 punch from 0x004769 to 0x01386a (0xf102 bytes) 000464 1788658721.663554 trunc from 0x02d31b to 0x0211fe (0xc11c bytes) 000465 1788658721.682028 read 0x01998e thru 0x020ef5 (0x7568 bytes) 000466 1788658721.682783 read 0x0160ee thru 0x01f310 (0x9223 bytes) 000467 1788658721.686467 MIRROR_RESYNC 7 mirrors 000468 1788658723.925089 mapread 0x01eaa2 thru 0x0211fe (0x275d bytes) 000472 1788658723.947838 trunc from 0x0211ff to 0x0147b8 (0xca46 bytes) 000473 1788658724.060375 mapwrite 0x00a0c1 thru 0x00f39e (0x52de bytes) 000474 1788658724.120720 mapwrite 0x01cb4b thru 0x01fc34 (0x30ea bytes) 000476 1788658724.223843 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1fc35): Mirror 3: 0x9ea2a10f (stale) Mirror 4: 0x9ea2a10f (stale) Mirror 5: 0x9ea2a10f (stale) Mirror 7: 0x9ea2a10f (stale) Mirror 8: 0x9ea2a10f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fc35): Mirror 3: 0x9ea2a10f (stale) Mirror 4: 0x9ea2a10f (stale) Mirror 5: 0x9ea2a10f (stale) Mirror 7: 0x9ea2a10f (stale) Mirror 8: 0x9ea2a10f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fc35: skipped 000478 1788658724.647674 read 0x01de6e thru 0x01fc34 (0x1dc7 bytes) 000480 1788658724.655939 write 0x02414a thru 0x032d3a (0xebf1 bytes) 000484 1788658724.660986 punch from 0x00dfe6 to 0x01ab47 (0xcb62 bytes) 000486 1788658724.677618 punch from 0x01daec to 0x02cfab (0xf4c0 bytes) 000488 1788658724.688606 mapread 0x01a921 thru 0x01bf61 (0x1641 bytes) 000490 1788658724.707057 trunc from 0x032d3b to 0x01c740 (0x165fa bytes) 000491 1788658724.725843 punch from 0x011185 to 0x01c740 (0xb5bc bytes) 000492 1788658724.731151 trunc from 0x01c741 to 0x0326f9 (0x15fb9 bytes) 000494 1788658724.761185 write 0x02da9a thru 0x031430 (0x3997 bytes) 000495 1788658724.775656 mapread 0x00ba7b thru 0x019ea1 (0xe427 bytes) 000497 1788658724.814989 write 0x00eb50 thru 0x017da9 (0x925a bytes) 000499 1788658724.828185 MIRROR_RESYNC 6 mirrors 000500 1788658726.643156 write 0x00a799 thru 0x01497a (0xa1e2 bytes) 000502 1788658726.766985 mapwrite 0x0176c4 thru 0x01fa80 (0x83bd bytes) 000503 1788658726.886481 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x326fa): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x326fa): Mirror 3: 0xcad6905e (stale) Mirror 4: 0xcad6905e (stale) Mirror 5: 0xcad6905e (stale) Mirror 7: 0xcad6905e (stale) Mirror 8: 0xcad6905e (stale) CRC-32 checksum value for chunk [0, 0x326fa): Mirror 1: 0x4048951c Mirror 9: 0x4048951c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x326fa: skipped 000504 1788658728.117975 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x326fa): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x326fa): Mirror 3: 0xcad6905e (stale) Mirror 4: 0xcad6905e (stale) Mirror 5: 0xcad6905e (stale) Mirror 7: 0xcad6905e (stale) Mirror 8: 0xcad6905e (stale) CRC-32 checksum value for chunk [0, 0x326fa): Mirror 1: 0x4048951c Mirror 9: 0x4048951c Mirror 10: 0x4048951c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x326fa: skipped 000505 1788658729.184345 read 0x02ab35 thru 0x032381 (0x784d bytes) 000506 1788658729.192188 trunc from 0x0326fa to 0x02039c (0x1235d bytes) 000507 1788658729.298073 write 0x02dd91 thru 0x03da98 (0xfd08 bytes) 000509 1788658729.301146 mapread 0x02fefe thru 0x0340b0 (0x41b3 bytes) 000510 1788658729.303812 trunc from 0x03da99 to 0x008953 (0x35145 bytes) 000511 1788658729.346745 write 0x03e503 thru 0x03ffff (0x1afd bytes) 000512 1788658729.347789 trunc from 0x040000 to 0x03c470 (0x3b8f bytes) 000513 1788658729.378014 write 0x031f25 thru 0x039731 (0x780d bytes) 000515 1788658729.400758 read 0x001b7f thru 0x0030f2 (0x1574 bytes) 000519 1788658729.437850 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c471): Mirror 3: 0x887e9626 (stale) Mirror 4: 0x887e9626 (stale) Mirror 5: 0x887e9626 (stale) Mirror 7: 0x887e9626 (stale) Mirror 8: 0x887e9626 (stale) Mirror 10: 0x3cb0009d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c471): Mirror 3: 0x887e9626 (stale) Mirror 4: 0x887e9626 (stale) Mirror 5: 0x887e9626 (stale) Mirror 7: 0x887e9626 (stale) Mirror 8: 0x887e9626 (stale) Mirror 10: 0x3cb0009d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c471: skipped 000520 1788658730.275845 read 0x02255f thru 0x027305 (0x4da7 bytes) 000521 1788658730.288822 write 0x021de3 thru 0x027535 (0x5753 bytes) 000522 1788658730.295862 punch from 0x007f58 to 0x00dd82 (0x5e2b bytes) 000523 1788658730.315449 read 0x029dd8 thru 0x02ec3d (0x4e66 bytes) 000524 1788658730.338640 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c471): Mirror 4: 0x887e9626 (stale) Mirror 5: 0x887e9626 (stale) Mirror 7: 0x887e9626 (stale) Mirror 8: 0x887e9626 (stale) Mirror 10: 0x3cb0009d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c471): Mirror 4: 0x887e9626 (stale) Mirror 5: 0x887e9626 (stale) Mirror 7: 0x887e9626 (stale) Mirror 8: 0x887e9626 (stale) Mirror 10: 0x3cb0009d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c471: skipped 000525 1788658731.144373 mapread 0x003f1e thru 0x0040df (0x1c2 bytes) 000528 1788658731.162972 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c471): Mirror 4: 0x887e9626 (stale) Mirror 5: 0x887e9626 (stale) Mirror 7: 0x887e9626 (stale) Mirror 10: 0x3cb0009d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c471): Mirror 4: 0x887e9626 (stale) Mirror 5: 0x887e9626 (stale) Mirror 7: 0x887e9626 (stale) Mirror 10: 0x3cb0009d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c471: skipped 000529 1788658731.657241 mapread 0x036902 thru 0x03a921 (0x4020 bytes) 000530 1788658731.675592 mapwrite 0x014727 thru 0x01f5c3 (0xae9d bytes) 000531 1788658731.753087 mapread 0x024ed0 thru 0x02668d (0x17be bytes) 000532 1788658731.769020 mapread 0x016784 thru 0x020182 (0x99ff bytes) 000533 1788658731.770014 punch from 0x00bf55 to 0x01bd16 (0xfdc2 bytes) 000534 1788658731.775710 mapread 0x01b2cb thru 0x024c1b (0x9951 bytes) 000535 1788658731.784768 read 0x024840 thru 0x02adcd (0x658e bytes) 000536 1788658731.786822 mapwrite 0x01b8df thru 0x024525 (0x8c47 bytes) 000537 1788658731.888580 write 0x024ee9 thru 0x025ba4 (0xcbc bytes) 000538 1788658731.889459 mapwrite 0x02c011 thru 0x033b48 (0x7b38 bytes) 000539 1788658731.937487 read 0x0188e5 thru 0x0226ff (0x9e1b bytes) 000541 1788658731.938628 trunc from 0x03c471 to 0x02ca19 (0xfa57 bytes) 000542 1788658731.972157 write 0x022c26 thru 0x028b94 (0x5f6f bytes) 000543 1788658731.973248 punch from 0x00cd55 to 0x019b74 (0xce20 bytes) 000544 1788658732.004897 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2ca1a): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ca1a): Mirror 4: 0x9e4a5d8b (stale) Mirror 5: 0x9e4a5d8b (stale) Mirror 7: 0x9e4a5d8b (stale) Mirror 10: 0xd020c723 (stale) CRC-32 checksum value for chunk [0, 0x2ca1a): Mirror 1: 0xde30a70a Mirror 11: 0xde30a70a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ca1a: skipped 000546 1788658733.486382 mapwrite 0x01e69e thru 0x02682c (0x818f bytes) 000548 1788658733.765632 punch from 0x009171 to 0x017332 (0xe1c2 bytes) 000550 1788658733.775907 mapwrite 0x0265ea thru 0x030ebf (0xa8d6 bytes) 000551 1788658733.946139 mapread 0x017750 thru 0x019812 (0x20c3 bytes) 000554 1788658733.972925 mapwrite 0x033e2b thru 0x039543 (0x5719 bytes) 000555 1788658734.127107 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39544): Mirror 5: 0xef2f22f5 (stale) Mirror 7: 0xef2f22f5 (stale) Mirror 10: 0x17543d7f (stale) Mirror 11: 0xe22bf71d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39544): Mirror 5: 0xef2f22f5 (stale) Mirror 7: 0xef2f22f5 (stale) Mirror 10: 0x17543d7f (stale) Mirror 11: 0xe22bf71d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39544: skipped 000557 1788658734.607989 punch from 0x030e76 to 0x0354ba (0x4645 bytes) 000560 1788658734.614547 mapread 0x01f2e3 thru 0x021484 (0x21a2 bytes) 000561 1788658734.640376 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x39544): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39544): Mirror 5: 0xef2f22f5 (stale) Mirror 7: 0xef2f22f5 (stale) Mirror 10: 0x17543d7f (stale) Mirror 11: 0xe22bf71d (stale) CRC-32 checksum value for chunk [0, 0x39544): Mirror 1: 0x8a89dfff Mirror 12: 0x8a89dfff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39544: skipped 000562 1788658735.745007 punch from 0x00cfbb to 0x010eb6 (0x3efc bytes) 000563 1788658735.855135 mapwrite 0x00bf20 thru 0x011205 (0x52e6 bytes) 000565 1788658736.052686 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x39544): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39544): Mirror 5: 0xef2f22f5 (stale) Mirror 7: 0xef2f22f5 (stale) Mirror 10: 0x17543d7f (stale) Mirror 11: 0xe22bf71d (stale) Mirror 12: 0x8a89dfff (stale) CRC-32 checksum value for chunk [0, 0x39544): Mirror 1: 0x66adef26 Mirror 13: 0x66adef26 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39544: skipped 000566 1788658737.403635 punch from 0x019bd0 to 0x021665 (0x7a96 bytes) 000568 1788658737.566456 mapread 0x021670 thru 0x02f598 (0xdf29 bytes) 000569 1788658737.592977 read 0x00be20 thru 0x019211 (0xd3f2 bytes) 000570 1788658737.603177 trunc from 0x039544 to 0x028375 (0x111ce bytes) 000571 1788658737.629838 trunc from 0x028376 to 0x0192c5 (0xf0b0 bytes) 000574 1788658737.674830 read 0x00ff6f thru 0x012969 (0x29fb bytes) 000575 1788658737.691138 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x192c6): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x192c6): Mirror 5: 0x534ac517 (stale) Mirror 7: 0x534ac517 (stale) Mirror 10: 0xe7421813 (stale) Mirror 11: 0xfeea274a (stale) Mirror 12: 0xfeea274a (stale) Mirror 13: 0x14fe4756 (stale) CRC-32 checksum value for chunk [0, 0x192c6): Mirror 1: 0x14fe4756 Mirror 14: 0x14fe4756 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x192c6: skipped 000576 1788658738.806799 mapwrite 0x03839d thru 0x03ffff (0x7c63 bytes) 000579 1788658739.030675 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x2fae6b28 (stale) Mirror 7: 0x2fae6b28 (stale) Mirror 10: 0x8aad039c (stale) Mirror 11: 0xf9dbf803 (stale) Mirror 12: 0x5d17aa83 (stale) Mirror 13: 0x16b90a64 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x2fae6b28 (stale) Mirror 7: 0x2fae6b28 (stale) Mirror 10: 0x8aad039c (stale) Mirror 11: 0xf9dbf803 (stale) Mirror 12: 0x5d17aa83 (stale) Mirror 13: 0x16b90a64 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000580 1788658739.485125 trunc from 0x040000 to 0x0114f5 (0x2eb0a bytes) 000581 1788658739.510545 mapwrite 0x001630 thru 0x0044d8 (0x2ea9 bytes) 000583 1788658739.646526 punch from 0x008030 to 0x00d6d2 (0x56a3 bytes) 000584 1788658739.653989 mapwrite 0x009a28 thru 0x019244 (0xf81d bytes) 000585 1788658739.833112 MIRROR_RESYNC 7 mirrors 000587 1788658741.109690 read 0x0126a7 thru 0x019244 (0x6b9e bytes) 000588 1788658741.119513 read 0x01258f thru 0x019244 (0x6cb6 bytes) 000590 1788658741.120068 trunc from 0x019245 to 0x0295c6 (0x10382 bytes) 000593 1788658741.210756 trunc from 0x0295c7 to 0x0352e1 (0xbd1b bytes) 000594 1788658741.235553 write 0x024e54 thru 0x026b69 (0x1d16 bytes) 000595 1788658741.267630 MIRROR_RESYNC 7 mirrors 000596 1788658743.538831 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) 000597 1788658744.033972 trunc from 0x0352e2 to 0x001cc0 (0x33621 bytes) 000599 1788658744.178508 mapwrite 0x03fcab thru 0x03ffff (0x355 bytes) 000600 1788658744.316437 mapwrite 0x032d28 thru 0x03ad95 (0x806e bytes) 000602 1788658744.423673 mapwrite 0x034607 thru 0x034c01 (0x5fb bytes) 000604 1788658744.496740 MIRROR_RESYNC 6 mirrors 000605 1788658746.113937 write 0x01d28b thru 0x02344f (0x61c5 bytes) 000606 1788658746.251174 write 0x01a786 thru 0x028332 (0xdbad bytes) 000607 1788658746.272578 punch from 0x0347c8 to 0x03ffff (0xb838 bytes) 000608 1788658746.309838 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe52c054f (stale) Mirror 7: 0xe52c054f (stale) Mirror 11: 0xe52c054f (stale) Mirror 13: 0xe52c054f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe52c054f (stale) Mirror 7: 0xe52c054f (stale) Mirror 11: 0xe52c054f (stale) Mirror 13: 0xe52c054f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000610 1788658746.859499 write 0x00f135 thru 0x01e03d (0xef09 bytes) 000613 1788658746.887469 read 0x029588 thru 0x02e91e (0x5397 bytes) 000615 1788658746.904243 mapread 0x01a0a8 thru 0x0251f2 (0xb14b bytes) 000616 1788658746.945132 trunc from 0x040000 to 0x03a1b5 (0x5e4a bytes) 000617 1788658746.961955 read 0x01ab65 thru 0x022443 (0x78df bytes) 000619 1788658746.963082 mapread 0x00bb73 thru 0x010845 (0x4cd3 bytes) 000620 1788658746.979933 punch from 0x00782e to 0x00992e (0x2101 bytes) 000621 1788658747.000073 trunc from 0x03a1b6 to 0x0391ba (0xffb bytes) 000622 1788658747.017757 mapwrite 0x025923 thru 0x02a89a (0x4f78 bytes) 000624 1788658747.115724 MIRROR_RESYNC 5 mirrors 000625 1788658749.211472 mapwrite 0x0372e4 thru 0x03ffff (0x8d1c bytes) 000627 1788658749.420456 trunc from 0x040000 to 0x0114b8 (0x2eb47 bytes) 000629 1788658749.465851 write 0x030045 thru 0x035686 (0x5642 bytes) 000630 1788658749.470980 trunc from 0x035687 to 0x00ddaf (0x278d7 bytes) 000631 1788658749.501233 punch from 0x00692d to 0x007648 (0xd1c bytes) 000632 1788658749.507247 mapwrite 0x01b5b3 thru 0x01c4a5 (0xef3 bytes) 000633 1788658749.577622 punch from 0x01435f to 0x01b894 (0x7536 bytes) 000634 1788658749.584816 punch from 0x004c13 to 0x004e9a (0x288 bytes) 000636 1788658749.593497 mapwrite 0x0164d5 thru 0x021e47 (0xb973 bytes) 000639 1788658749.691876 mapwrite 0x00eb8d thru 0x014178 (0x55ec bytes) 000640 1788658749.753460 mapread 0x0148c1 thru 0x021182 (0xc8c2 bytes) 000641 1788658749.755769 punch from 0x002cb9 to 0x0037c9 (0xb11 bytes) 000642 1788658749.763779 mapread 0x006b2b thru 0x00fa97 (0x8f6d bytes) 000643 1788658749.777108 mapwrite 0x0009d5 thru 0x00b9a9 (0xafd5 bytes) 000646 1788658749.828857 mapwrite 0x01d522 thru 0x01fd59 (0x2838 bytes) 000647 1788658749.895531 write 0x013a17 thru 0x019629 (0x5c13 bytes) 000648 1788658749.896780 punch from 0x00ba6b to 0x0170aa (0xb640 bytes) 000650 1788658749.920218 mapwrite 0x0346cd thru 0x03ae79 (0x67ad bytes) 000651 1788658750.005496 trunc from 0x03ae7a to 0x0179c3 (0x234b6 bytes) 000653 1788658750.040411 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x179c4): Mirror 7: 0xace6af27 (stale) Mirror 11: 0xace6af27 (stale) Mirror 13: 0xace6af27 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x179c4): Mirror 7: 0xace6af27 (stale) Mirror 11: 0xace6af27 (stale) Mirror 13: 0xace6af27 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x179c4: skipped 000654 1788658750.434962 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x179c4): Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x179c4): Mirror 7: 0xace6af27 (stale) Mirror 11: 0xace6af27 (stale) Mirror 13: 0xace6af27 (stale) CRC-32 checksum value for chunk [0, 0x179c4): Mirror 1: 0x2ee91122 Mirror 14: 0x2ee91122 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x179c4: skipped 000655 1788658751.657191 trunc from 0x0179c4 to 0x0339b2 (0x1bfef bytes) 000658 1788658751.734376 mapread 0x00c333 thru 0x01bb35 (0xf803 bytes) 000659 1788658751.765843 read 0x00ebdd thru 0x019b76 (0xaf9a bytes) 000661 1788658751.772537 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x339b3): Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x339b3): Mirror 7: 0xa5279a53 (stale) Mirror 11: 0xa5279a53 (stale) Mirror 13: 0xa5279a53 (stale) Mirror 14: 0xbd644940 (stale) CRC-32 checksum value for chunk [0, 0x339b3): Mirror 1: 0xbd644940 Mirror 15: 0xbd644940 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x339b3: skipped 000663 1788658753.156533 write 0x0162b7 thru 0x024893 (0xe5dd bytes) 000664 1788658753.254484 write 0x02c1cd thru 0x0322bc (0x60f0 bytes) 000666 1788658753.280603 trunc from 0x0339b3 to 0x00c331 (0x27681 bytes) 000667 1788658753.329977 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc332): Mirror 7: 0xb6b9e8c5 (stale) Mirror 11: 0xb6b9e8c5 (stale) Mirror 13: 0xb6b9e8c5 (stale) Mirror 14: 0x620a494a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc332): Mirror 7: 0xb6b9e8c5 (stale) Mirror 11: 0xb6b9e8c5 (stale) Mirror 13: 0xb6b9e8c5 (stale) Mirror 14: 0x620a494a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc332: skipped 000668 1788658753.792765 read 0x005d20 thru 0x0085da (0x28bb bytes) 000669 1788658753.797350 read 0x0020fd thru 0x00afaa (0x8eae bytes) 000670 1788658753.805612 read 0x00aab1 thru 0x00c331 (0x1881 bytes) 000671 1788658753.806731 trunc from 0x00c332 to 0x021393 (0x15062 bytes) 000672 1788658753.826673 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x21394): Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21394): Mirror 7: 0x7daa9991 (stale) Mirror 11: 0x7daa9991 (stale) Mirror 13: 0x7daa9991 (stale) Mirror 14: 0xce42bbcd (stale) CRC-32 checksum value for chunk [0, 0x21394): Mirror 1: 0x22db439e Mirror 15: 0x22db439e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21394: skipped 000674 1788658755.024547 mapwrite 0x03f719 thru 0x03ffff (0x8e7 bytes) 000675 1788658755.208043 punch from 0x03d14e to 0x03ffff (0x2eb2 bytes) 000677 1788658755.218161 mapread 0x0165d3 thru 0x01d5ab (0x6fd9 bytes) 000678 1788658755.238601 trunc from 0x040000 to 0x031ba5 (0xe45a bytes) 000679 1788658755.273177 trunc from 0x031ba6 to 0x037ad7 (0x5f32 bytes) 000682 1788658755.300627 mapwrite 0x00467e thru 0x00f92a (0xb2ad bytes) 000683 1788658755.364472 mapwrite 0x03fecb thru 0x03ffff (0x135 bytes) 000686 1788658755.453171 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x53ff19fc (stale) Mirror 11: 0x53ff19fc (stale) Mirror 13: 0x53ff19fc (stale) Mirror 14: 0xb579ace8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x53ff19fc (stale) Mirror 11: 0x53ff19fc (stale) Mirror 13: 0x53ff19fc (stale) Mirror 14: 0xb579ace8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000687 1788658755.864128 MIRROR_RESYNC 5 mirrors 000688 1788658757.198092 trunc from 0x040000 to 0x014757 (0x2b8a8 bytes) 000690 1788658757.297802 read 0x007c9e thru 0x00804f (0x3b2 bytes) 000691 1788658757.306693 mapwrite 0x034e95 thru 0x03a915 (0x5a81 bytes) 000692 1788658757.416485 trunc from 0x03a916 to 0x026fcf (0x13946 bytes) 000693 1788658757.441138 punch from 0x004337 to 0x00f969 (0xb633 bytes) 000694 1788658757.447473 mapwrite 0x0371e4 thru 0x03adea (0x3c07 bytes) 000696 1788658757.540609 read 0x02429d thru 0x034152 (0xfeb6 bytes) 000698 1788658757.551452 mapread 0x027933 thru 0x033799 (0xbe67 bytes) 000699 1788658757.552944 mapwrite 0x00ba92 thru 0x016897 (0xae06 bytes) 000700 1788658757.623498 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3adeb): Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3adeb): Mirror 7: 0x139f5ccb (stale) Mirror 11: 0x139f5ccb (stale) Mirror 13: 0x139f5ccb (stale) Mirror 14: 0x139f5ccb (stale) CRC-32 checksum value for chunk [0, 0x3adeb): Mirror 1: 0x8c0cc447 Mirror 15: 0x8c0cc447 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3adeb: skipped 000701 1788658758.586225 punch from 0x02faa7 to 0x03adea (0xb344 bytes) 000702 1788658758.669804 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3adeb): Mirror 7: 0x139f5ccb (stale) Mirror 11: 0x139f5ccb (stale) Mirror 13: 0x139f5ccb (stale) Mirror 15: 0x8c0cc447 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3adeb): Mirror 7: 0x139f5ccb (stale) Mirror 11: 0x139f5ccb (stale) Mirror 13: 0x139f5ccb (stale) Mirror 15: 0x8c0cc447 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3adeb: skipped 000703 1788658759.076593 trunc from 0x03adeb to 0x009ced (0x310fd bytes) 000704 1788658759.102043 mapread 0x007d09 thru 0x009ced (0x1fe5 bytes) 000705 1788658759.126080 trunc from 0x009cee to 0x025749 (0x1ba5c bytes) 000706 1788658759.144671 write 0x025982 thru 0x026cce (0x134d bytes) 000707 1788658759.150897 mapwrite 0x03aa32 thru 0x03ffff (0x55ce bytes) 000708 1788658759.254173 write 0x007ea7 thru 0x01207b (0xa1d5 bytes) 000710 1788658759.268443 MIRROR_RESYNC 5 mirrors 000711 1788658761.092232 write 0x032018 thru 0x0362e6 (0x42cf bytes) 000712 1788658761.180515 read 0x018e89 thru 0x01a576 (0x16ee bytes) 000714 1788658761.193623 read 0x031a33 thru 0x03a22b (0x87f9 bytes) 000715 1788658761.207793 read 0x01cbcb thru 0x02647d (0x98b3 bytes) 000716 1788658761.208650 trunc from 0x040000 to 0x003512 (0x3caed bytes) 000717 1788658761.231697 trunc from 0x003513 to 0x00c47a (0x8f68 bytes) 000720 1788658761.249134 write 0x00bb27 thru 0x00dae0 (0x1fba bytes) 000721 1788658761.258380 trunc from 0x00dae1 to 0x03e383 (0x308a3 bytes) 000722 1788658761.283702 trunc from 0x03e384 to 0x006d89 (0x375fa bytes) 000723 1788658761.307818 read 0x0014b9 thru 0x004f8c (0x3ad4 bytes) 000724 1788658761.308601 punch from 0x0060ce to 0x006d89 (0xcbc bytes) 000726 1788658761.313919 punch from 0x003f77 to 0x006d89 (0x2e13 bytes) 000728 1788658761.322886 write 0x020245 thru 0x0290e3 (0x8e9f bytes) 000730 1788658761.325710 trunc from 0x0290e4 to 0x0003f0 (0x28cf3 bytes) 000731 1788658761.348022 punch from 0x000002 to 0x0003f0 (0x3ef bytes) 000732 1788658761.353679 mapread 0x000266 thru 0x0003f0 (0x18b bytes) 000737 1788658761.362185 mapread 0x0000ec thru 0x0003f0 (0x305 bytes) 000738 1788658761.363474 punch from 0x00034b to 0x0003f0 (0xa6 bytes) 000739 1788658761.368573 mapread 0x000043 thru 0x0003f0 (0x3ae bytes) 000740 1788658761.373599 write 0x00810e thru 0x0175ce (0xf4c1 bytes) 000741 1788658761.377765 mapwrite 0x011bd0 thru 0x018b13 (0x6f44 bytes) 000743 1788658761.493518 trunc from 0x018b14 to 0x01bf32 (0x341f bytes) 000744 1788658761.532364 mapwrite 0x01adc5 thru 0x027d5b (0xcf97 bytes) 000745 1788658761.621756 trunc from 0x027d5c to 0x0026af (0x256ac bytes) 000746 1788658761.646649 mapread 0x001ec7 thru 0x0026af (0x7e9 bytes) 000749 1788658761.650560 mapwrite 0x00fd84 thru 0x01e0b5 (0xe332 bytes) 000750 1788658761.855802 mapread 0x01a5c6 thru 0x01e0b5 (0x3af0 bytes) 000753 1788658761.861406 read 0x01c991 thru 0x01e0b5 (0x1725 bytes) 000754 1788658761.868687 MIRROR_RESYNC 5 mirrors 000755 1788658763.246512 mapread 0x018344 thru 0x01e0b5 (0x5d72 bytes) 000756 1788658763.281196 punch from 0x01803a to 0x01e0b5 (0x607c bytes) 000757 1788658763.340542 mapread 0x008bee thru 0x01167e (0x8a91 bytes) 000758 1788658763.372446 trunc from 0x01e0b6 to 0x02bba1 (0xdaec bytes) 000759 1788658763.409870 read 0x028ea8 thru 0x02bba1 (0x2cfa bytes) 000761 1788658763.422525 punch from 0x01bacf to 0x023c9e (0x81d0 bytes) 000762 1788658763.427634 trunc from 0x02bba2 to 0x03a592 (0xe9f1 bytes) 000764 1788658763.442068 mapread 0x0127b3 thru 0x015a9b (0x32e9 bytes) 000765 1788658763.443111 mapread 0x0228f0 thru 0x02470f (0x1e20 bytes) 000767 1788658763.454751 mapread 0x0023d0 thru 0x004231 (0x1e62 bytes) 000770 1788658763.457370 mapread 0x004dc1 thru 0x00af76 (0x61b6 bytes) 000771 1788658763.458443 punch from 0x0324ce to 0x035b55 (0x3688 bytes) 000772 1788658763.463200 read 0x001468 thru 0x00eeb6 (0xda4f bytes) 000773 1788658763.467740 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a593): Mirror 7: 0x5ca0ad61 (stale) Mirror 13: 0x5ca0ad61 (stale) Mirror 15: 0x5ca0ad61 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a593): Mirror 7: 0x5ca0ad61 (stale) Mirror 13: 0x5ca0ad61 (stale) Mirror 15: 0x5ca0ad61 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a593: skipped 000774 1788658763.961613 punch from 0x02ae1f to 0x034358 (0x953a bytes) 000776 1788658763.981605 mapwrite 0x013eff thru 0x01f129 (0xb22b bytes) 000777 1788658764.111511 trunc from 0x03a593 to 0x02240e (0x18184 bytes) 000778 1788658764.148851 mapwrite 0x033be5 thru 0x0376da (0x3af6 bytes) 000779 1788658764.220375 punch from 0x00fab5 to 0x01e732 (0xec7e bytes) 000780 1788658764.228256 mapwrite 0x02516d thru 0x02bd67 (0x6bfb bytes) 000781 1788658764.330098 trunc from 0x0376db to 0x02e0ec (0x95ee bytes) 000782 1788658764.362051 trunc from 0x02e0ed to 0x0349f9 (0x690d bytes) 000784 1788658764.408502 mapread 0x01cc9a thru 0x0299bc (0xcd23 bytes) 000787 1788658764.425242 mapread 0x010be9 thru 0x019b13 (0x8f2b bytes) 000791 1788658764.446733 write 0x03c829 thru 0x03e3a6 (0x1b7e bytes) 000793 1788658764.456978 trunc from 0x03e3a7 to 0x02e5d8 (0xfdce bytes) 000795 1788658764.507951 MIRROR_RESYNC 4 mirrors 000796 1788658766.590144 write 0x00aad9 thru 0x012db5 (0x82dd bytes) 000798 1788658766.701730 trunc from 0x02e5d9 to 0x01a32a (0x142ae bytes) 000800 1788658766.721913 mapread 0x010e58 thru 0x01a32a (0x94d3 bytes) 000801 1788658766.741042 trunc from 0x01a32b to 0x02a0ff (0xfdd5 bytes) 000806 1788658766.761409 trunc from 0x02a100 to 0x023bbb (0x6544 bytes) 000807 1788658766.786566 mapwrite 0x03dbf2 thru 0x03ffff (0x240e bytes) 000808 1788658766.860800 read 0x030812 thru 0x03bb4b (0xb33a bytes) 000809 1788658766.877395 read 0x019d1d thru 0x022c04 (0x8ee8 bytes) 000810 1788658766.901602 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xefb47537 (stale) Mirror 13: 0xefb47537 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xefb47537 (stale) Mirror 13: 0xefb47537 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000812 1788658767.307524 MIRROR_RESYNC 3 mirrors 000816 1788658768.507321 mapread 0x026f0b thru 0x03646b (0xf561 bytes) 000817 1788658768.538191 write 0x03fa39 thru 0x03ffff (0x5c7 bytes) 000818 1788658768.611801 read 0x017cf8 thru 0x01ceb2 (0x51bb bytes) 000819 1788658768.624167 trunc from 0x040000 to 0x008c8b (0x37374 bytes) 000820 1788658768.643533 mapread 0x0080b2 thru 0x008c8b (0xbda bytes) 000821 1788658768.659793 punch from 0x00089c to 0x004212 (0x3977 bytes) 000822 1788658768.665619 trunc from 0x008c8c to 0x02abcf (0x21f44 bytes) 000823 1788658768.687980 write 0x00cbea thru 0x00fce7 (0x30fe bytes) 000825 1788658768.702996 punch from 0x01e961 to 0x023da6 (0x5446 bytes) 000826 1788658768.720839 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2abd0): Mirror 7: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2abd0): Mirror 7: 0x8d61897b (stale) Mirror 13: 0x8d61897b (stale) CRC-32 checksum value for chunk [0, 0x2abd0): Mirror 1: 0x338f783 Mirror 14: 0x338f783 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2abd0: skipped 000827 1788658769.801966 write 0x03972e thru 0x03d055 (0x3928 bytes) 000830 1788658769.893925 write 0x0270da thru 0x02e0cd (0x6ff4 bytes) 000831 1788658769.905217 punch from 0x0046a4 to 0x00a3ef (0x5d4c bytes) 000833 1788658769.919612 trunc from 0x03d056 to 0x02cf71 (0x100e4 bytes) 000834 1788658769.942664 trunc from 0x02cf72 to 0x00160f (0x2b962 bytes) 000837 1788658769.963663 punch from 0x000f5a to 0x00160f (0x6b6 bytes) 000839 1788658769.969955 write 0x02655d thru 0x02bbe7 (0x568b bytes) 000840 1788658769.971461 write 0x03d44d thru 0x03ffff (0x2bb3 bytes) 000842 1788658769.985246 read 0x00605e thru 0x0143ea (0xe38d bytes) 000843 1788658769.996774 read 0x027223 thru 0x0351b5 (0xdf93 bytes) 000844 1788658770.007046 write 0x006270 thru 0x00f0d9 (0x8e6a bytes) 000846 1788658770.010451 read 0x029126 thru 0x029ac8 (0x9a3 bytes) 000849 1788658770.012924 punch from 0x006dd7 to 0x011882 (0xaaac bytes) 000852 1788658770.034063 read 0x03fcd9 thru 0x03ffff (0x327 bytes) 000855 1788658770.036205 mapread 0x039813 thru 0x03cd07 (0x34f5 bytes) 000857 1788658770.038693 mapwrite 0x001e72 thru 0x00247d (0x60c bytes) 000859 1788658770.088966 mapwrite 0x00acec thru 0x01a34b (0xf660 bytes) 000861 1788658770.160409 punch from 0x02f4d9 to 0x02fed5 (0x9fd bytes) 000862 1788658770.166448 trunc from 0x040000 to 0x0217b3 (0x1e84c bytes) 000863 1788658770.186881 trunc from 0x0217b4 to 0x019c9c (0x7b17 bytes) 000864 1788658770.206857 write 0x000a67 thru 0x004a45 (0x3fdf bytes) 000865 1788658770.208112 punch from 0x00a657 to 0x00e2be (0x3c68 bytes) 000866 1788658770.222040 read 0x015ddd thru 0x019c9c (0x3ec0 bytes) 000867 1788658770.223149 mapread 0x0040c2 thru 0x005338 (0x1277 bytes) 000868 1788658770.224158 read 0x008103 thru 0x0131c3 (0xb0c1 bytes) 000869 1788658770.232123 mapread 0x002f5d thru 0x00c799 (0x983d bytes) 000871 1788658770.240737 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x19c9d): Mirror 7: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19c9d): Mirror 7: 0x13506bb3 (stale) Mirror 13: 0x13506bb3 (stale) Mirror 14: 0x7d681be1 (stale) CRC-32 checksum value for chunk [0, 0x19c9d): Mirror 1: 0x98d52cb Mirror 15: 0x98d52cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19c9d: skipped 000872 1788658771.208251 mapread 0x006ee9 thru 0x0129c8 (0xbae0 bytes) 000873 1788658771.253640 write 0x005b5a thru 0x012a9c (0xcf43 bytes) 000874 1788658771.358474 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19c9d): Mirror 7: 0x13506bb3 (stale) Mirror 14: 0x7d681be1 (stale) Mirror 15: 0x98d52cb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19c9d): Mirror 7: 0x13506bb3 (stale) Mirror 14: 0x7d681be1 (stale) Mirror 15: 0x98d52cb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19c9d: skipped 000875 1788658771.757228 mapread 0x007013 thru 0x00b5dc (0x45ca bytes) 000876 1788658771.774855 write 0x01c723 thru 0x0244fe (0x7ddc bytes) 000877 1788658771.776311 trunc from 0x0244ff to 0x008100 (0x1c3fe bytes) 000879 1788658771.830789 mapread 0x006a98 thru 0x008100 (0x1669 bytes) 000880 1788658771.838106 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x8101): Mirror 7: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8101): Mirror 7: 0xf62afa2f (stale) Mirror 14: 0xf62afa2f (stale) Mirror 15: 0xcec59b32 (stale) CRC-32 checksum value for chunk [0, 0x8101): Mirror 1: 0x15bded79 Mirror 16: 0x15bded79 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8101: skipped 000881 1788658772.984229 mapwrite 0x02ba0d thru 0x0364fd (0xaaf1 bytes) 000883 1788658773.272654 mapread 0x0200b3 thru 0x020dc9 (0xd17 bytes) 000885 1788658773.296534 mapread 0x01c43d thru 0x0221de (0x5da2 bytes) 000886 1788658773.298870 read 0x027952 thru 0x033f57 (0xc606 bytes) 000887 1788658773.300153 punch from 0x0179bc to 0x023668 (0xbcad bytes) 000888 1788658773.335094 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x364fe): Mirror 14: 0xb531df2a (stale) Mirror 15: 0x125468a9 (stale) Mirror 16: 0x55badccf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x364fe): Mirror 14: 0xb531df2a (stale) Mirror 15: 0x125468a9 (stale) Mirror 16: 0x55badccf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x364fe: skipped 000889 1788658773.748839 mapread 0x017ffb thru 0x01e78c (0x6792 bytes) 000890 1788658773.776513 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x364fe): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x364fe): Mirror 14: 0xb531df2a (stale) Mirror 15: 0x125468a9 (stale) Mirror 16: 0x55badccf (stale) CRC-32 checksum value for chunk [0, 0x364fe): Mirror 1: 0xabbfd9dd Mirror 17: 0xabbfd9dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x364fe: skipped 000891 1788658774.990733 mapwrite 0x01433f thru 0x0216b5 (0xd377 bytes) 000892 1788658775.253061 trunc from 0x0364fe to 0x034a66 (0x1a97 bytes) 000893 1788658775.279728 punch from 0x008c1a to 0x014c06 (0xbfed bytes) 000894 1788658775.290643 punch from 0x029e00 to 0x034a66 (0xac67 bytes) 000895 1788658775.307094 mapread 0x01bf7a thru 0x022d99 (0x6e20 bytes) 000898 1788658775.311009 trunc from 0x034a67 to 0x037a25 (0x2fbf bytes) 000899 1788658775.330039 mapwrite 0x024d5e thru 0x02b524 (0x67c7 bytes) 000900 1788658775.424686 punch from 0x00702b to 0x00be77 (0x4e4d bytes) 000901 1788658775.432113 write 0x018787 thru 0x019b90 (0x140a bytes) 000903 1788658775.435195 write 0x00436d thru 0x013441 (0xf0d5 bytes) 000904 1788658775.447867 mapread 0x012c14 thru 0x01dc6c (0xb059 bytes) 000905 1788658775.458593 mapwrite 0x033db7 thru 0x03ffff (0xc249 bytes) 000908 1788658775.632257 mapread 0x02065d thru 0x02d9a1 (0xd345 bytes) 000910 1788658775.635182 mapwrite 0x026692 thru 0x02ec9d (0x860c bytes) 000911 1788658775.743071 read 0x006caf thru 0x00e9b7 (0x7d09 bytes) 000912 1788658775.743788 mapread 0x01dd4c thru 0x01ee03 (0x10b8 bytes) 000913 1788658775.748040 mapwrite 0x03c634 thru 0x03ccbd (0x68a bytes) 000915 1788658775.875290 trunc from 0x040000 to 0x01144c (0x2ebb3 bytes) 000916 1788658775.907105 trunc from 0x01144d to 0x0222e1 (0x10e95 bytes) 000920 1788658775.934255 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x222e2): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x222e2): Mirror 14: 0xa9003241 (stale) Mirror 15: 0xf450343e (stale) Mirror 16: 0xb988c688 (stale) Mirror 17: 0xb988c688 (stale) CRC-32 checksum value for chunk [0, 0x222e2): Mirror 1: 0x7a0e1788 Mirror 18: 0x7a0e1788 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x222e2: skipped 000922 1788658777.159170 write 0x03c1d0 thru 0x03ffff (0x3e30 bytes) 000924 1788658777.257523 write 0x0106c6 thru 0x01ca0b (0xc346 bytes) 000925 1788658777.273983 mapread 0x0392c7 thru 0x03ffff (0x6d39 bytes) 000926 1788658777.293925 mapread 0x02ac66 thru 0x033b53 (0x8eee bytes) 000927 1788658777.306927 trunc from 0x040000 to 0x01f854 (0x207ab bytes) 000928 1788658777.334017 mapread 0x009423 thru 0x017756 (0xe334 bytes) 000929 1788658777.346704 mapread 0x01752f thru 0x01f854 (0x8326 bytes) 000932 1788658777.354241 write 0x02f314 thru 0x03f23e (0xff2b bytes) 000933 1788658777.361568 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f23f): Mirror 14: 0xe0a780d1 (stale) Mirror 15: 0x79bc5a74 (stale) Mirror 16: 0x83e218cd (stale) Mirror 17: 0x81f77209 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f23f): Mirror 14: 0xe0a780d1 (stale) Mirror 15: 0x79bc5a74 (stale) Mirror 16: 0x83e218cd (stale) Mirror 17: 0x81f77209 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f23f: skipped 000934 1788658777.898539 MIRROR_RESYNC 5 mirrors 000935 1788658779.700166 mapread 0x02fa8a thru 0x03ad98 (0xb30f bytes) 000936 1788658779.738909 write 0x001250 thru 0x003aa2 (0x2853 bytes) 000937 1788658779.818583 mapread 0x004331 thru 0x00a3c6 (0x6096 bytes) 000938 1788658779.819996 mapwrite 0x016c24 thru 0x0176bd (0xa9a bytes) 000941 1788658779.888741 mapread 0x02e748 thru 0x035cc5 (0x757e bytes) 000942 1788658779.922256 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f23f): Mirror 15: 0x8916c239 (stale) Mirror 16: 0x8916c239 (stale) Mirror 17: 0x8916c239 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f23f): Mirror 15: 0x8916c239 (stale) Mirror 16: 0x8916c239 (stale) Mirror 17: 0x8916c239 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f23f: skipped 000943 1788658780.619871 punch from 0x017c40 to 0x0242f9 (0xc6ba bytes) 000944 1788658780.628251 trunc from 0x03f23f to 0x037c5a (0x75e4 bytes) 000945 1788658780.656481 MIRROR_RESYNC 4 mirrors 000946 1788658782.154320 write 0x0004d0 thru 0x00f6de (0xf20f bytes) 000947 1788658782.273518 punch from 0x01a1e1 to 0x01c2d5 (0x20f5 bytes) 000948 1788658782.286552 punch from 0x027d5b to 0x033858 (0xbafe bytes) 000949 1788658782.291714 punch from 0x023e4b to 0x02ddbc (0x9f72 bytes) 000951 1788658782.298168 mapread 0x02a4e8 thru 0x02ea82 (0x459b bytes) 000952 1788658782.323186 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x37c5b): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37c5b): Mirror 15: 0xa550211 (stale) Mirror 16: 0xa550211 (stale) Mirror 17: 0xa550211 (stale) CRC-32 checksum value for chunk [0, 0x37c5b): Mirror 1: 0xa13c8361 Mirror 18: 0xa13c8361 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37c5b: skipped 000953 1788658783.923256 punch from 0x01d7e0 to 0x021775 (0x3f96 bytes) 000955 1788658784.161969 write 0x031f30 thru 0x03d5d5 (0xb6a6 bytes) 000956 1788658784.176992 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3d5d6): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d5d6): Mirror 15: 0x8d8052b (stale) Mirror 16: 0x8d8052b (stale) Mirror 17: 0x8d8052b (stale) Mirror 18: 0xb2eccb0b (stale) CRC-32 checksum value for chunk [0, 0x3d5d6): Mirror 1: 0x90e6ede1 Mirror 19: 0x90e6ede1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d5d6: skipped 000957 1788658785.573140 mapwrite 0x007831 thru 0x010dfb (0x95cb bytes) 000958 1788658785.818890 mapread 0x01a67d thru 0x01adfb (0x77f bytes) 000959 1788658785.848773 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d5d6): Mirror 15: 0x8d8052b (stale) Mirror 16: 0x8d8052b (stale) Mirror 17: 0x8d8052b (stale) Mirror 19: 0x90e6ede1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d5d6): Mirror 15: 0x8d8052b (stale) Mirror 16: 0x8d8052b (stale) Mirror 17: 0x8d8052b (stale) Mirror 19: 0x90e6ede1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d5d6: skipped 000960 1788658786.383050 write 0x015702 thru 0x02041b (0xad1a bytes) 000961 1788658786.406543 trunc from 0x03d5d6 to 0x028aae (0x14b27 bytes) 000962 1788658786.442676 read 0x004415 thru 0x00fa5f (0xb64b bytes) 000963 1788658786.461104 mapread 0x0282c4 thru 0x028aae (0x7eb bytes) 000965 1788658786.477036 mapread 0x01731d thru 0x01bba8 (0x488c bytes) 000967 1788658786.486729 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x28aaf): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28aaf): Mirror 15: 0x8d1937d8 (stale) Mirror 16: 0x8d1937d8 (stale) Mirror 17: 0x8d1937d8 (stale) Mirror 19: 0x6989203c (stale) CRC-32 checksum value for chunk [0, 0x28aaf): Mirror 1: 0xb7690295 Mirror 20: 0xb7690295 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28aaf: skipped 000970 1788658787.782149 mapread 0x01e0bf thru 0x028aae (0xa9f0 bytes) 000971 1788658787.804719 mapread 0x01a628 thru 0x02364a (0x9023 bytes) 000972 1788658787.805704 punch from 0x00b8dc to 0x00e990 (0x30b5 bytes) 000974 1788658787.886674 MIRROR_RESYNC 6 mirrors 000975 1788658789.283025 punch from 0x00ab90 to 0x00ce17 (0x2288 bytes) 000976 1788658789.389154 read 0x00336d thru 0x006176 (0x2e0a bytes) 000979 1788658789.400480 mapread 0x0263ba thru 0x028aae (0x26f5 bytes) 000980 1788658789.428610 write 0x0149cd thru 0x02123b (0xc86f bytes) 000982 1788658789.453966 punch from 0x0234fc to 0x025e05 (0x290a bytes) 000983 1788658789.500064 mapread 0x00afed thru 0x019cc8 (0xecdc bytes) 000984 1788658789.543185 write 0x020540 thru 0x02e4f6 (0xdfb7 bytes) 000985 1788658789.545068 trunc from 0x02e4f7 to 0x02101a (0xd4dc bytes) 000986 1788658789.585817 write 0x00f4db thru 0x01643e (0x6f64 bytes) 000987 1788658789.587911 trunc from 0x02101b to 0x0160fa (0xaf20 bytes) 000989 1788658789.611924 mapread 0x008175 thru 0x00e674 (0x6500 bytes) 000990 1788658789.613030 punch from 0x009de7 to 0x0160fa (0xc314 bytes) 000992 1788658789.636163 trunc from 0x0160fb to 0x0195f0 (0x34f6 bytes) 000994 1788658789.663755 write 0x00256f thru 0x0108f3 (0xe385 bytes) 000997 1788658789.688191 mapread 0x011939 thru 0x0195f0 (0x7cb8 bytes) 000998 1788658789.710476 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x195f1): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x195f1): Mirror 15: 0xe62d49a4 (stale) Mirror 16: 0xe62d49a4 (stale) Mirror 17: 0xe62d49a4 (stale) Mirror 19: 0xe62d49a4 (stale) Mirror 20: 0xe62d49a4 (stale) CRC-32 checksum value for chunk [0, 0x195f1): Mirror 1: 0xeafa5b9f Mirror 21: 0xeafa5b9f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x195f1: skipped 000999 1788658791.070475 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x195f1): Mirror 16: 0xe62d49a4 (stale) Mirror 17: 0xe62d49a4 (stale) Mirror 19: 0xe62d49a4 (stale) Mirror 20: 0xe62d49a4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x195f1): Mirror 16: 0xe62d49a4 (stale) Mirror 17: 0xe62d49a4 (stale) Mirror 19: 0xe62d49a4 (stale) Mirror 20: 0xe62d49a4 (stale) CRC-32 checksum value for chunk [0, 0x195f1): Mirror 1: 0xeafa5b9f Mirror 21: 0xeafa5b9f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x195f1: skipped 001000 1788658791.742184 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x195f1): Mirror 16: 0xe62d49a4 (stale) Mirror 17: 0xe62d49a4 (stale) Mirror 19: 0xe62d49a4 (stale) Mirror 20: 0xe62d49a4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x195f1): Mirror 16: 0xe62d49a4 (stale) Mirror 17: 0xe62d49a4 (stale) Mirror 19: 0xe62d49a4 (stale) Mirror 20: 0xe62d49a4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x195f1: skipped All operations completed A-OK! PASS 70a (117s) == sanity-flr test 71: check mirror extend parent fid ==== 21:39:57 (1788658797) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 21:40:00 (1788658800) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ee:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.327267 s, 6.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ee:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176939 s, 5.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ef:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ee:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ef:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ee:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 21:40:09 (1788658809) Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre2 Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1259168 .. Extending file size to 6035968 .. done Extending file size to 5313568 .. Extending file size to 6650336 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6980544 .. Extending file size to 7384928 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8600544 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7729056 .. Extending file size to 8614272 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8669152 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 57459 57460 57463 57464 57465 done 192.168.204.141@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg441-client.virtnet /mnt/lustre2 (opts:) 192.168.204.141@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg441-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (81s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 21:41:31 (1788658891) Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre2 Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 2794400 .. Extending file size to 5634848 .. failed Extending file size to 4282496 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5030016 .. Extending file size to 9128192 .. Extending file size to 7528096 .. Extending file size to 9388896 .. Extending file size to 8212256 .. failed Extending file size to 8653376 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8707712 .. Extending file size to 8976064 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58933 58935 58936 58937 58940 done 192.168.204.141@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg441-client.virtnet /mnt/lustre2 (opts:) 192.168.204.141@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg441-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (87s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 21:42:58 (1788658978) Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.158434 s, 13.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.423602 s, 5.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xcf3efcf7 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.141@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg441-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 21:43:18 (1788658998) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0899192 s, 23.3 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 21:43:25 (1788659005) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x215:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x305:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x305:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x306:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x321:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 21:43:34 (1788659014) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.287296 s, 7.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x307:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x322:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x323:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 21:43:44 (1788659024) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.280491 s, 7.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x309:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x309:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x325:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 21:43:54 (1788659034) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.269149 s, 7.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x326:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x326:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x327:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 21:44:28 (1788659068) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg441-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113889 s, 9.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x329:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x328:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x329:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x328:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30c:0x0] } PASS 204d (65s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 21:45:33 (1788659133) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg441-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.302477 s, 6.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (65s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 21:46:38 (1788659198) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg441-server mds-ost sync done. Creating new pool test_204f oleg441-server: Pool lustre.test_204f created Waiting 90s for '' Adding targets to pool oleg441-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg441-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.287012 s, 7.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg441-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg441-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg441-server: Pool lustre.test_204f destroyed PASS 204f (85s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 21:48:03 (1788659283) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 21:48:13 (1788659293) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 21:48:22 (1788659302) oleg441-server: Pool lustre.test_206 created Adding targets to pool oleg441-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg441-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Updated after 2s: want 'lustre-OST0000_UUID lustre-OST0001_UUID ' got 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg441-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg441-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg441-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 21:48:48 (1788659328) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119782 s, 8.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532935 s, 19.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x5bdd891 Mirror 3: 0x5bdd891 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 21:49:00 (1788659340) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 21:49:04 (1788659344) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 21:49:07 (1788659347) Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.111669 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x860000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1440000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1440000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1440000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xc30000 Read this (4 bytes): BBBB 192.168.204.141@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg441-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 21:49:46 (1788659386) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0244725 s, 335 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 21:49:55 (1788659395) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697501 s, 15.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x23a lmm_fid: [0x200000404:0x23a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x23a lmm_fid: [0x200000404:0x23a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x318:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834490 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 21:49:58.000000000 -0400 Modify: 2026-09-05 21:49:58.000000000 -0400 Change: 2026-09-05 21:49:59.000000000 -0400 Birth: 2026-09-05 21:49:58.000000000 -0400 PASS 210a (11s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 21:50:06 (1788659406) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 21:50:10 (1788659410) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.779147 s, 52.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0662065 s, 61.9 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 5 21:50 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23c:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 5 21:50 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 21:50:20 (1788659420) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 21:50:29 (1788659429) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1791 sec =========== 21:50:40 (1788659440) === sanity-flr: start cleanup 21:50:42 (1788659442) === === sanity-flr: finish cleanup 21:50:49 (1788659449) ===