-----============= acceptance-small: sanity-flr ============----- Mon Jun 1 10:26:13 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg245-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 10:26:27 (1780323987) === oleg245-client.virtnet: executing check_config_client /mnt/lustre oleg245-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg245-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff90e746ef3800.idle_timeout=debug osc.lustre-OST0001-osc-ffff90e746ef3800.idle_timeout=debug disable quota as required oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 10:26:48 (1780324008) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7590] == sanity-flr test 0a: lfs mirror create with -N option == 10:26:50 (1780324010) PASS 0a (12s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:27:02 (1780324022) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:27:04 (1780324024) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 10:27:07 (1780324027) PASS 0d (17s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:27:25 (1780324045) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:27:27 (1780324047) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 10:27:30 (1780324050) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff90e746ef3800.stats=clear osc.lustre-OST0001-osc-ffff90e746ef3800.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:27:36 (1780324056) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (7s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:27:44 (1780324064) PASS 0j (6s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:27:49 (1780324069) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 10:27:52 (1780324072) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (6s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:27:58 (1780324078) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (6s) == sanity-flr test 2: create components from existing files ========================================================== 10:28:04 (1780324084) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:28:10 (1780324090) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:28:13 (1780324093) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (5s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:28:19 (1780324099) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (5s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:28:26 (1780324106) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:28:32 (1780324112) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 1.65038 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11534 s, 9.1 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:28:43 (1780324123) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.386034 s, 43.5 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 10:28:50 (1780324130) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (8s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:28:58 (1780324138) Stopping /mnt/lustre-ost1 (opts:) on oleg245-server oleg245-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff90e746ef3800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90e746ef3800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg245-server: oleg245-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg245-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg245-server oleg245-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff90e746ef3800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90e746ef3800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg245-server: oleg245-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg245-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (51s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:29:49 (1780324189) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.266759 s, 23.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg245-server oleg245-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff90e746ef3800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90e746ef3800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg245-server: oleg245-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg245-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (31s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:30:20 (1780324220) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg245-server oleg245-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff90e746ef3800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90e746ef3800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg245-server: oleg245-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg245-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg245-server oleg245-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff90e746ef3800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90e746ef3800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg245-server: oleg245-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg245-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (58s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:31:18 (1780324278) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.62288 s, 14.5 MB/s -rw-r--r-- 1 root root 52428800 Jun 1 10:31 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.882s user 0m0.000s sys 0m0.181s Stopping /mnt/lustre-ost1 (opts:) on oleg245-server oleg245-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff90e746ef3800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90e746ef3800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.852s user 0m0.001s sys 0m0.176s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg245-server: oleg245-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg245-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg245-server oleg245-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff90e746ef3800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90e746ef3800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.800s user 0m0.001s sys 0m0.174s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg245-server: oleg245-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg245-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg245-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90e746ef3800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90e746ef3800.ost_server_uuid in FULL state after 0 sec PASS 33b (63s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:32:21 (1780324341) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:32:24 (1780324344) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:32:27 (1780324347) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 10:32:30 (1780324350) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 10:32:36 (1780324356) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.576528 s, 21.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (10s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:32:46 (1780324366) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238162 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252634 s, 41.5 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:32:53 (1780324373) PASS 36c (6s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:32:59 (1780324379) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00138716 s, 738 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000885213 s, 1.2 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0010277 s, 996 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00100537 s, 1.0 MB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 10:33:08 (1780324388) create test files with size 15752806 .. 7cec1408487b8759ee4810b648e28a56 - 2d0f56dfb73784cf7b4acb5222cc44b3 - c3e103ab964f77a4f74034486d4e2d71 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg245-client.virtnet /mnt/lustre (opts:) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 10:33:30 (1780324410) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:33:43 (1780324423) PASS 39 (5s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:33:49 (1780324429) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137037 s, 7.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305762 s, 34.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (7s) == sanity-flr test 41: lfs mirror resync check =========== 10:33:55 (1780324435) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.080821 s, 25.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.202207 s, 20.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.176746 s, 11.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (11s) == sanity-flr test 42: lfs mirror verify ================= 10:34:06 (1780324446) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 10:34:09 (1780324449) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:34:12 (1780324452) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480526 s, 21.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 44a: lfs mirror split check =========== 10:34:18 (1780324458) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:34:20 (1780324460) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (6s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:34:27 (1780324467) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.177034 s, 59.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg245-client.virtnet /mnt/lustre (opts:) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (7s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:34:34 (1780324474) llite.lustre-ffff90e74813a800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.153531 s, 68.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg245-client.virtnet /mnt/lustre (opts:) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff90e74373b800.xattr_cache=1 PASS 44e (10s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:34:45 (1780324485) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (26s) == sanity-flr test 46: Verify setstripe --copy option ==== 10:35:11 (1780324511) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (22s) == sanity-flr test 47: Verify mirror obj alloc =========== 10:35:33 (1780324533) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 10:35:35 (1780324535) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.139172 s, 22.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0991926 s, 31.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xe608b09b (nosync,stale) Mirror 2: 0xe608b09b (nosync,stale) ** verify mirror 2 doesn't change original checksum: b95064a0c6666c86a34ffc2f7044a28d - mirror 1 checksum: f4f2d1ace6804b00024cf2b9971c79bc - mirror 2 checksum: b95064a0c6666c86a34ffc2f7044a28d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Jun 1 10:35:37 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Jun 1 10:35:37 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: b95064a0c6666c86a34ffc2f7044a28d - mirror 1 checksum: f4f2d1ace6804b00024cf2b9971c79bc - mirror 2 checksum: f4f2d1ace6804b00024cf2b9971c79bc - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Jun 1 10:35:37 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Jun 1 10:35:37 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } PASS 48 (7s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:35:43 (1780324543) Failing mds1 on oleg245-server Stopping /mnt/lustre-mds1 (opts:) on oleg245-server 10:35:48 (1780324548) shut down facet: mds1 facet_host: oleg245-server facet_failover_host: oleg245-server Failover mds1 to oleg245-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 10:36:05 (1780324565) targets are mounted 10:36:05 (1780324565) facet_failover done oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185487 s, 56.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } PASS 50A (33s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 10:36:16 (1780324576) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 10:36:18 (1780324578) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:36:20 (1780324580) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00642291 s, 2.6 MB/s PASS 50c (4s) == sanity-flr test 50d: mirror rsync keep holes ========== 10:36:25 (1780324585) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:36:27 (1780324587) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:36:29 (1780324589) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (14s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:36:43 (1780324603) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (22s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 10:37:05 (1780324625) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (20s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 10:37:25 (1780324645) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 10:37:44 (1780324664) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (4s) == sanity-flr test 70: mirror create and split race ====== 10:37:48 (1780324668) mirror create pid 53474 mirror split pid 53475 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 53474 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 53475 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 10:38:51 (1780324731) Seed set to 4732 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x12133 000004 1780324732.834706 trunc from 00000000 to 0x012132 (0x12133 bytes) 000008 1780324732.848605 mapwrite 0x035060 thru 0x03ffff (0xafa0 bytes) 000009 1780324732.932715 read 0x016044 thru 0x0244c1 (0xe47e bytes) 000010 1780324732.948176 write 0x00039e thru 0x00199e (0x1601 bytes) 000012 1780324732.955777 mapwrite 0x03bffc thru 0x03ffff (0x4004 bytes) 000014 1780324732.993688 write 0x0152bc thru 0x023982 (0xe6c7 bytes) 000015 1780324733.006217 mapread 0x01db7a thru 0x023ee4 (0x636b bytes) 000021 1780324733.010501 read 0x01a583 thru 0x01eeff (0x497d bytes) 000022 1780324733.011260 read 0x0397bd thru 0x03b754 (0x1f98 bytes) 000023 1780324733.012108 trunc from 0x040000 to 0x003c68 (0x3c397 bytes) 000024 1780324733.031707 mapwrite 0x02179f thru 0x02ab52 (0x93b4 bytes) 000027 1780324733.112676 read 0x024c8b thru 0x025923 (0xc99 bytes) 000028 1780324733.113312 mapread 0x00639d thru 0x007282 (0xee6 bytes) 000032 1780324733.115653 mapread 0x018569 thru 0x01c68c (0x4124 bytes) 000033 1780324733.116618 mapwrite 0x009647 thru 0x009685 (0x3f bytes) 000035 1780324733.147317 read 0x020c9f thru 0x0229ea (0x1d4c bytes) 000037 1780324733.148603 trunc from 0x02ab53 to 0x006ffe (0x23b54 bytes) 000039 1780324733.166167 mapread 0x006bc1 thru 0x006ffe (0x43e bytes) truncating to largest ever: 0x3d2b6 000040 1780324733.167084 trunc from 0x006fff to 0x03d2b5 (0x362b7 bytes) 000043 1780324733.179971 read 0x01bbcf thru 0x021178 (0x55aa bytes) 000045 1780324733.188443 trunc from 0x03d2b6 to 0x00d9f7 (0x2f8be bytes) 000046 1780324733.199504 mapwrite 0x022667 thru 0x02d414 (0xadae bytes) 000048 1780324733.283476 mapwrite 0x03cf70 thru 0x03ffff (0x3090 bytes) 000051 1780324733.332221 trunc from 0x040000 to 0x01b078 (0x24f87 bytes) 000053 1780324733.345375 mapread 0x008680 thru 0x00e962 (0x62e3 bytes) 000055 1780324733.352147 mapread 0x0132ff thru 0x01974b (0x644d bytes) 000064 1780324733.385446 mapread 0x019b58 thru 0x01b078 (0x1521 bytes) 000066 1780324733.393343 mapread 0x01aa7b thru 0x01b078 (0x5fe bytes) 000068 1780324733.394657 mapread 0x019b84 thru 0x01b078 (0x14f5 bytes) 000069 1780324733.396841 mapwrite 0x024159 thru 0x02dd62 (0x9c0a bytes) 000071 1780324733.468075 read 0x015646 thru 0x02470f (0xf0ca bytes) 000073 1780324733.479825 mapread 0x02b783 thru 0x02c34c (0xbca bytes) 000074 1780324733.482182 mapwrite 0x03bd1c thru 0x03ffff (0x42e4 bytes) 000075 1780324733.525454 mapread 0x020f0b thru 0x029fee (0x90e4 bytes) 000077 1780324733.527420 mapread 0x00770b thru 0x016e80 (0xf776 bytes) 000079 1780324733.529310 write 0x02c46e thru 0x036184 (0x9d17 bytes) 000082 1780324733.531627 write 0x02f57f thru 0x0319a8 (0x242a bytes) 000083 1780324733.532817 write 0x013f16 thru 0x01e3ce (0xa4b9 bytes) truncating to largest ever: 0x3e3ad 000084 1780324733.533660 trunc from 0x040000 to 0x03e3ac (0x1c53 bytes) 000087 1780324733.556970 mapread 0x028ae6 thru 0x0296e6 (0xc01 bytes) 000090 1780324733.562036 MIRROR_EXTEND to 2 mirrors 000091 1780324733.908214 mapread 0x016f98 thru 0x026137 (0xf1a0 bytes) 000092 1780324733.940619 write 0x010856 thru 0x01378f (0x2f3a bytes) 000097 1780324734.034777 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e3ad): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e3ad): Mirror 2: 0x9991c642 (stale) CRC-32 checksum value for chunk [0, 0x3e3ad): Mirror 1: 0xf71d7496 Mirror 3: 0xf71d7496 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e3ad: skipped 000098 1780324734.618551 mapread 0x00c52d thru 0x00f67d (0x3151 bytes) 000099 1780324734.651809 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3e3ad): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e3ad): Mirror 2: 0x9991c642 (stale) CRC-32 checksum value for chunk [0, 0x3e3ad): Mirror 1: 0xf71d7496 Mirror 3: 0xf71d7496 Mirror 4: 0xf71d7496 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e3ad: skipped 000101 1780324735.235921 mapwrite 0x01207e thru 0x017594 (0x5517 bytes) 000102 1780324735.373046 mapwrite 0x037c09 thru 0x03851e (0x916 bytes) 000103 1780324735.431832 write 0x03e9d2 thru 0x03ffff (0x162e bytes) 000105 1780324735.433417 mapwrite 0x02598d thru 0x026230 (0x8a4 bytes) 000107 1780324735.476478 mapwrite 0x0126e6 thru 0x01532c (0x2c47 bytes) 000110 1780324735.513030 read 0x03852c thru 0x03ad12 (0x27e7 bytes) 000112 1780324735.519646 MIRROR_RESYNC 4 mirrors 000113 1780324735.992266 trunc from 0x040000 to 0x0141d7 (0x2be28 bytes) 000115 1780324736.060473 trunc from 0x0141d8 to 0x0335d3 (0x1f3fc bytes) 000116 1780324736.074278 write 0x01b384 thru 0x01c0d5 (0xd52 bytes) 000119 1780324736.081048 mapread 0x009ba8 thru 0x0105cf (0x6a28 bytes) 000121 1780324736.089190 mapwrite 0x0216ed thru 0x02dda2 (0xc6b6 bytes) 000122 1780324736.149571 trunc from 0x0335d4 to 0x01348e (0x20145 bytes) 000123 1780324736.171438 mapwrite 0x006aa5 thru 0x00ffb8 (0x9514 bytes) 000125 1780324736.205091 mapwrite 0x008087 thru 0x0105b1 (0x852b bytes) 000126 1780324736.243339 mapread 0x00df29 thru 0x00ee2a (0xf02 bytes) 000127 1780324736.244341 read 0x00db68 thru 0x00f880 (0x1d19 bytes) 000128 1780324736.245042 mapwrite 0x0208e0 thru 0x02848b (0x7bac bytes) 000133 1780324736.303600 MIRROR_RESYNC 4 mirrors 000136 1780324736.780159 read 0x010117 thru 0x018ba9 (0x8a93 bytes) 000137 1780324736.788028 mapwrite 0x03c155 thru 0x03ffff (0x3eab bytes) 000138 1780324736.930436 MIRROR_RESYNC 4 mirrors 000140 1780324737.387032 trunc from 0x040000 to 0x03d6be (0x2941 bytes) 000141 1780324737.445545 trunc from 0x03d6bf to 0x034c49 (0x8a75 bytes) 000142 1780324737.463866 write 0x035376 thru 0x03b648 (0x62d3 bytes) 000143 1780324737.470971 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3b649): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b649): Mirror 2: 0x764d9acd (stale) Mirror 3: 0x764d9acd (stale) Mirror 4: 0x764d9acd (stale) CRC-32 checksum value for chunk [0, 0x3b649): Mirror 1: 0xd9262e18 Mirror 5: 0xd9262e18 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b649: skipped 000144 1780324738.045175 write 0x02131c thru 0x0257c4 (0x44a9 bytes) 000145 1780324738.132352 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b649): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b649): Mirror 2: 0x764d9acd (stale) Mirror 3: 0x764d9acd (stale) Mirror 4: 0x764d9acd (stale) Mirror 5: 0xd9262e18 (stale) CRC-32 checksum value for chunk [0, 0x3b649): Mirror 1: 0x3f071519 Mirror 6: 0x3f071519 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b649: skipped 000146 1780324738.740252 trunc from 0x03b649 to 0x001360 (0x3a2e8 bytes) 000147 1780324738.789648 MIRROR_RESYNC 6 mirrors 000148 1780324739.486110 mapwrite 0x024702 thru 0x03399a (0xf299 bytes) 000151 1780324739.565484 mapwrite 0x0185a8 thru 0x021a3c (0x9495 bytes) 000152 1780324739.607933 trunc from 0x03399b to 0x026975 (0xd025 bytes) 000153 1780324739.635093 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x26976): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26976): Mirror 2: 0x5c889f61 (stale) Mirror 3: 0x5c889f61 (stale) Mirror 4: 0x5c889f61 (stale) Mirror 5: 0x5c889f61 (stale) Mirror 6: 0x5c889f61 (stale) CRC-32 checksum value for chunk [0, 0x26976): Mirror 1: 0x19cddf51 Mirror 7: 0x19cddf51 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26976: skipped 000157 1780324740.316099 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2e9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26976): Mirror 2: 0x5c889f61 (stale) Mirror 4: 0x5c889f61 (stale) Mirror 5: 0x5c889f61 (stale) Mirror 6: 0x5c889f61 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26976): Mirror 2: 0x5c889f61 (stale) Mirror 4: 0x5c889f61 (stale) Mirror 5: 0x5c889f61 (stale) Mirror 6: 0x5c889f61 (stale) CRC-32 checksum value for chunk [0, 0x26976): Mirror 1: 0x19cddf51 Mirror 7: 0x19cddf51 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26976: skipped 000159 1780324740.909963 write 0x00444d thru 0x0132d0 (0xee84 bytes) 000160 1780324740.956133 mapwrite 0x01753c thru 0x0210b2 (0x9b77 bytes) 000162 1780324741.016775 write 0x02998e thru 0x02cfcc (0x363f bytes) 000163 1780324741.017971 mapwrite 0x02f069 thru 0x03543a (0x63d2 bytes) 000164 1780324741.100342 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2e9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3543b): Mirror 2: 0xb137a18c (stale) Mirror 4: 0xb137a18c (stale) Mirror 5: 0xb137a18c (stale) Mirror 7: 0x310ba857 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3543b): Mirror 2: 0xb137a18c (stale) Mirror 4: 0xb137a18c (stale) Mirror 5: 0xb137a18c (stale) Mirror 7: 0x310ba857 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3543b: skipped 000165 1780324741.433976 read 0x010928 thru 0x01d4ae (0xcb87 bytes) 000166 1780324741.447230 read 0x020f0c thru 0x02fbbd (0xecb2 bytes) 000169 1780324741.464788 write 0x037cd9 thru 0x03fadc (0x7e04 bytes) 000171 1780324741.466475 mapread 0x02f8f8 thru 0x03622b (0x6934 bytes) 000172 1780324741.475159 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2e9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fadd): Mirror 2: 0x90e2e7ba (stale) Mirror 4: 0x90e2e7ba (stale) Mirror 7: 0xcde4b042 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fadd): Mirror 2: 0x90e2e7ba (stale) Mirror 4: 0x90e2e7ba (stale) Mirror 7: 0xcde4b042 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fadd: skipped 000173 1780324741.746688 write 0x03ec32 thru 0x03ee4e (0x21d bytes) 000174 1780324741.751589 write 0x027d0b thru 0x031799 (0x9a8f bytes) 000177 1780324741.767220 write 0x03affe thru 0x03df39 (0x2f3c bytes) 000179 1780324741.780085 mapread 0x024c89 thru 0x032948 (0xdcc0 bytes) 000180 1780324741.811392 read 0x03afcf thru 0x03fadc (0x4b0e bytes) 000182 1780324741.812640 trunc from 0x03fadd to 0x006305 (0x397d7 bytes) 000183 1780324741.841459 mapread 0x001129 thru 0x001f24 (0xdfc bytes) 000184 1780324741.857798 MIRROR_RESYNC 4 mirrors 000187 1780324742.614886 read 0x005533 thru 0x006305 (0xdd3 bytes) 000188 1780324742.621229 read 0x0041c1 thru 0x006305 (0x2145 bytes) 000192 1780324742.630156 trunc from 0x006306 to 0x02f2a3 (0x28f9e bytes) 000193 1780324742.679488 MIRROR_RESYNC 4 mirrors 000194 1780324743.302133 write 0x02715b thru 0x0303db (0x9281 bytes) 000195 1780324743.344317 mapwrite 0x01af38 thru 0x01f85b (0x4924 bytes) 000196 1780324743.392219 read 0x01f108 thru 0x021f76 (0x2e6f bytes) 000197 1780324743.401591 mapwrite 0x019c36 thru 0x025361 (0xb72c bytes) 000198 1780324743.458440 read 0x018e3f thru 0x024156 (0xb318 bytes) 000200 1780324743.459698 trunc from 0x0303dc to 0x0199ea (0x169f1 bytes) 000202 1780324743.478604 mapwrite 0x038435 thru 0x03cdea (0x49b6 bytes) 000204 1780324743.541443 trunc from 0x03cdeb to 0x03b8d1 (0x1519 bytes) 000205 1780324743.552085 trunc from 0x03b8d2 to 0x009b18 (0x31db9 bytes) 000208 1780324743.569591 write 0x00cde5 thru 0x011a0f (0x4c2b bytes) 000209 1780324743.575017 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2e9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11a10): Mirror 4: 0x351e01d5 (stale) Mirror 7: 0x351e01d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11a10): Mirror 4: 0x351e01d5 (stale) Mirror 7: 0x351e01d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11a10: skipped 000210 1780324743.836429 mapread 0x00d81a thru 0x011a0f (0x41f6 bytes) 000214 1780324743.876711 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2e9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11a10): Mirror 7: 0x351e01d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11a10): Mirror 7: 0x351e01d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11a10: skipped 000215 1780324744.138764 write 0x00536f thru 0x01403b (0xeccd bytes) 000220 1780324744.148778 trunc from 0x01403c to 0x01feb7 (0xbe7c bytes) 000222 1780324744.163884 write 0x0035e9 thru 0x00febf (0xc8d7 bytes) 000224 1780324744.170485 mapwrite 0x007150 thru 0x00bb99 (0x4a4a bytes) 000225 1780324744.218975 MIRROR_RESYNC 2 mirrors 000226 1780324744.720378 trunc from 0x01feb8 to 0x00c8d2 (0x135e5 bytes) 000228 1780324744.826071 MIRROR_RESYNC 2 mirrors 000231 1780324745.376240 trunc from 0x00c8d3 to 0x0082db (0x45f7 bytes) 000232 1780324745.442880 read 0x0053fb thru 0x0082db (0x2ee1 bytes) 000233 1780324745.448876 mapwrite 0x016a82 thru 0x021cc7 (0xb246 bytes) 000236 1780324745.543664 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2e9:0x0]: Invalid argument (22) 000238 1780324745.677737 mapwrite 0x016307 thru 0x02097e (0xa678 bytes) 000240 1780324745.715703 write 0x036065 thru 0x0367e2 (0x77e bytes) 000241 1780324745.716651 mapread 0x00d9cf thru 0x01a55d (0xcb8f bytes) 000243 1780324745.725216 mapread 0x01505f thru 0x01dfa3 (0x8f45 bytes) 000246 1780324745.726953 read 0x000d43 thru 0x00c9f5 (0xbcb3 bytes) 000248 1780324745.732664 mapread 0x00320c thru 0x010b00 (0xd8f5 bytes) 000249 1780324745.734145 write 0x007bcc thru 0x0124b0 (0xa8e5 bytes) 000250 1780324745.735326 mapread 0x0218a3 thru 0x02e989 (0xd0e7 bytes) 000253 1780324745.746123 read 0x026a40 thru 0x02b1d8 (0x4799 bytes) 000255 1780324745.747044 mapread 0x035e40 thru 0x0367e2 (0x9a3 bytes) 000257 1780324745.753282 mapwrite 0x035dbe thru 0x03f1dc (0x941f bytes) 000258 1780324745.854966 mapwrite 0x00a094 thru 0x01794d (0xd8ba bytes) 000259 1780324745.896794 read 0x02ebdb thru 0x03e0ba (0xf4e0 bytes) 000260 1780324745.897619 trunc from 0x03f1dd to 0x02b9af (0x1382d bytes) 000262 1780324745.924968 mapwrite 0x026280 thru 0x02ff13 (0x9c94 bytes) 000263 1780324746.009436 read 0x020f6c thru 0x02f04c (0xe0e1 bytes) 000264 1780324746.010331 mapwrite 0x014d41 thru 0x017d4f (0x300f bytes) 000265 1780324746.059280 write 0x00a0fe thru 0x00f2de (0x51e1 bytes) truncating to largest ever: 0x3f1df 000267 1780324746.069242 trunc from 0x02ff14 to 0x03f1de (0xf2cb bytes) 000269 1780324746.110033 read 0x008802 thru 0x014159 (0xb958 bytes) 000270 1780324746.116241 MIRROR_EXTEND to 2 mirrors 000271 1780324746.583896 mapread 0x0205d7 thru 0x02e3fe (0xde28 bytes) 000272 1780324746.605726 mapwrite 0x022567 thru 0x02deb1 (0xb94b bytes) 000274 1780324746.690121 trunc from 0x03f1df to 0x01ddce (0x21410 bytes) 000275 1780324746.718703 write 0x03f76c thru 0x03ffff (0x894 bytes) 000276 1780324746.719922 mapwrite 0x0210be thru 0x028558 (0x749b bytes) 000277 1780324746.777367 write 0x009b08 thru 0x00f3f7 (0x58f0 bytes) 000278 1780324746.794969 trunc from 0x040000 to 0x01fc46 (0x203b9 bytes) 000280 1780324746.808149 write 0x023db9 thru 0x025b67 (0x1daf bytes) 000282 1780324746.814697 write 0x035cfd thru 0x0370b5 (0x13b9 bytes) 000284 1780324746.816478 mapwrite 0x03d174 thru 0x03ffff (0x2e8c bytes) 000286 1780324746.886918 read 0x03bb64 thru 0x03ffff (0x449c bytes) 000287 1780324746.887717 trunc from 0x040000 to 0x03d1b9 (0x2e46 bytes) 000289 1780324746.911272 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d1ba): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d1ba): Mirror 2: 0x15e5a5f2 (stale) CRC-32 checksum value for chunk [0, 0x3d1ba): Mirror 1: 0xf7abdf94 Mirror 3: 0xf7abdf94 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d1ba: skipped 000291 1780324747.782912 mapwrite 0x032516 thru 0x03f8f1 (0xd3dc bytes) 000293 1780324747.944961 mapread 0x018525 thru 0x01e92f (0x640b bytes) 000298 1780324747.971115 trunc from 0x03f8f2 to 0x01ec8b (0x20c66 bytes) 000300 1780324747.998105 write 0x01201d thru 0x01258b (0x56f bytes) 000301 1780324747.999352 mapwrite 0x024437 thru 0x02b274 (0x6e3e bytes) 000302 1780324748.060458 write 0x00b056 thru 0x00e344 (0x32ef bytes) 000303 1780324748.066352 read 0x00c98f thru 0x01ace7 (0xe359 bytes) 000306 1780324748.068367 mapread 0x029f81 thru 0x02b274 (0x12f4 bytes) 000307 1780324748.069653 mapread 0x027b33 thru 0x02b274 (0x3742 bytes) 000309 1780324748.070951 write 0x02def2 thru 0x031d7a (0x3e89 bytes) 000313 1780324748.079926 MIRROR_RESYNC 3 mirrors 000314 1780324748.630852 trunc from 0x031d7b to 0x019ced (0x1808d bytes) 000315 1780324748.710012 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x19cee): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19cee): Mirror 2: 0x5c311d03 (stale) Mirror 3: 0x5c311d03 (stale) CRC-32 checksum value for chunk [0, 0x19cee): Mirror 1: 0x5c311d03 Mirror 4: 0x5c311d03 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19cee: skipped 000316 1780324749.281012 mapread 0x00c79b thru 0x00f50b (0x2d71 bytes) 000317 1780324749.301233 mapwrite 0x005964 thru 0x01502f (0xf6cc bytes) 000318 1780324749.460132 read 0x003591 thru 0x00a87b (0x72eb bytes) 000319 1780324749.461127 mapwrite 0x03cdef thru 0x03ffff (0x3211 bytes) 000321 1780324749.562386 mapread 0x022791 thru 0x029f35 (0x77a5 bytes) 000322 1780324749.575681 read 0x02c5c8 thru 0x034800 (0x8239 bytes) 000323 1780324749.585362 mapread 0x016e99 thru 0x0263b5 (0xf51d bytes) 000325 1780324749.587725 mapread 0x012c40 thru 0x018286 (0x5647 bytes) 000326 1780324749.589093 trunc from 0x040000 to 0x011147 (0x2eeb8 bytes) 000327 1780324749.605504 write 0x034404 thru 0x03ffff (0xbbfc bytes) 000328 1780324749.607289 trunc from 0x040000 to 0x034989 (0xb676 bytes) 000330 1780324749.626649 write 0x01b138 thru 0x02ab2f (0xf9f8 bytes) 000333 1780324749.637163 mapwrite 0x03fa75 thru 0x03ffff (0x58b bytes) 000335 1780324749.690169 mapwrite 0x012234 thru 0x01b546 (0x9313 bytes) 000336 1780324749.750896 trunc from 0x040000 to 0x00dc85 (0x3237a bytes) 000338 1780324749.765384 trunc from 0x00dc86 to 0x02b346 (0x1d6c1 bytes) 000340 1780324749.777221 write 0x0391b0 thru 0x03dd6a (0x4bbb bytes) 000341 1780324749.778570 mapwrite 0x025eb4 thru 0x035b8e (0xfcdb bytes) 000342 1780324749.927303 mapread 0x0047df thru 0x00b8fb (0x711d bytes) 000343 1780324749.928626 trunc from 0x03dd6b to 0x024ee6 (0x18e84 bytes) 000344 1780324749.940082 mapread 0x016199 thru 0x02308e (0xcef6 bytes) 000345 1780324750.003458 MIRROR_RESYNC 4 mirrors 000347 1780324750.511816 trunc from 0x024ee7 to 0x02fd38 (0xae52 bytes) 000349 1780324750.552564 mapread 0x00e2ca thru 0x0147d1 (0x6508 bytes) 000350 1780324750.573646 trunc from 0x02fd39 to 0x01e568 (0x117d0 bytes) 000351 1780324750.588575 read 0x001bb1 thru 0x0113a9 (0xf7f9 bytes) 000352 1780324750.597978 mapread 0x00d59a thru 0x01917d (0xbbe4 bytes) 000355 1780324750.605204 mapwrite 0x009950 thru 0x009ecf (0x580 bytes) 000356 1780324750.643187 trunc from 0x01e569 to 0x034777 (0x1620f bytes) 000358 1780324750.654270 mapread 0x01b0d5 thru 0x02b004 (0xff30 bytes) 000360 1780324750.665972 mapread 0x01106b thru 0x01147c (0x412 bytes) 000365 1780324750.668874 write 0x004166 thru 0x00622b (0x20c6 bytes) 000367 1780324750.669953 mapwrite 0x00d2a0 thru 0x012d45 (0x5aa6 bytes) 000368 1780324750.725879 read 0x00bf35 thru 0x013cae (0x7d7a bytes) 000371 1780324750.728016 mapread 0x031985 thru 0x034777 (0x2df3 bytes) 000373 1780324750.742321 mapwrite 0x01fce6 thru 0x02d924 (0xdc3f bytes) 000375 1780324750.786850 mapwrite 0x00bf8f thru 0x016ba2 (0xac14 bytes) 000376 1780324750.815275 read 0x014812 thru 0x01b217 (0x6a06 bytes) 000377 1780324750.815941 read 0x0131c2 thru 0x0217b8 (0xe5f7 bytes) 000380 1780324750.817135 trunc from 0x034778 to 0x009ef5 (0x2a882 bytes) 000381 1780324750.829384 mapwrite 0x005580 thru 0x00d6ef (0x8170 bytes) 000382 1780324750.904322 mapwrite 0x027da7 thru 0x037417 (0xf671 bytes) 000383 1780324750.959908 mapread 0x01be28 thru 0x01ccac (0xe85 bytes) 000384 1780324750.962477 read 0x01005e thru 0x01ebfd (0xeba0 bytes) 000385 1780324750.969759 MIRROR_RESYNC 4 mirrors 000387 1780324751.587710 mapread 0x004cf5 thru 0x008524 (0x3830 bytes) 000388 1780324751.600859 trunc from 0x037418 to 0x00bac7 (0x2b950 bytes) 000389 1780324751.660173 trunc from 0x00bac8 to 0x00fede (0x4417 bytes) 000391 1780324751.682856 mapwrite 0x0396f9 thru 0x03ffff (0x6907 bytes) 000392 1780324751.750556 trunc from 0x040000 to 0x02d48c (0x12b73 bytes) 000395 1780324751.776196 trunc from 0x02d48d to 0x0260e2 (0x73aa bytes) 000398 1780324751.795079 write 0x00fd48 thru 0x017784 (0x7a3d bytes) 000399 1780324751.808284 mapread 0x025f5a thru 0x0260e2 (0x189 bytes) 000400 1780324751.821468 read 0x0067ac thru 0x007f34 (0x1789 bytes) 000401 1780324751.827881 mapwrite 0x03dede thru 0x03ffff (0x2122 bytes) 000402 1780324751.908566 MIRROR_RESYNC 4 mirrors 000403 1780324752.431128 write 0x014ad5 thru 0x01a4a4 (0x59d0 bytes) 000404 1780324752.486155 trunc from 0x040000 to 0x024818 (0x1b7e7 bytes) 000410 1780324752.503740 trunc from 0x024819 to 0x02eb62 (0xa34a bytes) 000412 1780324752.519360 mapwrite 0x0047fe thru 0x0145d1 (0xfdd4 bytes) 000414 1780324752.580783 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2eb63): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2eb63): Mirror 2: 0x9e68f552 (stale) Mirror 3: 0x9e68f552 (stale) Mirror 4: 0x9e68f552 (stale) CRC-32 checksum value for chunk [0, 0x2eb63): Mirror 1: 0xb25f4627 Mirror 5: 0xb25f4627 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2eb63: skipped 000417 1780324753.105858 read 0x0267f4 thru 0x02eb62 (0x836f bytes) 000420 1780324753.127714 trunc from 0x02eb63 to 0x01eecf (0xfc93 bytes) 000424 1780324753.192736 write 0x030657 thru 0x03dd62 (0xd70c bytes) 000426 1780324753.196135 mapread 0x013e45 thru 0x0189b4 (0x4b70 bytes) 000427 1780324753.213988 mapwrite 0x01ee06 thru 0x02e53c (0xf737 bytes) 000428 1780324753.266193 trunc from 0x03dd63 to 0x009d08 (0x3405a bytes) 000429 1780324753.289228 trunc from 0x009d09 to 0x036f29 (0x2d221 bytes) 000432 1780324753.320131 mapread 0x00dade thru 0x01cfde (0xf501 bytes) 000434 1780324753.332902 mapread 0x02454a thru 0x028259 (0x3d10 bytes) 000437 1780324753.343779 mapwrite 0x00812b thru 0x017017 (0xeeed bytes) 000438 1780324753.371863 mapwrite 0x00a9a9 thru 0x01287c (0x7ed4 bytes) 000439 1780324753.400141 mapwrite 0x012fa6 thru 0x02212a (0xf185 bytes) 000442 1780324753.427890 read 0x028975 thru 0x035800 (0xce8c bytes) 000443 1780324753.432508 read 0x00f9c9 thru 0x011c30 (0x2268 bytes) 000444 1780324753.433065 trunc from 0x036f2a to 0x03d7ff (0x68d6 bytes) 000445 1780324753.438589 read 0x03962a thru 0x03d7ff (0x41d6 bytes) 000446 1780324753.447983 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3d800): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d800): Mirror 2: 0xa7e8f6af (stale) Mirror 3: 0xa7e8f6af (stale) Mirror 4: 0xa7e8f6af (stale) Mirror 5: 0x43e937e (stale) CRC-32 checksum value for chunk [0, 0x3d800): Mirror 1: 0xdf3628f1 Mirror 6: 0xdf3628f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d800: skipped 000447 1780324753.974052 trunc from 0x03d800 to 0x03d5b7 (0x248 bytes) 000449 1780324754.047175 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2e9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d5b8): Mirror 2: 0xfe2190b6 (stale) Mirror 3: 0xfe2190b6 (stale) Mirror 5: 0xc0218aae (stale) Mirror 6: 0x9452d866 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d5b8): Mirror 2: 0xfe2190b6 (stale) Mirror 3: 0xfe2190b6 (stale) Mirror 5: 0xc0218aae (stale) Mirror 6: 0x9452d866 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d5b8: skipped 000452 1780324754.279820 mapread 0x017b87 thru 0x023e69 (0xc2e3 bytes) 000453 1780324754.292289 mapwrite 0x03706b thru 0x03ff78 (0x8f0e bytes) 000454 1780324754.360652 read 0x00ec02 thru 0x019848 (0xac47 bytes) 000456 1780324754.370475 write 0x01b572 thru 0x02b37e (0xfe0d bytes) 000457 1780324754.375936 write 0x0097b4 thru 0x00d247 (0x3a94 bytes) 000458 1780324754.384937 read 0x026f34 thru 0x02a8f3 (0x39c0 bytes) 000460 1780324754.386258 read 0x00ed0d thru 0x012451 (0x3745 bytes) 000461 1780324754.387204 trunc from 0x03ff79 to 0x00642d (0x39b4b bytes) 000462 1780324754.400652 read 0x003aa1 thru 0x00642d (0x298d bytes) 000463 1780324754.401766 read 0x004fec thru 0x005409 (0x41e bytes) 000464 1780324754.403663 mapread 0x0033c8 thru 0x00642d (0x3066 bytes) 000465 1780324754.405444 write 0x03058d thru 0x03d50e (0xcf82 bytes) 000467 1780324754.410055 read 0x0123fb thru 0x017d4d (0x5953 bytes) 000469 1780324754.420923 write 0x011d28 thru 0x01f6f6 (0xd9cf bytes) 000475 1780324754.431569 MIRROR_RESYNC 5 mirrors 000477 1780324755.218046 read 0x024b38 thru 0x0307c6 (0xbc8f bytes) 000478 1780324755.230904 mapread 0x0016fd thru 0x00642d (0x4d31 bytes) 000480 1780324755.250739 read 0x029f43 thru 0x03890a (0xe9c8 bytes) 000483 1780324755.265189 read 0x012220 thru 0x01c08f (0x9e70 bytes) 000484 1780324755.269325 MIRROR_RESYNC 5 mirrors 000485 1780324755.699049 read 0x03ce75 thru 0x03d50e (0x69a bytes) 000487 1780324755.699619 read 0x01a600 thru 0x022756 (0x8157 bytes) 000488 1780324755.699890 mapread 0x01e14b thru 0x02b261 (0xd117 bytes) 000489 1780324755.700484 trunc from 0x03d50f to 0x014ab1 (0x28a5d bytes) 000493 1780324755.851255 read 0x003216 thru 0x00d68f (0xa47a bytes) 000494 1780324755.863436 read 0x010f20 thru 0x014ab1 (0x3b92 bytes) 000495 1780324755.864791 mapwrite 0x0071f7 thru 0x00c1f6 (0x5000 bytes) 000500 1780324755.952138 mapwrite 0x00f505 thru 0x01a593 (0xb08f bytes) 000501 1780324756.046081 mapwrite 0x024e13 thru 0x02d262 (0x8450 bytes) 000502 1780324756.132755 mapread 0x02b322 thru 0x02d262 (0x1f41 bytes) 000503 1780324756.133876 read 0x00e4fd thru 0x00f087 (0xb8b bytes) 000504 1780324756.134574 read 0x01b62f thru 0x01b95e (0x330 bytes) 000505 1780324756.135812 mapwrite 0x0244a9 thru 0x030a82 (0xc5da bytes) 000509 1780324756.194869 mapwrite 0x02a5a5 thru 0x02be54 (0x18b0 bytes) 000511 1780324756.231066 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2e9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30a83): Mirror 3: 0x5cecd501 (stale) Mirror 5: 0x5cecd501 (stale) Mirror 6: 0x5cecd501 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30a83): Mirror 3: 0x5cecd501 (stale) Mirror 5: 0x5cecd501 (stale) Mirror 6: 0x5cecd501 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30a83: skipped 000512 1780324756.467741 read 0x002284 thru 0x007585 (0x5302 bytes) 000513 1780324756.477744 write 0x01aa2a thru 0x028018 (0xd5ef bytes) 000514 1780324756.488577 read 0x00c9cd thru 0x01776f (0xada3 bytes) 000517 1780324756.497928 MIRROR_RESYNC 4 mirrors 000519 1780324757.017340 mapwrite 0x039ebb thru 0x03ffff (0x6145 bytes) 000520 1780324757.177500 read 0x03b81a thru 0x03ffff (0x47e6 bytes) 000521 1780324757.178163 write 0x019958 thru 0x01ddba (0x4463 bytes) 000523 1780324757.186999 read 0x00ec83 thru 0x01bbb9 (0xcf37 bytes) 000524 1780324757.193712 mapwrite 0x011c5f thru 0x01ed46 (0xd0e8 bytes) 000525 1780324757.224179 mapread 0x02f3ad thru 0x03ccf0 (0xd944 bytes) 000526 1780324757.225324 trunc from 0x040000 to 0x0270b2 (0x18f4d bytes) 000527 1780324757.239865 read 0x00f883 thru 0x01be6c (0xc5ea bytes) 000530 1780324757.244866 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2e9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x270b3): Mirror 5: 0xb393c231 (stale) Mirror 6: 0xb393c231 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x270b3): Mirror 5: 0xb393c231 (stale) Mirror 6: 0xb393c231 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x270b3: skipped 000532 1780324757.475474 write 0x033f96 thru 0x037226 (0x3291 bytes) 000534 1780324757.477566 mapwrite 0x01448c thru 0x023928 (0xf49d bytes) 000536 1780324757.537969 mapwrite 0x038634 thru 0x03a8d6 (0x22a3 bytes) 000537 1780324757.593921 write 0x03397b thru 0x03f579 (0xbbff bytes) 000540 1780324757.598172 mapread 0x03764a thru 0x03f579 (0x7f30 bytes) 000541 1780324757.599938 write 0x03d5ed thru 0x03ffff (0x2a13 bytes) 000542 1780324757.601166 mapread 0x0040b4 thru 0x0077f9 (0x3746 bytes) 000543 1780324757.615483 mapwrite 0x03cf2e thru 0x03d1c5 (0x298 bytes) 000544 1780324757.644722 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2e9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x33391875 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x33391875 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000545 1780324757.839655 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2e9:0x0]: Invalid argument (22) 000550 1780324757.942614 write 0x0379c4 thru 0x037b1a (0x157 bytes) 000551 1780324757.948143 read 0x003258 thru 0x009409 (0x61b2 bytes) 000555 1780324757.962404 mapwrite 0x01b252 thru 0x021ab6 (0x6865 bytes) 000556 1780324758.019315 mapread 0x007d4d thru 0x00ff64 (0x8218 bytes) 000564 1780324758.039416 trunc from 0x040000 to 0x00f126 (0x30ed9 bytes) 000566 1780324758.061651 read 0x00b82b thru 0x00c081 (0x857 bytes) 000567 1780324758.062188 trunc from 0x00f127 to 0x03e08c (0x2ef66 bytes) 000568 1780324758.078242 MIRROR_EXTEND to 2 mirrors 000569 1780324758.488400 MIRROR_EXTEND to 3 mirrors 000570 1780324758.870813 write 0x029cd1 thru 0x02ad95 (0x10c5 bytes) 000573 1780324758.911068 mapwrite 0x00d2ef thru 0x0183aa (0xb0bc bytes) 000574 1780324758.950886 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3e08d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e08d): Mirror 2: 0xb135a22d (stale) Mirror 3: 0xb135a22d (stale) CRC-32 checksum value for chunk [0, 0x3e08d): Mirror 1: 0x5f250380 Mirror 4: 0x5f250380 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e08d: skipped 000575 1780324759.474702 write 0x02b567 thru 0x038684 (0xd11e bytes) 000577 1780324759.552061 write 0x01fdea thru 0x02886c (0x8a83 bytes) 000578 1780324759.561536 write 0x0135c3 thru 0x017f94 (0x49d2 bytes) 000579 1780324759.568026 read 0x02f325 thru 0x031d75 (0x2a51 bytes) 000580 1780324759.568591 read 0x02afb5 thru 0x02f787 (0x47d3 bytes) 000581 1780324759.580217 MIRROR_RESYNC 4 mirrors 000582 1780324760.133063 write 0x00cb43 thru 0x00e607 (0x1ac5 bytes) 000583 1780324760.166820 mapwrite 0x022862 thru 0x0273b5 (0x4b54 bytes) 000585 1780324760.210967 trunc from 0x03e08d to 0x0337a9 (0xa8e3 bytes) 000586 1780324760.244154 mapwrite 0x0034e2 thru 0x012734 (0xf253 bytes) 000587 1780324760.301398 trunc from 0x0337aa to 0x0373c0 (0x3c17 bytes) 000588 1780324760.315831 read 0x009dfe thru 0x00fdf4 (0x5ff7 bytes) 000589 1780324760.316844 read 0x036369 thru 0x0373c0 (0x1058 bytes) 000590 1780324760.326178 trunc from 0x0373c1 to 0x03c6b7 (0x52f7 bytes) 000593 1780324760.346615 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3c6b8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c6b8): Mirror 2: 0x3b5cc694 (stale) Mirror 3: 0x3b5cc694 (stale) Mirror 4: 0x3b5cc694 (stale) CRC-32 checksum value for chunk [0, 0x3c6b8): Mirror 1: 0x7fe3eaa7 Mirror 5: 0x7fe3eaa7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c6b8: skipped 000594 1780324760.905987 mapwrite 0x01a528 thru 0x01d92a (0x3403 bytes) 000595 1780324761.051016 mapread 0x01474d thru 0x022048 (0xd8fc bytes) 000596 1780324761.052451 write 0x028fdd thru 0x02d7eb (0x480f bytes) 000597 1780324761.063141 mapwrite 0x034c78 thru 0x03b777 (0x6b00 bytes) 000598 1780324761.110425 mapwrite 0x031457 thru 0x037b84 (0x672e bytes) 000600 1780324761.154297 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3c6b8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c6b8): Mirror 2: 0x3b5cc694 (stale) Mirror 3: 0x3b5cc694 (stale) Mirror 4: 0x3b5cc694 (stale) Mirror 5: 0x7fe3eaa7 (stale) CRC-32 checksum value for chunk [0, 0x3c6b8): Mirror 1: 0x24adacf8 Mirror 6: 0x24adacf8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c6b8: skipped 000602 1780324761.785899 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2e9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c6b8): Mirror 2: 0x3b5cc694 (stale) Mirror 3: 0x3b5cc694 (stale) Mirror 5: 0x7fe3eaa7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c6b8): Mirror 2: 0x3b5cc694 (stale) Mirror 3: 0x3b5cc694 (stale) Mirror 5: 0x7fe3eaa7 (stale) CRC-32 checksum value for chunk [0, 0x3c6b8): Mirror 1: 0x24adacf8 Mirror 6: 0x24adacf8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c6b8: skipped 000607 1780324762.272425 read 0x003fad thru 0x00f768 (0xb7bc bytes) 000609 1780324762.292201 read 0x004f7b thru 0x009d60 (0x4de6 bytes) 000610 1780324762.300032 trunc from 0x03c6b8 to 0x0115fc (0x2b0bb bytes) 000613 1780324762.391763 mapread 0x00cc8c thru 0x0115fc (0x4971 bytes) 000615 1780324762.401315 read 0x00d90a thru 0x0115fc (0x3cf3 bytes) 000616 1780324762.402096 mapwrite 0x001f5e thru 0x004b51 (0x2bf4 bytes) 000618 1780324762.438541 mapread 0x00ea9f thru 0x0115fc (0x2b5e bytes) 000619 1780324762.439571 read 0x003167 thru 0x00ca1b (0x98b5 bytes) 000620 1780324762.442922 MIRROR_RESYNC 5 mirrors 000622 1780324763.155973 mapwrite 0x030cde thru 0x038969 (0x7c8c bytes) 000627 1780324763.339139 write 0x01bc35 thru 0x02b123 (0xf4ef bytes) 000628 1780324763.347463 write 0x034337 thru 0x0399e0 (0x56aa bytes) 000630 1780324763.348810 mapread 0x0187c9 thru 0x01fc89 (0x74c1 bytes) 000632 1780324763.355493 MIRROR_RESYNC 5 mirrors 000633 1780324764.079432 mapwrite 0x0249d3 thru 0x0259b2 (0xfe0 bytes) 000634 1780324764.199210 MIRROR_RESYNC 5 mirrors 000636 1780324764.841396 trunc from 0x0399e1 to 0x02fb35 (0x9eab bytes) 000637 1780324764.888278 mapread 0x00ffd5 thru 0x016b5c (0x6b88 bytes) 000639 1780324764.910092 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2e9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fb36): Mirror 2: 0x1406c3e3 (stale) Mirror 5: 0x1406c3e3 (stale) Mirror 6: 0x1406c3e3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fb36): Mirror 2: 0x1406c3e3 (stale) Mirror 5: 0x1406c3e3 (stale) Mirror 6: 0x1406c3e3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fb36: skipped 000640 1780324765.179160 write 0x01dd56 thru 0x023380 (0x562b bytes) 000641 1780324765.191409 write 0x02fe38 thru 0x03e790 (0xe959 bytes) 000642 1780324765.198471 write 0x016279 thru 0x0227af (0xc537 bytes) 000643 1780324765.212197 MIRROR_RESYNC 4 mirrors 000644 1780324765.871038 trunc from 0x03e791 to 0x00ff8b (0x2e805 bytes) 000645 1780324765.932114 trunc from 0x00ff8c to 0x0235ad (0x13622 bytes) 000646 1780324765.950035 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2e9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x235ae): Mirror 5: 0x3b026050 (stale) Mirror 6: 0x3b026050 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x235ae): Mirror 5: 0x3b026050 (stale) Mirror 6: 0x3b026050 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x235ae: skipped 000647 1780324766.185747 write 0x0163c0 thru 0x017a97 (0x16d8 bytes) 000650 1780324766.195506 read 0x015f7b thru 0x0235ad (0xd633 bytes) 000653 1780324766.205313 mapread 0x0081f6 thru 0x00b419 (0x3224 bytes) 000656 1780324766.214934 mapread 0x0234c3 thru 0x0235ad (0xeb bytes) 000659 1780324766.217352 mapwrite 0x018006 thru 0x026c23 (0xec1e bytes) 000660 1780324766.293686 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x26c24): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26c24): Mirror 5: 0xd6ac5302 (stale) Mirror 6: 0xd6ac5302 (stale) CRC-32 checksum value for chunk [0, 0x26c24): Mirror 1: 0x7023eff7 Mirror 7: 0x7023eff7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26c24: skipped 000662 1780324766.832433 mapwrite 0x01896b thru 0x01b2c5 (0x295b bytes) 000668 1780324766.974760 mapread 0x01cbd0 thru 0x0248de (0x7d0f bytes) 000671 1780324766.987329 mapread 0x0248fb thru 0x026c23 (0x2329 bytes) 000674 1780324766.989939 trunc from 0x026c24 to 0x010e91 (0x15d92 bytes) 000675 1780324767.014772 mapwrite 0x02d994 thru 0x0333cf (0x5a3c bytes) 000678 1780324767.079762 read 0x02b51b thru 0x0324a9 (0x6f8f bytes) 000681 1780324767.085125 MIRROR_RESYNC 4 mirrors 000683 1780324767.673611 mapread 0x013de0 thru 0x021212 (0xd433 bytes) 000684 1780324767.689865 write 0x01a42b thru 0x020a97 (0x666d bytes) 000689 1780324767.788285 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x333d0): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x333d0): Mirror 5: 0x81393919 (stale) Mirror 6: 0x81393919 (stale) Mirror 7: 0x81393919 (stale) CRC-32 checksum value for chunk [0, 0x333d0): Mirror 1: 0x63b8c7b6 Mirror 8: 0x63b8c7b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x333d0: skipped 000690 1780324768.432109 read 0x02446b thru 0x032752 (0xe2e8 bytes) 000691 1780324768.462994 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2e9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x333d0): Mirror 5: 0x81393919 (stale) Mirror 6: 0x81393919 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x333d0): Mirror 5: 0x81393919 (stale) Mirror 6: 0x81393919 (stale) CRC-32 checksum value for chunk [0, 0x333d0): Mirror 1: 0x63b8c7b6 Mirror 8: 0x63b8c7b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x333d0: skipped 000692 1780324769.034387 read 0x00500a thru 0x00d3aa (0x83a1 bytes) 000694 1780324769.064230 read 0x00072b thru 0x001268 (0xb3e bytes) 000695 1780324769.082955 write 0x02c8fe thru 0x0326b7 (0x5dba bytes) 000696 1780324769.150862 mapwrite 0x01673f thru 0x01dd34 (0x75f6 bytes) 000697 1780324769.240579 mapwrite 0x01c773 thru 0x0232b2 (0x6b40 bytes) 000698 1780324769.317310 mapwrite 0x0135d1 thru 0x01a378 (0x6da8 bytes) 000699 1780324769.364065 read 0x01f5cc thru 0x029564 (0x9f99 bytes) 000700 1780324769.377178 mapread 0x02e1f3 thru 0x0333cf (0x51dd bytes) 000702 1780324769.381293 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2e9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x333d0): Mirror 5: 0x81393919 (stale) Mirror 6: 0x81393919 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x333d0): Mirror 5: 0x81393919 (stale) Mirror 6: 0x81393919 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x333d0: skipped 000703 1780324769.574151 mapread 0x01a09b thru 0x01a97e (0x8e4 bytes) 000704 1780324769.579083 mapwrite 0x000fbd thru 0x00662d (0x5671 bytes) 000705 1780324769.627132 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2e9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x333d0): Mirror 6: 0x81393919 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x333d0): Mirror 6: 0x81393919 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x333d0: skipped 000707 1780324769.800920 write 0x030695 thru 0x0331da (0x2b46 bytes) 000711 1780324769.809900 mapwrite 0x013e60 thru 0x0141df (0x380 bytes) 000712 1780324769.837723 trunc from 0x0333d0 to 0x01e83f (0x14b90 bytes) 000714 1780324769.848575 mapread 0x0157cb thru 0x015a81 (0x2b7 bytes) 000717 1780324769.851088 read 0x0060db thru 0x00c92b (0x6851 bytes) 000720 1780324769.863007 read 0x00d89e thru 0x01bb80 (0xe2e3 bytes) 000721 1780324769.863798 mapwrite 0x031702 thru 0x034790 (0x308f bytes) 000722 1780324769.923962 mapwrite 0x027e0a thru 0x02e6cf (0x68c6 bytes) 000723 1780324769.961941 mapread 0x00f450 thru 0x01100e (0x1bbf bytes) 000727 1780324769.973260 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2e9:0x0]: Invalid argument (22) 000731 1780324770.084282 mapread 0x01fc81 thru 0x029374 (0x96f4 bytes) 000733 1780324770.100725 trunc from 0x034791 to 0x02788a (0xcf06 bytes) 000734 1780324770.118182 read 0x00fd52 thru 0x0182a7 (0x8556 bytes) 000738 1780324770.133621 write 0x023bcb thru 0x02549e (0x18d4 bytes) 000740 1780324770.135655 mapread 0x018060 thru 0x01f6ab (0x764c bytes) 000741 1780324770.142178 write 0x022d2c thru 0x029fcf (0x72a4 bytes) 000742 1780324770.143429 read 0x0133fc thru 0x013521 (0x126 bytes) 000743 1780324770.144177 read 0x007d61 thru 0x01382f (0xbacf bytes) 000745 1780324770.151343 mapread 0x012f2b thru 0x020ea9 (0xdf7f bytes) 000746 1780324770.153109 read 0x0102bb thru 0x010531 (0x277 bytes) 000747 1780324770.153931 write 0x007cd3 thru 0x007dd3 (0x101 bytes) 000749 1780324770.155933 write 0x02f574 thru 0x030796 (0x1223 bytes) 000751 1780324770.158228 mapwrite 0x02cb89 thru 0x02d127 (0x59f bytes) 000752 1780324770.208441 write 0x0281ca thru 0x034e31 (0xcc68 bytes) 000753 1780324770.209775 trunc from 0x034e32 to 0x002bfc (0x32235 bytes) 000759 1780324770.232005 trunc from 0x002bfd to 0x004d38 (0x213c bytes) 000760 1780324770.257405 mapwrite 0x016e2b thru 0x017fa7 (0x117d bytes) 000761 1780324770.320668 trunc from 0x017fa8 to 0x02603d (0xe096 bytes) 000762 1780324770.334821 trunc from 0x02603e to 0x023f03 (0x213a bytes) 000763 1780324770.349182 mapwrite 0x0145f2 thru 0x01c732 (0x8141 bytes) 000764 1780324770.400995 trunc from 0x023f04 to 0x01bb8d (0x8376 bytes) 000766 1780324770.412234 mapwrite 0x003438 thru 0x0048b9 (0x1482 bytes) 000767 1780324770.462579 mapread 0x010ffe thru 0x01bb8d (0xab90 bytes) 000769 1780324770.477324 mapread 0x00ad4f thru 0x00fb05 (0x4db7 bytes) 000770 1780324770.503184 mapread 0x014c63 thru 0x01bb8d (0x6f2b bytes) 000772 1780324770.504266 trunc from 0x01bb8e to 0x03b48d (0x1f900 bytes) 000773 1780324770.519013 read 0x002671 thru 0x0080ea (0x5a7a bytes) 000774 1780324770.526761 mapread 0x031f4a thru 0x03b48d (0x9544 bytes) 000777 1780324770.528663 trunc from 0x03b48e to 0x001bbf (0x398ce bytes) 000778 1780324770.539229 write 0x012097 thru 0x013139 (0x10a3 bytes) 000780 1780324770.541294 read 0x003412 thru 0x012e5b (0xfa4a bytes) 000781 1780324770.546124 trunc from 0x01313a to 0x019ab9 (0x6980 bytes) 000782 1780324770.556086 trunc from 0x019aba to 0x0348f6 (0x1ae3d bytes) 000783 1780324770.570914 write 0x02a76a thru 0x0311c2 (0x6a59 bytes) 000786 1780324770.581644 write 0x01e4f5 thru 0x024446 (0x5f52 bytes) 000787 1780324770.582982 mapread 0x02816c thru 0x02ca89 (0x491e bytes) 000793 1780324770.587379 write 0x0020cb thru 0x00aaee (0x8a24 bytes) 000794 1780324770.589370 mapwrite 0x037a07 thru 0x03cf47 (0x5541 bytes) 000795 1780324770.667259 read 0x02c6bd thru 0x035bfc (0x9540 bytes) 000796 1780324770.667840 write 0x001d10 thru 0x00fee0 (0xe1d1 bytes) 000797 1780324770.668827 mapread 0x011660 thru 0x01713a (0x5adb bytes) 000798 1780324770.672572 write 0x02bae2 thru 0x0384b9 (0xc9d8 bytes) 000799 1780324770.673877 read 0x014ab1 thru 0x018969 (0x3eb9 bytes) 000800 1780324770.675658 mapwrite 0x0066ff thru 0x009c2e (0x3530 bytes) 000801 1780324770.719038 mapwrite 0x01264b thru 0x01f5ef (0xcfa5 bytes) 000802 1780324770.755424 mapwrite 0x00b413 thru 0x0122a0 (0x6e8e bytes) 000805 1780324770.789742 mapread 0x00aaf6 thru 0x00ab2e (0x39 bytes) 000806 1780324770.790593 write 0x03eebd thru 0x03ffff (0x1143 bytes) 000807 1780324770.791396 trunc from 0x040000 to 0x0284da (0x17b25 bytes) 000809 1780324770.802101 write 0x0322d4 thru 0x03bafe (0x982b bytes) 000810 1780324770.803223 read 0x027fb8 thru 0x031e5f (0x9ea8 bytes) 000813 1780324770.810815 read 0x036ae5 thru 0x03ae70 (0x438c bytes) 000814 1780324770.811416 mapwrite 0x019d58 thru 0x023577 (0x9820 bytes) 000815 1780324770.850061 write 0x03e26b thru 0x03ffff (0x1d95 bytes) 000817 1780324770.852051 mapread 0x0148da thru 0x01d539 (0x8c60 bytes) 000818 1780324770.853108 read 0x01cec6 thru 0x01e005 (0x1140 bytes) 000820 1780324770.861303 write 0x02b900 thru 0x03780d (0xbf0e bytes) 000823 1780324770.864125 mapread 0x039dd8 thru 0x03b5f9 (0x1822 bytes) 000824 1780324770.865237 write 0x022e70 thru 0x031c05 (0xed96 bytes) 000825 1780324770.866492 mapread 0x01dc45 thru 0x01f6a4 (0x1a60 bytes) 000827 1780324770.868449 read 0x02586a thru 0x033583 (0xdd1a bytes) 000828 1780324770.871015 mapwrite 0x035549 thru 0x039f25 (0x49dd bytes) 000829 1780324770.896902 write 0x036129 thru 0x03665f (0x537 bytes) 000830 1780324770.897873 mapwrite 0x025f29 thru 0x02bbb6 (0x5c8e bytes) 000831 1780324770.929272 mapwrite 0x037377 thru 0x039f0e (0x2b98 bytes) 000832 1780324770.953118 write 0x001a43 thru 0x00bdc4 (0xa382 bytes) 000833 1780324770.954057 mapwrite 0x033e40 thru 0x03ffff (0xc1c0 bytes) 000834 1780324770.986695 read 0x030e1d thru 0x038eed (0x80d1 bytes) 000835 1780324770.987194 write 0x004dd7 thru 0x011895 (0xcabf bytes) 000837 1780324770.988443 read 0x037e20 thru 0x03ffff (0x81e0 bytes) 000838 1780324770.989016 write 0x01e5cc thru 0x029cc8 (0xb6fd bytes) 000842 1780324770.991187 read 0x014b18 thru 0x02189e (0xcd87 bytes) 000847 1780324770.993447 trunc from 0x040000 to 0x01a00c (0x25ff3 bytes) 000849 1780324771.008870 mapwrite 0x032605 thru 0x03655e (0x3f5a bytes) 000851 1780324771.077531 write 0x02a0be thru 0x030686 (0x65c9 bytes) 000855 1780324771.083922 MIRROR_EXTEND to 2 mirrors 000856 1780324771.454239 write 0x0390c0 thru 0x03c9d8 (0x3919 bytes) 000857 1780324771.483929 read 0x02260b thru 0x02aeab (0x88a1 bytes) 000858 1780324771.489459 mapread 0x0262eb thru 0x030181 (0x9e97 bytes) 000863 1780324771.505702 trunc from 0x03c9d9 to 0x02745d (0x1557b bytes) 000864 1780324771.531185 MIRROR_RESYNC 2 mirrors 000865 1780324771.849118 mapread 0x00f526 thru 0x015b1c (0x65f7 bytes) 000866 1780324771.869733 write 0x001fcf thru 0x010d1b (0xed4d bytes) 000870 1780324771.918713 read 0x010818 thru 0x01b743 (0xaf2c bytes) 000872 1780324771.925474 mapwrite 0x00aad2 thru 0x01103c (0x656b bytes) 000873 1780324771.950600 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2e9:0x0]: Invalid argument (22) 000875 1780324772.041959 trunc from 0x02745e to 0x039774 (0x12317 bytes) 000877 1780324772.061898 mapwrite 0x027359 thru 0x029e09 (0x2ab1 bytes) 000881 1780324772.109726 mapread 0x004394 thru 0x00e622 (0xa28f bytes) 000883 1780324772.120029 read 0x02218d thru 0x027ee6 (0x5d5a bytes) 000884 1780324772.121454 mapwrite 0x01e209 thru 0x02dea6 (0xfc9e bytes) 000887 1780324772.154197 mapread 0x03078d thru 0x039774 (0x8fe8 bytes) 000889 1780324772.161492 read 0x014438 thru 0x01f3b2 (0xaf7b bytes) 000890 1780324772.177572 mapwrite 0x03f69f thru 0x03ffff (0x961 bytes) 000891 1780324772.210996 read 0x01c5d4 thru 0x020fc7 (0x49f4 bytes) 000892 1780324772.214018 MIRROR_EXTEND to 2 mirrors 000893 1780324772.539952 mapread 0x007ee4 thru 0x0128b9 (0xa9d6 bytes) 000896 1780324772.566681 MIRROR_RESYNC 2 mirrors 000898 1780324772.751228 MIRROR_RESYNC 2 mirrors 000899 1780324772.936290 write 0x03b736 thru 0x03ee48 (0x3713 bytes) 000901 1780324772.979769 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2e9:0x0]: Invalid argument (22) 000905 1780324773.069135 read 0x00e109 thru 0x017043 (0x8f3b bytes) 000909 1780324773.082458 read 0x00bba0 thru 0x015976 (0x9dd7 bytes) 000910 1780324773.096759 MIRROR_EXTEND to 2 mirrors 000911 1780324773.432028 mapwrite 0x000146 thru 0x007e8c (0x7d47 bytes) 000913 1780324773.572252 trunc from 0x040000 to 0x00c324 (0x33cdb bytes) 000914 1780324773.592291 mapwrite 0x003b91 thru 0x00c34f (0x87bf bytes) 000917 1780324773.637168 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xc350): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc350): Mirror 2: 0x8397250 (stale) CRC-32 checksum value for chunk [0, 0xc350): Mirror 1: 0xba70d491 Mirror 3: 0xba70d491 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc350: skipped 000918 1780324774.124305 mapread 0x006d14 thru 0x00740e (0x6fb bytes) 000920 1780324774.146220 trunc from 0x00c350 to 0x024029 (0x17cda bytes) 000921 1780324774.242967 write 0x000108 thru 0x00451d (0x4416 bytes) 000922 1780324774.253979 mapread 0x017521 thru 0x01c0c6 (0x4ba6 bytes) 000923 1780324774.265474 trunc from 0x02402a to 0x018023 (0xc006 bytes) 000924 1780324774.286710 mapwrite 0x0056ba thru 0x013aec (0xe433 bytes) 000927 1780324774.350419 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x18024): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18024): Mirror 2: 0x57e97e9a (stale) Mirror 3: 0xb9e1c3de (stale) CRC-32 checksum value for chunk [0, 0x18024): Mirror 1: 0x6f845043 Mirror 4: 0x6f845043 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18024: skipped 000931 1780324774.913459 mapwrite 0x037c14 thru 0x03ffff (0x83ec bytes) 000932 1780324775.102775 mapread 0x0042f0 thru 0x011e93 (0xdba4 bytes) 000933 1780324775.123395 write 0x033fd2 thru 0x034b0d (0xb3c bytes) 000934 1780324775.124190 mapwrite 0x02e517 thru 0x0326d5 (0x41bf bytes) 000935 1780324775.201971 trunc from 0x040000 to 0x02483a (0x1b7c5 bytes) 000937 1780324775.218323 mapread 0x0207e3 thru 0x02483a (0x4058 bytes) 000938 1780324775.228774 read 0x013018 thru 0x01e2c9 (0xb2b2 bytes) 000940 1780324775.229878 trunc from 0x02483b to 0x039169 (0x1492f bytes) 000941 1780324775.239485 mapwrite 0x0211af thru 0x026a25 (0x5877 bytes) 000942 1780324775.282195 MIRROR_RESYNC 4 mirrors 000943 1780324775.961986 read 0x038374 thru 0x039169 (0xdf6 bytes) 000944 1780324775.966028 trunc from 0x03916a to 0x02f15b (0xa00e bytes) 000946 1780324776.045813 trunc from 0x02f15c to 0x02cd16 (0x2445 bytes) 000947 1780324776.060986 read 0x01ac6e thru 0x021c19 (0x6fac bytes) 000949 1780324776.071704 write 0x0268fe thru 0x02b867 (0x4f6a bytes) 000950 1780324776.075043 trunc from 0x02cd17 to 0x01cf85 (0xfd91 bytes) 000951 1780324776.096410 mapread 0x012357 thru 0x01973e (0x73e8 bytes) 000953 1780324776.107927 MIRROR_RESYNC 4 mirrors 000955 1780324776.535144 trunc from 0x01cf86 to 0x00142b (0x1bb5a bytes) 000956 1780324776.634762 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x142c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x142c): Mirror 2: 0x479a05cf (stale) Mirror 3: 0x479a05cf (stale) Mirror 4: 0x479a05cf (stale) CRC-32 checksum value for chunk [0, 0x142c): Mirror 1: 0x479a05cf Mirror 5: 0x479a05cf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x142c: skipped 000958 1780324777.051176 mapwrite 0x033407 thru 0x03eaec (0xb6e6 bytes) 000959 1780324777.175217 mapread 0x022112 thru 0x02a182 (0x8071 bytes) 000965 1780324777.189437 write 0x0117e1 thru 0x0143a4 (0x2bc4 bytes) 000966 1780324777.198617 trunc from 0x03eaed to 0x038ccd (0x5e1f bytes) 000967 1780324777.218050 mapread 0x035ca8 thru 0x038ccd (0x3026 bytes) 000968 1780324777.222233 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x38cce): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38cce): Mirror 2: 0x663598a7 (stale) Mirror 3: 0x663598a7 (stale) Mirror 4: 0x663598a7 (stale) Mirror 5: 0xe1674591 (stale) CRC-32 checksum value for chunk [0, 0x38cce): Mirror 1: 0x3445ab4e Mirror 6: 0x3445ab4e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38cce: skipped 000970 1780324777.727155 read 0x014bcb thru 0x020acb (0xbf01 bytes) 000972 1780324777.739079 write 0x03af03 thru 0x03c3f0 (0x14ee bytes) 000974 1780324777.795561 MIRROR_RESYNC 6 mirrors 000975 1780324778.481745 mapread 0x0130e9 thru 0x0164db (0x33f3 bytes) 000977 1780324778.509095 mapread 0x001d87 thru 0x00753d (0x57b7 bytes) 000978 1780324778.516961 mapread 0x015df6 thru 0x024232 (0xe43d bytes) 000980 1780324778.524608 trunc from 0x03c3f1 to 0x03c59d (0x1ad bytes) 000981 1780324778.594670 mapread 0x00db51 thru 0x01c0f7 (0xe5a7 bytes) 000982 1780324778.609694 mapwrite 0x03f49d thru 0x03ffff (0xb63 bytes) 000985 1780324778.659276 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x24b50867 (stale) Mirror 3: 0x24b50867 (stale) Mirror 4: 0x24b50867 (stale) Mirror 5: 0x24b50867 (stale) Mirror 6: 0x24b50867 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x268bddf6 Mirror 7: 0x268bddf6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000986 1780324779.200411 mapread 0x026844 thru 0x0285a0 (0x1d5d bytes) 000987 1780324779.216931 write 0x019f1f thru 0x0244a3 (0xa585 bytes) 000990 1780324779.263180 read 0x02e44b thru 0x03ce8b (0xea41 bytes) 000991 1780324779.272989 MIRROR_RESYNC 7 mirrors 000993 1780324779.956818 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2e9:0x0]: Invalid argument (22) 000996 1780324780.255701 trunc from 0x040000 to 0x01a23c (0x25dc3 bytes) 000997 1780324780.327587 read 0x00b73a thru 0x00f747 (0x400e bytes) 000998 1780324780.338181 mapread 0x013f8a thru 0x01a23c (0x62b3 bytes) All operations completed A-OK! PASS 70a (52s) == sanity-flr test 71: check mirror extend parent fid ==== 10:39:43 (1780324783) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 72: check append on FLR file ========== 10:39:45 (1780324785) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x491:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4ad:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.112648 s, 18.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x491:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x492:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4ad:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105568 s, 9.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x491:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x492:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4ae:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4ad:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x491:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x492:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4ae:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4ad:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x493:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (4s) == sanity-flr test 200a: stress test ===================== 10:39:49 (1780324789) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre2 Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7454368 .. Extending file size to 7401728 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8807872 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8564800 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8770592 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8909248 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9140512 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Waiting 57426 57427 57428 57430 57433 192.168.202.145@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg245-client.virtnet /mnt/lustre2 (opts:) 192.168.202.145@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg245-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (69s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 10:40:58 (1780324858) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre2 Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5315264 .. Extending file size to 6380736 .. Extending file size to 6852800 .. Extending file size to 6108704 .. Extending file size to 7514752 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7539936 .. Extending file size to 8209920 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9193216 .. Extending file size to 8743104 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9013024 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59239 59240 59241 59244 59245 failed 192.168.202.145@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg245-client.virtnet /mnt/lustre2 (opts:) 192.168.202.145@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg245-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (67s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 10:42:05 (1780324925) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2944 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2944 2205440 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.112939 s, 18.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.172267 s, 12.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xab763df4 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.145@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg245-client.virtnet /mnt/lustre2 (opts:) PASS 200c (9s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 10:42:15 (1780324935) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0427747 s, 49.0 MB/s PASS 202 (4s) == sanity-flr test 203: mirror file preserve mirror ID === 10:42:19 (1780324939) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x329:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c4:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c4:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c5:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x4a8:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 10:42:22 (1780324942) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.127045 s, 16.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4a9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4aa:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (4s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 10:42:26 (1780324946) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.105284 s, 19.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4ab:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4ab:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4ac:0x0] } PASS 204b (5s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 10:42:31 (1780324951) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.143024 s, 14.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4ad:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4c9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4ad:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4ae:0x0] } PASS 204c (27s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 10:42:58 (1780324978) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg245-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997404 s, 10.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4ca:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4b0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4af:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4ca:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4b0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4af:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4cb:0x0] } PASS 204d (49s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 10:43:47 (1780325027) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg245-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4cd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.127676 s, 16.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4cd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4b1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4cd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4b1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 10:44:34 (1780325074) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg245-server mds-ost sync done. Creating new pool test_204f oleg245-server: Pool lustre.test_204f created Adding targets to pool oleg245-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg245-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0807903 s, 26.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4b2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4b2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg245-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg245-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg245-server: Pool lustre.test_204f destroyed PASS 204f (60s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 10:45:33 (1780325133) PASS 205a (4s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 10:45:37 (1780325137) PASS 205b (4s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 10:45:41 (1780325141) oleg245-server: Pool lustre.test_206 created Adding targets to pool oleg245-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg245-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg245-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg245-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg245-server: Pool lustre.test_206 destroyed PASS 206 (16s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 10:45:57 (1780325157) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280735 s, 37.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172066 s, 60.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x4b71a59a Mirror 3: 0x4b71a59a PASS 207 (5s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 10:46:02 (1780325162) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 10:46:04 (1780325164) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 10:46:06 (1780325166) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0351632 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2390000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xe90000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xe90000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xe90000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1230000 Read this (4 bytes): BBBB 192.168.202.145@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg245-client.virtnet /mnt/lustre2 (opts:) PASS 209a (20s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 10:46:26 (1780325186) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00625895 s, 1.3 MB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 10:46:29 (1780325189) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292393 s, 35.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x34e lmm_fid: [0x200000404:0x34e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x34e lmm_fid: [0x200000404:0x34e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4bd:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834766 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-01 10:46:29.000000000 -0400 Modify: 2026-06-01 10:46:29.000000000 -0400 Change: 2026-06-01 10:46:30.000000000 -0400 Birth: 2026-06-01 10:46:30.000000000 -0400 PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 10:46:33 (1780325193) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 10:46:34 (1780325194) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.18796 s, 218 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0202075 s, 203 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 1 10:46 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x350:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 1 10:46 /mnt/lustre/f211.sanity-flr PASS 211 (4s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 10:46:38 (1780325198) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (3s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 10:46:41 (1780325201) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (4s) == sanity-flr test complete, duration 1231 sec =========== 10:46:45 (1780325205) === sanity-flr: start cleanup 10:46:46 (1780325206) === === sanity-flr: finish cleanup 10:46:49 (1780325209) ===