-----============= acceptance-small: sanity-flr ============----- Tue Jun 2 02:11:35 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg150-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 02:11:55 (1780380715) === oleg150-client.virtnet: executing check_config_client /mnt/lustre oleg150-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg150-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff976282c40800.idle_timeout=debug osc.lustre-OST0001-osc-ffff976282c40800.idle_timeout=debug disable quota as required oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 02:12:29 (1780380749) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7579] == sanity-flr test 0a: lfs mirror create with -N option == 02:12:32 (1780380752) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 02:12:51 (1780380771) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 02:12:55 (1780380775) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 02:12:59 (1780380779) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 02:13:24 (1780380804) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 02:13:27 (1780380807) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 02:13:31 (1780380811) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff976282c40800.stats=clear osc.lustre-OST0001-osc-ffff976282c40800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 02:13:40 (1780380820) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 02:13:52 (1780380832) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 02:14:01 (1780380841) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 02:14:05 (1780380845) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 02:14:12 (1780380852) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 02:14:21 (1780380861) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 02:14:28 (1780380868) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 02:14:31 (1780380871) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 02:14:39 (1780380879) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 02:14:49 (1780380889) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 02:14:55 (1780380895) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 1.04804 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949926 s, 11.0 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 02:15:06 (1780380906) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.14923 s, 42.2 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 02:15:14 (1780380914) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 02:15:25 (1780380925) Stopping /mnt/lustre-ost1 (opts:) on oleg150-server oleg150-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff976282c40800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff976282c40800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg150-server: oleg150-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg150-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg150-server oleg150-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff976282c40800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff976282c40800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg150-server: oleg150-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg150-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (69s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 02:16:34 (1780380994) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.72698 s, 15.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg150-server oleg150-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff976282c40800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff976282c40800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg150-server: oleg150-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg150-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (41s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 02:17:15 (1780381035) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg150-server oleg150-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff976282c40800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff976282c40800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg150-server: oleg150-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg150-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg150-server oleg150-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff976282c40800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff976282c40800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg150-server: oleg150-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg150-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (79s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 02:18:34 (1780381114) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 6.12217 s, 8.6 MB/s -rw-r--r-- 1 root root 52428800 Jun 2 02:18 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.252s user 0m0.001s sys 0m0.171s Stopping /mnt/lustre-ost1 (opts:) on oleg150-server oleg150-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff976282c40800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff976282c40800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m2.096s user 0m0.001s sys 0m0.191s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg150-server: oleg150-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg150-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg150-server oleg150-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff976282c40800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff976282c40800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.226s user 0m0.001s sys 0m0.156s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg150-server: oleg150-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg150-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg150-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff976282c40800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff976282c40800.ost_server_uuid in FULL state after 0 sec PASS 33b (105s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 02:20:20 (1780381220) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 02:20:23 (1780381223) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 02:20:27 (1780381227) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 02:20:31 (1780381231) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 02:20:39 (1780381239) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.403917 s, 31.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 02:20:51 (1780381251) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355025 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048129 s, 21.8 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 02:20:59 (1780381259) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 02:21:07 (1780381267) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00105801 s, 968 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00213996 s, 479 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00114046 s, 898 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00121829 s, 841 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 02:21:18 (1780381278) create test files with size 15760891 .. 99ab784204f8189e8a00aca5c268bd48 - 199eecbbddf44ac0df6096d63bec074d - 7abbf998bc89c5a8bc72178db01e1ce0 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg150-client.virtnet /mnt/lustre (opts:) Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 02:21:40 (1780381300) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 02:21:53 (1780381313) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 02:21:59 (1780381319) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121306 s, 8.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xb8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xb9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xd6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238551 s, 44.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xd7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xd8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 02:22:09 (1780381329) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.187454 s, 11.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.419044 s, 10.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.156136 s, 13.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 02:22:24 (1780381344) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 02:22:27 (1780381347) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 02:22:31 (1780381351) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417645 s, 25.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 44a: lfs mirror split check =========== 02:22:41 (1780381361) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 02:22:45 (1780381365) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc1:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc1:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (12s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 02:22:58 (1780381378) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.244414 s, 42.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg150-client.virtnet /mnt/lustre (opts:) Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 02:23:12 (1780381392) llite.lustre-ffff976282a0e800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.334435 s, 31.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg150-client.virtnet /mnt/lustre (opts:) Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff976288bd8000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 02:23:27 (1780381407) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (40s) == sanity-flr test 46: Verify setstripe --copy option ==== 02:24:07 (1780381447) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (39s) == sanity-flr test 47: Verify mirror obj alloc =========== 02:24:47 (1780381487) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 02:24:50 (1780381490) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.212938 s, 14.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.129769 s, 24.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x5b2dd517 (nosync,stale) Mirror 2: 0x5b2dd517 (nosync,stale) ** verify mirror 2 doesn't change original checksum: fb7914054b236bbde35c34241c0b1bc5 - mirror 1 checksum: 725c3725c208d5fd6af1affcc3bfa6dd - mirror 2 checksum: fb7914054b236bbde35c34241c0b1bc5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Jun 2 02:24:52 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Jun 2 02:24:52 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf6:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: fb7914054b236bbde35c34241c0b1bc5 - mirror 1 checksum: 725c3725c208d5fd6af1affcc3bfa6dd - mirror 2 checksum: 725c3725c208d5fd6af1affcc3bfa6dd - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Jun 2 02:24:52 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Jun 2 02:24:52 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf6:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 02:25:02 (1780381502) Failing mds1 on oleg150-server Stopping /mnt/lustre-mds1 (opts:) on oleg150-server 02:25:09 (1780381509) shut down facet: mds1 facet_host: oleg150-server facet_failover_host: oleg150-server Failover mds1 to oleg150-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-MDT0000 02:25:32 (1780381532) targets are mounted 02:25:32 (1780381532) facet_failover done oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306537 s, 34.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf7:0x0] } PASS 50A (47s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 02:25:50 (1780381550) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 02:25:54 (1780381554) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 02:25:57 (1780381557) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00655295 s, 2.5 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 02:26:06 (1780381566) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (5s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 02:26:11 (1780381571) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 02:26:14 (1780381574) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 02:26:39 (1780381599) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 02:27:08 (1780381628) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 02:27:32 (1780381652) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 02:27:57 (1780381677) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 02:28:05 (1780381685) mirror create pid 53547 mirror split pid 53548 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7730: 53548 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 53547 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 02:29:12 (1780381752) Seed set to 1753 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1780381753.892811 mapwrite 0x00cbbd thru 0x012eb8 (0x62fc bytes) 000007 1780381754.005319 write 0x03e5bb thru 0x03ffff (0x1a45 bytes) 000008 1780381754.006705 mapread 0x031013 thru 0x03fe78 (0xee66 bytes) truncating to largest ever: 0x279d4 000009 1780381754.026724 trunc from 0x040000 to 0x0279d3 (0x1862c bytes) 000010 1780381754.044354 read 0x0059c3 thru 0x00e182 (0x87c0 bytes) 000011 1780381754.049770 trunc from 0x0279d4 to 0x01d080 (0xa953 bytes) 000012 1780381754.075213 mapwrite 0x012ed0 thru 0x01590a (0x2a3b bytes) 000013 1780381754.144540 MIRROR_EXTEND to 2 mirrors 000016 1780381754.587720 trunc from 0x01d081 to 0x023fb9 (0x6f39 bytes) 000017 1780381754.668218 mapwrite 0x02b39c thru 0x033993 (0x85f8 bytes) 000018 1780381754.748323 trunc from 0x033994 to 0x018bf2 (0x1ada1 bytes) 000022 1780381754.772060 mapread 0x0098e8 thru 0x00b3c8 (0x1ae1 bytes) 000023 1780381754.779917 read 0x000478 thru 0x002b87 (0x2710 bytes) truncating to largest ever: 0x2f922 000024 1780381754.784506 trunc from 0x018bf3 to 0x02f921 (0x16d2f bytes) 000025 1780381754.801617 read 0x0218f0 thru 0x0251ca (0x38db bytes) 000027 1780381754.818212 mapwrite 0x014621 thru 0x0153c5 (0xda5 bytes) 000029 1780381754.856353 write 0x0374f9 thru 0x03ffff (0x8b07 bytes) 000030 1780381754.857718 trunc from 0x040000 to 0x019b94 (0x2646b bytes) 000035 1780381754.871949 mapread 0x01548e thru 0x019b94 (0x4707 bytes) 000038 1780381754.873653 write 0x01db0a thru 0x02bdb0 (0xe2a7 bytes) 000039 1780381754.875190 trunc from 0x02bdb1 to 0x0252cd (0x6ae3 bytes) 000041 1780381754.897256 MIRROR_RESYNC 2 mirrors 000042 1780381755.469172 mapread 0x02262c thru 0x0232a3 (0xc78 bytes) 000043 1780381755.479772 mapread 0x012687 thru 0x01b2c6 (0x8c40 bytes) 000046 1780381755.497733 read 0x011fc3 thru 0x0217d7 (0xf815 bytes) truncating to largest ever: 0x331ec 000047 1780381755.499126 trunc from 0x0252ce to 0x0331eb (0xdf1e bytes) 000048 1780381755.569904 read 0x00e52b thru 0x0143f0 (0x5ec6 bytes) 000049 1780381755.580338 trunc from 0x0331ec to 0x023955 (0xf896 bytes) 000050 1780381755.604161 MIRROR_RESYNC 2 mirrors 000051 1780381756.125982 read 0x01e3be thru 0x01f9c6 (0x1609 bytes) 000052 1780381756.131298 write 0x03e9c9 thru 0x03ffff (0x1637 bytes) 000053 1780381756.185243 write 0x01fbb0 thru 0x026b36 (0x6f87 bytes) 000054 1780381756.195300 mapread 0x037e20 thru 0x03ffff (0x81e0 bytes) 000056 1780381756.209121 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x197:0x0]: Invalid argument (22) 000057 1780381756.354849 trunc from 0x040000 to 0x007900 (0x386ff bytes) 000058 1780381756.395605 mapwrite 0x017704 thru 0x025519 (0xde16 bytes) 000059 1780381756.484937 mapwrite 0x038a06 thru 0x03cac6 (0x40c1 bytes) 000060 1780381756.542343 read 0x01ca15 thru 0x022e7d (0x6469 bytes) 000062 1780381756.547741 MIRROR_EXTEND to 2 mirrors 000063 1780381757.062138 mapwrite 0x012f40 thru 0x01bd45 (0x8e06 bytes) 000065 1780381757.328373 write 0x00a8d4 thru 0x00cfd1 (0x26fe bytes) 000066 1780381757.337361 mapread 0x034fb3 thru 0x03cac6 (0x7b14 bytes) 000068 1780381757.365917 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x197:0x0]: Invalid argument (22) 000069 1780381757.525316 read 0x02675f thru 0x02a845 (0x40e7 bytes) 000070 1780381757.534958 read 0x008081 thru 0x01243d (0xa3bd bytes) 000071 1780381757.545577 mapwrite 0x01e0e5 thru 0x029cba (0xbbd6 bytes) truncating to largest ever: 0x394bb 000072 1780381757.611054 trunc from 0x03cac7 to 0x0394ba (0x360c bytes) 000076 1780381757.657481 mapwrite 0x03e2e9 thru 0x03ffff (0x1d17 bytes) 000077 1780381757.719552 read 0x002b5a thru 0x007362 (0x4809 bytes) 000079 1780381757.721192 read 0x003af7 thru 0x00b202 (0x770c bytes) 000081 1780381757.723483 write 0x011120 thru 0x01eb62 (0xda43 bytes) 000083 1780381757.725962 mapwrite 0x01f2a3 thru 0x026346 (0x70a4 bytes) 000085 1780381757.772080 trunc from 0x040000 to 0x0166e0 (0x2991f bytes) 000087 1780381757.800713 MIRROR_EXTEND to 2 mirrors 000092 1780381758.327307 mapwrite 0x014f70 thru 0x01d292 (0x8323 bytes) 000093 1780381758.536153 read 0x00feb3 thru 0x018797 (0x88e5 bytes) 000094 1780381758.538355 mapread 0x01a992 thru 0x01d292 (0x2901 bytes) 000095 1780381758.542874 trunc from 0x01d293 to 0x001c98 (0x1b5fa bytes) 000096 1780381758.580516 trunc from 0x001c99 to 0x037c3a (0x35fa2 bytes) 000099 1780381758.604595 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37c3b): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37c3b): Mirror 2: 0xbbb37aed (stale) CRC-32 checksum value for chunk [0, 0x37c3b): Mirror 1: 0x14f8d5b0 Mirror 3: 0x14f8d5b0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37c3b: skipped 000100 1780381759.583431 MIRROR_RESYNC 3 mirrors 000102 1780381760.269673 mapwrite 0x00fcd6 thru 0x017aff (0x7e2a bytes) 000103 1780381760.455638 write 0x02c1e0 thru 0x03bd18 (0xfb39 bytes) 000104 1780381760.469425 write 0x0261fa thru 0x0338b2 (0xd6b9 bytes) 000105 1780381760.481293 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3bd19): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd19): Mirror 2: 0x6f010219 (stale) Mirror 3: 0x6f010219 (stale) CRC-32 checksum value for chunk [0, 0x3bd19): Mirror 1: 0xa05c6d99 Mirror 4: 0xa05c6d99 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd19: skipped 000106 1780381761.145872 write 0x00ae9e thru 0x0159f4 (0xab57 bytes) 000107 1780381761.285257 write 0x01d4e3 thru 0x01e203 (0xd21 bytes) 000109 1780381761.325951 mapwrite 0x00ca35 thru 0x01b9ef (0xefbb bytes) 000110 1780381761.393918 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3bd19): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd19): Mirror 2: 0x6f010219 (stale) Mirror 3: 0x6f010219 (stale) Mirror 4: 0xa05c6d99 (stale) CRC-32 checksum value for chunk [0, 0x3bd19): Mirror 1: 0xf46146eb Mirror 5: 0xf46146eb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd19: skipped 000111 1780381762.037394 read 0x00f580 thru 0x012550 (0x2fd1 bytes) 000112 1780381762.046968 trunc from 0x03bd19 to 0x029cfd (0x1201b bytes) 000114 1780381762.153994 MIRROR_RESYNC 5 mirrors 000115 1780381762.914612 read 0x02065c thru 0x024b45 (0x44ea bytes) 000116 1780381762.920540 trunc from 0x029cfe to 0x02298e (0x736f bytes) 000117 1780381762.983378 write 0x01cea7 thru 0x02594c (0x8aa6 bytes) 000118 1780381762.989873 mapread 0x023f24 thru 0x02594c (0x1a29 bytes) 000119 1780381762.991032 write 0x004438 thru 0x00d850 (0x9419 bytes) 000121 1780381763.003981 trunc from 0x02594d to 0x02273a (0x3212 bytes) 000123 1780381763.029620 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2273b): Mirror 3: 0xcacb3d85 (stale) Mirror 4: 0xcacb3d85 (stale) Mirror 5: 0xcacb3d85 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2273b): Mirror 3: 0xcacb3d85 (stale) Mirror 4: 0xcacb3d85 (stale) Mirror 5: 0xcacb3d85 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2273b: skipped 000125 1780381763.363638 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2273b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2273b): Mirror 3: 0xcacb3d85 (stale) Mirror 4: 0xcacb3d85 (stale) Mirror 5: 0xcacb3d85 (stale) CRC-32 checksum value for chunk [0, 0x2273b): Mirror 1: 0x5017d78e Mirror 6: 0x5017d78e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2273b: skipped 000126 1780381764.198862 write 0x02392d thru 0x02494c (0x1020 bytes) 000127 1780381764.245845 mapwrite 0x00bf02 thru 0x012328 (0x6427 bytes) 000129 1780381764.290379 mapwrite 0x01a008 thru 0x021d6f (0x7d68 bytes) 000130 1780381764.338051 mapwrite 0x03ef87 thru 0x03ffff (0x1079 bytes) 000135 1780381764.419029 read 0x028a9a thru 0x03220b (0x9772 bytes) 000136 1780381764.431311 mapread 0x0078f3 thru 0x00fc62 (0x8370 bytes) 000137 1780381764.437844 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x20644f80 (stale) Mirror 4: 0x20644f80 (stale) Mirror 5: 0x20644f80 (stale) Mirror 6: 0xa6d15cc5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc34a8af2 Mirror 7: 0xc34a8af2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000138 1780381765.080904 write 0x0073c4 thru 0x00f287 (0x7ec4 bytes) 000139 1780381765.177294 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x20644f80 (stale) Mirror 4: 0x20644f80 (stale) Mirror 5: 0x20644f80 (stale) Mirror 6: 0xa6d15cc5 (stale) Mirror 7: 0xc34a8af2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4f0323c6 Mirror 8: 0x4f0323c6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000140 1780381765.917860 write 0x00c08b thru 0x01921e (0xd194 bytes) 000141 1780381765.990845 write 0x00a5c6 thru 0x01312e (0x8b69 bytes) 000142 1780381765.996702 mapwrite 0x02f860 thru 0x038bb0 (0x9351 bytes) 000144 1780381766.056599 mapwrite 0x00b836 thru 0x01104b (0x5816 bytes) 000145 1780381766.098845 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x20644f80 (stale) Mirror 4: 0x20644f80 (stale) Mirror 5: 0x20644f80 (stale) Mirror 6: 0xa6d15cc5 (stale) Mirror 8: 0x4f0323c6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x20644f80 (stale) Mirror 4: 0x20644f80 (stale) Mirror 5: 0x20644f80 (stale) Mirror 6: 0xa6d15cc5 (stale) Mirror 8: 0x4f0323c6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000147 1780381766.484282 read 0x029b54 thru 0x034372 (0xa81f bytes) 000153 1780381766.492297 mapwrite 0x00b35e thru 0x01a575 (0xf218 bytes) 000154 1780381766.548065 read 0x018bf3 thru 0x01904c (0x45a bytes) 000156 1780381766.549138 trunc from 0x040000 to 0x026ba3 (0x1945c bytes) 000158 1780381766.563422 mapread 0x025aa6 thru 0x026ba3 (0x10fe bytes) 000159 1780381766.574713 mapwrite 0x005dc4 thru 0x00f1dd (0x941a bytes) 000161 1780381766.619224 write 0x02aa77 thru 0x02fa17 (0x4fa1 bytes) 000162 1780381766.621361 write 0x031098 thru 0x03c5b5 (0xb51e bytes) 000164 1780381766.624891 trunc from 0x03c5b6 to 0x028a59 (0x13b5c bytes) 000167 1780381766.651110 write 0x022429 thru 0x02dfa5 (0xbb7d bytes) 000169 1780381766.654776 mapwrite 0x039b40 thru 0x039bd1 (0x92 bytes) 000170 1780381766.759645 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39bd2): Mirror 4: 0xe364cb6c (stale) Mirror 5: 0xe364cb6c (stale) Mirror 6: 0xd7312be8 (stale) Mirror 8: 0xf976c6fd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39bd2): Mirror 4: 0xe364cb6c (stale) Mirror 5: 0xe364cb6c (stale) Mirror 6: 0xd7312be8 (stale) Mirror 8: 0xf976c6fd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39bd2: skipped 000172 1780381767.186987 mapwrite 0x00bce3 thru 0x01020a (0x4528 bytes) 000173 1780381767.268439 mapwrite 0x0068c1 thru 0x013f79 (0xd6b9 bytes) 000174 1780381767.310422 mapwrite 0x0276a2 thru 0x03141f (0x9d7e bytes) 000177 1780381767.399288 mapwrite 0x01e2b8 thru 0x0249eb (0x6734 bytes) 000180 1780381767.462858 read 0x025a86 thru 0x03319c (0xd717 bytes) 000182 1780381767.464093 mapwrite 0x018da5 thru 0x01b316 (0x2572 bytes) 000186 1780381767.499400 mapwrite 0x02e786 thru 0x030e45 (0x26c0 bytes) 000188 1780381767.533210 trunc from 0x039bd2 to 0x019ecf (0x1fd02 bytes) 000190 1780381767.544785 mapread 0x00110a thru 0x00ebe9 (0xdae0 bytes) 000191 1780381767.551614 write 0x028d86 thru 0x0302ee (0x7569 bytes) 000193 1780381767.558023 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x302ef): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x302ef): Mirror 4: 0x8b304038 (stale) Mirror 5: 0x8b304038 (stale) Mirror 6: 0xa7683abb (stale) Mirror 8: 0x61e5cbb6 (stale) CRC-32 checksum value for chunk [0, 0x302ef): Mirror 1: 0x57dd5b17 Mirror 9: 0x57dd5b17 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x302ef: skipped 000194 1780381768.378651 mapread 0x0043be thru 0x00e408 (0xa04b bytes) 000196 1780381768.403353 write 0x03a76c thru 0x03f57b (0x4e10 bytes) 000197 1780381768.491375 trunc from 0x03f57c to 0x00f6b0 (0x2fecb bytes) 000200 1780381768.552737 write 0x007e17 thru 0x00f36c (0x7556 bytes) 000204 1780381768.572396 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf6b1): Mirror 4: 0x605115be (stale) Mirror 5: 0x605115be (stale) Mirror 6: 0xa8d87383 (stale) Mirror 8: 0x8730c04a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf6b1): Mirror 4: 0x605115be (stale) Mirror 5: 0x605115be (stale) Mirror 6: 0xa8d87383 (stale) Mirror 8: 0x8730c04a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf6b1: skipped 000205 1780381769.035185 read 0x006aca thru 0x00f6b0 (0x8be7 bytes) 000206 1780381769.042125 mapread 0x00905a thru 0x00a5f8 (0x159f bytes) 000208 1780381769.043762 read 0x005f1b thru 0x00f325 (0x940b bytes) 000210 1780381769.055929 write 0x01501b thru 0x02132a (0xc310 bytes) truncating to largest ever: 0x3a10b 000211 1780381769.061726 trunc from 0x02132b to 0x03a10a (0x18de0 bytes) 000212 1780381769.086894 write 0x01baf6 thru 0x01dcf3 (0x21fe bytes) 000214 1780381769.092969 MIRROR_RESYNC 5 mirrors 000215 1780381770.321194 read 0x03482b thru 0x03a10a (0x58e0 bytes) 000217 1780381770.332371 mapread 0x0095b5 thru 0x010478 (0x6ec4 bytes) 000218 1780381770.351260 mapread 0x0005da thru 0x0007f7 (0x21e bytes) 000219 1780381770.360103 MIRROR_RESYNC 5 mirrors 000220 1780381770.704787 mapread 0x0030d4 thru 0x007026 (0x3f53 bytes) 000221 1780381770.709795 MIRROR_RESYNC 5 mirrors 000224 1780381771.052774 read 0x027a2b thru 0x02b0dd (0x36b3 bytes) truncating to largest ever: 0x3c6d9 000226 1780381771.060221 trunc from 0x03a10b to 0x03c6d8 (0x25ce bytes) 000227 1780381771.157177 read 0x031f18 thru 0x0324cd (0x5b6 bytes) 000229 1780381771.171491 MIRROR_RESYNC 5 mirrors 000231 1780381772.032586 write 0x013118 thru 0x01f07b (0xbf64 bytes) 000235 1780381772.133871 trunc from 0x03c6d9 to 0x013c8c (0x28a4c bytes) 000236 1780381772.190116 read 0x00c2bc thru 0x01119b (0x4ee0 bytes) 000238 1780381772.199785 trunc from 0x013c8d to 0x01ac16 (0x6f8a bytes) 000239 1780381772.235622 mapwrite 0x009115 thru 0x00ab6a (0x1a56 bytes) 000241 1780381772.293795 trunc from 0x01ac17 to 0x0111ac (0x9a6a bytes) 000245 1780381772.317125 mapread 0x00f013 thru 0x0111ac (0x219a bytes) 000246 1780381772.318827 mapread 0x00b902 thru 0x00d7d5 (0x1ed4 bytes) 000247 1780381772.320213 read 0x0013ab thru 0x009ecf (0x8b25 bytes) 000248 1780381772.326592 write 0x016beb thru 0x0259e8 (0xedfe bytes) 000249 1780381772.329633 read 0x015864 thru 0x01d1bd (0x795a bytes) 000250 1780381772.335424 mapread 0x00fc61 thru 0x01d861 (0xdc01 bytes) 000251 1780381772.337259 trunc from 0x0259e9 to 0x006ccc (0x1ed1c bytes) 000255 1780381772.355158 trunc from 0x006ccd to 0x00a4ba (0x37ee bytes) 000256 1780381772.371813 MIRROR_RESYNC 5 mirrors 000257 1780381773.142437 mapwrite 0x01f81d thru 0x021011 (0x17f5 bytes) 000258 1780381773.327711 mapwrite 0x00483a thru 0x012d6e (0xe535 bytes) 000260 1780381773.421695 MIRROR_RESYNC 5 mirrors 000262 1780381774.208600 write 0x00e1bc thru 0x01ad71 (0xcbb6 bytes) 000263 1780381774.298632 mapread 0x0038b2 thru 0x0040b1 (0x800 bytes) 000264 1780381774.311670 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21012): Mirror 5: 0xb3f56b9e (stale) Mirror 6: 0xb3f56b9e (stale) Mirror 8: 0xb3f56b9e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21012): Mirror 5: 0xb3f56b9e (stale) Mirror 6: 0xb3f56b9e (stale) Mirror 8: 0xb3f56b9e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21012: skipped 000266 1780381774.671504 mapwrite 0x012036 thru 0x01e9f3 (0xc9be bytes) 000268 1780381774.742444 trunc from 0x021012 to 0x01f3b5 (0x1c5c bytes) 000270 1780381774.778435 write 0x01b0b9 thru 0x01bd04 (0xc4c bytes) 000272 1780381774.780045 write 0x02eb7b thru 0x0387c5 (0x9c4b bytes) 000274 1780381774.784786 mapwrite 0x00aecb thru 0x00c082 (0x11b8 bytes) 000276 1780381774.844824 trunc from 0x0387c6 to 0x01c95a (0x1be6b bytes) 000278 1780381774.886517 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c95b): Mirror 6: 0x60b216f6 (stale) Mirror 8: 0x60b216f6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c95b): Mirror 6: 0x60b216f6 (stale) Mirror 8: 0x60b216f6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c95b: skipped 000279 1780381775.178117 write 0x019d80 thru 0x01cb69 (0x2dea bytes) 000281 1780381775.188836 trunc from 0x01cb6a to 0x03909c (0x1c533 bytes) 000282 1780381775.206308 write 0x013074 thru 0x01a2de (0x726b bytes) 000284 1780381775.216686 read 0x0176f1 thru 0x021409 (0x9d19 bytes) 000285 1780381775.226293 mapwrite 0x02648c thru 0x026efb (0xa70 bytes) 000287 1780381775.294721 trunc from 0x03909d to 0x03014f (0x8f4d bytes) 000290 1780381775.319797 mapwrite 0x00c349 thru 0x011a27 (0x56df bytes) 000294 1780381775.369025 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30150): Mirror 6: 0x7bb46ed5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30150): Mirror 6: 0x7bb46ed5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30150: skipped 000302 1780381775.687201 write 0x0396f6 thru 0x03ffff (0x690a bytes) 000303 1780381775.688900 read 0x039498 thru 0x03ffff (0x6b68 bytes) 000304 1780381775.689633 mapread 0x02ad88 thru 0x034d01 (0x9f7a bytes) 000305 1780381775.721056 trunc from 0x040000 to 0x02d636 (0x129c9 bytes) 000306 1780381775.737895 mapread 0x00c3a1 thru 0x01a211 (0xde71 bytes) 000307 1780381775.754940 read 0x00cfb1 thru 0x018fe9 (0xc039 bytes) 000308 1780381775.756025 write 0x01b9c0 thru 0x025757 (0x9d98 bytes) 000310 1780381775.762304 MIRROR_RESYNC 2 mirrors 000313 1780381776.366162 mapread 0x01f4ea thru 0x029acf (0xa5e6 bytes) 000315 1780381776.384689 trunc from 0x02d637 to 0x0002a1 (0x2d395 bytes) 000316 1780381776.461281 read 0x00003b thru 0x0002a1 (0x267 bytes) 000319 1780381776.473564 write 0x00729a thru 0x010fcd (0x9d34 bytes) 000321 1780381776.476125 trunc from 0x010fce to 0x02e874 (0x1d8a7 bytes) 000323 1780381776.493069 write 0x003eb3 thru 0x00c677 (0x87c5 bytes) 000327 1780381776.502325 write 0x0361ba thru 0x036660 (0x4a7 bytes) 000329 1780381776.504508 trunc from 0x036661 to 0x030041 (0x661f bytes) 000334 1780381776.554367 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x30042): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30042): Mirror 6: 0xcfed1153 (stale) CRC-32 checksum value for chunk [0, 0x30042): Mirror 1: 0x7b6d38b Mirror 7: 0x7b6d38b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30042: skipped 000339 1780381777.156623 MIRROR_RESYNC 3 mirrors 000340 1780381777.661368 mapread 0x012676 thru 0x01716b (0x4af6 bytes) 000343 1780381777.684163 mapwrite 0x00d6f9 thru 0x01421d (0x6b25 bytes) 000347 1780381777.879218 mapwrite 0x02f06b thru 0x02f35a (0x2f0 bytes) 000349 1780381777.945099 MIRROR_RESYNC 3 mirrors 000351 1780381778.701573 mapwrite 0x032c2c thru 0x03eece (0xc2a3 bytes) 000353 1780381778.839921 mapwrite 0x024b1e thru 0x0340f2 (0xf5d5 bytes) 000354 1780381778.889236 trunc from 0x03eecf to 0x00fc7c (0x2f252 bytes) 000357 1780381778.912730 write 0x03de84 thru 0x03f5b4 (0x1731 bytes) 000358 1780381778.914673 read 0x015c57 thru 0x01a609 (0x49b3 bytes) 000359 1780381778.925680 read 0x00ee8b thru 0x01d0c9 (0xe23f bytes) 000360 1780381778.940751 trunc from 0x03f5b5 to 0x02c43c (0x13178 bytes) 000361 1780381778.965881 mapread 0x01ae81 thru 0x022bf6 (0x7d76 bytes) 000362 1780381778.967098 mapwrite 0x0284be thru 0x02ba7b (0x35be bytes) 000364 1780381779.017236 trunc from 0x02c43d to 0x02a19c (0x22a0 bytes) 000365 1780381779.029736 mapwrite 0x008546 thru 0x013a79 (0xb534 bytes) 000366 1780381779.094147 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a19d): Mirror 7: 0x1cb83b8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a19d): Mirror 7: 0x1cb83b8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a19d: skipped 000367 1780381779.348554 mapread 0x00cc31 thru 0x0160bf (0x948f bytes) 000368 1780381779.367222 read 0x02288d thru 0x02a19c (0x7910 bytes) 000370 1780381779.375777 read 0x0007b1 thru 0x0087e4 (0x8034 bytes) 000371 1780381779.384657 write 0x03c9eb thru 0x03ca16 (0x2c bytes) 000372 1780381779.389209 MIRROR_RESYNC 2 mirrors 000375 1780381779.881045 read 0x032458 thru 0x03ca16 (0xa5bf bytes) 000377 1780381779.894189 read 0x0048a8 thru 0x00c5d6 (0x7d2f bytes) 000379 1780381779.918867 read 0x03a9a0 thru 0x03afd1 (0x632 bytes) 000380 1780381779.919601 mapwrite 0x022548 thru 0x0285b1 (0x606a bytes) truncating to largest ever: 0x3f3eb 000383 1780381780.061735 trunc from 0x03ca17 to 0x03f3ea (0x29d4 bytes) 000386 1780381780.117128 MIRROR_RESYNC 2 mirrors 000387 1780381780.714329 mapwrite 0x01d8b7 thru 0x0260d6 (0x8820 bytes) 000392 1780381780.921178 read 0x030942 thru 0x03e536 (0xdbf5 bytes) 000394 1780381780.937563 mapwrite 0x001b9b thru 0x00a004 (0x846a bytes) 000395 1780381781.009137 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x197:0x0]: Invalid argument (22) 000396 1780381781.109148 read 0x000ac9 thru 0x00caf8 (0xc030 bytes) 000397 1780381781.120320 write 0x036f21 thru 0x03ffff (0x90df bytes) 000398 1780381781.127200 mapread 0x03fe1a thru 0x03ffff (0x1e6 bytes) 000399 1780381781.128481 write 0x03d0f1 thru 0x03ffff (0x2f0f bytes) 000402 1780381781.135192 write 0x036bcd thru 0x03ffff (0x9433 bytes) 000404 1780381781.137435 read 0x001e62 thru 0x0109c2 (0xeb61 bytes) 000406 1780381781.161204 trunc from 0x040000 to 0x037be5 (0x841a bytes) 000407 1780381781.179044 write 0x02145d thru 0x02a882 (0x9426 bytes) 000408 1780381781.179964 trunc from 0x037be6 to 0x03b011 (0x342c bytes) 000409 1780381781.196750 mapwrite 0x012194 thru 0x020fd2 (0xee3f bytes) 000414 1780381781.264608 mapread 0x0318ef thru 0x03b011 (0x9723 bytes) 000415 1780381781.296666 mapwrite 0x031898 thru 0x032d41 (0x14aa bytes) 000416 1780381781.332786 write 0x03f44c thru 0x03ffff (0xbb4 bytes) 000418 1780381781.335120 mapread 0x01cc20 thru 0x01f507 (0x28e8 bytes) 000420 1780381781.339083 mapread 0x03b42a thru 0x03ffff (0x4bd6 bytes) 000421 1780381781.354947 mapwrite 0x00beb6 thru 0x019adc (0xdc27 bytes) 000422 1780381781.405113 trunc from 0x040000 to 0x00dd0e (0x322f1 bytes) 000426 1780381781.430759 read 0x00a2b0 thru 0x00b4f6 (0x1247 bytes) 000429 1780381781.433715 read 0x00ab80 thru 0x00dd0e (0x318f bytes) 000432 1780381781.435829 trunc from 0x00dd0f to 0x03a0a4 (0x2c396 bytes) 000434 1780381781.451034 mapread 0x035eb8 thru 0x03a0a4 (0x41ed bytes) 000435 1780381781.469280 mapwrite 0x02a23a thru 0x02a4ea (0x2b1 bytes) 000436 1780381781.507668 mapwrite 0x013fd6 thru 0x014f26 (0xf51 bytes) truncating to largest ever: 0x3ffcc 000438 1780381781.537302 trunc from 0x03a0a5 to 0x03ffcb (0x5f27 bytes) 000442 1780381781.555429 read 0x01183b thru 0x01a347 (0x8b0d bytes) 000443 1780381781.556529 write 0x032a05 thru 0x03aadc (0x80d8 bytes) 000444 1780381781.558128 mapread 0x03a1cd thru 0x03f899 (0x56cd bytes) 000445 1780381781.567897 mapread 0x01806d thru 0x02656d (0xe501 bytes) 000446 1780381781.569980 mapread 0x0353b4 thru 0x03ffcb (0xac18 bytes) 000447 1780381781.571495 mapread 0x01f9a5 thru 0x02bc7a (0xc2d6 bytes) 000448 1780381781.573089 mapread 0x017ae2 thru 0x01c8a3 (0x4dc2 bytes) 000450 1780381781.575891 write 0x001f53 thru 0x0030f6 (0x11a4 bytes) 000451 1780381781.577209 mapwrite 0x02f9e9 thru 0x037a6a (0x8082 bytes) 000452 1780381781.636549 write 0x02c65b thru 0x032543 (0x5ee9 bytes) 000455 1780381781.639913 mapread 0x007cc6 thru 0x012aba (0xadf5 bytes) 000459 1780381781.644360 write 0x03080e thru 0x0396c1 (0x8eb4 bytes) 000460 1780381781.645852 mapread 0x039a61 thru 0x03c563 (0x2b03 bytes) 000461 1780381781.649554 write 0x00b994 thru 0x00c080 (0x6ed bytes) 000462 1780381781.650961 write 0x034153 thru 0x03f89f (0xb74d bytes) 000463 1780381781.652934 trunc from 0x03ffcc to 0x027fae (0x1801d bytes) 000465 1780381781.673675 MIRROR_EXTEND to 2 mirrors 000466 1780381782.278072 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x197:0x0]: Invalid argument (22) 000467 1780381782.459851 write 0x030a29 thru 0x0371f8 (0x67d0 bytes) 000469 1780381782.484947 mapwrite 0x00691a thru 0x010f3f (0xa626 bytes) 000474 1780381782.583553 mapread 0x0311cf thru 0x0371f8 (0x602a bytes) 000480 1780381782.595699 MIRROR_EXTEND to 2 mirrors 000481 1780381783.218218 read 0x035a00 thru 0x0371f8 (0x17f9 bytes) 000483 1780381783.227114 mapwrite 0x02467f thru 0x0328cb (0xe24d bytes) 000484 1780381783.406653 read 0x01f0da thru 0x020714 (0x163b bytes) 000486 1780381783.409039 mapread 0x0014aa thru 0x006fdc (0x5b33 bytes) 000488 1780381783.420788 read 0x003b5b thru 0x00afdb (0x7481 bytes) 000489 1780381783.422071 mapwrite 0x01128a thru 0x01e900 (0xd677 bytes) 000490 1780381783.481826 mapwrite 0x020bdb thru 0x023377 (0x279d bytes) 000491 1780381783.520486 trunc from 0x0371f9 to 0x014071 (0x23187 bytes) 000493 1780381783.554849 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x14072): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14072): Mirror 2: 0xdfc5ffa8 (stale) CRC-32 checksum value for chunk [0, 0x14072): Mirror 1: 0x4c08138b Mirror 3: 0x4c08138b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14072: skipped 000494 1780381784.193338 mapread 0x001faa thru 0x00c206 (0xa25d bytes) 000497 1780381784.219015 read 0x000f02 thru 0x00e0f7 (0xd1f6 bytes) 000499 1780381784.224769 read 0x008d23 thru 0x010a8d (0x7d6b bytes) 000500 1780381784.225354 write 0x025ad6 thru 0x03154e (0xba79 bytes) 000501 1780381784.304500 mapread 0x01ca81 thru 0x021dfb (0x537b bytes) 000502 1780381784.317345 write 0x016dd6 thru 0x023a2b (0xcc56 bytes) 000503 1780381784.323633 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3154f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3154f): Mirror 2: 0x1a7b9f13 (stale) Mirror 3: 0x1493a7a5 (stale) CRC-32 checksum value for chunk [0, 0x3154f): Mirror 1: 0x319ad28d Mirror 4: 0x319ad28d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3154f: skipped 000504 1780381784.931518 MIRROR_RESYNC 4 mirrors 000505 1780381785.601036 trunc from 0x03154f to 0x023f67 (0xd5e7 bytes) 000508 1780381785.722957 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23f68): Mirror 2: 0x8796be55 (stale) Mirror 3: 0x8796be55 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23f68): Mirror 2: 0x8796be55 (stale) Mirror 3: 0x8796be55 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23f68: skipped 000509 1780381785.957060 trunc from 0x023f68 to 0x00d8ec (0x1667b bytes) 000512 1780381785.973606 read 0x003b5f thru 0x0068fa (0x2d9c bytes) 000514 1780381785.981682 trunc from 0x00d8ed to 0x007e37 (0x5ab5 bytes) 000515 1780381785.995087 write 0x032ea8 thru 0x03fe00 (0xcf59 bytes) 000518 1780381785.999999 trunc from 0x03fe01 to 0x03f795 (0x66b bytes) 000519 1780381786.018852 mapread 0x032b80 thru 0x03a35d (0x77de bytes) 000521 1780381786.022330 mapread 0x038440 thru 0x039019 (0xbda bytes) 000523 1780381786.025578 mapwrite 0x03b0fc thru 0x03ffff (0x4f04 bytes) 000524 1780381786.113809 mapread 0x02a54a thru 0x038189 (0xdc40 bytes) 000527 1780381786.139427 mapread 0x012dcc thru 0x012ea0 (0xd5 bytes) 000528 1780381786.160120 trunc from 0x040000 to 0x035b1b (0xa4e4 bytes) 000529 1780381786.179024 mapwrite 0x00e228 thru 0x017fcb (0x9da4 bytes) 000530 1780381786.222871 mapwrite 0x00b47f thru 0x010d0a (0x588c bytes) 000532 1780381786.259114 write 0x002dc2 thru 0x006b8f (0x3dce bytes) 000533 1780381786.276607 read 0x0151f2 thru 0x018ab4 (0x38c3 bytes) 000536 1780381786.278317 mapread 0x0211ef thru 0x027a32 (0x6844 bytes) 000537 1780381786.295412 read 0x01c7b2 thru 0x0255a5 (0x8df4 bytes) 000539 1780381786.297835 mapwrite 0x0296d3 thru 0x02e412 (0x4d40 bytes) 000541 1780381786.354258 MIRROR_RESYNC 3 mirrors 000544 1780381787.127806 trunc from 0x035b1c to 0x033881 (0x229a bytes) 000545 1780381787.241082 trunc from 0x033882 to 0x0087d6 (0x2b0ab bytes) 000546 1780381787.269088 mapwrite 0x039236 thru 0x03b273 (0x203e bytes) 000547 1780381787.357746 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b274): Mirror 3: 0x656d7f0b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b274): Mirror 3: 0x656d7f0b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b274: skipped 000548 1780381787.582775 write 0x016944 thru 0x025442 (0xeaff bytes) 000549 1780381787.597159 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x197:0x0]: Invalid argument (22) 000551 1780381787.714972 mapwrite 0x031016 thru 0x0367ce (0x57b9 bytes) 000552 1780381787.774767 trunc from 0x03b274 to 0x039448 (0x1e2b bytes) 000558 1780381787.801776 MIRROR_EXTEND to 2 mirrors 000560 1780381788.396292 read 0x03465f thru 0x039448 (0x4dea bytes) 000562 1780381788.405010 read 0x0189aa thru 0x025562 (0xcbb9 bytes) 000564 1780381788.414577 write 0x019564 thru 0x01ab14 (0x15b1 bytes) 000565 1780381788.531146 mapwrite 0x00e0fc thru 0x016b40 (0x8a45 bytes) 000567 1780381788.702312 MIRROR_RESYNC 2 mirrors 000569 1780381789.499818 write 0x034966 thru 0x03ffff (0xb69a bytes) 000570 1780381789.561883 read 0x028267 thru 0x02c20a (0x3fa4 bytes) 000572 1780381789.575439 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x42f3cbb4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfbad7460 Mirror 3: 0xfbad7460 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000573 1780381790.230344 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x42f3cbb4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfbad7460 Mirror 3: 0xfbad7460 Mirror 4: 0xfbad7460 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000576 1780381790.941932 mapread 0x00319e thru 0x0052d6 (0x2139 bytes) 000577 1780381790.968421 MIRROR_RESYNC 4 mirrors 000578 1780381791.523969 write 0x03d300 thru 0x03ffff (0x2d00 bytes) 000579 1780381791.585919 read 0x03caf3 thru 0x03ffff (0x350d bytes) 000580 1780381791.595844 MIRROR_RESYNC 4 mirrors 000582 1780381792.222302 mapwrite 0x03211d thru 0x03cefe (0xade2 bytes) 000583 1780381792.519121 write 0x02c780 thru 0x02e22b (0x1aac bytes) 000584 1780381792.522077 read 0x03f1e4 thru 0x03ffff (0xe1c bytes) 000585 1780381792.532923 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6108b601 (stale) Mirror 3: 0x6108b601 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6108b601 (stale) Mirror 3: 0x6108b601 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000587 1780381793.079908 trunc from 0x040000 to 0x018b3c (0x274c3 bytes) 000593 1780381793.133701 trunc from 0x018b3d to 0x0065cc (0x12570 bytes) 000594 1780381793.148741 write 0x039cd0 thru 0x03ffff (0x6330 bytes) 000595 1780381793.150158 mapread 0x008087 thru 0x00832d (0x2a7 bytes) 000596 1780381793.157329 read 0x039696 thru 0x03e21b (0x4b86 bytes) 000597 1780381793.158259 mapread 0x03bdf5 thru 0x03ffff (0x420b bytes) 000599 1780381793.161173 mapwrite 0x02ff8c thru 0x038f24 (0x8f99 bytes) 000600 1780381793.247903 mapread 0x006a81 thru 0x00ec64 (0x81e4 bytes) 000602 1780381793.261191 mapread 0x027db0 thru 0x02b70f (0x3960 bytes) 000603 1780381793.281976 MIRROR_RESYNC 3 mirrors 000604 1780381793.833771 read 0x01908f thru 0x022d0d (0x9c7f bytes) 000609 1780381793.844303 trunc from 0x040000 to 0x0393e7 (0x6c18 bytes) 000612 1780381793.948684 read 0x0212ea thru 0x02877c (0x7493 bytes) 000613 1780381793.960960 read 0x0355e1 thru 0x0393e7 (0x3e07 bytes) 000615 1780381793.962927 mapread 0x00ac34 thru 0x0165a3 (0xb970 bytes) 000617 1780381793.969395 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x393e8): Mirror 3: 0x8118871e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x393e8): Mirror 3: 0x8118871e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x393e8: skipped 000619 1780381794.225995 mapread 0x02dff3 thru 0x032c38 (0x4c46 bytes) 000621 1780381794.252780 MIRROR_RESYNC 2 mirrors 000627 1780381794.822275 mapread 0x01060d thru 0x010f8b (0x97f bytes) 000628 1780381794.828396 read 0x03886f thru 0x0393e7 (0xb79 bytes) 000629 1780381794.835245 mapwrite 0x0314e7 thru 0x03815f (0x6c79 bytes) 000632 1780381794.955880 write 0x020df7 thru 0x02a4fe (0x9708 bytes) 000634 1780381794.964279 mapread 0x0319f2 thru 0x037d7e (0x638d bytes) 000636 1780381794.965859 mapread 0x01af35 thru 0x025bfd (0xacc9 bytes) 000637 1780381794.966852 read 0x003eac thru 0x00687f (0x29d4 bytes) 000638 1780381794.973709 write 0x03610d thru 0x03ffff (0x9ef3 bytes) 000639 1780381794.975021 read 0x01912e thru 0x0275af (0xe482 bytes) 000640 1780381794.979179 MIRROR_RESYNC 2 mirrors 000644 1780381795.509174 MIRROR_EXTEND to 3 mirrors 000645 1780381796.106924 write 0x00de7e thru 0x0176ac (0x982f bytes) 000646 1780381796.180441 read 0x039e5a thru 0x03ffff (0x61a6 bytes) 000647 1780381796.192426 MIRROR_RESYNC 3 mirrors 000648 1780381796.667417 mapwrite 0x026f32 thru 0x02ed90 (0x7e5f bytes) 000649 1780381796.804765 trunc from 0x040000 to 0x00bdd9 (0x34226 bytes) 000651 1780381796.889659 trunc from 0x00bdda to 0x01b79c (0xf9c3 bytes) 000652 1780381796.926896 mapread 0x00160b thru 0x003646 (0x203c bytes) 000653 1780381797.010679 mapread 0x0041fb thru 0x0054d1 (0x12d7 bytes) 000655 1780381797.030534 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1b79d): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b79d): Mirror 3: 0xc8eae392 (stale) Mirror 4: 0xc8eae392 (stale) CRC-32 checksum value for chunk [0, 0x1b79d): Mirror 1: 0x9c92e8c8 Mirror 5: 0x9c92e8c8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b79d: skipped 000657 1780381798.372694 trunc from 0x01b79d to 0x03861e (0x1ce82 bytes) 000658 1780381798.496276 trunc from 0x03861f to 0x01e698 (0x19f86 bytes) 000659 1780381798.526898 trunc from 0x01e699 to 0x0360b3 (0x17a1b bytes) 000661 1780381798.545175 trunc from 0x0360b4 to 0x03d337 (0x7284 bytes) 000662 1780381798.562108 write 0x0334b1 thru 0x03ffff (0xcb4f bytes) 000663 1780381798.568723 write 0x02cfc2 thru 0x03789f (0xa8de bytes) 000665 1780381798.575326 read 0x038cff thru 0x03ffff (0x7301 bytes) 000666 1780381798.576208 write 0x016030 thru 0x023c9c (0xdc6d bytes) 000668 1780381798.595068 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xbfbfc628 (stale) Mirror 5: 0x9529bf2e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xbfbfc628 (stale) Mirror 5: 0x9529bf2e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000669 1780381798.893752 trunc from 0x040000 to 0x025352 (0x1acad bytes) 000670 1780381798.924143 mapread 0x008f7e thru 0x00e9b3 (0x5a36 bytes) 000671 1780381798.946970 write 0x018595 thru 0x01cfa1 (0x4a0d bytes) 000672 1780381798.959568 mapread 0x0240c3 thru 0x025352 (0x1290 bytes) 000673 1780381798.968262 mapread 0x0214dd thru 0x0236b3 (0x21d7 bytes) 000676 1780381798.970490 write 0x005928 thru 0x00d75f (0x7e38 bytes) 000678 1780381798.977760 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25353): Mirror 5: 0xe6a850db (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25353): Mirror 5: 0xe6a850db (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25353: skipped 000679 1780381799.423269 read 0x016b04 thru 0x0183c9 (0x18c6 bytes) 000680 1780381799.430676 write 0x015237 thru 0x018f9d (0x3d67 bytes) 000684 1780381799.459179 trunc from 0x025353 to 0x00557f (0x1fdd3 bytes) 000685 1780381799.535296 mapwrite 0x03af74 thru 0x03ffff (0x508c bytes) 000686 1780381799.738419 write 0x035e92 thru 0x03ffff (0xa16e bytes) 000687 1780381799.745088 read 0x0355a8 thru 0x038e10 (0x3869 bytes) 000688 1780381799.747659 mapwrite 0x0386a0 thru 0x03a88d (0x21ee bytes) 000689 1780381799.858280 trunc from 0x040000 to 0x00371b (0x3c8e4 bytes) 000690 1780381799.875559 trunc from 0x00371c to 0x035abb (0x323a0 bytes) 000693 1780381799.903850 read 0x02a4b2 thru 0x030637 (0x6186 bytes) 000694 1780381799.923385 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x35abc): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35abc): Mirror 5: 0x325b51ed (stale) CRC-32 checksum value for chunk [0, 0x35abc): Mirror 1: 0x686a2d1e Mirror 6: 0x686a2d1e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35abc: skipped 000696 1780381800.705323 trunc from 0x035abc to 0x01d24a (0x18871 bytes) 000698 1780381800.847093 mapwrite 0x00ce7e thru 0x015a9d (0x8c20 bytes) 000699 1780381800.952384 mapread 0x00b382 thru 0x010088 (0x4d07 bytes) 000701 1780381800.954083 read 0x00bc45 thru 0x01a8cb (0xec87 bytes) 000702 1780381800.979946 trunc from 0x01d24b to 0x037a8d (0x1a843 bytes) 000703 1780381801.025905 write 0x0332f3 thru 0x036a90 (0x379e bytes) 000704 1780381801.054531 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37a8e): Mirror 5: 0xba314bdd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a8e): Mirror 5: 0xba314bdd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a8e: skipped 000706 1780381801.376970 write 0x010f2e thru 0x0126c2 (0x1795 bytes) 000707 1780381801.390370 mapwrite 0x004897 thru 0x009b02 (0x526c bytes) 000708 1780381801.481819 mapread 0x00708c thru 0x00f2e6 (0x825b bytes) 000709 1780381801.484529 mapread 0x036bfe thru 0x036d4f (0x152 bytes) 000713 1780381801.507337 write 0x003298 thru 0x00b016 (0x7d7f bytes) 000715 1780381801.512068 trunc from 0x037a8e to 0x0368f3 (0x119a bytes) 000716 1780381801.536405 mapread 0x022048 thru 0x02242a (0x3e3 bytes) 000718 1780381801.551975 write 0x01b9ac thru 0x01bebf (0x514 bytes) 000721 1780381801.563396 trunc from 0x0368f4 to 0x03a6e4 (0x3df1 bytes) 000722 1780381801.581296 mapwrite 0x0200bc thru 0x027890 (0x77d5 bytes) 000723 1780381801.710122 mapread 0x00dd5d thru 0x01c3ac (0xe650 bytes) 000724 1780381801.728715 mapwrite 0x02bc4f thru 0x02da2e (0x1de0 bytes) 000725 1780381801.783343 mapread 0x01e4b6 thru 0x0254c3 (0x700e bytes) 000726 1780381801.785101 mapwrite 0x03f010 thru 0x03ffff (0xff0 bytes) 000727 1780381801.844800 read 0x01881d thru 0x0251a8 (0xc98c bytes) 000728 1780381801.846132 read 0x029915 thru 0x03959a (0xfc86 bytes) 000729 1780381801.853696 trunc from 0x040000 to 0x03bb26 (0x44d9 bytes) 000732 1780381801.869148 trunc from 0x03bb27 to 0x03395b (0x81cb bytes) 000736 1780381801.886005 write 0x00a463 thru 0x00ea70 (0x460e bytes) 000737 1780381801.893183 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3395c): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3395c): Mirror 5: 0xabffc61 (stale) CRC-32 checksum value for chunk [0, 0x3395c): Mirror 1: 0xdff7ba7b Mirror 6: 0xdff7ba7b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3395c: skipped 000738 1780381802.606357 mapwrite 0x027035 thru 0x02dc9d (0x6c69 bytes) 000740 1780381802.840145 read 0x02ea7e thru 0x03270a (0x3c8d bytes) 000741 1780381802.847586 read 0x01a8b2 thru 0x01c7f0 (0x1f3f bytes) 000742 1780381802.856402 mapread 0x02b134 thru 0x02cc07 (0x1ad4 bytes) 000743 1780381802.857348 read 0x012abb thru 0x020bf6 (0xe13c bytes) 000746 1780381802.859776 mapwrite 0x02cdc7 thru 0x032eb1 (0x60eb bytes) 000749 1780381802.922672 write 0x03c2c0 thru 0x03ffff (0x3d40 bytes) 000750 1780381802.924758 mapread 0x024ef3 thru 0x02da43 (0x8b51 bytes) 000751 1780381802.927166 trunc from 0x040000 to 0x038213 (0x7dec bytes) 000752 1780381802.946287 trunc from 0x038214 to 0x006872 (0x319a1 bytes) 000753 1780381802.965086 write 0x01fb21 thru 0x0298c6 (0x9da6 bytes) 000754 1780381802.966534 mapwrite 0x013fd1 thru 0x02137a (0xd3aa bytes) 000756 1780381803.028118 trunc from 0x0298c7 to 0x02e91b (0x5055 bytes) 000759 1780381803.054012 MIRROR_RESYNC 3 mirrors 000761 1780381803.684087 trunc from 0x02e91c to 0x01855b (0x163c0 bytes) 000763 1780381803.790068 MIRROR_RESYNC 3 mirrors 000765 1780381804.655952 trunc from 0x01855c to 0x024910 (0xc3b5 bytes) 000766 1780381804.753261 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x24911): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24911): Mirror 5: 0x840055be (stale) Mirror 6: 0x840055be (stale) CRC-32 checksum value for chunk [0, 0x24911): Mirror 1: 0x840055be Mirror 7: 0x840055be lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24911: skipped 000767 1780381805.491481 trunc from 0x024911 to 0x03666f (0x11d5f bytes) 000768 1780381805.574922 read 0x009aa9 thru 0x00c160 (0x26b8 bytes) 000769 1780381805.585794 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36670): Mirror 5: 0xa3d88af4 (stale) Mirror 6: 0xa3d88af4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36670): Mirror 5: 0xa3d88af4 (stale) Mirror 6: 0xa3d88af4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36670: skipped 000770 1780381805.852832 trunc from 0x036670 to 0x007164 (0x2f50b bytes) 000771 1780381805.869610 write 0x036791 thru 0x0396e4 (0x2f54 bytes) 000773 1780381805.873020 mapwrite 0x009fd8 thru 0x012b9d (0x8bc6 bytes) 000775 1780381805.967610 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x396e5): Mirror 5: 0xe1da41a7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x396e5): Mirror 5: 0xe1da41a7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x396e5: skipped 000776 1780381806.303220 mapwrite 0x01a25e thru 0x0238d2 (0x9675 bytes) 000777 1780381806.397609 trunc from 0x0396e5 to 0x00db4d (0x2bb97 bytes) 000779 1780381806.433048 mapwrite 0x020cd3 thru 0x0302ee (0xf61c bytes) 000781 1780381806.606299 read 0x00c57f thru 0x012807 (0x6289 bytes) 000783 1780381806.619419 read 0x0301bb thru 0x0302ee (0x134 bytes) 000785 1780381806.626912 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x302ef): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x302ef): Mirror 5: 0x27e7f9f1 (stale) CRC-32 checksum value for chunk [0, 0x302ef): Mirror 1: 0x79e8dc7e Mirror 6: 0x79e8dc7e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x302ef: skipped 000789 1780381807.335953 mapwrite 0x013255 thru 0x023104 (0xfeb0 bytes) 000791 1780381807.564313 read 0x005535 thru 0x0121b9 (0xcc85 bytes) 000792 1780381807.577831 read 0x01e904 thru 0x02960f (0xad0c bytes) 000793 1780381807.593640 write 0x01fd32 thru 0x020fb7 (0x1286 bytes) 000795 1780381807.602505 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x302ef): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x302ef): Mirror 5: 0x27e7f9f1 (stale) Mirror 6: 0x79e8dc7e (stale) CRC-32 checksum value for chunk [0, 0x302ef): Mirror 1: 0x60a992ea Mirror 7: 0x60a992ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x302ef: skipped 000796 1780381808.515484 read 0x019749 thru 0x0256ac (0xbf64 bytes) 000804 1780381808.557058 write 0x004db3 thru 0x00dfa5 (0x91f3 bytes) 000805 1780381808.626734 trunc from 0x0302ef to 0x02f2b6 (0x1038 bytes) 000808 1780381808.649912 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f2b7): Mirror 6: 0x4366d73f (stale) Mirror 7: 0xa80300ea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f2b7): Mirror 6: 0x4366d73f (stale) Mirror 7: 0xa80300ea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f2b7: skipped 000810 1780381808.991639 mapread 0x02694a thru 0x02d2c5 (0x697c bytes) 000813 1780381809.018566 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2f2b7): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f2b7): Mirror 6: 0x4366d73f (stale) Mirror 7: 0xa80300ea (stale) CRC-32 checksum value for chunk [0, 0x2f2b7): Mirror 1: 0xba5c6fcd Mirror 8: 0xba5c6fcd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f2b7: skipped 000815 1780381810.116338 read 0x01d259 thru 0x01eb5a (0x1902 bytes) 000817 1780381810.132614 read 0x0121f3 thru 0x017c24 (0x5a32 bytes) 000818 1780381810.149618 trunc from 0x02f2b7 to 0x03f762 (0x104ac bytes) 000819 1780381810.226782 read 0x0314fd thru 0x03be81 (0xa985 bytes) 000820 1780381810.236985 mapread 0x0206a1 thru 0x02acda (0xa63a bytes) 000822 1780381810.240071 trunc from 0x03f763 to 0x0237d6 (0x1bf8c bytes) 000826 1780381810.260615 write 0x039ff5 thru 0x03ffff (0x600b bytes) 000827 1780381810.262124 mapread 0x00a32d thru 0x0148c0 (0xa594 bytes) 000829 1780381810.264762 read 0x0311b6 thru 0x03e891 (0xd6dc bytes) 000830 1780381810.275777 MIRROR_RESYNC 4 mirrors 000831 1780381811.011175 mapread 0x02c3db thru 0x02dea2 (0x1ac8 bytes) 000833 1780381811.023810 trunc from 0x040000 to 0x018869 (0x27796 bytes) 000834 1780381811.135889 mapwrite 0x02eaac thru 0x03383b (0x4d90 bytes) 000835 1780381811.250032 read 0x01b156 thru 0x02629f (0xb14a bytes) 000836 1780381811.260404 mapread 0x017197 thru 0x021003 (0x9e6d bytes) 000837 1780381811.270878 trunc from 0x03383c to 0x016551 (0x1d2ea bytes) 000840 1780381811.295546 mapread 0x0098ce thru 0x00b919 (0x204c bytes) 000841 1780381811.306661 mapwrite 0x03bdb0 thru 0x03ffff (0x4250 bytes) 000842 1780381811.366661 write 0x02cbbc thru 0x036542 (0x9987 bytes) 000843 1780381811.387910 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x98893d7 (stale) Mirror 8: 0x98893d7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x98893d7 (stale) Mirror 8: 0x98893d7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000844 1780381811.866639 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x98893d7 (stale) Mirror 8: 0x98893d7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb2ba3eed Mirror 9: 0xb2ba3eed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000845 1780381813.043054 write 0x00fbe2 thru 0x018ce0 (0x90ff bytes) 000846 1780381813.120099 mapwrite 0x033da6 thru 0x03ffff (0xc25a bytes) 000847 1780381813.195972 mapwrite 0x032af6 thru 0x03d912 (0xae1d bytes) 000849 1780381813.245969 read 0x013fee thru 0x01ce60 (0x8e73 bytes) 000850 1780381813.250978 read 0x03219d thru 0x03cb6f (0xa9d3 bytes) 000851 1780381813.251717 mapwrite 0x034208 thru 0x03ffff (0xbdf8 bytes) 000853 1780381813.288010 trunc from 0x040000 to 0x03cb75 (0x348a bytes) 000855 1780381813.306590 trunc from 0x03cb76 to 0x03fc52 (0x30dd bytes) 000857 1780381813.326150 mapread 0x00cf04 thru 0x019984 (0xca81 bytes) 000858 1780381813.338221 mapwrite 0x025db1 thru 0x02fe66 (0xa0b6 bytes) 000860 1780381813.425137 mapread 0x029b3e thru 0x02df12 (0x43d5 bytes) 000862 1780381813.429148 mapread 0x03d203 thru 0x03fc52 (0x2a50 bytes) 000863 1780381813.431714 mapwrite 0x02467e thru 0x027683 (0x3006 bytes) 000864 1780381813.557398 mapwrite 0x0025b3 thru 0x00ac4a (0x8698 bytes) 000867 1780381813.626101 read 0x0219b5 thru 0x02ae58 (0x94a4 bytes) 000868 1780381813.626873 read 0x03baf8 thru 0x03fc52 (0x415b bytes) 000870 1780381813.629310 mapwrite 0x00eaa5 thru 0x00f006 (0x562 bytes) 000873 1780381813.688975 mapread 0x01e94e thru 0x01fa9a (0x114d bytes) 000875 1780381813.691379 mapwrite 0x02e002 thru 0x02f520 (0x151f bytes) 000876 1780381813.730886 read 0x01b3a3 thru 0x02516d (0x9dcb bytes) 000877 1780381813.732811 mapwrite 0x00a6a2 thru 0x014fd9 (0xa938 bytes) 000880 1780381813.839275 MIRROR_RESYNC 4 mirrors 000881 1780381815.116188 mapwrite 0x03f1ff thru 0x03ffff (0xe01 bytes) 000882 1780381815.280353 mapwrite 0x03a753 thru 0x03ffff (0x58ad bytes) 000885 1780381815.417893 trunc from 0x040000 to 0x01bf23 (0x240dc bytes) 000886 1780381815.450392 trunc from 0x01bf24 to 0x02f7b2 (0x1388f bytes) 000889 1780381815.468210 read 0x00a557 thru 0x018ea4 (0xe94e bytes) 000892 1780381815.495548 MIRROR_RESYNC 4 mirrors 000894 1780381817.033023 mapread 0x022d85 thru 0x02c8dd (0x9b59 bytes) 000897 1780381817.069992 read 0x004686 thru 0x012808 (0xe183 bytes) 000899 1780381817.093863 mapwrite 0x02fb30 thru 0x036d16 (0x71e7 bytes) 000902 1780381817.466784 write 0x00e588 thru 0x013f26 (0x599f bytes) 000904 1780381817.493870 read 0x01c02c thru 0x02b060 (0xf035 bytes) 000905 1780381817.513067 trunc from 0x036d17 to 0x02389c (0x1347a bytes) 000910 1780381817.542900 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x197:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2389d): Mirror 8: 0xfd88d262 (stale) Mirror 9: 0xfd88d262 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2389d): Mirror 8: 0xfd88d262 (stale) Mirror 9: 0xfd88d262 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2389d: skipped 000911 1780381817.952230 write 0x035947 thru 0x03ffff (0xa6b9 bytes) 000912 1780381817.962705 MIRROR_RESYNC 3 mirrors 000913 1780381819.139833 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x197:0x0]: Invalid argument (22) 000915 1780381819.607419 mapread 0x020c31 thru 0x02228f (0x165f bytes) 000916 1780381819.649180 mapread 0x0150db thru 0x01a3ce (0x52f4 bytes) 000920 1780381819.666414 trunc from 0x040000 to 0x0360f3 (0x9f0c bytes) 000923 1780381819.828872 mapwrite 0x01a65d thru 0x0205d2 (0x5f76 bytes) 000925 1780381820.006560 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x197:0x0]: Invalid argument (22) 000927 1780381820.207687 mapwrite 0x01f5ac thru 0x023b90 (0x45e5 bytes) 000930 1780381820.378032 trunc from 0x0360f4 to 0x0342cb (0x1e28 bytes) 000931 1780381820.410346 write 0x010a6e thru 0x0196f3 (0x8c86 bytes) 000932 1780381820.456982 write 0x008bb0 thru 0x00fb49 (0x6f9a bytes) 000933 1780381820.473232 mapread 0x014a7b thru 0x021591 (0xcb17 bytes) 000934 1780381820.475014 write 0x02d018 thru 0x0390a8 (0xc091 bytes) 000937 1780381820.498348 mapread 0x014561 thru 0x018ea8 (0x4948 bytes) 000938 1780381820.499556 mapread 0x00b5c6 thru 0x0158be (0xa2f9 bytes) 000940 1780381820.505496 read 0x01f226 thru 0x029c47 (0xaa22 bytes) 000942 1780381820.507979 write 0x009098 thru 0x00a07f (0xfe8 bytes) 000944 1780381820.510715 mapread 0x006780 thru 0x00fe47 (0x96c8 bytes) 000945 1780381820.512528 write 0x03a5ca thru 0x03ffff (0x5a36 bytes) 000948 1780381820.516181 trunc from 0x040000 to 0x01ae0d (0x251f2 bytes) 000950 1780381820.535509 read 0x0060e5 thru 0x015a65 (0xf981 bytes) 000951 1780381820.536393 read 0x01505d thru 0x01ae0d (0x5db1 bytes) 000955 1780381820.540165 mapread 0x0149db thru 0x01ae0d (0x6433 bytes) 000956 1780381820.541539 read 0x010f97 thru 0x01a4bd (0x9527 bytes) 000957 1780381820.543500 write 0x03a667 thru 0x03b3ee (0xd88 bytes) 000958 1780381820.544971 mapwrite 0x007b25 thru 0x009eaa (0x2386 bytes) 000959 1780381820.609033 trunc from 0x03b3ef to 0x00014a (0x3b2a4 bytes) 000960 1780381820.633194 write 0x01779f thru 0x019c3d (0x249f bytes) 000961 1780381820.634238 trunc from 0x019c3e to 0x018399 (0x18a4 bytes) 000962 1780381820.652167 trunc from 0x01839a to 0x022d03 (0xa96a bytes) 000966 1780381820.676316 trunc from 0x022d04 to 0x01254f (0x107b4 bytes) 000968 1780381820.694359 mapwrite 0x017158 thru 0x018ae1 (0x198a bytes) 000970 1780381820.802843 write 0x013c03 thru 0x01f143 (0xb541 bytes) 000971 1780381820.804248 trunc from 0x01f144 to 0x02b4d5 (0xc392 bytes) 000976 1780381820.836280 mapwrite 0x01ca8e thru 0x023a0a (0x6f7d bytes) 000978 1780381820.904057 mapread 0x01bda1 thru 0x023789 (0x79e9 bytes) 000980 1780381820.905754 mapwrite 0x011433 thru 0x01d03a (0xbc08 bytes) 000981 1780381820.957903 read 0x01654b thru 0x01b275 (0x4d2b bytes) 000982 1780381820.958923 trunc from 0x02b4d6 to 0x0218cc (0x9c09 bytes) 000983 1780381820.979140 read 0x009179 thru 0x00d2bb (0x4143 bytes) 000984 1780381820.981372 read 0x006bb3 thru 0x01119e (0xa5ec bytes) 000985 1780381820.983379 trunc from 0x0218cd to 0x00339d (0x1e52f bytes) 000986 1780381821.040951 read 0x00054d thru 0x00339d (0x2e51 bytes) 000989 1780381821.053181 read 0x001389 thru 0x00339d (0x2015 bytes) 000990 1780381821.056810 write 0x0321ca thru 0x03df21 (0xbd58 bytes) 000991 1780381821.063362 mapread 0x001133 thru 0x001212 (0xe0 bytes) 000995 1780381821.070835 mapread 0x03711a thru 0x03df21 (0x6e08 bytes) 000997 1780381821.074379 read 0x006128 thru 0x013489 (0xd362 bytes) 000998 1780381821.093027 trunc from 0x03df22 to 0x000925 (0x3d5fc bytes) 000999 1780381821.113980 write 0x0003a3 thru 0x00ab7d (0xa7db bytes) 001000 1780381821.119645 trunc from 0x00ab7e to 0x001ce0 (0x8e9d bytes) All operations completed A-OK! PASS 70a (74s) == sanity-flr test 71: check mirror extend parent fid ==== 02:30:26 (1780381826) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 02:30:30 (1780381830) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.182889 s, 11.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182494 s, 5.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 02:30:38 (1780381838) Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre2 Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5694912 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 2486624 .. Extending file size to 8659584 .. Extending file size to 8181472 .. Extending file size to 8575808 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8848512 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9105088 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8935072 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 56635 56636 56639 56641 56645 done 192.168.201.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg150-client.virtnet /mnt/lustre2 (opts:) 192.168.201.150@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg150-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (81s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 02:31:59 (1780381919) Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre2 Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 2728320 .. Extending file size to 1325440 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6295072 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7442848 .. Extending file size to 8837792 .. Extending file size to 8869440 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9160704 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8920960 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 58076 58077 58078 58080 58082 192.168.201.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg150-client.virtnet /mnt/lustre2 (opts:) 192.168.201.150@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg150-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (76s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 02:33:15 (1780381995) Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0953181 s, 22.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.287095 s, 7.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x66a31885 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg150-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 02:33:37 (1780382017) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.109355 s, 19.2 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 02:33:45 (1780382025) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1de:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cf:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cf:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d0:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2f3:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 02:33:53 (1780382033) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.1617 s, 13.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 02:34:00 (1780382040) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.141715 s, 14.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 02:34:07 (1780382047) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.227724 s, 9.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f9:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 02:34:41 (1780382081) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg150-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949783 s, 11.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d6:0x0] } PASS 204d (59s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 02:35:40 (1780382140) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg150-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.190291 s, 11.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (56s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 02:36:36 (1780382196) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg150-server mds-ost sync done. Creating new pool test_204f oleg150-server: Pool lustre.test_204f created Adding targets to pool oleg150-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg150-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.392542 s, 5.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg150-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg150-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg150-server: Pool lustre.test_204f destroyed PASS 204f (82s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 02:37:58 (1780382278) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 02:38:06 (1780382286) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 02:38:14 (1780382294) oleg150-server: Pool lustre.test_206 created Adding targets to pool oleg150-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg150-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Updated after 2s: want 'lustre-OST0000_UUID lustre-OST0001_UUID ' got 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg150-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg150-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg150-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 02:38:40 (1780382320) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963993 s, 10.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271647 s, 38.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xabdfe96e Mirror 3: 0xabdfe96e PASS 207 (14s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 02:38:54 (1780382334) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 02:38:58 (1780382338) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 02:39:02 (1780382342) Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.156693 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1ae0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1780000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1780000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1780000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2560000 Read this (4 bytes): BBBB 192.168.201.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg150-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 02:39:39 (1780382379) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0270966 s, 302 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 02:39:47 (1780382387) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356601 s, 29.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x203 lmm_fid: [0x200000404:0x203:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x203 lmm_fid: [0x200000404:0x203:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x307:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834435 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-02 02:39:48.000000000 -0400 Modify: 2026-06-02 02:39:48.000000000 -0400 Change: 2026-06-02 02:39:49.000000000 -0400 Birth: 2026-06-02 02:39:47.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 02:39:54 (1780382394) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 02:39:57 (1780382397) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.655856 s, 62.5 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0654409 s, 62.6 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 2 02:40 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x205:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 2 02:40 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 02:40:05 (1780382405) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 02:40:14 (1780382414) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1725 sec =========== 02:40:21 (1780382421) === sanity-flr: start cleanup 02:40:23 (1780382423) === === sanity-flr: finish cleanup 02:40:28 (1780382428) ===