-----============= acceptance-small: sanity-flr ============----- Wed Apr 29 16:29:18 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg413-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 16:29:39 (1777494579) === oleg413-client.virtnet: executing check_config_client /mnt/lustre oleg413-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg413-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8df35080a800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8df35080a800.idle_timeout=debug disable quota as required oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 16:30:10 (1777494610) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7543] == sanity-flr test 0a: lfs mirror create with -N option == 16:30:12 (1777494612) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 16:30:29 (1777494629) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 16:30:32 (1777494632) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 16:30:35 (1777494635) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 16:31:02 (1777494662) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 16:31:05 (1777494665) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 16:31:09 (1777494669) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8df35080a800.stats=clear osc.lustre-OST0001-osc-ffff8df35080a800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 16:31:16 (1777494676) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 16:31:26 (1777494686) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 16:31:34 (1777494694) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 16:31:37 (1777494697) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 16:31:44 (1777494704) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 16:31:54 (1777494714) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 16:32:01 (1777494721) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 16:32:04 (1777494724) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 16:32:12 (1777494732) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 16:32:21 (1777494741) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 16:32:27 (1777494747) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 1.73889 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152254 s, 6.9 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 16:32:37 (1777494757) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.057264 s, 36.6 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 16:32:45 (1777494765) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 16:32:55 (1777494775) Stopping /mnt/lustre-ost1 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8df35080a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8df35080a800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg413-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8df35080a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8df35080a800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg413-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (59s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 16:33:53 (1777494833) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.421503 s, 19.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8df35080a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8df35080a800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg413-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (36s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 16:34:29 (1777494869) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8df35080a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8df35080a800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg413-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8df35080a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8df35080a800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg413-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (73s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 16:35:42 (1777494942) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.39124 s, 11.9 MB/s -rw-r--r-- 1 root root 52428800 Apr 29 16:35 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.973s user 0m0.001s sys 0m0.191s Stopping /mnt/lustre-ost1 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8df35080a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8df35080a800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.986s user 0m0.001s sys 0m0.194s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg413-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8df35080a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8df35080a800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.874s user 0m0.001s sys 0m0.185s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg413-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg413-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8df35080a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8df35080a800.ost_server_uuid in FULL state after 0 sec PASS 33b (72s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 16:36:54 (1777495014) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 16:36:57 (1777495017) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 16:37:00 (1777495020) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 16:37:03 (1777495023) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 16:37:09 (1777495029) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.564681 s, 22.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 16:37:20 (1777495040) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557216 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313654 s, 33.4 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 16:37:27 (1777495047) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 16:37:35 (1777495055) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00955914 s, 107 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00344317 s, 297 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00233042 s, 439 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00215494 s, 475 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 16:37:45 (1777495065) create test files with size 15751750 .. bf718acfeb1eace1dea17ee1aab49b6b - fa00b4ca14db9878c0a0bae0b6653c99 - 6cb88df1873f385534db01d55420e065 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg413-client.virtnet /mnt/lustre (opts:) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 16:38:09 (1777495089) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 16:38:22 (1777495102) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 16:38:29 (1777495109) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140269 s, 7.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261504 s, 40.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 16:38:37 (1777495117) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.235042 s, 8.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.446907 s, 9.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.205238 s, 10.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 16:38:53 (1777495133) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 16:38:57 (1777495137) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 16:39:00 (1777495140) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329393 s, 31.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 16:39:08 (1777495148) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 16:39:11 (1777495151) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 16:39:21 (1777495161) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.24933 s, 42.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg413-client.virtnet /mnt/lustre (opts:) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 16:39:32 (1777495172) llite.lustre-ffff8df34529c800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.29772 s, 35.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg413-client.virtnet /mnt/lustre (opts:) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8df34c98f000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 16:39:47 (1777495187) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (37s) == sanity-flr test 46: Verify setstripe --copy option ==== 16:40:24 (1777495224) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (39s) == sanity-flr test 47: Verify mirror obj alloc =========== 16:41:03 (1777495263) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 16:41:06 (1777495266) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.209742 s, 15.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.157119 s, 20.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x144cd2d0 (stale) Mirror 2: 0x144cd2d0 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 8e7125b9602605b73905e881a2380cb9 - mirror 1 checksum: 97624aeba87d3cefe0833213a43f59b4 - mirror 2 checksum: 8e7125b9602605b73905e881a2380cb9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Apr 29 16:41:10 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Apr 29 16:41:10 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 8e7125b9602605b73905e881a2380cb9 - mirror 1 checksum: 97624aeba87d3cefe0833213a43f59b4 - mirror 2 checksum: 97624aeba87d3cefe0833213a43f59b4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Apr 29 16:41:10 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Apr 29 16:41:10 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 16:41:18 (1777495278) Failing mds1 on oleg413-server Stopping /mnt/lustre-mds1 (opts:) on oleg413-server 16:41:29 (1777495289) shut down facet: mds1 facet_host: oleg413-server facet_failover_host: oleg413-server Failover mds1 to oleg413-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-MDT0000 16:41:49 (1777495309) targets are mounted 16:41:49 (1777495309) facet_failover done oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514849 s, 20.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } PASS 50A (44s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 16:42:02 (1777495322) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 16:42:06 (1777495326) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 16:42:09 (1777495329) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.01534 s, 1.1 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 16:42:17 (1777495337) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 16:42:20 (1777495340) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 16:42:23 (1777495343) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 16:42:46 (1777495366) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 16:43:16 (1777495396) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 16:43:42 (1777495422) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 16:44:07 (1777495447) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 16:44:17 (1777495457) mirror create pid 53479 mirror split pid 53481 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4993: 53479 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4993: 53481 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 16:45:23 (1777495523) Seed set to 5525 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000004 1777495525.869260 mapwrite 0x00a6e8 thru 0x016f94 (0xc8ad bytes) 000005 1777495526.057507 mapread 0x008563 thru 0x00ba16 (0x34b4 bytes) 000007 1777495526.069174 MIRROR_EXTEND to 2 mirrors 000010 1777495526.950048 mapread 0x011d91 thru 0x01340b (0x167b bytes) 000011 1777495526.995967 MIRROR_RESYNC 2 mirrors 000015 1777495527.423172 mapwrite 0x03c65f thru 0x03ffff (0x39a1 bytes) truncating to largest ever: 0x1070c 000016 1777495527.615946 trunc from 0x040000 to 0x01070b (0x2f8f4 bytes) 000017 1777495527.651126 mapread 0x00b239 thru 0x01070b (0x54d3 bytes) 000018 1777495527.691968 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) 000021 1777495527.914836 MIRROR_EXTEND to 2 mirrors 000022 1777495528.715456 mapread 0x0059ad thru 0x006bc6 (0x121a bytes) 000023 1777495528.729557 write 0x002ef3 thru 0x00f0f7 (0xc205 bytes) 000024 1777495528.870102 read 0x00ef7d thru 0x01070b (0x178f bytes) 000030 1777495528.886051 write 0x022c94 thru 0x028dbe (0x612b bytes) 000031 1777495528.887737 mapwrite 0x00fbbc thru 0x01244c (0x2891 bytes) 000034 1777495528.956079 mapread 0x00baed thru 0x00df38 (0x244c bytes) 000039 1777495528.960043 mapwrite 0x02f29e thru 0x036492 (0x71f5 bytes) truncating to largest ever: 0x32925 000041 1777495529.055301 trunc from 0x036493 to 0x032924 (0x3b6e bytes) 000042 1777495529.088695 mapread 0x0132f0 thru 0x015ec2 (0x2bd3 bytes) 000045 1777495529.109199 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) 000046 1777495529.264030 write 0x0039a5 thru 0x009dfb (0x6457 bytes) 000047 1777495529.281022 write 0x0036af thru 0x01257c (0xeece bytes) 000049 1777495529.294302 trunc from 0x032925 to 0x02ecbc (0x3c68 bytes) 000050 1777495529.322730 write 0x01c0a5 thru 0x01cb70 (0xacc bytes) 000052 1777495529.340367 trunc from 0x02ecbd to 0x01c13e (0x12b7e bytes) 000054 1777495529.368953 mapwrite 0x014d84 thru 0x01a733 (0x59b0 bytes) 000056 1777495529.481429 trunc from 0x01c13f to 0x013c26 (0x8518 bytes) 000059 1777495529.515890 read 0x004005 thru 0x0044c9 (0x4c5 bytes) 000060 1777495529.517356 read 0x00e375 thru 0x013c26 (0x58b2 bytes) 000062 1777495529.519229 mapread 0x0008e8 thru 0x006ce4 (0x63fd bytes) 000065 1777495529.528568 mapwrite 0x02e15d thru 0x03d221 (0xf0c5 bytes) 000068 1777495529.651525 trunc from 0x03d222 to 0x004fe1 (0x38240 bytes) 000072 1777495529.684534 mapread 0x004b47 thru 0x004fe1 (0x49b bytes) 000075 1777495529.692019 MIRROR_EXTEND to 2 mirrors 000076 1777495530.385806 mapread 0x001492 thru 0x004fe1 (0x3b50 bytes) 000078 1777495530.404622 read 0x004696 thru 0x004fe1 (0x94c bytes) 000081 1777495530.405975 read 0x0013d6 thru 0x004fe1 (0x3c0c bytes) 000083 1777495530.408149 mapwrite 0x033f31 thru 0x034ccb (0xd9b bytes) 000084 1777495530.663853 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) 000085 1777495530.861241 mapread 0x002ced thru 0x007430 (0x4744 bytes) 000086 1777495530.902164 read 0x02afbc thru 0x02e170 (0x31b5 bytes) 000087 1777495530.918798 trunc from 0x034ccc to 0x020a20 (0x142ab bytes) 000088 1777495530.941627 read 0x00c7ab thru 0x015bf5 (0x944b bytes) 000089 1777495530.965279 read 0x013253 thru 0x020a20 (0xd7ce bytes) 000090 1777495530.970280 read 0x00ef77 thru 0x01e1a1 (0xf22b bytes) 000091 1777495530.973959 mapwrite 0x017769 thru 0x0234af (0xbd47 bytes) 000094 1777495531.094418 read 0x01b1b5 thru 0x0234af (0x82fb bytes) 000095 1777495531.097278 read 0x001413 thru 0x008e08 (0x79f6 bytes) 000096 1777495531.106195 MIRROR_EXTEND to 2 mirrors 000097 1777495531.807932 mapread 0x017726 thru 0x020b8c (0x9467 bytes) 000098 1777495531.847468 read 0x009a96 thru 0x00a0e7 (0x652 bytes) 000099 1777495531.860122 mapread 0x0081d2 thru 0x009690 (0x14bf bytes) truncating to largest ever: 0x3a284 000101 1777495531.880342 trunc from 0x0234b0 to 0x03a283 (0x16dd4 bytes) truncating to largest ever: 0x3bf2f 000102 1777495532.092465 trunc from 0x03a284 to 0x03bf2e (0x1cab bytes) 000104 1777495532.131044 read 0x00a553 thru 0x00e158 (0x3c06 bytes) 000105 1777495532.150171 mapwrite 0x017edf thru 0x01f7c1 (0x78e3 bytes) 000106 1777495532.284433 mapwrite 0x0329b2 thru 0x03c3b2 (0x9a01 bytes) 000107 1777495532.450639 mapread 0x0370b7 thru 0x03a19f (0x30e9 bytes) 000108 1777495532.452173 trunc from 0x03c3b3 to 0x030375 (0xc03d bytes) 000109 1777495532.479068 write 0x01debf thru 0x027674 (0x97b6 bytes) 000110 1777495532.485107 read 0x006334 thru 0x00ba4f (0x571c bytes) 000115 1777495532.492540 write 0x024c9c thru 0x031ed3 (0xd238 bytes) 000118 1777495532.508670 write 0x0224f7 thru 0x030ad2 (0xe5dc bytes) 000120 1777495532.510728 read 0x00df9b thru 0x018b57 (0xabbd bytes) 000121 1777495532.516598 mapread 0x026fef thru 0x02adaf (0x3dc1 bytes) 000122 1777495532.519134 trunc from 0x031ed4 to 0x006e53 (0x2b080 bytes) 000123 1777495532.576302 mapwrite 0x03e6a6 thru 0x03edd0 (0x72b bytes) 000127 1777495532.745725 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) 000129 1777495532.952398 read 0x01fdfe thru 0x02391c (0x3b1f bytes) 000130 1777495532.971152 mapwrite 0x03ebdd thru 0x03ffff (0x1423 bytes) 000131 1777495533.087340 trunc from 0x040000 to 0x006ef6 (0x39109 bytes) 000132 1777495533.110982 mapwrite 0x00bb0d thru 0x01960e (0xdb02 bytes) 000135 1777495533.228691 write 0x0112cf thru 0x01aeaa (0x9bdc bytes) 000136 1777495533.230078 mapread 0x0064f4 thru 0x00e52e (0x803b bytes) 000137 1777495533.231206 trunc from 0x01aeab to 0x007b6b (0x1333f bytes) 000138 1777495533.288186 trunc from 0x007b6c to 0x035f70 (0x2e405 bytes) 000139 1777495533.316272 trunc from 0x035f71 to 0x029e06 (0xc16a bytes) 000142 1777495533.354483 MIRROR_EXTEND to 2 mirrors 000146 1777495534.125764 mapread 0x026e47 thru 0x029e06 (0x2fc0 bytes) 000147 1777495534.151632 mapwrite 0x02cdcd thru 0x038ed7 (0xc10b bytes) 000149 1777495534.360779 mapwrite 0x0126d6 thru 0x013a73 (0x139e bytes) 000150 1777495534.433883 read 0x01f333 thru 0x027c59 (0x8927 bytes) 000152 1777495534.444092 mapread 0x033485 thru 0x038ed7 (0x5a53 bytes) 000153 1777495534.445983 mapwrite 0x00925e thru 0x010588 (0x732b bytes) 000155 1777495534.526202 write 0x0394ce thru 0x03ffff (0x6b32 bytes) 000156 1777495534.527552 write 0x001f3a thru 0x00ac3e (0x8d05 bytes) 000159 1777495534.559366 write 0x00bde4 thru 0x016637 (0xa854 bytes) 000161 1777495534.564237 read 0x01ba60 thru 0x01d002 (0x15a3 bytes) 000163 1777495534.598998 mapread 0x0245df thru 0x02e264 (0x9c86 bytes) 000164 1777495534.600335 mapread 0x00daf3 thru 0x0109a2 (0x2eb0 bytes) 000165 1777495534.609615 read 0x00b5d5 thru 0x017588 (0xbfb4 bytes) 000170 1777495534.618995 trunc from 0x040000 to 0x039923 (0x66dc bytes) 000171 1777495534.642991 mapread 0x000d2b thru 0x0104d6 (0xf7ac bytes) 000172 1777495534.644202 read 0x0032a0 thru 0x00f3a0 (0xc101 bytes) 000179 1777495534.650637 write 0x013b6d thru 0x014b08 (0xf9c bytes) 000183 1777495534.654954 trunc from 0x039924 to 0x01cdcf (0x1cb54 bytes) 000185 1777495534.673786 write 0x030b43 thru 0x0391fb (0x86b9 bytes) 000186 1777495534.681437 MIRROR_RESYNC 2 mirrors 000188 1777495535.432632 MIRROR_RESYNC 2 mirrors 000190 1777495535.942473 mapread 0x013884 thru 0x016542 (0x2cbf bytes) 000191 1777495535.989117 write 0x01659c thru 0x01da52 (0x74b7 bytes) 000192 1777495536.130458 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x391fc): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x391fc): Mirror 2: 0x3d2d28a3 (stale) CRC-32 checksum value for chunk [0, 0x391fc): Mirror 1: 0x53b2aac5 Mirror 3: 0x53b2aac5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x391fc: skipped 000193 1777495537.159195 trunc from 0x0391fc to 0x0019d7 (0x37824 bytes) 000194 1777495537.286988 read 0x000f23 thru 0x0019d7 (0xab5 bytes) 000197 1777495537.294637 mapwrite 0x00a009 thru 0x010773 (0x676b bytes) 000198 1777495537.471320 MIRROR_RESYNC 3 mirrors 000199 1777495538.337180 mapread 0x00d2ab thru 0x010773 (0x34c9 bytes) 000201 1777495538.356902 write 0x022a74 thru 0x02368f (0xc1c bytes) 000205 1777495538.450732 mapwrite 0x021032 thru 0x022a32 (0x1a01 bytes) 000206 1777495538.541778 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x23690): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23690): Mirror 2: 0x3930076a (stale) Mirror 3: 0x3930076a (stale) CRC-32 checksum value for chunk [0, 0x23690): Mirror 1: 0x559327ae Mirror 4: 0x559327ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23690: skipped 000208 1777495539.306004 mapread 0x01f9c6 thru 0x02368f (0x3cca bytes) 000209 1777495539.350576 trunc from 0x023690 to 0x016a85 (0xcc0a bytes) 000211 1777495539.481020 trunc from 0x016a86 to 0x0057ab (0x112da bytes) 000214 1777495539.509897 read 0x00272f thru 0x004e6a (0x273c bytes) 000215 1777495539.518163 trunc from 0x0057ac to 0x03699d (0x311f2 bytes) 000218 1777495539.545907 read 0x028eb7 thru 0x03699d (0xdae7 bytes) 000219 1777495539.565511 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3699e): Mirror 2: 0xba8f89c5 (stale) Mirror 3: 0xba8f89c5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3699e): Mirror 2: 0xba8f89c5 (stale) Mirror 3: 0xba8f89c5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3699e: skipped 000221 1777495539.941558 mapread 0x014301 thru 0x016ae7 (0x27e7 bytes) 000224 1777495539.977552 write 0x0234ad thru 0x02d7cc (0xa320 bytes) 000226 1777495540.009868 trunc from 0x03699e to 0x016304 (0x20699 bytes) 000227 1777495540.046121 write 0x02f061 thru 0x03112f (0x20cf bytes) 000228 1777495540.060548 MIRROR_RESYNC 3 mirrors 000229 1777495540.928584 mapwrite 0x00186c thru 0x00cb2c (0xb2c1 bytes) 000230 1777495541.182088 mapwrite 0x011a38 thru 0x0150db (0x36a4 bytes) 000231 1777495541.278822 read 0x015741 thru 0x0249d2 (0xf292 bytes) 000233 1777495541.295487 read 0x00664e thru 0x016467 (0xfe1a bytes) 000234 1777495541.296675 write 0x02838f thru 0x03633c (0xdfae bytes) 000236 1777495541.318539 write 0x025183 thru 0x0339c4 (0xe842 bytes) 000239 1777495541.336235 MIRROR_RESYNC 3 mirrors 000240 1777495542.250567 read 0x01494d thru 0x01a5ce (0x5c82 bytes) 000241 1777495542.263944 trunc from 0x03633d to 0x008217 (0x2e125 bytes) 000243 1777495542.472429 mapread 0x00814c thru 0x008217 (0xcc bytes) 000244 1777495542.485861 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x8218): Mirror 3: 0xb7cbc4d2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8218): Mirror 3: 0xb7cbc4d2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8218: skipped 000247 1777495542.847349 trunc from 0x008218 to 0x00c64f (0x4438 bytes) 000249 1777495542.869155 write 0x019642 thru 0x024205 (0xabc4 bytes) 000250 1777495542.879139 trunc from 0x024206 to 0x030370 (0xc16b bytes) 000251 1777495542.920958 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x30371): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30371): Mirror 3: 0xaf686861 (stale) CRC-32 checksum value for chunk [0, 0x30371): Mirror 1: 0xb50d4794 Mirror 4: 0xb50d4794 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30371: skipped 000253 1777495544.186970 write 0x0142c5 thru 0x018ecb (0x4c07 bytes) 000257 1777495544.357919 trunc from 0x030371 to 0x025d8c (0xa5e4 bytes) 000259 1777495544.377754 write 0x014061 thru 0x014795 (0x735 bytes) 000260 1777495544.378497 mapread 0x00703b thru 0x01034b (0x9311 bytes) 000262 1777495544.402865 MIRROR_RESYNC 3 mirrors 000263 1777495545.239304 read 0x00f852 thru 0x0193a0 (0x9b4f bytes) 000264 1777495545.268795 read 0x00e375 thru 0x016255 (0x7ee1 bytes) 000265 1777495545.287865 trunc from 0x025d8d to 0x03729a (0x1150e bytes) 000268 1777495545.452754 read 0x019c4e thru 0x0277c8 (0xdb7b bytes) 000269 1777495545.466537 mapread 0x009dc5 thru 0x014842 (0xaa7e bytes) 000271 1777495545.469017 read 0x01fe30 thru 0x023976 (0x3b47 bytes) 000272 1777495545.470440 read 0x015897 thru 0x024049 (0xe7b3 bytes) 000275 1777495545.473177 read 0x025aab thru 0x0293c2 (0x3918 bytes) 000276 1777495545.474109 read 0x034b31 thru 0x03729a (0x276a bytes) 000277 1777495545.480864 MIRROR_RESYNC 3 mirrors 000279 1777495546.290589 read 0x034851 thru 0x03729a (0x2a4a bytes) 000280 1777495546.300517 trunc from 0x03729b to 0x015e69 (0x21431 bytes) 000281 1777495546.535440 write 0x01c0f7 thru 0x02689c (0xa7a6 bytes) 000282 1777495546.539117 trunc from 0x02689d to 0x031bc5 (0xb329 bytes) 000283 1777495546.578216 read 0x015779 thru 0x01ca22 (0x72aa bytes) 000285 1777495546.600499 write 0x02f051 thru 0x032202 (0x31b2 bytes) 000289 1777495546.614093 mapread 0x010847 thru 0x01ed77 (0xe531 bytes) 000291 1777495546.618730 trunc from 0x032203 to 0x02b46e (0x6d94 bytes) 000293 1777495546.665362 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b46f): Mirror 4: 0x228b120d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b46f): Mirror 4: 0x228b120d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b46f: skipped 000295 1777495547.107712 read 0x021845 thru 0x0248a3 (0x305f bytes) 000298 1777495547.132867 MIRROR_RESYNC 2 mirrors 000300 1777495547.884986 write 0x017b8f thru 0x025bb6 (0xe028 bytes) 000302 1777495547.976476 mapwrite 0x013e60 thru 0x018f78 (0x5119 bytes) 000304 1777495548.040435 read 0x016128 thru 0x01cde8 (0x6cc1 bytes) 000308 1777495548.043807 mapwrite 0x011659 thru 0x0166bd (0x5065 bytes) 000309 1777495548.086284 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) 000314 1777495548.238104 trunc from 0x02b46f to 0x030aad (0x563f bytes) 000315 1777495548.263041 trunc from 0x030aae to 0x01cbd3 (0x13eda bytes) 000316 1777495548.282710 mapread 0x005cd9 thru 0x009331 (0x3659 bytes) 000317 1777495548.303995 mapread 0x012451 thru 0x01cbd3 (0xa783 bytes) 000318 1777495548.327506 trunc from 0x01cbd4 to 0x00f464 (0xd76f bytes) 000320 1777495548.378097 write 0x004c1a thru 0x010cc6 (0xc0ad bytes) 000321 1777495548.386638 trunc from 0x010cc7 to 0x01097f (0x347 bytes) 000322 1777495548.415052 trunc from 0x010980 to 0x023c11 (0x13292 bytes) 000323 1777495548.440721 mapread 0x004b55 thru 0x012c5e (0xe10a bytes) 000324 1777495548.464514 trunc from 0x023c12 to 0x0075c4 (0x1c64d bytes) 000326 1777495548.492345 write 0x03ee3a thru 0x03ef89 (0x150 bytes) 000329 1777495548.496212 read 0x0092ec thru 0x00eeb4 (0x5bc9 bytes) 000330 1777495548.514781 write 0x036596 thru 0x038107 (0x1b72 bytes) 000332 1777495548.530384 read 0x009f84 thru 0x018e4e (0xeecb bytes) 000333 1777495548.541676 mapwrite 0x01720d thru 0x0239af (0xc7a3 bytes) 000337 1777495548.608882 mapread 0x01cc14 thru 0x021e17 (0x5204 bytes) 000338 1777495548.611586 write 0x01ddfe thru 0x024b93 (0x6d96 bytes) 000339 1777495548.614449 mapread 0x00a4e8 thru 0x00a916 (0x42f bytes) 000346 1777495548.624443 mapwrite 0x02b6d8 thru 0x034994 (0x92bd bytes) 000348 1777495548.696722 mapread 0x037738 thru 0x038a44 (0x130d bytes) 000353 1777495548.710753 mapwrite 0x0375f7 thru 0x03ffff (0x8a09 bytes) 000355 1777495548.862904 mapwrite 0x006870 thru 0x010e57 (0xa5e8 bytes) 000356 1777495548.966137 trunc from 0x040000 to 0x03ba74 (0x458b bytes) 000357 1777495548.992953 write 0x0155db thru 0x021ec5 (0xc8eb bytes) 000358 1777495548.997706 mapread 0x021c52 thru 0x02dca3 (0xc052 bytes) 000361 1777495549.001402 write 0x03127b thru 0x034552 (0x32d8 bytes) 000363 1777495549.012843 mapwrite 0x025ea0 thru 0x033d20 (0xde81 bytes) 000367 1777495549.085725 write 0x037ef0 thru 0x03ffff (0x8110 bytes) 000368 1777495549.090932 write 0x0091c6 thru 0x00b348 (0x2183 bytes) 000371 1777495549.101927 write 0x02036f thru 0x0262f3 (0x5f85 bytes) 000373 1777495549.105186 mapread 0x00475d thru 0x00a0b9 (0x595d bytes) 000374 1777495549.106402 mapread 0x02c9b7 thru 0x03b95f (0xefa9 bytes) 000375 1777495549.120038 trunc from 0x040000 to 0x00f73e (0x308c1 bytes) 000378 1777495549.166143 read 0x0073dc thru 0x00f52b (0x8150 bytes) 000380 1777495549.172016 mapread 0x002cb1 thru 0x00f73e (0xca8e bytes) 000381 1777495549.174154 mapread 0x007145 thru 0x00f320 (0x81dc bytes) 000382 1777495549.176036 write 0x01fb51 thru 0x026e39 (0x72e9 bytes) 000383 1777495549.178159 trunc from 0x026e3a to 0x008bbc (0x1e27d bytes) 000384 1777495549.222024 read 0x0055c7 thru 0x008bbc (0x35f6 bytes) 000386 1777495549.229633 read 0x008872 thru 0x008bbc (0x34b bytes) 000387 1777495549.234486 mapread 0x0026f0 thru 0x008bbc (0x64cd bytes) 000388 1777495549.237808 write 0x029de6 thru 0x0318bb (0x7ad6 bytes) 000389 1777495549.246719 write 0x0256db thru 0x030231 (0xab57 bytes) 000398 1777495549.281028 write 0x00140d thru 0x008fae (0x7ba2 bytes) 000399 1777495549.282536 mapread 0x018a84 thru 0x023e4e (0xb3cb bytes) 000400 1777495549.283794 mapread 0x0250b7 thru 0x02df0b (0x8e55 bytes) 000402 1777495549.287466 trunc from 0x0318bc to 0x0164f7 (0x1b3c4 bytes) 000404 1777495549.310473 mapwrite 0x01a711 thru 0x01dbfd (0x34ed bytes) 000405 1777495549.512944 mapwrite 0x00813b thru 0x011784 (0x964a bytes) 000407 1777495549.642116 write 0x008cff thru 0x00b481 (0x2783 bytes) 000411 1777495549.646057 mapread 0x01d935 thru 0x01dbfd (0x2c9 bytes) 000413 1777495549.647898 mapread 0x00ed68 thru 0x01b464 (0xc6fd bytes) 000416 1777495549.683908 mapwrite 0x0123b4 thru 0x013ec4 (0x1b11 bytes) 000417 1777495549.844154 mapwrite 0x019921 thru 0x01e8cd (0x4fad bytes) 000419 1777495550.029199 mapwrite 0x0349f3 thru 0x03e868 (0x9e76 bytes) 000421 1777495550.164422 mapread 0x0334b5 thru 0x039e06 (0x6952 bytes) 000423 1777495550.170913 mapread 0x0068eb thru 0x00ba12 (0x5128 bytes) 000427 1777495550.178630 write 0x0106c0 thru 0x012b9a (0x24db bytes) 000429 1777495550.180225 read 0x03e77c thru 0x03e868 (0xed bytes) 000431 1777495550.198775 read 0x000abd thru 0x00a420 (0x9964 bytes) 000436 1777495550.208923 mapread 0x035386 thru 0x03e868 (0x94e3 bytes) 000437 1777495550.211498 trunc from 0x03e869 to 0x012fd4 (0x2b894 bytes) truncating to largest ever: 0x3bf39 000439 1777495550.264435 trunc from 0x012fd5 to 0x03bf38 (0x28f64 bytes) 000443 1777495550.308489 mapwrite 0x02250e thru 0x02cdd8 (0xa8cb bytes) 000445 1777495550.461162 write 0x01d106 thru 0x0242a6 (0x71a1 bytes) 000446 1777495550.462729 mapread 0x02aa17 thru 0x03576d (0xad57 bytes) 000448 1777495550.468321 mapread 0x02b835 thru 0x02fff1 (0x47bd bytes) 000450 1777495550.472956 read 0x03312c thru 0x03469a (0x156f bytes) 000451 1777495550.474356 trunc from 0x03bf39 to 0x0289c9 (0x1356f bytes) 000452 1777495550.507812 mapwrite 0x028dd0 thru 0x02c0b4 (0x32e5 bytes) 000453 1777495550.592700 mapwrite 0x015f72 thru 0x0220a5 (0xc134 bytes) 000455 1777495550.662123 write 0x023736 thru 0x02b88a (0x8155 bytes) 000457 1777495550.665549 read 0x0279a0 thru 0x02c0b4 (0x4715 bytes) 000458 1777495550.673506 MIRROR_EXTEND to 2 mirrors 000459 1777495551.377456 trunc from 0x02c0b5 to 0x005053 (0x27061 bytes) 000460 1777495551.498497 write 0x02ba18 thru 0x036eac (0xb495 bytes) 000466 1777495551.504093 trunc from 0x036ead to 0x01b0e4 (0x1bdc8 bytes) 000467 1777495551.545577 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) 000468 1777495551.718352 trunc from 0x01b0e5 to 0x00c212 (0xeed2 bytes) 000469 1777495551.736563 mapread 0x00a56a thru 0x00c212 (0x1ca9 bytes) 000470 1777495551.756092 trunc from 0x00c213 to 0x03389a (0x27688 bytes) 000471 1777495551.771633 read 0x009efc thru 0x013817 (0x991c bytes) 000473 1777495551.794343 mapwrite 0x028cdf thru 0x02f613 (0x6935 bytes) 000474 1777495551.868663 write 0x03eb7d thru 0x03ffff (0x1483 bytes) 000477 1777495551.872248 mapwrite 0x02f0ae thru 0x035fbd (0x6f10 bytes) 000478 1777495551.928453 mapwrite 0x02c55d thru 0x03a327 (0xddcb bytes) 000479 1777495552.067638 write 0x009493 thru 0x011add (0x864b bytes) 000480 1777495552.073890 MIRROR_EXTEND to 2 mirrors 000481 1777495552.834519 mapwrite 0x018e07 thru 0x0199de (0xbd8 bytes) 000482 1777495553.067839 mapread 0x01b647 thru 0x025464 (0x9e1e bytes) 000485 1777495553.094358 read 0x000db6 thru 0x007847 (0x6a92 bytes) 000489 1777495553.111062 mapread 0x025f52 thru 0x0339ba (0xda69 bytes) 000491 1777495553.142392 mapwrite 0x028525 thru 0x0357e6 (0xd2c2 bytes) 000493 1777495553.289802 write 0x0141dd thru 0x01e179 (0x9f9d bytes) 000494 1777495553.291084 write 0x030db4 thru 0x036f01 (0x614e bytes) 000497 1777495553.298329 mapwrite 0x01bd13 thru 0x01d1a4 (0x1492 bytes) 000498 1777495553.389752 write 0x020afe thru 0x023d64 (0x3267 bytes) 000500 1777495553.396161 write 0x00121b thru 0x002a11 (0x17f7 bytes) 000503 1777495553.409191 write 0x00b35e thru 0x0134f8 (0x819b bytes) 000508 1777495553.432490 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) 000510 1777495553.658457 mapread 0x017812 thru 0x01a3fa (0x2be9 bytes) 000511 1777495553.680193 mapread 0x015b1f thru 0x01e848 (0x8d2a bytes) 000513 1777495553.682197 mapwrite 0x0384bc thru 0x03ffff (0x7b44 bytes) 000514 1777495553.735359 mapread 0x02ccf9 thru 0x039c59 (0xcf61 bytes) 000517 1777495553.748467 mapwrite 0x02ef3a thru 0x03eecf (0xff96 bytes) 000518 1777495553.797462 mapwrite 0x030585 thru 0x03d912 (0xd38e bytes) 000522 1777495553.910376 mapwrite 0x00e11b thru 0x018514 (0xa3fa bytes) 000524 1777495554.006994 write 0x0255d4 thru 0x02607a (0xaa7 bytes) 000525 1777495554.010030 write 0x0338ac thru 0x036999 (0x30ee bytes) 000526 1777495554.016945 write 0x02f118 thru 0x03dc7f (0xeb68 bytes) 000531 1777495554.029180 read 0x00f894 thru 0x012d56 (0x34c3 bytes) 000532 1777495554.030597 write 0x028f68 thru 0x02d3b3 (0x444c bytes) 000534 1777495554.034432 trunc from 0x040000 to 0x033f76 (0xc089 bytes) 000535 1777495554.061233 MIRROR_EXTEND to 2 mirrors 000538 1777495555.048334 read 0x00971d thru 0x0190c5 (0xf9a9 bytes) 000542 1777495555.079472 trunc from 0x033f77 to 0x031873 (0x2703 bytes) 000545 1777495555.212792 read 0x00f432 thru 0x01c594 (0xd163 bytes) 000546 1777495555.244410 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x31874): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31874): Mirror 2: 0x856434fb (stale) CRC-32 checksum value for chunk [0, 0x31874): Mirror 1: 0x856434fb Mirror 3: 0x856434fb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31874: skipped 000547 1777495556.271364 mapwrite 0x0027b8 thru 0x00d634 (0xae7d bytes) 000548 1777495556.558623 mapwrite 0x00d9bf thru 0x018379 (0xa9bb bytes) 000549 1777495556.650213 mapwrite 0x00d7dd thru 0x01cb67 (0xf38b bytes) 000550 1777495556.755903 read 0x018507 thru 0x01d08a (0x4b84 bytes) 000552 1777495556.762905 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31874): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31874): Mirror 2: 0x856434fb (stale) Mirror 3: 0x856434fb (stale) CRC-32 checksum value for chunk [0, 0x31874): Mirror 1: 0x7707d750 Mirror 4: 0x7707d750 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31874: skipped 000553 1777495558.073592 read 0x0230a3 thru 0x028925 (0x5883 bytes) 000554 1777495558.086742 mapwrite 0x0079d0 thru 0x00bf0c (0x453d bytes) 000556 1777495558.342450 write 0x014b40 thru 0x01e8f2 (0x9db3 bytes) 000559 1777495558.393456 trunc from 0x031874 to 0x014fc8 (0x1c8ab bytes) 000560 1777495558.465454 trunc from 0x014fc9 to 0x038019 (0x23051 bytes) 000563 1777495558.523309 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3801a): Mirror 2: 0x13d9c91e (stale) Mirror 3: 0xfb7014eb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3801a): Mirror 2: 0x13d9c91e (stale) Mirror 3: 0xfb7014eb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3801a: skipped 000565 1777495559.131778 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3801a): Mirror 3: 0xfb7014eb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3801a): Mirror 3: 0xfb7014eb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3801a: skipped 000566 1777495559.555148 read 0x0036fb thru 0x00c9a9 (0x92af bytes) 000571 1777495559.572205 trunc from 0x03801a to 0x01703c (0x20fdd bytes) 000572 1777495559.600716 mapread 0x0096b3 thru 0x01703c (0xd98a bytes) 000573 1777495559.621150 mapread 0x01226f thru 0x01703c (0x4dce bytes) 000574 1777495559.622495 mapwrite 0x03c5f5 thru 0x03ffff (0x3a0b bytes) 000575 1777495559.735062 trunc from 0x040000 to 0x00f313 (0x30cec bytes) 000577 1777495559.765290 mapwrite 0x01cf44 thru 0x02c405 (0xf4c2 bytes) 000579 1777495559.945445 trunc from 0x02c406 to 0x028a1b (0x39ea bytes) 000580 1777495559.971784 mapread 0x00f501 thru 0x016ea5 (0x79a5 bytes) 000581 1777495559.981809 mapread 0x003d62 thru 0x00c505 (0x87a4 bytes) 000583 1777495559.983538 mapwrite 0x00db90 thru 0x0135f7 (0x5a68 bytes) 000584 1777495560.086040 mapwrite 0x03e26c thru 0x03ffff (0x1d94 bytes) 000585 1777495560.205489 write 0x025329 thru 0x031519 (0xc1f1 bytes) 000586 1777495560.220514 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) 000588 1777495560.478617 write 0x0087a9 thru 0x0108f4 (0x814c bytes) 000589 1777495560.502970 read 0x024a52 thru 0x02a313 (0x58c2 bytes) 000590 1777495560.520931 mapread 0x02b1a0 thru 0x03051d (0x537e bytes) 000592 1777495560.553084 write 0x03ff83 thru 0x03ffff (0x7d bytes) 000593 1777495560.569855 write 0x000bcf thru 0x0065e3 (0x5a15 bytes) 000595 1777495560.594192 mapread 0x01f720 thru 0x02e69e (0xef7f bytes) 000596 1777495560.617609 write 0x03b11b thru 0x03ffff (0x4ee5 bytes) 000597 1777495560.622467 mapread 0x00ab96 thru 0x00f526 (0x4991 bytes) 000599 1777495560.635433 MIRROR_EXTEND to 2 mirrors 000600 1777495561.499040 trunc from 0x040000 to 0x02ad71 (0x1528e bytes) 000601 1777495561.643927 mapwrite 0x00446f thru 0x007de0 (0x3972 bytes) 000602 1777495561.753793 mapread 0x02810a thru 0x02ad71 (0x2c68 bytes) 000603 1777495561.767354 trunc from 0x02ad72 to 0x02b1fa (0x489 bytes) 000605 1777495561.801815 trunc from 0x02b1fb to 0x00aa7c (0x2077e bytes) 000606 1777495561.834358 trunc from 0x00aa7d to 0x031145 (0x266c9 bytes) 000607 1777495561.872587 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) 000612 1777495562.139173 write 0x03ee16 thru 0x03ffff (0x11ea bytes) 000613 1777495562.142765 mapread 0x01ac0a thru 0x01ddcc (0x31c3 bytes) 000615 1777495562.181519 mapread 0x01f90a thru 0x026e9c (0x7593 bytes) 000616 1777495562.206270 mapwrite 0x005ea9 thru 0x0145c8 (0xe720 bytes) 000617 1777495562.373537 mapwrite 0x031300 thru 0x035870 (0x4571 bytes) 000618 1777495562.522093 trunc from 0x040000 to 0x016a42 (0x295bd bytes) 000621 1777495562.563846 trunc from 0x016a43 to 0x033e5b (0x1d419 bytes) 000623 1777495562.615223 write 0x00e099 thru 0x010ed0 (0x2e38 bytes) 000634 1777495562.648255 read 0x013073 thru 0x01b66e (0x85fc bytes) 000638 1777495562.668552 trunc from 0x033e5c to 0x010d10 (0x2314b bytes) 000643 1777495562.706586 trunc from 0x010d11 to 0x01e6e6 (0xd9d6 bytes) 000645 1777495562.732845 write 0x019317 thru 0x025d9a (0xca84 bytes) 000646 1777495562.746563 mapwrite 0x03453b thru 0x03ffff (0xbac5 bytes) 000647 1777495562.880267 mapwrite 0x02067a thru 0x026e5a (0x67e1 bytes) 000651 1777495562.958117 mapread 0x013a17 thru 0x02367b (0xfc65 bytes) 000652 1777495562.962117 read 0x004554 thru 0x005e9c (0x1949 bytes) 000653 1777495562.963292 write 0x00cec6 thru 0x0108ff (0x3a3a bytes) 000654 1777495562.965791 mapread 0x02a460 thru 0x0389cd (0xe56e bytes) 000656 1777495562.969111 mapread 0x00fa77 thru 0x01321a (0x37a4 bytes) 000657 1777495562.979548 write 0x018339 thru 0x01de2f (0x5af7 bytes) 000661 1777495562.987273 trunc from 0x040000 to 0x0067fe (0x39801 bytes) 000662 1777495563.011254 mapread 0x005c67 thru 0x0067fe (0xb98 bytes) 000664 1777495563.013080 write 0x011e38 thru 0x01b956 (0x9b1f bytes) 000665 1777495563.018171 mapread 0x00c5c2 thru 0x01b0e6 (0xeb25 bytes) 000666 1777495563.047183 read 0x00c53b thru 0x011b7a (0x5640 bytes) 000671 1777495563.058167 MIRROR_EXTEND to 2 mirrors 000673 1777495564.055500 read 0x00e4dd thru 0x00fa87 (0x15ab bytes) 000675 1777495564.073013 read 0x01194e thru 0x0186cc (0x6d7f bytes) 000676 1777495564.087507 mapwrite 0x03d7d9 thru 0x03ffff (0x2827 bytes) 000677 1777495564.378270 mapwrite 0x03f824 thru 0x03ffff (0x7dc bytes) 000678 1777495564.501629 read 0x03b263 thru 0x03ffff (0x4d9d bytes) 000680 1777495564.507269 mapread 0x018c0b thru 0x01a99a (0x1d90 bytes) 000681 1777495564.527231 mapwrite 0x005a61 thru 0x011752 (0xbcf2 bytes) 000682 1777495564.686255 trunc from 0x040000 to 0x03126f (0xed90 bytes) 000686 1777495564.756364 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x31270): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31270): Mirror 2: 0xe8d61ca1 (stale) CRC-32 checksum value for chunk [0, 0x31270): Mirror 1: 0xe36e1caa Mirror 3: 0xe36e1caa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31270: skipped 000687 1777495566.007086 mapread 0x008fc7 thru 0x00f160 (0x619a bytes) 000690 1777495566.054836 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) 000691 1777495566.530525 MIRROR_EXTEND to 3 mirrors 000692 1777495567.530153 trunc from 0x031270 to 0x030500 (0xd6f bytes) 000693 1777495567.660871 read 0x01007e thru 0x013490 (0x3413 bytes) 000698 1777495567.700540 trunc from 0x030501 to 0x010529 (0x1ffd7 bytes) 000699 1777495567.742517 mapread 0x00ea43 thru 0x010529 (0x1ae7 bytes) 000701 1777495567.768691 mapwrite 0x027c7c thru 0x02a792 (0x2b17 bytes) 000702 1777495567.873576 MIRROR_RESYNC 3 mirrors 000704 1777495569.179132 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) truncating to largest ever: 0x3d37e 000706 1777495569.705636 trunc from 0x02a793 to 0x03d37d (0x12beb bytes) 000707 1777495569.810961 mapread 0x0174ee thru 0x0270d3 (0xfbe6 bytes) 000709 1777495569.845186 write 0x017206 thru 0x01f120 (0x7f1b bytes) 000710 1777495569.847307 mapread 0x0245e6 thru 0x02a5bc (0x5fd7 bytes) 000711 1777495569.849274 write 0x02cfad thru 0x02d492 (0x4e6 bytes) 000713 1777495569.866263 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) 000715 1777495570.072589 read 0x01280c thru 0x021dc7 (0xf5bc bytes) 000717 1777495570.087386 read 0x0324e3 thru 0x03d37d (0xae9b bytes) 000718 1777495570.105470 MIRROR_EXTEND to 2 mirrors 000721 1777495571.067529 mapwrite 0x015047 thru 0x01de90 (0x8e4a bytes) 000722 1777495571.355733 read 0x031fa3 thru 0x03a476 (0x84d4 bytes) 000723 1777495571.365638 mapwrite 0x01a399 thru 0x01de56 (0x3abe bytes) 000724 1777495571.426076 write 0x00980d thru 0x00e0f6 (0x48ea bytes) 000727 1777495571.448604 MIRROR_RESYNC 2 mirrors 000728 1777495572.363470 mapwrite 0x0388c5 thru 0x03ffff (0x773b bytes) 000729 1777495572.654047 read 0x005e52 thru 0x00f572 (0x9721 bytes) 000731 1777495572.665741 mapwrite 0x02d101 thru 0x02f1bf (0x20bf bytes) 000734 1777495572.786330 read 0x00f836 thru 0x01b808 (0xbfd3 bytes) 000735 1777495572.818943 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) 000736 1777495573.030331 mapread 0x0199a0 thru 0x01e6ea (0x4d4b bytes) 000737 1777495573.084456 mapwrite 0x00f42f thru 0x016fe2 (0x7bb4 bytes) 000740 1777495573.259785 mapwrite 0x011df7 thru 0x018999 (0x6ba3 bytes) 000741 1777495573.406255 trunc from 0x040000 to 0x024a12 (0x1b5ed bytes) truncating to largest ever: 0x3fa40 000742 1777495573.441450 trunc from 0x024a13 to 0x03fa3f (0x1b02d bytes) 000743 1777495573.492668 mapread 0x00d77b thru 0x00f4f6 (0x1d7c bytes) 000744 1777495573.502396 write 0x010281 thru 0x011608 (0x1388 bytes) 000746 1777495573.510436 mapread 0x00da63 thru 0x01a0d2 (0xc670 bytes) 000747 1777495573.511726 read 0x026db9 thru 0x02dc22 (0x6e6a bytes) 000748 1777495573.534596 write 0x005022 thru 0x01245e (0xd43d bytes) 000750 1777495573.560673 write 0x02cbf8 thru 0x03c0d2 (0xf4db bytes) 000751 1777495573.573272 mapwrite 0x0243fe thru 0x031f1d (0xdb20 bytes) 000752 1777495573.750067 mapwrite 0x010ca8 thru 0x01d458 (0xc7b1 bytes) 000753 1777495573.895237 mapread 0x03090b thru 0x03c20f (0xb905 bytes) 000758 1777495573.904159 mapwrite 0x03b147 thru 0x03ffff (0x4eb9 bytes) 000759 1777495574.051233 read 0x03882b thru 0x03b6b2 (0x2e88 bytes) 000760 1777495574.051936 read 0x020075 thru 0x02df8f (0xdf1b bytes) 000764 1777495574.057017 mapwrite 0x01f64b thru 0x02394f (0x4305 bytes) 000767 1777495574.160840 mapread 0x00cb27 thru 0x011da8 (0x5282 bytes) 000768 1777495574.172428 mapwrite 0x035e9e thru 0x03d88f (0x79f2 bytes) 000769 1777495574.279862 read 0x004666 thru 0x005e28 (0x17c3 bytes) 000770 1777495574.281478 mapwrite 0x0107c8 thru 0x01edec (0xe625 bytes) 000771 1777495574.401287 write 0x03e8aa thru 0x03ffff (0x1756 bytes) 000776 1777495574.416179 write 0x00db76 thru 0x010fae (0x3439 bytes) 000778 1777495574.423349 read 0x0301b2 thru 0x03cd5c (0xcbab bytes) 000779 1777495574.428222 write 0x000360 thru 0x0067e6 (0x6487 bytes) 000787 1777495574.460225 mapwrite 0x016019 thru 0x018a72 (0x2a5a bytes) 000788 1777495574.578152 mapwrite 0x02c966 thru 0x039678 (0xcd13 bytes) 000790 1777495574.744410 mapwrite 0x013c1b thru 0x014ca7 (0x108d bytes) 000791 1777495574.792757 write 0x035889 thru 0x03ffff (0xa777 bytes) 000794 1777495574.812833 MIRROR_EXTEND to 2 mirrors 000796 1777495575.944554 write 0x025745 thru 0x0296c4 (0x3f80 bytes) 000798 1777495576.120623 trunc from 0x040000 to 0x00f888 (0x30777 bytes) 000799 1777495576.194998 write 0x02e055 thru 0x033ed8 (0x5e84 bytes) 000801 1777495576.207311 mapwrite 0x02f258 thru 0x03d096 (0xde3f bytes) 000804 1777495576.415422 mapwrite 0x01eb16 thru 0x028cfe (0xa1e9 bytes) 000806 1777495576.540245 read 0x0254f1 thru 0x02a04d (0x4b5d bytes) 000811 1777495576.544102 write 0x03c767 thru 0x03ffff (0x3899 bytes) 000814 1777495576.551785 MIRROR_RESYNC 2 mirrors 000815 1777495577.253435 write 0x01abfb thru 0x01b95e (0xd64 bytes) 000817 1777495577.351248 read 0x0169de thru 0x0236f3 (0xcd16 bytes) 000820 1777495577.369290 mapwrite 0x0071f5 thru 0x0120a7 (0xaeb3 bytes) 000822 1777495577.470408 mapread 0x01dc68 thru 0x02a3eb (0xc784 bytes) 000823 1777495577.488058 read 0x02efaa thru 0x033a70 (0x4ac7 bytes) 000825 1777495577.518305 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) 000828 1777495577.759261 read 0x004c1c thru 0x01168a (0xca6f bytes) 000829 1777495577.780345 mapwrite 0x0183b5 thru 0x01cf90 (0x4bdc bytes) 000830 1777495577.953262 read 0x023538 thru 0x02af9e (0x7a67 bytes) 000832 1777495577.968284 read 0x011ecf thru 0x01b6f7 (0x9829 bytes) 000834 1777495577.971909 mapread 0x036b08 thru 0x03fe45 (0x933e bytes) 000837 1777495577.975976 trunc from 0x040000 to 0x019363 (0x26c9c bytes) 000838 1777495578.042031 mapwrite 0x035870 thru 0x03c35b (0x6aec bytes) 000839 1777495578.230282 mapwrite 0x0108ad thru 0x01ceb4 (0xc608 bytes) 000840 1777495578.315518 read 0x02f22b thru 0x0366a1 (0x7477 bytes) 000841 1777495578.316504 mapread 0x021f65 thru 0x02bfb7 (0xa053 bytes) 000842 1777495578.319489 mapwrite 0x016f91 thru 0x01bafb (0x4b6b bytes) 000843 1777495578.420744 read 0x02f7ae thru 0x03218c (0x29df bytes) 000844 1777495578.422658 mapread 0x02fde4 thru 0x033568 (0x3785 bytes) 000845 1777495578.425712 read 0x005382 thru 0x014308 (0xef87 bytes) 000848 1777495578.429443 mapread 0x006758 thru 0x009c88 (0x3531 bytes) 000849 1777495578.431453 mapread 0x007da6 thru 0x00df33 (0x618e bytes) 000850 1777495578.433571 read 0x00acc9 thru 0x015b50 (0xae88 bytes) 000852 1777495578.435290 write 0x03573a thru 0x03d1be (0x7a85 bytes) 000853 1777495578.437355 mapwrite 0x025fd5 thru 0x02d3a9 (0x73d5 bytes) 000854 1777495578.512544 mapread 0x004161 thru 0x0095e9 (0x5489 bytes) 000855 1777495578.514557 write 0x026369 thru 0x030343 (0x9fdb bytes) 000856 1777495578.516703 mapwrite 0x03d7c7 thru 0x03ffff (0x2839 bytes) 000857 1777495578.633481 mapread 0x00a41f thru 0x00be69 (0x1a4b bytes) 000859 1777495578.637848 write 0x030c01 thru 0x039be3 (0x8fe3 bytes) 000861 1777495578.642962 trunc from 0x040000 to 0x03d021 (0x2fde bytes) 000862 1777495578.670369 mapwrite 0x033ea0 thru 0x03b61e (0x777f bytes) 000864 1777495578.767994 read 0x017090 thru 0x02298d (0xb8fe bytes) 000865 1777495578.769529 trunc from 0x03d022 to 0x010139 (0x2cee8 bytes) 000868 1777495578.800314 mapread 0x00242a thru 0x00fa03 (0xd5da bytes) 000869 1777495578.808883 MIRROR_EXTEND to 2 mirrors 000870 1777495579.916196 mapread 0x00f698 thru 0x010139 (0xaa2 bytes) 000873 1777495579.939217 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) 000874 1777495580.174488 read 0x003699 thru 0x0047f1 (0x1159 bytes) 000879 1777495580.199000 write 0x03ed63 thru 0x03ffff (0x129d bytes) 000880 1777495580.210955 mapread 0x016b54 thru 0x0194c9 (0x2976 bytes) 000882 1777495580.229622 write 0x01ff08 thru 0x0265bd (0x66b6 bytes) 000883 1777495580.237917 write 0x03a6a9 thru 0x03b677 (0xfcf bytes) 000886 1777495580.255386 mapwrite 0x01c6f6 thru 0x022160 (0x5a6b bytes) 000889 1777495580.401226 write 0x023313 thru 0x02664c (0x333a bytes) 000894 1777495580.408895 mapread 0x03ba42 thru 0x03ffff (0x45be bytes) 000896 1777495580.411740 write 0x032a71 thru 0x033997 (0xf27 bytes) 000897 1777495580.433199 mapread 0x018ca1 thru 0x023f25 (0xb285 bytes) 000899 1777495580.445053 mapread 0x039d52 thru 0x03add3 (0x1082 bytes) 000900 1777495580.449362 trunc from 0x040000 to 0x01456b (0x2ba94 bytes) 000901 1777495580.493156 write 0x008876 thru 0x01359a (0xad25 bytes) 000903 1777495580.524991 mapread 0x004669 thru 0x0119f0 (0xd388 bytes) 000904 1777495580.534042 read 0x008fb4 thru 0x01456b (0xb5b8 bytes) 000905 1777495580.550002 mapwrite 0x027076 thru 0x02a5c6 (0x3551 bytes) 000907 1777495580.706147 mapwrite 0x02985a thru 0x02f3eb (0x5b92 bytes) 000911 1777495580.888186 mapwrite 0x005714 thru 0x006b58 (0x1445 bytes) 000912 1777495580.987063 mapread 0x0171db thru 0x01fd3e (0x8b64 bytes) 000913 1777495580.988532 trunc from 0x02f3ec to 0x039fb9 (0xabce bytes) 000915 1777495581.029151 write 0x0046b0 thru 0x00761e (0x2f6f bytes) 000916 1777495581.037421 MIRROR_EXTEND to 2 mirrors 000921 1777495582.079321 write 0x014b8f thru 0x01ba72 (0x6ee4 bytes) 000922 1777495582.181402 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) 000925 1777495582.400765 read 0x038046 thru 0x039fb9 (0x1f74 bytes) 000926 1777495582.411491 write 0x00f2d8 thru 0x017a94 (0x87bd bytes) 000927 1777495582.428439 write 0x00d572 thru 0x01a320 (0xcdaf bytes) 000928 1777495582.450946 mapread 0x0081b1 thru 0x00cea9 (0x4cf9 bytes) 000929 1777495582.453895 write 0x019b1b thru 0x028949 (0xee2f bytes) 000932 1777495582.468008 read 0x01b023 thru 0x01ed67 (0x3d45 bytes) 000935 1777495582.472466 read 0x010ac4 thru 0x01c993 (0xbed0 bytes) 000936 1777495582.474744 mapread 0x038387 thru 0x039fb9 (0x1c33 bytes) 000937 1777495582.476924 mapwrite 0x01dca2 thru 0x027373 (0x96d2 bytes) 000939 1777495582.538284 read 0x014445 thru 0x016953 (0x250f bytes) 000940 1777495582.539105 trunc from 0x039fba to 0x0084c4 (0x31af5 bytes) 000941 1777495582.571139 mapread 0x004738 thru 0x007c07 (0x34d0 bytes) 000942 1777495582.599811 mapread 0x0081e9 thru 0x0084c4 (0x2dc bytes) 000945 1777495582.605338 write 0x03d152 thru 0x03ffff (0x2eae bytes) 000946 1777495582.614033 trunc from 0x040000 to 0x01db53 (0x224ac bytes) 000948 1777495582.649768 mapread 0x006f04 thru 0x0168bb (0xf9b8 bytes) 000949 1777495582.671800 read 0x000f3f thru 0x000fb2 (0x74 bytes) 000950 1777495582.681580 MIRROR_EXTEND to 2 mirrors 000951 1777495583.677368 mapread 0x008d0c thru 0x00e3cb (0x56c0 bytes) 000954 1777495583.741477 MIRROR_EXTEND to 3 mirrors 000956 1777495584.503443 MIRROR_EXTEND to 4 mirrors 000957 1777495585.525808 trunc from 0x01db54 to 0x018c09 (0x4f4a bytes) 000958 1777495585.756972 read 0x013885 thru 0x0160bd (0x2839 bytes) 000961 1777495585.778171 write 0x0173a5 thru 0x019097 (0x1cf3 bytes) 000962 1777495585.793696 trunc from 0x019098 to 0x034f38 (0x1bea1 bytes) 000963 1777495585.842800 write 0x02e818 thru 0x030098 (0x1881 bytes) 000964 1777495585.881139 read 0x025581 thru 0x032e7a (0xd8fa bytes) 000965 1777495585.895721 mapwrite 0x0142c6 thru 0x01fcca (0xba05 bytes) 000966 1777495586.042306 mapread 0x0067fb thru 0x010ee9 (0xa6ef bytes) 000969 1777495586.050368 read 0x026905 thru 0x030a17 (0xa113 bytes) 000970 1777495586.054218 mapwrite 0x0057ad thru 0x0131f1 (0xda45 bytes) 000971 1777495586.222236 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34f39): Mirror 3: 0xd71a0291 (stale) Mirror 4: 0xd71a0291 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34f39): Mirror 3: 0xd71a0291 (stale) Mirror 4: 0xd71a0291 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34f39: skipped 000974 1777495586.655857 mapwrite 0x034560 thru 0x03ffff (0xbaa0 bytes) 000975 1777495586.754057 mapwrite 0x036f31 thru 0x03d6fa (0x67ca bytes) 000978 1777495586.806280 read 0x005709 thru 0x00f24f (0x9b47 bytes) 000979 1777495586.819872 trunc from 0x040000 to 0x0013ad (0x3ec52 bytes) 000982 1777495586.849472 mapwrite 0x01dde5 thru 0x01f4cd (0x16e9 bytes) 000983 1777495586.942991 mapread 0x01082e thru 0x0169ae (0x6181 bytes) 000986 1777495586.964493 read 0x001a81 thru 0x0060b4 (0x4634 bytes) 000988 1777495586.994040 trunc from 0x01f4ce to 0x000445 (0x1f088 bytes) 000989 1777495587.016523 read 0x0002ef thru 0x000445 (0x157 bytes) 000991 1777495587.017978 trunc from 0x000446 to 0x002bd3 (0x278e bytes) 000992 1777495587.044383 mapwrite 0x03d44c thru 0x03ffff (0x2bb4 bytes) 000995 1777495587.218636 write 0x001742 thru 0x008bf1 (0x74b0 bytes) 000997 1777495587.229831 read 0x015e2d thru 0x01c49d (0x6671 bytes) 000998 1777495587.230725 write 0x004490 thru 0x012936 (0xe4a7 bytes) 000999 1777495587.238325 trunc from 0x040000 to 0x0240d2 (0x1bf2d bytes) 001000 1777495587.338495 trunc from 0x0240d3 to 0x0077be (0x1c914 bytes) All operations completed A-OK! PASS 70a (72s) == sanity-flr test 71: check mirror extend parent fid ==== 16:46:35 (1777495595) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 16:46:39 (1777495599) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x295:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x279:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.377337 s, 5.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x295:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x296:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x279:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339616 s, 3.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x295:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x296:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27a:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x279:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x295:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x296:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27a:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x279:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x297:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 16:46:50 (1777495610) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre2 Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2735808 .. Extending file size to 5510784 .. done Extending file size to 6826592 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9130752 .. Extending file size to 5577728 .. Extending file size to 7992288 .. Extending file size to 8564352 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9134496 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9332608 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 56396 56397 56398 56402 56403 done 192.168.204.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre2 (opts:) 192.168.204.113@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (84s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 16:48:14 (1777495694) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre2 Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5467808 .. Extending file size to 4981408 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7886368 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8390368 .. failed Extending file size to 8533536 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9068576 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8979360 .. Waiting 57993 57995 57997 57998 57999 done 192.168.204.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre2 (opts:) 192.168.204.113@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (85s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 16:49:40 (1777495780) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.112427 s, 18.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.480892 s, 4.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x3e566adb (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 16:50:01 (1777495801) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.155171 s, 13.5 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 16:50:08 (1777495808) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x290:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x290:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x291:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2ac:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 16:50:15 (1777495815) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.270254 s, 7.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x292:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ad:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 16:50:24 (1777495824) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.192675 s, 10.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x293:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x294:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2af:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x293:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x294:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2af:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b0:0x0] } PASS 204b (11s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 16:50:35 (1777495835) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.316282 s, 6.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x295:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x295:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b2:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 16:51:10 (1777495870) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg413-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179735 s, 5.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x296:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x296:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x297:0x0] } PASS 204d (63s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 16:52:13 (1777495933) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg413-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.431766 s, 4.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x298:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x298:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (61s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 16:53:14 (1777495994) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg413-server mds-ost sync done. Creating new pool test_204f oleg413-server: Pool lustre.test_204f created Adding targets to pool oleg413-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg413-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.204712 s, 10.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x299:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x299:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg413-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg413-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg413-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 16:54:34 (1777496074) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 16:54:44 (1777496084) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 16:54:53 (1777496093) oleg413-server: Pool lustre.test_206 created Adding targets to pool oleg413-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg413-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg413-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg413-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg413-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 16:55:21 (1777496121) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100381 s, 10.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05598 s, 18.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xf1de7647 Mirror 3: 0xf1de7647 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 16:55:34 (1777496134) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 16:55:37 (1777496137) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 16:55:41 (1777496141) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.114326 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xa70000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x20c0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x20c0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x20c0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1e70000 Read this (4 bytes): BBBB 192.168.204.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre2 (opts:) PASS 209a (44s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 16:56:25 (1777496185) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0197994 s, 414 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 16:56:34 (1777496194) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323116 s, 32.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1b7 lmm_fid: [0x200000404:0x1b7:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1b7 lmm_fid: [0x200000404:0x1b7:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a4:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834359 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-29 16:56:36.000000000 -0400 Modify: 2026-04-29 16:56:36.000000000 -0400 Change: 2026-04-29 16:56:37.000000000 -0400 Birth: 2026-04-29 16:56:37.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 16:56:44 (1777496204) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 16:56:48 (1777496208) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 1.13014 s, 36.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0607012 s, 67.5 kB/s size after second write -rw-r--r-- 1 root root 4096 Apr 29 16:56 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b9:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Apr 29 16:56 /mnt/lustre/f211.sanity-flr PASS 211 (14s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 16:57:03 (1777496223) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (10s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 16:57:12 (1777496232) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1683 sec =========== 16:57:23 (1777496243) === sanity-flr: start cleanup 16:57:25 (1777496245) === === sanity-flr: finish cleanup 16:57:33 (1777496253) ===