-----============= acceptance-small: sanity-flr ============----- Fri Nov 14 15:56:41 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg130-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 15:56:55 (1763153815) === oleg130-client.virtnet: executing check_config_client /mnt/lustre oleg130-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg130-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9985c75c3000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9985c75c3000.idle_timeout=debug disable quota as required oleg130-server: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 15:57:17 (1763153837) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7550] == sanity-flr test 0a: lfs mirror create with -N option == 15:57:18 (1763153838) PASS 0a (12s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:57:30 (1763153850) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:57:33 (1763153853) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 15:57:35 (1763153855) PASS 0d (21s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:57:56 (1763153876) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:57:59 (1763153879) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 15:58:02 (1763153882) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9985c75c3000.stats=clear osc.lustre-OST0001-osc-ffff9985c75c3000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:58:09 (1763153889) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (7s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:58:16 (1763153896) PASS 0j (6s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:58:22 (1763153902) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 15:58:24 (1763153904) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (7s) == sanity-flr test 2: create components from existing files ========================================================== 15:58:31 (1763153911) PASS 2 (5s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:58:36 (1763153916) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:58:39 (1763153919) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:58:45 (1763153925) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (5s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 15:58:53 (1763153933) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:58:58 (1763153938) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.268678 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12246 s, 8.6 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:59:05 (1763153945) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.193858 s, 48.7 MB/s PASS 22 (6s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:59:11 (1763153951) Stopping /mnt/lustre-ost1 (opts:) on oleg130-server oleg130-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9985c75c3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9985c75c3000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg130-server: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg130-server: oleg130-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg130-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg130-server oleg130-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9985c75c3000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9985c75c3000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg130-server: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg130-server: oleg130-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg130-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (53s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 16:00:04 (1763154004) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.222562 s, 28.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg130-server oleg130-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9985c75c3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9985c75c3000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg130-server: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg130-server: oleg130-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg130-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (27s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 16:00:31 (1763154031) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg130-server oleg130-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9985c75c3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9985c75c3000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg130-server: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg130-server: oleg130-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg130-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg130-server oleg130-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9985c75c3000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9985c75c3000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg130-server: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg130-server: oleg130-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg130-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (54s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 16:01:25 (1763154085) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.50205 s, 15.0 MB/s -rw-r--r-- 1 root root 52428800 Nov 14 16:01 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.922s user 0m0.002s sys 0m0.178s Stopping /mnt/lustre-ost1 (opts:) on oleg130-server oleg130-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9985c75c3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9985c75c3000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.909s user 0m0.000s sys 0m0.194s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg130-server: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg130-server: oleg130-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg130-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg130-server oleg130-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9985c75c3000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9985c75c3000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.880s user 0m0.000s sys 0m0.169s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg130-server: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg130-server: oleg130-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg130-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg130-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9985c75c3000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9985c75c3000.ost_server_uuid in FULL state after 0 sec PASS 33b (62s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 16:02:27 (1763154147) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 16:02:29 (1763154149) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 16:02:32 (1763154152) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 16:02:34 (1763154154) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 16:02:40 (1763154160) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.51278 s, 24.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 16:02:52 (1763154172) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234541 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174886 s, 60.0 MB/s PASS 36b (5s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 16:02:57 (1763154177) PASS 36c (5s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 16:03:02 (1763154182) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000899714 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00119861 s, 854 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00248056 s, 413 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00101521 s, 1.0 MB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 16:03:10 (1763154190) create test files with size 15741897 .. 1450f8fac99eade360290d4ff102df6a - e2abaa4bb5e5f9d6bf17a16a7e7de6b2 - 25604fd842fcab7fc8fb3a4fa127191d - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg130-client.virtnet /mnt/lustre (opts:) Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre PASS 37 (17s) == sanity-flr test 38: resync ============================ 16:03:27 (1763154207) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (11s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 16:03:38 (1763154218) PASS 39 (5s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 16:03:44 (1763154224) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109022 s, 9.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283749 s, 37.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 16:03:52 (1763154232) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.15135 s, 13.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.408124 s, 10.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.27398 s, 7.7 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (11s) == sanity-flr test 42: lfs mirror verify ================= 16:04:03 (1763154243) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 16:04:05 (1763154245) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 16:04:08 (1763154248) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171314 s, 61.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 44a: lfs mirror split check =========== 16:04:14 (1763154254) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 16:04:17 (1763154257) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (6s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 16:04:24 (1763154264) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.218368 s, 48.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg130-client.virtnet /mnt/lustre (opts:) Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (8s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 16:04:32 (1763154272) llite.lustre-ffff9985db555800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.188149 s, 55.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg130-client.virtnet /mnt/lustre (opts:) Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9985d0fd2800.xattr_cache=1 PASS 44e (9s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 16:04:41 (1763154281) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (28s) == sanity-flr test 46: Verify setstripe --copy option ==== 16:05:09 (1763154309) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (30s) == sanity-flr test 47: Verify mirror obj alloc =========== 16:05:39 (1763154339) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 16:05:43 (1763154343) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.214893 s, 14.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.229921 s, 13.7 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: e111a5e9a3f0cefebceb04370a3b71a5 - mirror 1 checksum: eea8e1962d0340cb631c79d5a510e9ac - mirror 2 checksum: e111a5e9a3f0cefebceb04370a3b71a5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Nov 14 16:05:45 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Nov 14 16:05:45 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: e111a5e9a3f0cefebceb04370a3b71a5 - mirror 1 checksum: eea8e1962d0340cb631c79d5a510e9ac - mirror 2 checksum: eea8e1962d0340cb631c79d5a510e9ac - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Nov 14 16:05:45 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Nov 14 16:05:45 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 16:05:56 (1763154356) Failing mds1 on oleg130-server Stopping /mnt/lustre-mds1 (opts:) on oleg130-server 16:06:03 (1763154363) shut down facet: mds1 facet_host: oleg130-server facet_failover_host: oleg130-server Failover mds1 to oleg130-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg130-server: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-MDT0000 16:06:26 (1763154386) targets are mounted 16:06:26 (1763154386) facet_failover done oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273918 s, 38.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } PASS 50A (45s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 16:06:41 (1763154401) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 16:06:44 (1763154404) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 16:06:47 (1763154407) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0112115 s, 1.5 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 16:06:55 (1763154415) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 16:06:58 (1763154418) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 16:07:02 (1763154422) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 16:07:24 (1763154444) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (26s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 16:07:50 (1763154470) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 16:08:13 (1763154493) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (22s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 16:08:35 (1763154515) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 16:08:41 (1763154521) mirror create pid 52262 mirror split pid 52263 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 52262 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 52263 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 16:09:48 (1763154588) Seed set to 4590 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1763154590.439419 mapwrite 0x037b81 thru 0x038846 (0xcc6 bytes) truncating to largest ever: 0x1ced6 000003 1763154590.567773 trunc from 0x038847 to 0x01ced5 (0x1b971 bytes) 000004 1763154590.606508 read 0x01c13f thru 0x01ced5 (0xd97 bytes) 000005 1763154590.612114 mapread 0x0118d7 thru 0x0133bb (0x1ae5 bytes) truncating to largest ever: 0x3069c 000010 1763154590.640432 trunc from 0x01ced6 to 0x03069b (0x137c6 bytes) 000012 1763154590.681291 mapwrite 0x002e41 thru 0x009a18 (0x6bd8 bytes) 000013 1763154590.772859 read 0x0155d1 thru 0x017a51 (0x2481 bytes) 000018 1763154590.775199 mapread 0x020f6a thru 0x02b0a7 (0xa13e bytes) 000020 1763154590.792864 write 0x032d64 thru 0x037ea0 (0x513d bytes) 000022 1763154590.794378 read 0x018234 thru 0x019ddf (0x1bac bytes) 000023 1763154590.795282 trunc from 0x037ea1 to 0x00f67b (0x28825 bytes) truncating to largest ever: 0x3dd50 000024 1763154590.825367 trunc from 0x00f67c to 0x03dd4f (0x2e6d4 bytes) 000025 1763154590.854673 MIRROR_EXTEND to 2 mirrors 000028 1763154591.629888 MIRROR_RESYNC 2 mirrors 000029 1763154591.852174 trunc from 0x03dd50 to 0x0118b6 (0x2c499 bytes) 000032 1763154591.972032 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d3:0x0]: Invalid argument (22) 000033 1763154592.175826 read 0x00ba8b thru 0x00f3a5 (0x391b bytes) 000035 1763154592.193626 read 0x00c2da thru 0x0118b6 (0x55dd bytes) 000036 1763154592.205472 read 0x00bc95 thru 0x0118b6 (0x5c22 bytes) 000038 1763154592.206502 write 0x03aa5c thru 0x03ffff (0x55a4 bytes) 000042 1763154592.209445 write 0x012ded thru 0x01f05c (0xc270 bytes) 000043 1763154592.226760 MIRROR_EXTEND to 2 mirrors 000044 1763154592.878700 read 0x033e58 thru 0x03ffff (0xc1a8 bytes) 000047 1763154592.908749 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d3:0x0]: Invalid argument (22) 000048 1763154593.063099 read 0x02ee65 thru 0x03a653 (0xb7ef bytes) 000052 1763154593.083546 mapwrite 0x03b580 thru 0x03ffff (0x4a80 bytes) 000053 1763154593.253896 read 0x01d179 thru 0x02bb03 (0xe98b bytes) 000054 1763154593.280884 MIRROR_EXTEND to 2 mirrors 000056 1763154593.844150 trunc from 0x040000 to 0x012625 (0x2d9da bytes) 000058 1763154593.985748 read 0x00b285 thru 0x010ea6 (0x5c22 bytes) 000059 1763154594.000877 read 0x0106cf thru 0x012625 (0x1f57 bytes) 000060 1763154594.031816 mapread 0x0061ae thru 0x009f46 (0x3d99 bytes) 000062 1763154594.033161 trunc from 0x012626 to 0x000f03 (0x11722 bytes) 000063 1763154594.071094 write 0x01a99b thru 0x0267b6 (0xbe1c bytes) 000065 1763154594.078495 mapread 0x02088d thru 0x0246a6 (0x3e1a bytes) 000066 1763154594.081013 read 0x01dbab thru 0x0267b6 (0x8c0c bytes) 000067 1763154594.083549 write 0x02a6de thru 0x033e16 (0x9739 bytes) 000068 1763154594.084914 mapwrite 0x02b276 thru 0x02f37b (0x4106 bytes) 000069 1763154594.198944 mapread 0x006a3f thru 0x009348 (0x290a bytes) 000070 1763154594.217302 trunc from 0x033e17 to 0x018c11 (0x1b205 bytes) 000072 1763154594.276796 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d3:0x0]: Invalid argument (22) 000073 1763154594.483156 write 0x030465 thru 0x03ffff (0xfb9b bytes) 000076 1763154594.494850 read 0x00b6d3 thru 0x01b4ae (0xfddc bytes) 000077 1763154594.503420 write 0x028394 thru 0x03817f (0xfdec bytes) 000078 1763154594.516898 read 0x00686b thru 0x00d3b2 (0x6b48 bytes) 000079 1763154594.526888 mapwrite 0x014e49 thru 0x01cc3d (0x7df5 bytes) 000082 1763154594.585699 read 0x028078 thru 0x02d239 (0x51c2 bytes) 000086 1763154594.587600 read 0x0143c9 thru 0x0166bb (0x22f3 bytes) 000087 1763154594.588367 mapwrite 0x0188d9 thru 0x01c79d (0x3ec5 bytes) 000090 1763154594.641536 write 0x02181f thru 0x02aaa9 (0x928b bytes) 000092 1763154594.643610 trunc from 0x040000 to 0x02a9ff (0x15600 bytes) 000096 1763154594.665883 trunc from 0x02aa00 to 0x015840 (0x151bf bytes) 000097 1763154594.691306 read 0x000838 thru 0x0081c6 (0x798f bytes) 000100 1763154594.692802 write 0x017e2c thru 0x01b16c (0x3341 bytes) 000101 1763154594.694366 mapread 0x0106c5 thru 0x019ce4 (0x9620 bytes) 000103 1763154594.702738 write 0x03d2d7 thru 0x03ffff (0x2d29 bytes) 000107 1763154594.706735 trunc from 0x040000 to 0x02293b (0x1d6c4 bytes) 000108 1763154594.722236 mapread 0x01ec03 thru 0x0203bb (0x17b9 bytes) 000110 1763154594.734964 trunc from 0x02293c to 0x00f36e (0x135cd bytes) 000111 1763154594.756401 read 0x00e310 thru 0x00f36e (0x105f bytes) 000112 1763154594.757007 read 0x004979 thru 0x00d274 (0x88fc bytes) 000113 1763154594.757462 trunc from 0x00f36f to 0x01001a (0xcac bytes) 000115 1763154594.782623 mapwrite 0x0220bb thru 0x023506 (0x144c bytes) 000116 1763154594.866434 write 0x00b158 thru 0x010699 (0x5542 bytes) 000118 1763154594.867990 mapread 0x009924 thru 0x00dda9 (0x4486 bytes) 000120 1763154594.869262 write 0x020590 thru 0x0302f7 (0xfd68 bytes) 000121 1763154594.873616 mapwrite 0x01e694 thru 0x022f02 (0x486f bytes) 000124 1763154594.933510 trunc from 0x0302f8 to 0x03db5d (0xd866 bytes) 000125 1763154594.959361 read 0x00373a thru 0x0107fa (0xd0c1 bytes) 000126 1763154594.960016 trunc from 0x03db5e to 0x0266da (0x17483 bytes) 000128 1763154594.982549 read 0x012e9f thru 0x015845 (0x29a7 bytes) 000131 1763154594.985074 mapwrite 0x0374a8 thru 0x03ffff (0x8b58 bytes) 000139 1763154595.100174 write 0x016727 thru 0x022e15 (0xc6ef bytes) 000140 1763154595.102649 trunc from 0x040000 to 0x021261 (0x1ed9e bytes) 000142 1763154595.133762 mapwrite 0x018673 thru 0x018b31 (0x4bf bytes) 000143 1763154595.188101 trunc from 0x021262 to 0x03a3bd (0x1915c bytes) 000144 1763154595.207329 read 0x038d50 thru 0x03a3bd (0x166e bytes) 000145 1763154595.221792 mapread 0x0009a4 thru 0x00e2bd (0xd91a bytes) 000147 1763154595.223505 mapwrite 0x014419 thru 0x01b8d6 (0x74be bytes) 000150 1763154595.280496 MIRROR_EXTEND to 2 mirrors 000153 1763154595.883281 trunc from 0x03a3be to 0x00fc8f (0x2a72e bytes) 000154 1763154596.007633 mapwrite 0x009381 thru 0x00a318 (0xf98 bytes) 000155 1763154596.088336 mapread 0x004249 thru 0x00c7f3 (0x85ab bytes) 000158 1763154596.098142 read 0x00456b thru 0x008001 (0x3a97 bytes) 000160 1763154596.099057 trunc from 0x00fc90 to 0x005c79 (0xa016 bytes) 000161 1763154596.128869 write 0x0107b7 thru 0x0113b7 (0xc01 bytes) 000162 1763154596.133636 read 0x00a935 thru 0x00d1a0 (0x286c bytes) 000163 1763154596.148497 mapwrite 0x0017cc thru 0x010dc4 (0xf5f9 bytes) 000166 1763154596.197531 mapwrite 0x019369 thru 0x01bc45 (0x28dd bytes) 000168 1763154596.308881 MIRROR_EXTEND to 3 mirrors 000171 1763154596.911001 mapread 0x0166bd thru 0x01bc45 (0x5589 bytes) 000174 1763154596.951629 MIRROR_RESYNC 3 mirrors 000176 1763154597.621365 read 0x007921 thru 0x00b1b3 (0x3893 bytes) 000179 1763154597.634261 MIRROR_RESYNC 3 mirrors 000181 1763154597.900864 mapread 0x00471f thru 0x013990 (0xf272 bytes) 000184 1763154597.954837 read 0x018183 thru 0x01bc45 (0x3ac3 bytes) 000185 1763154597.963370 mapwrite 0x014048 thru 0x021f4f (0xdf08 bytes) 000191 1763154598.182295 read 0x00e150 thru 0x00f940 (0x17f1 bytes) 000193 1763154598.183312 mapwrite 0x03bd60 thru 0x03ffff (0x42a0 bytes) 000194 1763154598.271449 mapwrite 0x0134b9 thru 0x01cd66 (0x98ae bytes) 000198 1763154598.356048 mapwrite 0x031dcc thru 0x036e38 (0x506d bytes) 000199 1763154598.394623 mapread 0x01cb37 thru 0x028e4f (0xc319 bytes) 000200 1763154598.396729 mapread 0x01a233 thru 0x020a54 (0x6822 bytes) 000201 1763154598.397564 mapwrite 0x02cf07 thru 0x03b1ea (0xe2e4 bytes) 000202 1763154598.456165 read 0x03e099 thru 0x03fb8a (0x1af2 bytes) 000204 1763154598.459414 trunc from 0x040000 to 0x00d24f (0x32db0 bytes) 000207 1763154598.480767 mapread 0x009ff6 thru 0x00d24f (0x325a bytes) 000208 1763154598.492685 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d3:0x0]: Invalid argument (22) 000209 1763154598.752435 write 0x025a2a thru 0x02ffad (0xa584 bytes) 000211 1763154598.759305 MIRROR_EXTEND to 3 mirrors 000213 1763154599.464407 trunc from 0x02ffae to 0x03c91e (0xc971 bytes) 000217 1763154599.541896 read 0x022633 thru 0x0309f6 (0xe3c4 bytes) 000218 1763154599.557277 mapread 0x017157 thru 0x01baf6 (0x49a0 bytes) 000221 1763154599.608786 write 0x014dfb thru 0x021d62 (0xcf68 bytes) 000222 1763154599.610025 mapread 0x02d269 thru 0x02f563 (0x22fb bytes) 000223 1763154599.613114 read 0x02cf30 thru 0x033558 (0x6629 bytes) 000224 1763154599.627956 read 0x00a1b4 thru 0x018b80 (0xe9cd bytes) 000226 1763154599.630237 write 0x02bd85 thru 0x02e999 (0x2c15 bytes) 000227 1763154599.631230 mapread 0x0032dd thru 0x009d7e (0x6aa2 bytes) 000229 1763154599.635848 mapwrite 0x02bcc2 thru 0x0382b5 (0xc5f4 bytes) 000230 1763154599.705741 mapwrite 0x00a6e3 thru 0x0140be (0x99dc bytes) 000232 1763154599.752768 mapwrite 0x013a37 thru 0x01446b (0xa35 bytes) 000234 1763154599.808412 mapwrite 0x007d2c thru 0x015d63 (0xe038 bytes) 000235 1763154599.879096 trunc from 0x03c91f to 0x009b84 (0x32d9a bytes) 000236 1763154599.898475 write 0x015161 thru 0x017b98 (0x2a38 bytes) 000238 1763154599.899997 read 0x01796a thru 0x017b98 (0x22f bytes) 000242 1763154599.903620 read 0x0022bc thru 0x00ad76 (0x8abb bytes) 000243 1763154599.923037 MIRROR_RESYNC 3 mirrors 000244 1763154600.826281 write 0x0125b3 thru 0x019dd0 (0x781e bytes) 000245 1763154600.899862 mapread 0x016c27 thru 0x0191e3 (0x25bd bytes) 000247 1763154600.901518 mapread 0x0173e7 thru 0x019dd0 (0x29ea bytes) 000248 1763154600.909723 MIRROR_RESYNC 3 mirrors 000249 1763154601.722968 mapwrite 0x03b846 thru 0x03ffff (0x47ba bytes) 000250 1763154601.882237 read 0x0053f3 thru 0x01017e (0xad8c bytes) 000252 1763154601.893052 trunc from 0x040000 to 0x027dac (0x18253 bytes) 000253 1763154601.915154 write 0x021ff7 thru 0x031088 (0xf092 bytes) 000254 1763154601.929899 MIRROR_RESYNC 3 mirrors 000255 1763154602.699869 MIRROR_RESYNC 3 mirrors 000256 1763154602.926129 write 0x008839 thru 0x0100a8 (0x7870 bytes) 000258 1763154603.040283 write 0x0231c6 thru 0x02a862 (0x769d bytes) 000260 1763154603.060498 mapread 0x019acf thru 0x01b4a2 (0x19d4 bytes) 000261 1763154603.075150 mapwrite 0x005775 thru 0x010cb7 (0xb543 bytes) 000262 1763154603.137554 mapread 0x0045e2 thru 0x006035 (0x1a54 bytes) 000263 1763154603.138897 mapread 0x003369 thru 0x010124 (0xcdbc bytes) 000264 1763154603.141233 mapwrite 0x03af06 thru 0x03ff12 (0x500d bytes) 000265 1763154603.220639 read 0x006c8e thru 0x016b36 (0xfea9 bytes) 000268 1763154603.228562 trunc from 0x03ff13 to 0x01a97b (0x25597 bytes) 000269 1763154603.242154 mapread 0x001a28 thru 0x008bd2 (0x71ab bytes) 000270 1763154603.243329 trunc from 0x01a97c to 0x010a16 (0x9f65 bytes) 000271 1763154603.268027 read 0x0007ef thru 0x010050 (0xf862 bytes) 000274 1763154603.275273 read 0x00a0fb thru 0x010a16 (0x691c bytes) 000275 1763154603.275952 trunc from 0x010a17 to 0x0307a2 (0x1fd8c bytes) 000278 1763154603.295838 write 0x02f3f3 thru 0x032124 (0x2d32 bytes) 000280 1763154603.318266 mapwrite 0x02b106 thru 0x03a795 (0xf690 bytes) 000283 1763154603.413904 write 0x0226f2 thru 0x02d5d2 (0xaee1 bytes) 000287 1763154603.417062 mapread 0x0353e8 thru 0x03a795 (0x53ae bytes) 000290 1763154603.425194 MIRROR_RESYNC 3 mirrors 000293 1763154604.209692 MIRROR_EXTEND to 4 mirrors 000295 1763154604.914129 write 0x02e8ad thru 0x030f51 (0x26a5 bytes) 000296 1763154605.031363 mapwrite 0x0069ac thru 0x006ac9 (0x11e bytes) 000297 1763154605.087289 write 0x0116fa thru 0x015c6e (0x4575 bytes) 000299 1763154605.098523 mapwrite 0x02da88 thru 0x02f645 (0x1bbe bytes) 000303 1763154605.175376 MIRROR_RESYNC 4 mirrors 000304 1763154605.952638 mapread 0x0001bb thru 0x00250e (0x2354 bytes) 000305 1763154605.983061 MIRROR_RESYNC 4 mirrors 000306 1763154606.290387 read 0x005e69 thru 0x00a548 (0x46e0 bytes) 000307 1763154606.299220 mapread 0x011572 thru 0x01fd10 (0xe79f bytes) 000310 1763154606.329807 MIRROR_EXTEND to 5 mirrors 000311 1763154606.938961 mapread 0x00788f thru 0x007fac (0x71e bytes) 000312 1763154606.970067 write 0x029983 thru 0x039719 (0xfd97 bytes) 000313 1763154607.077715 MIRROR_RESYNC 5 mirrors 000315 1763154607.926650 write 0x0075cf thru 0x01341f (0xbe51 bytes) 000317 1763154608.034236 mapread 0x00ab21 thru 0x00e4d1 (0x39b1 bytes) 000319 1763154608.035500 trunc from 0x03a796 to 0x02ba91 (0xed04 bytes) 000320 1763154608.058706 read 0x01c7ab thru 0x027a76 (0xb2cc bytes) 000321 1763154608.066376 write 0x029cdd thru 0x0304fc (0x6820 bytes) 000322 1763154608.073062 mapwrite 0x0157ca thru 0x023561 (0xdd98 bytes) 000323 1763154608.162183 mapread 0x001c55 thru 0x0088e7 (0x6c93 bytes) 000325 1763154608.183769 trunc from 0x0304fd to 0x0281a0 (0x835c bytes) 000327 1763154608.218779 MIRROR_EXTEND to 6 mirrors 000328 1763154608.739732 mapwrite 0x01beb8 thru 0x02b7ea (0xf933 bytes) 000329 1763154609.031939 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d3:0x0]: Invalid argument (22) 000331 1763154609.348748 read 0x0164a8 thru 0x016e8d (0x9e6 bytes) 000333 1763154609.358658 write 0x00dcb5 thru 0x01bcba (0xe006 bytes) 000334 1763154609.370506 read 0x001000 thru 0x00cca2 (0xbca3 bytes) 000335 1763154609.383546 mapread 0x00498f thru 0x0050e6 (0x758 bytes) 000336 1763154609.391218 MIRROR_EXTEND to 6 mirrors 000339 1763154610.142630 write 0x01f8d5 thru 0x020c49 (0x1375 bytes) 000340 1763154610.240425 MIRROR_RESYNC 6 mirrors 000341 1763154611.230429 trunc from 0x02b7eb to 0x03494e (0x9164 bytes) 000343 1763154611.335616 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d3:0x0]: Invalid argument (22) 000344 1763154611.552083 MIRROR_RESYNC 5 mirrors 000347 1763154612.354168 mapread 0x004cb1 thru 0x0115d5 (0xc925 bytes) 000348 1763154612.408332 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d3:0x0]: Invalid argument (22) 000349 1763154612.941004 mapwrite 0x03d658 thru 0x03ffff (0x29a8 bytes) 000350 1763154613.088015 trunc from 0x040000 to 0x022d1c (0x1d2e3 bytes) 000351 1763154613.114062 read 0x014fa9 thru 0x0191b3 (0x420b bytes) 000352 1763154613.123735 mapwrite 0x0084be thru 0x00e7b1 (0x62f4 bytes) 000353 1763154613.180883 MIRROR_RESYNC 4 mirrors 000354 1763154613.998930 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d3:0x0]: Invalid argument (22) 000355 1763154614.342385 MIRROR_EXTEND to 4 mirrors 000356 1763154614.918528 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d3:0x0]: Invalid argument (22) 000358 1763154615.327882 mapwrite 0x02e984 thru 0x031a59 (0x30d6 bytes) 000359 1763154615.501262 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d3:0x0]: Invalid argument (22) 000361 1763154615.736488 read 0x0154e2 thru 0x02460e (0xf12d bytes) 000364 1763154615.749497 mapread 0x02e962 thru 0x031a59 (0x30f8 bytes) 000366 1763154615.769153 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d3:0x0]: Invalid argument (22) truncating to largest ever: 0x3deee 000369 1763154615.934266 trunc from 0x031a5a to 0x03deed (0xc494 bytes) truncating to largest ever: 0x3fc70 000370 1763154615.961809 trunc from 0x03deee to 0x03fc6f (0x1d82 bytes) 000373 1763154615.992511 write 0x01c568 thru 0x027bc1 (0xb65a bytes) 000376 1763154616.008069 trunc from 0x03fc70 to 0x01c41c (0x23853 bytes) 000380 1763154616.028513 read 0x00f8c0 thru 0x010ece (0x160f bytes) 000382 1763154616.039196 MIRROR_EXTEND to 2 mirrors 000384 1763154616.666820 read 0x015b1d thru 0x01beb7 (0x639b bytes) 000387 1763154616.675200 read 0x01108f thru 0x014f8c (0x3efe bytes) 000389 1763154616.697476 read 0x0007e3 thru 0x00a897 (0xa0b5 bytes) 000393 1763154616.701182 trunc from 0x01c41d to 0x017457 (0x4fc5 bytes) 000394 1763154616.839703 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d3:0x0]: Invalid argument (22) 000395 1763154616.993757 write 0x02de93 thru 0x03324b (0x53b9 bytes) 000399 1763154616.997724 write 0x009ed6 thru 0x018a87 (0xebb2 bytes) 000400 1763154617.005821 mapread 0x015083 thru 0x015a58 (0x9d6 bytes) 000401 1763154617.006818 mapread 0x011a7b thru 0x01330f (0x1895 bytes) 000404 1763154617.008621 mapwrite 0x03b720 thru 0x03c85a (0x113b bytes) 000405 1763154617.096852 mapread 0x0208ef thru 0x02f645 (0xed57 bytes) 000407 1763154617.111810 trunc from 0x03c85b to 0x02117b (0x1b6df bytes) 000408 1763154617.149160 mapread 0x016593 thru 0x02117b (0xabe9 bytes) 000411 1763154617.152825 mapread 0x010cb6 thru 0x019122 (0x846d bytes) 000413 1763154617.156007 read 0x00f986 thru 0x01e8c5 (0xef40 bytes) 000414 1763154617.156791 write 0x03afa7 thru 0x03f60e (0x4668 bytes) 000415 1763154617.158145 trunc from 0x03f60f to 0x01c291 (0x2337d bytes) 000417 1763154617.178656 trunc from 0x01c292 to 0x02415f (0x7ece bytes) 000419 1763154617.199613 mapread 0x00807e thru 0x01172d (0x96b0 bytes) 000421 1763154617.210345 mapread 0x0182bc thru 0x0184ea (0x22f bytes) 000422 1763154617.218784 MIRROR_EXTEND to 2 mirrors 000423 1763154617.759350 mapwrite 0x00f83b thru 0x012cf8 (0x34be bytes) 000427 1763154617.904019 read 0x00fe6f thru 0x0139b0 (0x3b42 bytes) 000430 1763154617.906897 write 0x03a3b1 thru 0x03b62b (0x127b bytes) 000433 1763154617.909970 read 0x02c1bc thru 0x02f5eb (0x3430 bytes) 000436 1763154617.926872 mapwrite 0x02d5ad thru 0x032839 (0x528d bytes) 000437 1763154617.995519 write 0x03ab99 thru 0x03bb54 (0xfbc bytes) 000438 1763154617.996723 read 0x021038 thru 0x0301a7 (0xf170 bytes) 000440 1763154617.997836 mapwrite 0x0219b9 thru 0x02f698 (0xdce0 bytes) 000442 1763154618.070866 read 0x01a89c thru 0x01d16b (0x28d0 bytes) 000443 1763154618.071546 mapread 0x01e560 thru 0x026fb5 (0x8a56 bytes) 000445 1763154618.077077 mapwrite 0x0285b8 thru 0x02d5ad (0x4ff6 bytes) 000447 1763154618.127223 mapread 0x036475 thru 0x03bb54 (0x56e0 bytes) 000448 1763154618.128269 write 0x006db1 thru 0x00cdc3 (0x6013 bytes) 000455 1763154618.133523 mapwrite 0x03c866 thru 0x03ffff (0x379a bytes) 000456 1763154618.225968 mapread 0x02b36c thru 0x02d94e (0x25e3 bytes) 000461 1763154618.229317 mapread 0x006cd7 thru 0x00d1c0 (0x64ea bytes) 000462 1763154618.230856 read 0x03c355 thru 0x03ffff (0x3cab bytes) 000464 1763154618.231619 write 0x009b5e thru 0x00d2d0 (0x3773 bytes) 000465 1763154618.232923 trunc from 0x040000 to 0x00a5e8 (0x35a17 bytes) 000466 1763154618.267451 trunc from 0x00a5e9 to 0x001068 (0x9580 bytes) 000467 1763154618.296026 write 0x033c9f thru 0x034662 (0x9c4 bytes) 000468 1763154618.297227 trunc from 0x034663 to 0x012726 (0x21f3c bytes) 000469 1763154618.315725 write 0x02bacf thru 0x02d12f (0x1661 bytes) 000471 1763154618.317436 mapwrite 0x01f4fe thru 0x025ca2 (0x67a5 bytes) 000472 1763154618.367102 mapwrite 0x03f34a thru 0x03ffff (0xcb6 bytes) 000474 1763154618.457560 mapwrite 0x01bd8c thru 0x01bf53 (0x1c8 bytes) 000476 1763154618.503775 mapwrite 0x01065d thru 0x016fef (0x6993 bytes) 000477 1763154618.557039 trunc from 0x040000 to 0x0075c8 (0x38a37 bytes) 000478 1763154618.579887 read 0x001d22 thru 0x0075c8 (0x58a7 bytes) 000480 1763154618.585856 mapread 0x0073fb thru 0x0075c8 (0x1ce bytes) 000481 1763154618.588614 mapread 0x0007ae thru 0x0075c8 (0x6e1b bytes) 000482 1763154618.589710 read 0x00162a thru 0x0075c8 (0x5f9f bytes) 000484 1763154618.594772 write 0x01254a thru 0x014242 (0x1cf9 bytes) 000488 1763154618.610442 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d3:0x0]: Invalid argument (22) 000490 1763154618.752556 trunc from 0x014243 to 0x038b65 (0x24923 bytes) 000491 1763154618.769444 trunc from 0x038b66 to 0x03ac94 (0x212f bytes) 000492 1763154618.783192 trunc from 0x03ac95 to 0x015c17 (0x2507d bytes) 000495 1763154618.801390 mapwrite 0x027a66 thru 0x02a87a (0x2e15 bytes) 000499 1763154618.896191 read 0x01689f thru 0x01cf91 (0x66f3 bytes) 000507 1763154618.910215 read 0x01b958 thru 0x01bd1c (0x3c5 bytes) 000509 1763154618.911294 read 0x00d55b thru 0x0160f8 (0x8b9e bytes) 000511 1763154618.921835 mapwrite 0x0264f6 thru 0x02bc3c (0x5747 bytes) 000513 1763154619.005779 mapread 0x024c06 thru 0x027f87 (0x3382 bytes) 000515 1763154619.008399 read 0x00778e thru 0x007a85 (0x2f8 bytes) 000518 1763154619.010953 mapwrite 0x034405 thru 0x03b6c1 (0x72bd bytes) 000520 1763154619.081376 mapread 0x038d0a thru 0x03b6c1 (0x29b8 bytes) 000522 1763154619.083805 trunc from 0x03b6c2 to 0x00e701 (0x2cfc0 bytes) 000524 1763154619.110009 mapwrite 0x00f27c thru 0x016418 (0x719d bytes) 000526 1763154619.235776 read 0x0089b2 thru 0x016068 (0xd6b7 bytes) 000529 1763154619.237480 read 0x009325 thru 0x016418 (0xd0f4 bytes) 000530 1763154619.238607 mapwrite 0x0218a5 thru 0x03100c (0xf768 bytes) 000533 1763154619.327103 trunc from 0x03100d to 0x00444b (0x2cbc1 bytes) 000534 1763154619.363331 write 0x035efa thru 0x03d5fe (0x7705 bytes) 000536 1763154619.368314 read 0x015fd5 thru 0x02281b (0xc847 bytes) 000538 1763154619.387083 trunc from 0x03d5ff to 0x00b4df (0x3211f bytes) 000540 1763154619.415454 mapread 0x0040c2 thru 0x00b4df (0x741e bytes) 000541 1763154619.417850 mapread 0x007419 thru 0x00b4df (0x40c7 bytes) 000542 1763154619.419836 trunc from 0x00b4e0 to 0x00ed90 (0x38b1 bytes) 000545 1763154619.441797 read 0x00c369 thru 0x00ed90 (0x2a28 bytes) 000548 1763154619.452785 write 0x0189a6 thru 0x024601 (0xbc5c bytes) 000550 1763154619.454951 read 0x019845 thru 0x024601 (0xadbd bytes) 000551 1763154619.455900 trunc from 0x024602 to 0x00fe59 (0x147a8 bytes) 000553 1763154619.472941 trunc from 0x00fe5a to 0x01e86f (0xea16 bytes) 000554 1763154619.494072 mapread 0x0052dd thru 0x00d0f3 (0x7e17 bytes) 000555 1763154619.495189 read 0x01a1c1 thru 0x01e86f (0x46af bytes) 000556 1763154619.510116 trunc from 0x01e870 to 0x00b135 (0x1373a bytes) 000560 1763154619.533723 read 0x00a5db thru 0x00b135 (0xb5b bytes) 000563 1763154619.536622 read 0x004703 thru 0x004fbf (0x8bd bytes) 000565 1763154619.537759 write 0x037091 thru 0x03d87d (0x67ed bytes) 000568 1763154619.539919 trunc from 0x03d87e to 0x035afa (0x7d83 bytes) 000570 1763154619.561624 write 0x02b3d7 thru 0x03036a (0x4f94 bytes) 000571 1763154619.581613 write 0x0311d7 thru 0x034b5f (0x3989 bytes) 000573 1763154619.583167 trunc from 0x035afb to 0x023071 (0x12a89 bytes) 000574 1763154619.611191 read 0x00156f thru 0x0047de (0x3270 bytes) 000575 1763154619.611916 mapread 0x010adf thru 0x01ad33 (0xa255 bytes) 000577 1763154619.617531 trunc from 0x023072 to 0x01b70d (0x7964 bytes) 000579 1763154619.634098 mapread 0x0120c2 thru 0x01b70d (0x964c bytes) 000582 1763154619.645786 read 0x0070f0 thru 0x00d967 (0x6878 bytes) 000586 1763154619.653137 mapread 0x01211f thru 0x01ae28 (0x8d0a bytes) 000589 1763154619.656809 mapread 0x016b45 thru 0x01b70d (0x4bc9 bytes) 000592 1763154619.662096 mapread 0x0037fc thru 0x01234a (0xeb4f bytes) 000593 1763154619.667287 mapwrite 0x015451 thru 0x017065 (0x1c15 bytes) 000594 1763154619.765666 trunc from 0x01b70e to 0x0222d7 (0x6bca bytes) 000595 1763154619.785009 mapwrite 0x018c54 thru 0x022833 (0x9be0 bytes) 000596 1763154619.874813 write 0x0318ad thru 0x03ffff (0xe753 bytes) 000597 1763154619.885707 MIRROR_EXTEND to 2 mirrors 000598 1763154620.619060 write 0x00f429 thru 0x01cf52 (0xdb2a bytes) 000599 1763154620.747003 mapwrite 0x017be4 thru 0x01fec9 (0x82e6 bytes) 000600 1763154620.812997 mapwrite 0x00f365 thru 0x019c10 (0xa8ac bytes) 000601 1763154620.865340 trunc from 0x040000 to 0x021371 (0x1ec8e bytes) 000602 1763154620.890957 mapread 0x015157 thru 0x01ab1d (0x59c7 bytes) 000603 1763154620.892170 read 0x014850 thru 0x019fae (0x575f bytes) 000604 1763154620.893188 mapwrite 0x037696 thru 0x03ffff (0x896a bytes) 000606 1763154620.983818 MIRROR_RESYNC 2 mirrors 000608 1763154621.674434 trunc from 0x040000 to 0x00a2e3 (0x35d1c bytes) 000609 1763154621.790285 mapwrite 0x008ab9 thru 0x00c564 (0x3aac bytes) 000610 1763154621.925196 read 0x0030e6 thru 0x004361 (0x127c bytes) 000611 1763154621.925984 read 0x0006b9 thru 0x00c564 (0xbeac bytes) 000612 1763154621.931964 mapwrite 0x03f906 thru 0x03ffff (0x6fa bytes) 000614 1763154622.035485 mapread 0x023472 thru 0x029f48 (0x6ad7 bytes) 000615 1763154622.037329 write 0x03d35d thru 0x03ffff (0x2ca3 bytes) 000616 1763154622.040927 trunc from 0x040000 to 0x003b31 (0x3c4ce bytes) 000617 1763154622.073429 mapwrite 0x0283af thru 0x0381ac (0xfdfe bytes) 000618 1763154622.291924 MIRROR_EXTEND to 3 mirrors 000619 1763154622.784465 read 0x000185 thru 0x00dae8 (0xd964 bytes) 000620 1763154622.801684 trunc from 0x0381ad to 0x01ab85 (0x1d627 bytes) 000622 1763154622.948925 mapread 0x0115a6 thru 0x01ab85 (0x95e0 bytes) 000624 1763154622.981822 mapwrite 0x014035 thru 0x01c78b (0x8757 bytes) 000626 1763154623.103458 mapread 0x01af77 thru 0x01b441 (0x4cb bytes) 000627 1763154623.104388 read 0x015cb2 thru 0x0166cb (0xa1a bytes) 000628 1763154623.105870 write 0x036e26 thru 0x03ffff (0x91da bytes) 000629 1763154623.107301 write 0x01d8a6 thru 0x029a63 (0xc1be bytes) 000631 1763154623.118027 read 0x03de30 thru 0x03ffff (0x21d0 bytes) 000633 1763154623.125075 MIRROR_EXTEND to 4 mirrors 000635 1763154623.755607 mapread 0x0279ea thru 0x027b43 (0x15a bytes) 000636 1763154623.765946 mapwrite 0x00e880 thru 0x01ae22 (0xc5a3 bytes) 000638 1763154623.958167 write 0x004710 thru 0x007b15 (0x3406 bytes) 000639 1763154623.971389 mapwrite 0x02e177 thru 0x03d5f2 (0xf47c bytes) 000641 1763154624.042675 mapread 0x00444a thru 0x010e09 (0xc9c0 bytes) 000642 1763154624.045288 mapread 0x00799c thru 0x0081ff (0x864 bytes) 000643 1763154624.046123 write 0x02c56f thru 0x03bcf6 (0xf788 bytes) 000645 1763154624.049557 trunc from 0x040000 to 0x016008 (0x29ff7 bytes) 000646 1763154624.104298 trunc from 0x016009 to 0x03db7d (0x27b75 bytes) 000649 1763154624.123827 trunc from 0x03db7e to 0x0331c2 (0xa9bb bytes) 000650 1763154624.144490 mapread 0x01b12a thru 0x021583 (0x645a bytes) 000653 1763154624.155136 mapwrite 0x00e922 thru 0x013b45 (0x5224 bytes) 000654 1763154624.214156 MIRROR_RESYNC 4 mirrors 000655 1763154625.055463 trunc from 0x0331c3 to 0x0346d1 (0x150f bytes) 000656 1763154625.156523 mapwrite 0x021222 thru 0x02e9b1 (0xd790 bytes) 000658 1763154625.253420 mapread 0x02fe1d thru 0x0346d1 (0x48b5 bytes) 000660 1763154625.268682 MIRROR_RESYNC 4 mirrors 000662 1763154626.004476 mapwrite 0x006c42 thru 0x00bfb3 (0x5372 bytes) 000664 1763154626.190526 read 0x0129ab thru 0x01e89f (0xbef5 bytes) 000669 1763154626.206960 read 0x02f32c thru 0x0346d1 (0x53a6 bytes) 000671 1763154626.236394 MIRROR_EXTEND to 5 mirrors 000672 1763154626.882466 mapwrite 0x01ba75 thru 0x0230e1 (0x766d bytes) 000673 1763154627.150537 mapread 0x021700 thru 0x02dc08 (0xc509 bytes) 000674 1763154627.160894 read 0x000b3e thru 0x001ede (0x13a1 bytes) 000675 1763154627.172890 MIRROR_EXTEND to 6 mirrors 000677 1763154627.825935 trunc from 0x0346d2 to 0x00da2c (0x26ca5 bytes) 000678 1763154627.940722 trunc from 0x00da2d to 0x03c7bc (0x2ed90 bytes) 000679 1763154627.971074 read 0x03aa05 thru 0x03c60c (0x1c08 bytes) 000680 1763154627.979627 read 0x005f73 thru 0x011e9d (0xbf2b bytes) 000682 1763154627.993717 read 0x03122a thru 0x03ba66 (0xa83d bytes) 000684 1763154627.994950 mapwrite 0x00c842 thru 0x019e90 (0xd64f bytes) 000685 1763154628.066528 trunc from 0x03c7bd to 0x005ddc (0x369e0 bytes) 000692 1763154628.103455 read 0x005366 thru 0x005ddc (0xa77 bytes) 000693 1763154628.121907 MIRROR_RESYNC 6 mirrors 000695 1763154629.101507 mapwrite 0x019615 thru 0x01cbe0 (0x35cc bytes) 000696 1763154629.282551 MIRROR_EXTEND to 7 mirrors 000697 1763154629.904096 trunc from 0x01cbe1 to 0x02c716 (0xfb36 bytes) 000698 1763154629.998018 mapwrite 0x0245ef thru 0x031cb6 (0xd6c8 bytes) 000699 1763154630.116209 MIRROR_RESYNC 7 mirrors 000700 1763154631.195954 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d3:0x0]: Invalid argument (22) 000702 1763154631.600128 trunc from 0x031cb7 to 0x01a2dd (0x179d9 bytes) 000703 1763154631.736865 write 0x022ae7 thru 0x028738 (0x5c52 bytes) 000704 1763154631.738342 mapwrite 0x01ae7a thru 0x023bbb (0x8d42 bytes) 000705 1763154631.842758 mapwrite 0x025170 thru 0x02651a (0x13ab bytes) 000707 1763154631.910344 write 0x00dc32 thru 0x01b0b7 (0xd486 bytes) 000708 1763154631.923386 mapread 0x00101b thru 0x009387 (0x836d bytes) 000709 1763154631.942429 trunc from 0x028739 to 0x0119a5 (0x16d93 bytes) 000711 1763154631.978275 trunc from 0x0119a6 to 0x00d75d (0x4248 bytes) 000715 1763154632.009653 mapread 0x00d664 thru 0x00d75d (0xfa bytes) 000716 1763154632.010562 trunc from 0x00d75e to 0x02a83c (0x1d0df bytes) 000717 1763154632.030150 trunc from 0x02a83d to 0x012e61 (0x179db bytes) 000721 1763154632.049495 read 0x011fb8 thru 0x012e61 (0xeaa bytes) 000722 1763154632.057030 mapwrite 0x02731d thru 0x029fd5 (0x2cb9 bytes) 000723 1763154632.169798 mapwrite 0x003460 thru 0x007efb (0x4a9c bytes) 000724 1763154632.237272 mapwrite 0x0314f0 thru 0x034201 (0x2d12 bytes) 000725 1763154632.324134 MIRROR_RESYNC 6 mirrors 000729 1763154633.475799 mapread 0x00b83a thru 0x010de5 (0x55ac bytes) 000730 1763154633.502149 mapwrite 0x015700 thru 0x021a77 (0xc378 bytes) 000732 1763154633.670259 MIRROR_EXTEND to 7 mirrors 000733 1763154634.320909 write 0x001f52 thru 0x0037ca (0x1879 bytes) 000735 1763154634.407947 mapread 0x01065a thru 0x012339 (0x1ce0 bytes) 000736 1763154634.415899 trunc from 0x034202 to 0x03a17a (0x5f79 bytes) 000740 1763154634.439704 read 0x02f0a1 thru 0x039d14 (0xac74 bytes) 000741 1763154634.446547 mapread 0x002c87 thru 0x00504f (0x23c9 bytes) 000742 1763154634.453499 write 0x02716d thru 0x02fd20 (0x8bb4 bytes) 000743 1763154634.478391 mapwrite 0x035d25 thru 0x03ffff (0xa2db bytes) 000744 1763154634.594512 mapwrite 0x032a8a thru 0x0396ba (0x6c31 bytes) 000745 1763154634.675095 MIRROR_RESYNC 7 mirrors 000746 1763154635.861600 mapwrite 0x008fd8 thru 0x0170b7 (0xe0e0 bytes) 000747 1763154636.023005 mapwrite 0x01f44c thru 0x02bdc3 (0xc978 bytes) 000748 1763154636.101682 trunc from 0x040000 to 0x027aef (0x18510 bytes) 000750 1763154636.132773 mapread 0x022cfe thru 0x027aef (0x4df2 bytes) 000751 1763154636.133839 mapwrite 0x006c9e thru 0x0150ae (0xe411 bytes) 000752 1763154636.211240 write 0x009380 thru 0x015d17 (0xc998 bytes) 000753 1763154636.212660 read 0x018d32 thru 0x0243a7 (0xb676 bytes) 000754 1763154636.213332 mapwrite 0x0019d1 thru 0x00e75e (0xcd8e bytes) 000756 1763154636.289361 write 0x0349e5 thru 0x03eed7 (0xa4f3 bytes) 000757 1763154636.294670 mapwrite 0x00347f thru 0x005a5c (0x25de bytes) 000760 1763154636.342900 mapwrite 0x0199da thru 0x024091 (0xa6b8 bytes) 000761 1763154636.413181 trunc from 0x03eed8 to 0x0262cf (0x18c08 bytes) 000765 1763154636.447364 trunc from 0x0262d0 to 0x02e079 (0x7daa bytes) 000766 1763154636.471997 MIRROR_EXTEND to 8 mirrors 000767 1763154637.093092 read 0x01b145 thru 0x01ea24 (0x38e0 bytes) 000768 1763154637.108311 read 0x00926c thru 0x009687 (0x41c bytes) 000770 1763154637.147403 MIRROR_RESYNC 8 mirrors 000771 1763154638.187996 write 0x03bef7 thru 0x03ffff (0x4109 bytes) 000772 1763154638.276772 trunc from 0x040000 to 0x02b418 (0x14be7 bytes) 000773 1763154638.302609 write 0x02d6f2 thru 0x03abf7 (0xd506 bytes) 000774 1763154638.309550 MIRROR_RESYNC 8 mirrors 000775 1763154639.496898 read 0x0341b7 thru 0x03760f (0x3459 bytes) 000776 1763154639.502319 trunc from 0x03abf8 to 0x020495 (0x1a762 bytes) 000778 1763154639.598340 trunc from 0x020496 to 0x010f61 (0xf534 bytes) 000779 1763154639.634325 mapwrite 0x019425 thru 0x022389 (0x8f65 bytes) 000780 1763154639.830033 trunc from 0x02238a to 0x025bb7 (0x382e bytes) 000785 1763154639.854829 write 0x033b3c thru 0x033fd7 (0x49c bytes) 000786 1763154639.857672 read 0x022bfc thru 0x029fd1 (0x73d6 bytes) 000788 1763154639.875708 mapread 0x02d2d5 thru 0x033fd7 (0x6d03 bytes) 000790 1763154639.877598 mapwrite 0x02020e thru 0x025995 (0x5788 bytes) 000791 1763154639.942488 mapwrite 0x0218e6 thru 0x027535 (0x5c50 bytes) 000792 1763154639.984032 trunc from 0x033fd8 to 0x03ded1 (0x9efa bytes) 000793 1763154640.012328 read 0x03da7c thru 0x03ded1 (0x456 bytes) 000794 1763154640.022387 mapread 0x0011e5 thru 0x00aeba (0x9cd6 bytes) 000796 1763154640.024761 mapwrite 0x01d092 thru 0x024486 (0x73f5 bytes) 000797 1763154640.074684 trunc from 0x03ded2 to 0x033ca9 (0xa228 bytes) 000798 1763154640.098256 write 0x023bb1 thru 0x024509 (0x959 bytes) 000800 1763154640.099643 write 0x0051dc thru 0x00df19 (0x8d3e bytes) 000803 1763154640.103613 write 0x000139 thru 0x003068 (0x2f30 bytes) 000805 1763154640.106149 mapread 0x01cfc1 thru 0x026823 (0x9863 bytes) 000809 1763154640.109295 read 0x025752 thru 0x02c78c (0x703b bytes) 000810 1763154640.110294 write 0x0025f6 thru 0x00f014 (0xca1f bytes) 000812 1763154640.112019 mapread 0x01d07a thru 0x02ca24 (0xf9ab bytes) 000814 1763154640.113522 trunc from 0x033caa to 0x038e9b (0x51f2 bytes) 000815 1763154640.135000 read 0x014414 thru 0x014ff3 (0xbe0 bytes) 000818 1763154640.136461 trunc from 0x038e9c to 0x033554 (0x5947 bytes) 000821 1763154640.163146 MIRROR_RESYNC 8 mirrors 000822 1763154641.235672 write 0x03b10c thru 0x03e79f (0x3694 bytes) 000824 1763154641.297182 trunc from 0x03e7a0 to 0x00400b (0x3a794 bytes) 000825 1763154641.354506 write 0x02a4fb thru 0x02eae9 (0x45ef bytes) 000826 1763154641.355682 read 0x014598 thru 0x023d6c (0xf7d5 bytes) 000827 1763154641.362365 trunc from 0x02eaea to 0x02ec18 (0x12f bytes) 000829 1763154641.379968 mapwrite 0x027d52 thru 0x02b5ec (0x389b bytes) 000831 1763154641.463772 MIRROR_EXTEND to 9 mirrors 000832 1763154641.984440 write 0x009e3e thru 0x00baee (0x1cb1 bytes) 000834 1763154642.091238 write 0x01ef43 thru 0x02700a (0x80c8 bytes) 000835 1763154642.112476 trunc from 0x02ec19 to 0x026f96 (0x7c82 bytes) 000840 1763154642.156039 trunc from 0x026f97 to 0x005444 (0x21b52 bytes) 000842 1763154642.194688 mapwrite 0x002374 thru 0x00f156 (0xcde3 bytes) truncating to largest ever: 0x3fe35 000846 1763154642.337506 trunc from 0x00f157 to 0x03fe34 (0x30cde bytes) 000847 1763154642.351187 read 0x020593 thru 0x02e806 (0xe274 bytes) 000849 1763154642.367843 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d3:0x0]: Invalid argument (22) 000850 1763154642.614813 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d3:0x0]: Invalid argument (22) 000851 1763154642.820622 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d3:0x0]: Invalid argument (22) 000852 1763154643.087809 MIRROR_EXTEND to 7 mirrors 000853 1763154643.744471 mapwrite 0x022170 thru 0x028313 (0x61a4 bytes) 000854 1763154643.970062 read 0x02a14a thru 0x032f82 (0x8e39 bytes) 000855 1763154643.997143 MIRROR_RESYNC 7 mirrors 000856 1763154645.183204 MIRROR_EXTEND to 8 mirrors 000859 1763154645.989303 read 0x027323 thru 0x032c95 (0xb973 bytes) 000860 1763154645.998111 trunc from 0x03fe35 to 0x017dd7 (0x2805d bytes) 000861 1763154646.083967 trunc from 0x017dd8 to 0x03a1fa (0x22423 bytes) 000863 1763154646.118522 read 0x001510 thru 0x010725 (0xf216 bytes) 000868 1763154646.138658 trunc from 0x03a1fb to 0x03f6e4 (0x54ea bytes) 000871 1763154646.167194 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d3:0x0]: Invalid argument (22) 000873 1763154646.403254 read 0x0229fd thru 0x0326fd (0xfd01 bytes) 000874 1763154646.414128 write 0x005c29 thru 0x011a5a (0xbe32 bytes) 000876 1763154646.427987 mapread 0x02554e thru 0x02d9d3 (0x8486 bytes) 000879 1763154646.437112 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d3:0x0]: Invalid argument (22) 000882 1763154646.723658 read 0x008ea5 thru 0x01852a (0xf686 bytes) 000883 1763154646.743949 read 0x02297a thru 0x02db3a (0xb1c1 bytes) 000884 1763154646.765333 mapwrite 0x033b6f thru 0x03fd70 (0xc202 bytes) 000885 1763154646.888297 mapread 0x028e2f thru 0x03463e (0xb810 bytes) 000886 1763154646.896009 MIRROR_RESYNC 6 mirrors 000887 1763154647.984176 MIRROR_EXTEND to 7 mirrors 000888 1763154648.846971 write 0x037224 thru 0x03ffff (0x8ddc bytes) 000889 1763154648.939602 mapwrite 0x01c2e3 thru 0x01ea47 (0x2765 bytes) 000892 1763154649.018681 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d3:0x0]: Invalid argument (22) 000894 1763154649.260967 trunc from 0x040000 to 0x02446c (0x1bb93 bytes) 000897 1763154649.289797 read 0x01c1d5 thru 0x022b74 (0x69a0 bytes) 000898 1763154649.307820 write 0x025300 thru 0x032ee8 (0xdbe9 bytes) 000900 1763154649.309995 trunc from 0x032ee9 to 0x01a621 (0x188c7 bytes) 000901 1763154649.335151 read 0x007b1d thru 0x00f300 (0x77e4 bytes) 000904 1763154649.350134 MIRROR_RESYNC 6 mirrors 000905 1763154650.525631 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d3:0x0]: Invalid argument (22) 000906 1763154650.865826 write 0x00005c thru 0x00f64d (0xf5f2 bytes) 000909 1763154650.944193 mapread 0x00c0ea thru 0x01a621 (0xe538 bytes) 000918 1763154650.968159 trunc from 0x01a622 to 0x02cae2 (0x124c1 bytes) 000919 1763154650.994154 write 0x021e67 thru 0x02bf3a (0xa0d4 bytes) 000921 1763154651.009928 mapread 0x01210f thru 0x01650f (0x4401 bytes) 000923 1763154651.012192 mapread 0x011070 thru 0x012eb6 (0x1e47 bytes) 000924 1763154651.013431 write 0x02ebc4 thru 0x031119 (0x2556 bytes) 000926 1763154651.014945 mapwrite 0x03dab0 thru 0x03ffff (0x2550 bytes) 000928 1763154651.104035 trunc from 0x040000 to 0x0390fa (0x6f05 bytes) 000930 1763154651.125516 trunc from 0x0390fb to 0x017ed1 (0x21229 bytes) 000933 1763154651.158486 write 0x003950 thru 0x00aec9 (0x757a bytes) 000934 1763154651.160770 read 0x007557 thru 0x014b47 (0xd5f1 bytes) 000935 1763154651.161415 trunc from 0x017ed2 to 0x0088bc (0xf615 bytes) 000936 1763154651.189996 mapwrite 0x006349 thru 0x007e55 (0x1b0d bytes) 000937 1763154651.240063 MIRROR_EXTEND to 6 mirrors 000938 1763154651.903544 trunc from 0x0088bd to 0x0215c1 (0x18d05 bytes) 000939 1763154652.028008 MIRROR_RESYNC 6 mirrors 000941 1763154653.121615 write 0x005b40 thru 0x014e66 (0xf327 bytes) 000942 1763154653.225385 write 0x03dd64 thru 0x03ffff (0x229c bytes) 000943 1763154653.226391 trunc from 0x040000 to 0x01d3e2 (0x22c1d bytes) 000944 1763154653.251410 mapread 0x000eab thru 0x00970d (0x8863 bytes) 000946 1763154653.262157 mapread 0x002476 thru 0x00c2c4 (0x9e4f bytes) 000951 1763154653.270829 MIRROR_EXTEND to 7 mirrors 000952 1763154653.808957 trunc from 0x01d3e3 to 0x028c86 (0xb8a4 bytes) 000953 1763154653.939742 mapwrite 0x021ba3 thru 0x02a1a0 (0x85fe bytes) 000955 1763154654.044590 trunc from 0x02a1a1 to 0x01fe44 (0xa35c bytes) 000956 1763154654.068760 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d3:0x0]: Invalid argument (22) 000957 1763154654.338248 trunc from 0x01fe45 to 0x01e452 (0x19f2 bytes) 000959 1763154654.375008 MIRROR_RESYNC 6 mirrors 000960 1763154655.407299 mapread 0x0113e0 thru 0x013056 (0x1c77 bytes) 000961 1763154655.435896 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d3:0x0]: Invalid argument (22) 000962 1763154655.869305 mapread 0x01ad1c thru 0x01e452 (0x3737 bytes) 000964 1763154655.898966 read 0x00449d thru 0x010994 (0xc4f8 bytes) 000966 1763154655.908852 read 0x013329 thru 0x01a746 (0x741e bytes) 000967 1763154655.926280 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d3:0x0]: Invalid argument (22) 000970 1763154656.312797 MIRROR_RESYNC 4 mirrors 000972 1763154656.611261 mapwrite 0x027848 thru 0x035117 (0xd8d0 bytes) 000974 1763154656.861513 trunc from 0x035118 to 0x002379 (0x32d9e bytes) 000975 1763154656.913518 mapread 0x00101e thru 0x002379 (0x135c bytes) 000977 1763154656.922280 mapwrite 0x0046ad thru 0x0091db (0x4b2f bytes) 000978 1763154657.021203 MIRROR_RESYNC 4 mirrors 000980 1763154657.882850 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d3:0x0]: Invalid argument (22) 000981 1763154658.165221 mapread 0x000a6c thru 0x0091db (0x8770 bytes) 000982 1763154658.195747 mapread 0x002468 thru 0x0091db (0x6d74 bytes) 000983 1763154658.196498 read 0x007831 thru 0x0091db (0x19ab bytes) 000984 1763154658.196985 mapwrite 0x0026b6 thru 0x011a6e (0xf3b9 bytes) 000985 1763154658.386395 read 0x00f56a thru 0x010730 (0x11c7 bytes) 000986 1763154658.391751 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d3:0x0]: Invalid argument (22) 000987 1763154658.610174 write 0x01c649 thru 0x02a023 (0xd9db bytes) 000989 1763154658.612143 trunc from 0x02a024 to 0x02ffde (0x5fbb bytes) 000991 1763154658.636341 trunc from 0x02ffdf to 0x003120 (0x2cebe bytes) 000993 1763154658.675144 MIRROR_EXTEND to 3 mirrors 000994 1763154659.330058 read 0x00191e thru 0x003120 (0x1803 bytes) 000995 1763154659.348986 write 0x0058ce thru 0x00ee38 (0x956b bytes) 000996 1763154659.445403 MIRROR_EXTEND to 4 mirrors 000998 1763154660.202942 trunc from 0x00ee39 to 0x00f68e (0x856 bytes) 000999 1763154660.301362 MIRROR_RESYNC 4 mirrors All operations completed A-OK! PASS 70a (76s) == sanity-flr test 71: check mirror extend parent fid ==== 16:11:05 (1763154665) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 200a: stress test ===================== 16:11:07 (1763154667) Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre2 Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 1313088 .. Extending file size to 4844960 .. failed Extending file size to 4964416 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5348384 .. Extending file size to 5883936 .. Extending file size to 7943648 .. Extending file size to 4942400 .. Extending file size to 8990496 .. Extending file size to 8165920 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8795776 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9024064 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9188416 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 54901 54902 54903 54906 54909 done 192.168.201.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg130-client.virtnet /mnt/lustre2 (opts:) 192.168.201.130@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg130-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (77s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 16:12:24 (1763154744) Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre2 Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3025920 .. Extending file size to 7358208 .. failed Extending file size to 5662816 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8012032 .. Extending file size to 8003584 .. Extending file size to 8075648 .. Extending file size to 9059968 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8919296 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9095104 .. Extending file size to 9316608 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9241760 .. Waiting 56992 56993 56995 56996 56997 done 192.168.201.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg130-client.virtnet /mnt/lustre2 (opts:) 192.168.201.130@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg130-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (75s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 16:13:39 (1763154819) Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4224 2204288 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4224 2204288 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131426 s, 16.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.318113 s, 6.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg130-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 16:13:57 (1763154837) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.122099 s, 17.2 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 16:14:04 (1763154844) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x218:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b3:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b3:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b4:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2ae:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 16:14:11 (1763154851) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.266564 s, 7.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2af:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 16:14:19 (1763154859) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.19784 s, 10.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b2:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 16:14:28 (1763154868) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.224705 s, 9.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b4:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 16:15:00 (1763154900) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg130-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947062 s, 11.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ba:0x0] } PASS 204d (59s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 16:15:59 (1763154959) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg130-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.245086 s, 8.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (58s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 16:16:58 (1763155018) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg130-server mds-ost sync done. Creating new pool test_204f oleg130-server: Pool lustre.test_204f created Adding targets to pool oleg130-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg130-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.194895 s, 10.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg130-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg130-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg130-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 16:18:18 (1763155098) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 16:18:25 (1763155105) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 16:18:32 (1763155112) oleg130-server: Pool lustre.test_206 created Adding targets to pool oleg130-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg130-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg130-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg130-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg130-server: Pool lustre.test_206 destroyed PASS 206 (21s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 16:18:53 (1763155133) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749367 s, 14.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486249 s, 21.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x66fd8900 Mirror 3: 0x66fd8900 PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 16:19:02 (1763155142) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 16:19:05 (1763155145) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 16:19:08 (1763155148) Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.106746 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xc50000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2300000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2300000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2300000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1470000 Read this (4 bytes): BBBB 192.168.201.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg130-client.virtnet /mnt/lustre2 (opts:) PASS 209a (32s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 16:19:40 (1763155180) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0180728 s, 453 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 16:19:47 (1763155187) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438693 s, 23.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x23d lmm_fid: [0x200000404:0x23d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x23d lmm_fid: [0x200000404:0x23d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834493 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-14 16:19:48.000000000 -0500 Modify: 2025-11-14 16:19:48.000000000 -0500 Change: 2025-11-14 16:19:49.000000000 -0500 Birth: 2025-11-14 16:19:47.000000000 -0500 PASS 210a (6s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 16:19:53 (1763155193) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 16:19:56 (1763155196) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.571389 s, 71.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0688206 s, 59.5 kB/s size after second write -rw-r--r-- 1 root root 4096 Nov 14 16:19 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23f:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Nov 14 16:19 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test complete, duration 1402 sec =========== 16:20:05 (1763155205) === sanity-flr: start cleanup 16:20:08 (1763155208) === === sanity-flr: finish cleanup 16:20:15 (1763155215) ===