-----============= acceptance-small: sanity-flr ============----- Wed Apr 30 23:21:30 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg302-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 23:21:40 (1746069700) === oleg302-client.virtnet: executing check_config_client /mnt/lustre oleg302-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg302-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b62e7800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b62e7800.idle_timeout=debug disable quota as required oleg302-server: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 23:21:55 (1746069715) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7034] == sanity-flr test 0a: lfs mirror create with -N option == 23:21:56 (1746069716) PASS 0a (8s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 23:22:04 (1746069724) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 23:22:06 (1746069726) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 23:22:07 (1746069727) PASS 0d (10s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 23:22:17 (1746069737) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 23:22:19 (1746069739) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 23:22:21 (1746069741) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b62e7800.stats=clear osc.lustre-OST0001-osc-ffff8800b62e7800.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 23:22:24 (1746069744) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (5s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 23:22:29 (1746069749) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 23:22:33 (1746069753) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 23:22:35 (1746069755) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (5s) == sanity-flr test 2: create components from existing files ========================================================== 23:22:40 (1746069760) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 23:22:44 (1746069764) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 23:22:46 (1746069766) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 23:22:50 (1746069770) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (4s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 23:22:56 (1746069776) PASS 7 (4s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 23:23:00 (1746069780) 15+0 records in 15+0 records out 15728640 bytes (16 MB) copied, 1.26905 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.133956 s, 7.8 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 23:23:07 (1746069787) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0552356 s, 38.0 MB/s PASS 22 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 23:23:11 (1746069791) Stopping /mnt/lustre-ost1 (opts:) on oleg302-server oleg302-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b62e7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b62e7800.ost_server_uuid in DISCONN state after 0 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg302-server: oleg302-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg302-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg302-server oleg302-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b62e7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b62e7800.ost_server_uuid in DISCONN state after 0 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg302-server: oleg302-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg302-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (35s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 23:23:46 (1746069826) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.142289 s, 36.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg302-server oleg302-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b62e7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b62e7800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg302-server: oleg302-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg302-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (20s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 23:24:06 (1746069846) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg302-server oleg302-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b62e7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b62e7800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg302-server: oleg302-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg302-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg302-server oleg302-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b62e7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b62e7800.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg302-server: oleg302-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg302-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (37s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 23:24:43 (1746069883) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 1.6681 s, 31.4 MB/s -rw-r--r-- 1 root root 52428800 Apr 30 23:24 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.906s user 0m0.000s sys 0m0.153s Stopping /mnt/lustre-ost1 (opts:) on oleg302-server oleg302-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b62e7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b62e7800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.729s user 0m0.001s sys 0m0.147s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg302-server: oleg302-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg302-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg302-server oleg302-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b62e7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b62e7800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.861s user 0m0.000s sys 0m0.147s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg302-server: oleg302-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg302-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg302-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b62e7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b62e7800.ost_server_uuid in FULL state after 0 sec PASS 33b (51s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 23:25:34 (1746069934) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 23:25:36 (1746069936) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 23:25:38 (1746069938) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 23:25:40 (1746069940) PASS 35 (4s) == sanity-flr test 36a: write to mirrored files ========== 23:25:44 (1746069944) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.405914 s, 31.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (9s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 23:25:53 (1746069953) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0344148 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.017394 s, 60.3 MB/s PASS 36b (5s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 23:25:58 (1746069958) PASS 36c (5s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 23:26:03 (1746069963) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00084429 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00222723 s, 460 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.0012034 s, 851 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00433396 s, 236 kB/s flv=10 olv=10 PASS 36d (7s) == sanity-flr test 37: mirror I/O API verification ======= 23:26:10 (1746069970) create test files with size 15749914 .. 5b09e2f60dfc2c4347da93f7106d0c44 - b5844d9bde202358ebae37f7560f786d - 78c79846bde9c9537f0bfe97c7033f8d - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg302-client.virtnet /mnt/lustre (opts:) Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre PASS 37 (18s) == sanity-flr test 38: resync ============================ 23:26:28 (1746069988) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (8s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 23:26:36 (1746069996) PASS 39 (4s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 23:26:40 (1746070000) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.078794 s, 13.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x44:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x46:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0183938 s, 57.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (5s) == sanity-flr test 41: lfs mirror resync check =========== 23:26:45 (1746070005) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.126192 s, 16.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.258978 s, 16.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.209769 s, 10.0 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (8s) == sanity-flr test 42: lfs mirror verify ================= 23:26:53 (1746070013) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 23:26:56 (1746070016) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 23:26:58 (1746070018) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.018288 s, 57.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (5s) == sanity-flr test 44a: lfs mirror split check =========== 23:27:03 (1746070023) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 23:27:05 (1746070025) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4f:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4d:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4d:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (6s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 23:27:12 (1746070032) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.196437 s, 53.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg302-client.virtnet /mnt/lustre (opts:) Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 23:27:23 (1746070043) llite.lustre-ffff8800b62fe000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.259398 s, 40.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg302-client.virtnet /mnt/lustre (opts:) Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800a6222800.xattr_cache=1 PASS 44e (10s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 23:27:33 (1746070053) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (23s) == sanity-flr test 46: Verify setstripe --copy option ==== 23:27:56 (1746070076) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (20s) == sanity-flr test 47: Verify mirror obj alloc =========== 23:28:16 (1746070096) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 23:28:18 (1746070098) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.13783 s, 22.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.10457 s, 30.1 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 4cd1f5c9b1c0204ff39aeeb2e1f97128 - mirror 1 checksum: a526d74ad4d51d869caabc5f9c830f3c - mirror 2 checksum: 4cd1f5c9b1c0204ff39aeeb2e1f97128 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Apr 30 23:28:19 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Apr 30 23:28:19 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 4cd1f5c9b1c0204ff39aeeb2e1f97128 - mirror 1 checksum: a526d74ad4d51d869caabc5f9c830f3c - mirror 2 checksum: a526d74ad4d51d869caabc5f9c830f3c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Apr 30 23:28:19 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Apr 30 23:28:19 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } PASS 48 (6s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 23:28:25 (1746070105) Failing mds1 on oleg302-server Stopping /mnt/lustre-mds1 (opts:) on oleg302-server 23:28:29 (1746070109) shut down facet: mds1 facet_host: oleg302-server facet_failover_host: oleg302-server Failover mds1 to oleg302-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg302-server: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-MDT0000 23:28:46 (1746070126) targets are mounted 23:28:46 (1746070126) facet_failover done oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0179749 s, 58.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } PASS 50A (31s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 23:28:56 (1746070136) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 23:28:58 (1746070138) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 23:29:00 (1746070140) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.0136349 s, 1.2 MB/s PASS 50c (5s) == sanity-flr test 50d: mirror rsync keep holes ========== 23:29:05 (1746070145) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 23:29:06 (1746070146) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 23:29:08 (1746070148) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (12s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 23:29:21 (1746070161) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (22s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 23:29:43 (1746070183) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (20s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 23:30:03 (1746070203) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (20s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 23:30:23 (1746070223) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (4s) == sanity-flr test 70: mirror create and split race ====== 23:30:27 (1746070227) mirror create pid 22630 mirror split pid 22632 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 22630 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 22632 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (65s) == sanity-flr test 70a: flr mode fsx test ================ 23:31:32 (1746070292) Seed set to 293 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x3889d 000010 1746070293.650653 trunc from 00000000 to 0x03889c (0x3889d bytes) 000011 1746070293.672438 mapread 0x02e19f thru 0x03889c (0xa6fe bytes) 000012 1746070293.699114 mapwrite 0x03216c thru 0x03e519 (0xc3ae bytes) 000013 1746070293.869509 read 0x004455 thru 0x009437 (0x4fe3 bytes) 000016 1746070293.877568 trunc from 0x03e51a to 0x01a85f (0x23cba bytes) 000018 1746070293.892021 read 0x0094f1 thru 0x00eb92 (0x56a2 bytes) 000025 1746070293.911117 trunc from 0x01a860 to 0x00431a (0x16545 bytes) 000027 1746070293.969396 write 0x007606 thru 0x012af1 (0xb4ec bytes) 000029 1746070293.971131 write 0x022c64 thru 0x031ae9 (0xee86 bytes) 000031 1746070293.972602 trunc from 0x031aea to 0x00e40a (0x236df bytes) 000036 1746070293.990796 mapread 0x0093a3 thru 0x00bcc1 (0x291f bytes) 000037 1746070293.992000 read 0x002ef4 thru 0x00e40a (0xb517 bytes) 000038 1746070293.996187 read 0x007472 thru 0x00c748 (0x52d7 bytes) 000041 1746070293.998073 mapwrite 0x018db4 thru 0x023d75 (0xafc2 bytes) 000042 1746070294.108747 read 0x0002c3 thru 0x0095c2 (0x9300 bytes) 000044 1746070294.109525 mapwrite 0x01492c thru 0x018fbe (0x4693 bytes) 000045 1746070294.159381 write 0x019f55 thru 0x01ec1c (0x4cc8 bytes) 000050 1746070294.161384 mapwrite 0x02648b thru 0x032fa4 (0xcb1a bytes) 000052 1746070294.250338 mapread 0x02dc11 thru 0x02f1d8 (0x15c8 bytes) 000053 1746070294.250974 mapread 0x024279 thru 0x0261d7 (0x1f5f bytes) 000054 1746070294.257155 MIRROR_EXTEND to 2 mirrors 000055 1746070294.706441 read 0x028af7 thru 0x032fa4 (0xa4ae bytes) 000056 1746070294.719907 write 0x038173 thru 0x03d8e1 (0x576f bytes) 000057 1746070294.805923 mapwrite 0x01934f thru 0x01b30a (0x1fbc bytes) 000058 1746070294.932034 trunc from 0x03d8e2 to 0x00347c (0x3a465 bytes) 000059 1746070294.997149 read 0x000a05 thru 0x00347c (0x2a78 bytes) 000061 1746070295.006237 trunc from 0x00347d to 0x01b460 (0x17fe4 bytes) 000062 1746070295.035166 mapwrite 0x0214d1 thru 0x0291d7 (0x7d07 bytes) 000064 1746070295.177450 read 0x0137d4 thru 0x01db61 (0xa38e bytes) 000065 1746070295.178841 trunc from 0x0291d8 to 0x027baa (0x162d bytes) 000069 1746070295.207674 write 0x027410 thru 0x03138c (0x9f7d bytes) 000070 1746070295.214774 MIRROR_RESYNC 2 mirrors 000071 1746070295.645341 write 0x02615c thru 0x02ce18 (0x6cbd bytes) 000072 1746070295.758945 mapwrite 0x0188d3 thru 0x019ddc (0x150a bytes) 000075 1746070295.826934 MIRROR_RESYNC 2 mirrors 000076 1746070296.211407 read 0x0071aa thru 0x009a87 (0x28de bytes) 000079 1746070296.219872 write 0x026d2c thru 0x02aa95 (0x3d6a bytes) 000081 1746070296.288724 write 0x006def thru 0x0076a5 (0x8b7 bytes) 000082 1746070296.305581 mapread 0x00f75a thru 0x01ba60 (0xc307 bytes) 000084 1746070296.323221 read 0x01d0a5 thru 0x020002 (0x2f5e bytes) 000085 1746070296.325374 trunc from 0x03138d to 0x008c99 (0x286f3 bytes) 000086 1746070296.345650 write 0x01c8e6 thru 0x020e4e (0x4569 bytes) 000088 1746070296.359058 MIRROR_EXTEND to 3 mirrors 000089 1746070296.883278 read 0x017c05 thru 0x020e4e (0x924a bytes) 000094 1746070296.947375 MIRROR_RESYNC 3 mirrors 000095 1746070297.386220 trunc from 0x020e4f to 0x03232a (0x114dc bytes) 000096 1746070297.445331 mapread 0x0292d2 thru 0x02cc3d (0x396c bytes) 000097 1746070297.461243 read 0x01eff1 thru 0x02e3d7 (0xf3e7 bytes) 000098 1746070297.469765 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x276:0x0]: Invalid argument (22) 000103 1746070297.610616 mapwrite 0x02cf24 thru 0x03b8c7 (0xe9a4 bytes) 000105 1746070297.703614 trunc from 0x03b8c8 to 0x021b1e (0x19da9 bytes) 000107 1746070297.717656 mapread 0x01192e thru 0x0141d5 (0x28a8 bytes) truncating to largest ever: 0x3c550 000108 1746070297.726040 trunc from 0x021b1f to 0x03c54f (0x1aa31 bytes) 000110 1746070297.738945 mapread 0x0324d6 thru 0x03a105 (0x7c30 bytes) 000111 1746070297.749458 write 0x00d541 thru 0x01424f (0x6d0f bytes) 000114 1746070297.752875 trunc from 0x03c550 to 0x02481a (0x17d35 bytes) 000115 1746070297.763038 trunc from 0x02481b to 0x03356d (0xed53 bytes) 000116 1746070297.775627 mapread 0x0201cf thru 0x02edcf (0xec01 bytes) 000117 1746070297.790913 mapwrite 0x03c858 thru 0x03ffff (0x37a8 bytes) 000118 1746070297.894574 mapwrite 0x005c15 thru 0x007722 (0x1b0e bytes) 000120 1746070297.975524 mapwrite 0x01e7a7 thru 0x01fb7e (0x13d8 bytes) 000122 1746070298.024254 MIRROR_RESYNC 2 mirrors 000123 1746070298.392297 trunc from 0x040000 to 0x03afc7 (0x5038 bytes) 000124 1746070298.464058 trunc from 0x03afc8 to 0x004cf4 (0x362d3 bytes) 000125 1746070298.488040 mapwrite 0x00d4e9 thru 0x0144b4 (0x6fcc bytes) 000126 1746070298.600089 read 0x004334 thru 0x00a39c (0x6069 bytes) 000128 1746070298.606712 mapwrite 0x01076b thru 0x01be4f (0xb6e5 bytes) 000129 1746070298.691739 trunc from 0x01be50 to 0x012c3e (0x9211 bytes) 000130 1746070298.703327 mapwrite 0x021f1e thru 0x02a5cc (0x86af bytes) 000131 1746070298.793645 read 0x003fac thru 0x0083bd (0x4412 bytes) 000132 1746070298.801429 write 0x01d259 thru 0x02475a (0x7502 bytes) 000133 1746070298.802279 write 0x018b73 thru 0x01c157 (0x35e5 bytes) 000135 1746070298.803236 mapwrite 0x0370ad thru 0x03d323 (0x6277 bytes) 000136 1746070298.906218 mapwrite 0x005296 thru 0x00dc2e (0x8999 bytes) 000137 1746070298.980391 mapread 0x022f4b thru 0x031e99 (0xef4f bytes) 000140 1746070298.981674 mapread 0x01e0bc thru 0x02397f (0x58c4 bytes) 000143 1746070298.982725 write 0x021008 thru 0x02fded (0xede6 bytes) 000145 1746070298.985973 mapwrite 0x01bda6 thru 0x0254d5 (0x9730 bytes) 000146 1746070299.067905 read 0x00f6cc thru 0x019c43 (0xa578 bytes) 000148 1746070299.069995 trunc from 0x03d324 to 0x020bd6 (0x1c74d bytes) 000158 1746070299.097754 write 0x028024 thru 0x02ed98 (0x6d75 bytes) 000159 1746070299.098734 write 0x012e84 thru 0x01a76b (0x78e8 bytes) 000160 1746070299.106466 MIRROR_RESYNC 2 mirrors 000161 1746070299.558925 trunc from 0x02ed99 to 0x01b211 (0x13b87 bytes) 000162 1746070299.641232 read 0x0038e7 thru 0x00ff51 (0xc66b bytes) 000163 1746070299.649324 mapwrite 0x018095 thru 0x01ffa9 (0x7f15 bytes) 000165 1746070299.748471 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x276:0x0]: Invalid argument (22) 000167 1746070299.873154 mapread 0x0090f4 thru 0x018d2f (0xfc3c bytes) 000170 1746070299.893430 trunc from 0x01ffaa to 0x0142d7 (0xbcd2 bytes) 000171 1746070299.907631 trunc from 0x0142d8 to 0x010020 (0x42b7 bytes) 000172 1746070299.920402 write 0x0398fe thru 0x03ffff (0x6702 bytes) 000174 1746070299.922611 read 0x00960a thru 0x00f30e (0x5d05 bytes) 000179 1746070299.925162 write 0x01aa6d thru 0x02a8b3 (0xfe47 bytes) 000181 1746070299.936408 mapwrite 0x039da6 thru 0x03ffff (0x625a bytes) 000182 1746070299.981372 write 0x02af25 thru 0x02f213 (0x42ef bytes) 000183 1746070299.986570 trunc from 0x040000 to 0x031de4 (0xe21b bytes) 000185 1746070300.000025 write 0x009e96 thru 0x00a8e9 (0xa54 bytes) 000186 1746070300.000715 trunc from 0x031de5 to 0x03be9e (0xa0ba bytes) 000187 1746070300.018936 mapread 0x0051f1 thru 0x01181f (0xc62f bytes) 000188 1746070300.038029 read 0x02926b thru 0x02ff30 (0x6cc6 bytes) 000190 1746070300.040773 read 0x0151ef thru 0x021fdd (0xcdef bytes) 000193 1746070300.042128 mapwrite 0x025119 thru 0x0296a4 (0x458c bytes) 000195 1746070300.109782 trunc from 0x03be9f to 0x003fac (0x37ef2 bytes) 000197 1746070300.126068 mapwrite 0x03034e thru 0x036420 (0x60d3 bytes) 000198 1746070300.308680 write 0x0224f2 thru 0x02964a (0x7159 bytes) 000200 1746070300.329445 MIRROR_EXTEND to 2 mirrors 000201 1746070300.739305 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x276:0x0]: Invalid argument (22) 000202 1746070300.844735 trunc from 0x036421 to 0x01907e (0x1d3a2 bytes) 000204 1746070300.883469 mapread 0x015c7c thru 0x01907e (0x3403 bytes) 000208 1746070300.897303 mapwrite 0x006036 thru 0x014494 (0xe45f bytes) 000209 1746070300.967528 read 0x008043 thru 0x017bc3 (0xfb81 bytes) 000210 1746070300.968111 trunc from 0x01907f to 0x02c755 (0x136d7 bytes) 000214 1746070300.999885 mapread 0x02873d thru 0x02c4b9 (0x3d7d bytes) 000215 1746070301.018331 trunc from 0x02c756 to 0x00627e (0x264d7 bytes) 000216 1746070301.035132 trunc from 0x00627f to 0x03827f (0x32001 bytes) 000217 1746070301.046962 mapwrite 0x005e7a thru 0x00af26 (0x50ad bytes) 000218 1746070301.154661 write 0x03217e thru 0x03867f (0x6502 bytes) 000219 1746070301.172015 mapwrite 0x00c1fb thru 0x017e2d (0xbc33 bytes) 000220 1746070301.260843 mapwrite 0x011954 thru 0x01991c (0x7fc9 bytes) 000222 1746070301.319686 read 0x021863 thru 0x02284d (0xfeb bytes) 000226 1746070301.329128 mapread 0x00db96 thru 0x01d1de (0xf649 bytes) 000228 1746070301.331327 read 0x0165e6 thru 0x024a12 (0xe42d bytes) 000232 1746070301.333876 mapwrite 0x013f8b thru 0x01b93e (0x79b4 bytes) 000233 1746070301.394170 mapread 0x02fc05 thru 0x036bcb (0x6fc7 bytes) 000237 1746070301.396649 read 0x0043c1 thru 0x0056ed (0x132d bytes) 000239 1746070301.402080 read 0x03615e thru 0x03867f (0x2522 bytes) 000240 1746070301.402597 mapread 0x030a81 thru 0x03867f (0x7bff bytes) 000241 1746070301.404098 mapwrite 0x02e5d5 thru 0x039221 (0xac4d bytes) 000242 1746070301.526890 mapread 0x02dade thru 0x039221 (0xb744 bytes) 000243 1746070301.528162 read 0x00bf13 thru 0x01625b (0xa349 bytes) 000244 1746070301.528586 write 0x03036a thru 0x0346d4 (0x436b bytes) 000246 1746070301.530717 mapwrite 0x020281 thru 0x02c7f1 (0xc571 bytes) 000250 1746070301.581102 trunc from 0x039222 to 0x035c27 (0x35fa bytes) 000251 1746070301.612312 mapwrite 0x022780 thru 0x02682f (0x40b0 bytes) 000253 1746070301.681329 read 0x022d28 thru 0x028bdb (0x5eb4 bytes) 000254 1746070301.681825 mapwrite 0x02c7d5 thru 0x0358c4 (0x90f0 bytes) 000255 1746070301.776693 read 0x0285c4 thru 0x02c1d7 (0x3c14 bytes) 000256 1746070301.778290 write 0x02df13 thru 0x03b767 (0xd855 bytes) 000260 1746070301.784462 mapread 0x03b66a thru 0x03b767 (0xfe bytes) 000262 1746070301.795858 read 0x0352a4 thru 0x03b767 (0x64c4 bytes) 000263 1746070301.796295 write 0x01c15b thru 0x024bfb (0x8aa1 bytes) 000265 1746070301.809920 write 0x03f7c1 thru 0x03ffff (0x83f bytes) 000266 1746070301.812273 write 0x020992 thru 0x03051d (0xfb8c bytes) 000270 1746070301.820246 mapread 0x03b1fc thru 0x03ffff (0x4e04 bytes) 000271 1746070301.827916 mapread 0x038b34 thru 0x03df88 (0x5455 bytes) 000272 1746070301.828721 read 0x006c6a thru 0x0082ce (0x1665 bytes) 000273 1746070301.829884 write 0x0346fa thru 0x0374ee (0x2df5 bytes) 000274 1746070301.830986 read 0x02a69c thru 0x030030 (0x5995 bytes) 000275 1746070301.831361 mapread 0x03428d thru 0x03ffff (0xbd73 bytes) 000277 1746070301.839082 MIRROR_EXTEND to 2 mirrors 000281 1746070302.488579 mapwrite 0x00ea35 thru 0x018e88 (0xa454 bytes) 000283 1746070302.665693 mapwrite 0x00910c thru 0x009a50 (0x945 bytes) 000284 1746070302.709168 read 0x03e735 thru 0x03ffff (0x18cb bytes) 000285 1746070302.714890 write 0x0368ab thru 0x03e8f5 (0x804b bytes) 000288 1746070302.726722 MIRROR_RESYNC 2 mirrors 000289 1746070303.247237 write 0x0336af thru 0x03c3ea (0x8d3c bytes) 000290 1746070303.314431 trunc from 0x040000 to 0x037e3a (0x81c5 bytes) 000291 1746070303.332278 mapwrite 0x03387c thru 0x03d9e9 (0xa16e bytes) 000292 1746070303.432075 trunc from 0x03d9ea to 0x015737 (0x282b2 bytes) 000293 1746070303.474269 read 0x007659 thru 0x00ff49 (0x88f1 bytes) 000298 1746070303.485652 mapwrite 0x014c03 thru 0x01d46e (0x886c bytes) 000300 1746070303.600553 mapread 0x00f1ba thru 0x011adb (0x2922 bytes) 000302 1746070303.601604 mapwrite 0x03a267 thru 0x03ffff (0x5d99 bytes) 000304 1746070303.736985 mapread 0x026353 thru 0x028014 (0x1cc2 bytes) 000306 1746070303.759841 mapwrite 0x01409f thru 0x015256 (0x11b8 bytes) 000308 1746070303.828467 write 0x0304a2 thru 0x039030 (0x8b8f bytes) 000309 1746070303.844361 MIRROR_EXTEND to 3 mirrors 000310 1746070304.349011 read 0x03bcdd thru 0x03ffff (0x4323 bytes) 000314 1746070304.415710 trunc from 0x040000 to 0x00ee1c (0x311e3 bytes) 000315 1746070304.497734 mapread 0x00e8c3 thru 0x00ee1c (0x55a bytes) 000316 1746070304.511979 trunc from 0x00ee1d to 0x0186dc (0x98c0 bytes) 000317 1746070304.543551 trunc from 0x0186dd to 0x026d40 (0xe664 bytes) 000318 1746070304.567995 mapread 0x0268b5 thru 0x026d40 (0x48c bytes) 000320 1746070304.576384 read 0x00b4d0 thru 0x00c05d (0xb8e bytes) 000321 1746070304.585819 read 0x02336f thru 0x023fbe (0xc50 bytes) 000323 1746070304.586575 mapread 0x015186 thru 0x0179e7 (0x2862 bytes) 000324 1746070304.587392 mapwrite 0x02e100 thru 0x037191 (0x9092 bytes) 000326 1746070304.648732 read 0x021893 thru 0x02b708 (0x9e76 bytes) 000329 1746070304.652018 read 0x019195 thru 0x01f0e8 (0x5f54 bytes) 000331 1746070304.653644 mapread 0x02958e thru 0x030aeb (0x755e bytes) 000332 1746070304.654634 mapwrite 0x00d27e thru 0x01168a (0x440d bytes) 000333 1746070304.702354 read 0x01cc31 thru 0x02c8e7 (0xfcb7 bytes) 000334 1746070304.704449 mapread 0x019809 thru 0x01e811 (0x5009 bytes) 000335 1746070304.705334 mapread 0x011ec4 thru 0x01bff3 (0xa130 bytes) 000338 1746070304.709059 trunc from 0x037192 to 0x02ab46 (0xc64b bytes) 000339 1746070304.739601 MIRROR_RESYNC 3 mirrors 000342 1746070305.292568 write 0x00dfb6 thru 0x019a61 (0xbaac bytes) 000344 1746070305.377568 mapwrite 0x00cf28 thru 0x0170d4 (0xa1ad bytes) 000345 1746070305.472182 trunc from 0x02ab47 to 0x02c4a3 (0x195d bytes) 000346 1746070305.492071 mapread 0x00ab3b thru 0x011047 (0x650d bytes) 000347 1746070305.494397 mapwrite 0x03029c thru 0x0396bf (0x9424 bytes) 000350 1746070305.588749 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x276:0x0]: Invalid argument (22) 000351 1746070305.787186 MIRROR_RESYNC 2 mirrors 000353 1746070306.329289 read 0x00366e thru 0x00e351 (0xace4 bytes) 000354 1746070306.339892 trunc from 0x0396c0 to 0x027ce4 (0x119db bytes) 000355 1746070306.410869 read 0x01a97b thru 0x01c98e (0x2014 bytes) 000357 1746070306.425784 MIRROR_RESYNC 2 mirrors 000358 1746070306.761394 trunc from 0x027ce5 to 0x0114d4 (0x16810 bytes) 000360 1746070306.857391 write 0x003b30 thru 0x00d586 (0x9a57 bytes) 000361 1746070306.869804 MIRROR_RESYNC 2 mirrors 000364 1746070307.275804 mapwrite 0x021e81 thru 0x024f4a (0x30ca bytes) 000366 1746070307.498465 mapread 0x01defc thru 0x024f4a (0x704f bytes) 000369 1746070307.507362 trunc from 0x024f4b to 0x02b17c (0x6232 bytes) 000370 1746070307.571237 write 0x025d7f thru 0x02775e (0x19e0 bytes) 000372 1746070307.590381 read 0x01ec53 thru 0x02aa88 (0xbe36 bytes) 000373 1746070307.603311 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x276:0x0]: Invalid argument (22) 000375 1746070307.847458 mapwrite 0x00f686 thru 0x01805b (0x89d6 bytes) 000376 1746070308.002160 mapwrite 0x00001a thru 0x007a55 (0x7a3c bytes) 000379 1746070308.059272 mapwrite 0x02da4a thru 0x0378bb (0x9e72 bytes) 000381 1746070308.166413 write 0x025fb8 thru 0x033f43 (0xdf8c bytes) 000382 1746070308.176229 mapread 0x010a14 thru 0x01da2f (0xd01c bytes) 000384 1746070308.183431 read 0x021e76 thru 0x02fac6 (0xdc51 bytes) 000387 1746070308.184869 mapwrite 0x0290a1 thru 0x036b6b (0xdacb bytes) 000392 1746070308.248980 MIRROR_EXTEND to 2 mirrors 000393 1746070308.680934 write 0x0388d1 thru 0x03ffff (0x772f bytes) 000395 1746070308.740148 mapread 0x036719 thru 0x03ffff (0x98e7 bytes) 000396 1746070308.748600 trunc from 0x040000 to 0x02f7f3 (0x1080c bytes) 000398 1746070308.784039 read 0x00c5af thru 0x01a426 (0xde78 bytes) 000399 1746070308.798811 read 0x018b0a thru 0x027b0d (0xf004 bytes) 000400 1746070308.814862 mapwrite 0x018d05 thru 0x01c261 (0x355d bytes) 000401 1746070308.864977 write 0x01cc25 thru 0x02b920 (0xecfc bytes) 000402 1746070308.866385 trunc from 0x02f7f4 to 0x022729 (0xd0ca bytes) 000403 1746070308.885692 mapwrite 0x005fa5 thru 0x0100d6 (0xa132 bytes) 000404 1746070308.960607 mapread 0x015b70 thru 0x021502 (0xb993 bytes) 000407 1746070308.962088 mapwrite 0x0066d5 thru 0x00abb7 (0x44e3 bytes) 000408 1746070309.013993 read 0x0036ca thru 0x00bb9d (0x84d4 bytes) 000411 1746070309.015180 mapread 0x009c00 thru 0x0144f0 (0xa8f1 bytes) 000412 1746070309.017277 write 0x0081d7 thru 0x00e870 (0x669a bytes) 000413 1746070309.018012 write 0x015348 thru 0x021d80 (0xca39 bytes) 000415 1746070309.019213 write 0x037a87 thru 0x03ffff (0x8579 bytes) 000417 1746070309.035397 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x276:0x0]: Invalid argument (22) 000419 1746070309.218586 trunc from 0x040000 to 0x02b33f (0x14cc0 bytes) 000420 1746070309.234135 write 0x00e58b thru 0x0133ce (0x4e44 bytes) 000421 1746070309.243627 write 0x010d4b thru 0x017628 (0x68de bytes) 000423 1746070309.248975 trunc from 0x02b340 to 0x02661a (0x4d25 bytes) 000424 1746070309.261823 mapread 0x000bd1 thru 0x00bd20 (0xb150 bytes) 000425 1746070309.275485 mapread 0x01164c thru 0x01bc78 (0xa62d bytes) 000426 1746070309.283198 mapread 0x01b59e thru 0x01d8fe (0x2361 bytes) 000429 1746070309.284584 read 0x007f9a thru 0x01699a (0xea01 bytes) 000432 1746070309.285660 mapwrite 0x007104 thru 0x00f23a (0x8137 bytes) 000435 1746070309.355720 mapwrite 0x035873 thru 0x03b157 (0x58e5 bytes) 000438 1746070309.425972 mapread 0x03aa5a thru 0x03b157 (0x6fe bytes) 000441 1746070309.428639 read 0x01d6fa thru 0x029188 (0xba8f bytes) 000443 1746070309.437948 mapread 0x0370d4 thru 0x037e7f (0xdac bytes) 000444 1746070309.438876 mapwrite 0x002619 thru 0x00b778 (0x9160 bytes) 000447 1746070309.498738 read 0x030853 thru 0x03b157 (0xa905 bytes) truncating to largest ever: 0x3fd2d 000448 1746070309.499218 trunc from 0x03b158 to 0x03fd2c (0x4bd5 bytes) 000452 1746070309.512827 mapwrite 0x039c17 thru 0x03a705 (0xaef bytes) 000453 1746070309.563592 read 0x0290bc thru 0x0306f3 (0x7638 bytes) 000455 1746070309.564410 mapread 0x03791a thru 0x03bc5e (0x4345 bytes) 000457 1746070309.565333 trunc from 0x03fd2d to 0x020598 (0x1f794 bytes) 000459 1746070309.578232 write 0x038ea2 thru 0x03bd40 (0x2e9f bytes) 000460 1746070309.579486 read 0x0171a2 thru 0x01eb7a (0x79d9 bytes) 000461 1746070309.579870 read 0x01b183 thru 0x01edf6 (0x3c74 bytes) 000465 1746070309.581623 read 0x01f188 thru 0x028aab (0x9924 bytes) 000466 1746070309.591621 read 0x0373f1 thru 0x03bd40 (0x4950 bytes) 000469 1746070309.592684 read 0x00e546 thru 0x010566 (0x2021 bytes) 000470 1746070309.593077 read 0x00251e thru 0x006457 (0x3f3a bytes) 000471 1746070309.593440 write 0x02e14a thru 0x03d41a (0xf2d1 bytes) 000472 1746070309.594440 write 0x0089e6 thru 0x0163c3 (0xd9de bytes) 000476 1746070309.598727 trunc from 0x03d41b to 0x0013ab (0x3c06f bytes) 000477 1746070309.618399 read 0x000631 thru 0x0013ab (0xd7b bytes) 000478 1746070309.619384 mapwrite 0x01da95 thru 0x01e618 (0xb84 bytes) 000479 1746070309.691190 trunc from 0x01e619 to 0x0365b6 (0x17f9e bytes) 000480 1746070309.706113 write 0x013c5c thru 0x01c0fb (0x84a0 bytes) 000481 1746070309.707048 write 0x02250f thru 0x02780b (0x52fd bytes) 000484 1746070309.716283 mapread 0x02770d thru 0x033e56 (0xc74a bytes) 000486 1746070309.717560 mapread 0x019b06 thru 0x023873 (0x9d6e bytes) 000487 1746070309.718276 mapread 0x00b611 thru 0x01089f (0x528f bytes) 000488 1746070309.718979 read 0x01a488 thru 0x01cac7 (0x2640 bytes) 000491 1746070309.725464 mapwrite 0x02a79e thru 0x035423 (0xac86 bytes) 000494 1746070309.793114 read 0x018d12 thru 0x01c492 (0x3781 bytes) 000496 1746070309.793897 mapread 0x03182b thru 0x0365b6 (0x4d8c bytes) 000497 1746070309.794545 write 0x021464 thru 0x030c19 (0xf7b6 bytes) 000498 1746070309.795539 write 0x007fd3 thru 0x010acd (0x8afb bytes) 000500 1746070309.797757 mapwrite 0x01be0e thru 0x02b575 (0xf768 bytes) 000502 1746070309.863217 write 0x0067fa thru 0x00fcc3 (0x94ca bytes) 000504 1746070309.864301 mapread 0x0268cd thru 0x02b18f (0x48c3 bytes) 000506 1746070309.865167 mapwrite 0x0338a5 thru 0x0339ac (0x108 bytes) 000507 1746070309.926522 read 0x034339 thru 0x035a21 (0x16e9 bytes) 000508 1746070309.926950 mapread 0x003724 thru 0x009ed2 (0x67af bytes) 000509 1746070309.927530 trunc from 0x0365b7 to 0x030cb3 (0x5903 bytes) 000510 1746070309.948328 write 0x03d99e thru 0x03ffff (0x2662 bytes) 000513 1746070309.953920 MIRROR_EXTEND to 2 mirrors 000515 1746070310.430782 MIRROR_EXTEND to 3 mirrors 000516 1746070310.888917 write 0x03e0a1 thru 0x03ffff (0x1f5f bytes) 000517 1746070310.974069 trunc from 0x040000 to 0x00dc7f (0x32380 bytes) 000518 1746070311.019384 MIRROR_RESYNC 3 mirrors 000519 1746070311.707810 mapread 0x00a4b3 thru 0x00dc7f (0x37cd bytes) 000522 1746070311.756943 read 0x001673 thru 0x00dc7f (0xc60d bytes) 000523 1746070311.760361 trunc from 0x00dc80 to 0x028c42 (0x1afc3 bytes) 000526 1746070311.845473 write 0x012900 thru 0x015baf (0x32b0 bytes) 000531 1746070311.862150 mapread 0x010b30 thru 0x01d244 (0xc715 bytes) 000533 1746070311.873530 trunc from 0x028c43 to 0x031cbb (0x9079 bytes) 000534 1746070311.893330 trunc from 0x031cbc to 0x017de9 (0x19ed2 bytes) 000535 1746070311.910669 write 0x01dfda thru 0x026d0b (0x8d32 bytes) 000536 1746070311.913032 mapread 0x020685 thru 0x026d0b (0x6687 bytes) 000538 1746070311.914438 trunc from 0x026d0c to 0x00b356 (0x1b9b5 bytes) 000540 1746070311.929426 mapwrite 0x00be7b thru 0x01073a (0x48c0 bytes) 000541 1746070312.013415 mapread 0x009322 thru 0x01073a (0x7419 bytes) 000543 1746070312.014433 trunc from 0x01073b to 0x0393a6 (0x28c6c bytes) 000548 1746070312.032953 mapwrite 0x023d5c thru 0x0275b3 (0x3858 bytes) 000550 1746070312.111310 MIRROR_RESYNC 3 mirrors 000552 1746070312.653393 write 0x018160 thru 0x01a38d (0x222e bytes) 000554 1746070312.719329 MIRROR_RESYNC 3 mirrors 000555 1746070313.240903 write 0x00333a thru 0x00c447 (0x910e bytes) 000557 1746070313.346185 write 0x02e0c2 thru 0x030c9e (0x2bdd bytes) 000561 1746070313.356584 write 0x0010fb thru 0x00eec3 (0xddc9 bytes) 000562 1746070313.369765 MIRROR_EXTEND to 4 mirrors 000566 1746070313.904873 MIRROR_RESYNC 4 mirrors 000567 1746070314.348577 mapread 0x01c18b thru 0x0266bf (0xa535 bytes) 000568 1746070314.367599 write 0x0087df thru 0x014701 (0xbf23 bytes) 000570 1746070314.444365 mapwrite 0x02318e thru 0x031c21 (0xea94 bytes) 000571 1746070314.528334 mapread 0x0381eb thru 0x0393a6 (0x11bc bytes) 000573 1746070314.537964 trunc from 0x0393a7 to 0x02ec85 (0xa721 bytes) 000574 1746070314.559154 MIRROR_EXTEND to 5 mirrors 000575 1746070314.997127 mapwrite 0x01d36e thru 0x0265d0 (0x9263 bytes) 000577 1746070315.170205 mapwrite 0x03d545 thru 0x03ffff (0x2abb bytes) 000584 1746070315.250149 mapwrite 0x017bc1 thru 0x024c66 (0xd0a6 bytes) 000587 1746070315.320233 read 0x02e6f8 thru 0x036553 (0x7e5c bytes) 000589 1746070315.334403 trunc from 0x040000 to 0x037732 (0x88cd bytes) 000592 1746070315.354264 write 0x01dbb8 thru 0x0229e7 (0x4e30 bytes) 000593 1746070315.355106 read 0x011925 thru 0x016aec (0x51c8 bytes) 000594 1746070315.374960 read 0x02ac7f thru 0x02f22d (0x45af bytes) 000595 1746070315.375398 write 0x02b20a thru 0x03051c (0x5313 bytes) 000598 1746070315.378110 trunc from 0x037733 to 0x01706f (0x206c3 bytes) 000599 1746070315.401366 mapread 0x016bcb thru 0x01706f (0x4a5 bytes) 000602 1746070315.404276 mapwrite 0x001902 thru 0x00a2cb (0x89ca bytes) 000604 1746070315.482984 read 0x002d72 thru 0x00b8d2 (0x8b61 bytes) 000606 1746070315.486446 MIRROR_EXTEND to 6 mirrors 000607 1746070316.008608 mapread 0x00c0c3 thru 0x016949 (0xa887 bytes) 000611 1746070316.058888 MIRROR_RESYNC 6 mirrors 000613 1746070316.724516 mapwrite 0x03d263 thru 0x03e829 (0x15c7 bytes) 000614 1746070316.973116 read 0x00f7db thru 0x01a515 (0xad3b bytes) 000617 1746070316.981322 read 0x02d68c thru 0x035e95 (0x880a bytes) 000620 1746070316.992368 read 0x00e6df thru 0x013536 (0x4e58 bytes) 000621 1746070316.994294 read 0x01bd77 thru 0x01c70f (0x999 bytes) 000625 1746070316.997004 write 0x01ac27 thru 0x01c1e5 (0x15bf bytes) 000626 1746070316.999315 read 0x02cee1 thru 0x02deab (0xfcb bytes) 000627 1746070317.000382 mapwrite 0x0084a3 thru 0x0105df (0x813d bytes) 000630 1746070317.061152 trunc from 0x03e82a to 0x028a63 (0x15dc6 bytes) 000632 1746070317.086254 read 0x02615f thru 0x028a63 (0x2905 bytes) 000634 1746070317.091956 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x276:0x0]: Invalid argument (22) 000635 1746070317.305380 write 0x021a88 thru 0x0258d0 (0x3e49 bytes) 000636 1746070317.352832 trunc from 0x028a64 to 0x0260d1 (0x2992 bytes) 000637 1746070317.380912 mapwrite 0x038e46 thru 0x03ffff (0x71ba bytes) 000640 1746070317.492901 trunc from 0x040000 to 0x02d1e5 (0x12e1a bytes) 000642 1746070317.513387 mapread 0x02be59 thru 0x02d1e5 (0x138d bytes) 000643 1746070317.526377 MIRROR_RESYNC 5 mirrors 000644 1746070318.110150 mapread 0x0145c5 thru 0x01a921 (0x635d bytes) 000645 1746070318.125945 mapread 0x0164ad thru 0x0187e6 (0x233a bytes) 000647 1746070318.130522 MIRROR_RESYNC 5 mirrors 000648 1746070318.325978 mapwrite 0x0037d7 thru 0x00e376 (0xaba0 bytes) 000649 1746070318.595467 trunc from 0x02d1e6 to 0x0110bd (0x1c128 bytes) 000650 1746070318.671026 mapread 0x00c3e4 thru 0x0110bd (0x4cda bytes) 000651 1746070318.677533 read 0x00c14c thru 0x0110bd (0x4f72 bytes) 000655 1746070318.691034 MIRROR_RESYNC 5 mirrors 000657 1746070319.312973 write 0x00e1e3 thru 0x014664 (0x6482 bytes) 000658 1746070319.383147 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x276:0x0]: Invalid argument (22) 000659 1746070319.558955 trunc from 0x014665 to 0x019608 (0x4fa4 bytes) 000660 1746070319.583387 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x276:0x0]: Invalid argument (22) 000661 1746070319.783943 mapwrite 0x02b00d thru 0x035e1d (0xae11 bytes) 000662 1746070319.871999 read 0x0149ae thru 0x020b05 (0xc158 bytes) 000665 1746070319.882397 read 0x01f1e9 thru 0x023620 (0x4438 bytes) 000667 1746070319.898315 read 0x019d6c thru 0x01f0a0 (0x5335 bytes) 000668 1746070319.898830 mapwrite 0x00e383 thru 0x017442 (0x90c0 bytes) 000669 1746070319.972382 mapread 0x028886 thru 0x02a5c4 (0x1d3f bytes) 000670 1746070319.974773 write 0x033df0 thru 0x03c5a8 (0x87b9 bytes) 000671 1746070319.978293 mapread 0x031513 thru 0x03b7bd (0xa2ab bytes) 000675 1746070319.980029 mapread 0x028f25 thru 0x038990 (0xfa6c bytes) 000677 1746070319.981078 trunc from 0x03c5a9 to 0x02351b (0x1908d bytes) 000678 1746070320.011864 read 0x01573a thru 0x0189fd (0x32c4 bytes) 000680 1746070320.012597 trunc from 0x02351c to 0x03e46e (0x1af53 bytes) 000681 1746070320.041067 MIRROR_EXTEND to 4 mirrors 000684 1746070320.546458 trunc from 0x03e46f to 0x0149ad (0x29ac1 bytes) 000686 1746070320.610400 mapwrite 0x006dd8 thru 0x00ff11 (0x913a bytes) 000687 1746070320.677607 write 0x021449 thru 0x0216b6 (0x26e bytes) 000689 1746070320.679048 read 0x008c42 thru 0x00ada6 (0x2165 bytes) 000693 1746070320.683963 MIRROR_RESYNC 4 mirrors 000697 1746070321.164945 write 0x010aab thru 0x02070d (0xfc63 bytes) 000698 1746070321.230692 write 0x013652 thru 0x01f4fb (0xbeaa bytes) 000699 1746070321.231700 trunc from 0x0216b7 to 0x007197 (0x1a51f bytes) 000702 1746070321.269903 write 0x018948 thru 0x0224e7 (0x9ba0 bytes) 000703 1746070321.272023 mapread 0x0144dd thru 0x01990a (0x542e bytes) 000705 1746070321.279481 read 0x01e665 thru 0x0224e7 (0x3e83 bytes) 000707 1746070321.280574 mapwrite 0x03d680 thru 0x03ffff (0x2980 bytes) 000710 1746070321.394006 trunc from 0x040000 to 0x01c49f (0x23b60 bytes) 000711 1746070321.419530 mapwrite 0x008856 thru 0x01823c (0xf9e7 bytes) 000713 1746070321.504841 write 0x034219 thru 0x039bcc (0x59b4 bytes) 000714 1746070321.505788 read 0x01ae0f thru 0x02440e (0x9600 bytes) 000716 1746070321.521131 trunc from 0x039bcd to 0x032d52 (0x6e7a bytes) 000717 1746070321.546187 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x276:0x0]: Invalid argument (22) 000720 1746070321.791999 trunc from 0x032d53 to 0x0082b8 (0x2aa9a bytes) 000721 1746070321.820224 read 0x007a7c thru 0x0082b8 (0x83d bytes) 000722 1746070321.827066 read 0x003475 thru 0x006683 (0x320f bytes) 000723 1746070321.841439 write 0x039dd4 thru 0x03ffff (0x622c bytes) 000725 1746070321.847142 mapread 0x01e51c thru 0x02993e (0xb423 bytes) 000726 1746070321.869875 MIRROR_EXTEND to 4 mirrors 000727 1746070322.465049 mapread 0x00e4a0 thru 0x011a78 (0x35d9 bytes) 000728 1746070322.499255 trunc from 0x040000 to 0x00c3e5 (0x33c1a bytes) 000729 1746070322.587184 MIRROR_RESYNC 4 mirrors 000732 1746070323.152818 read 0x00a8ed thru 0x00c3e5 (0x1af9 bytes) 000733 1746070323.158041 mapread 0x008620 thru 0x00c3e5 (0x3dc6 bytes) 000734 1746070323.165731 read 0x008c50 thru 0x009584 (0x935 bytes) 000737 1746070323.166300 trunc from 0x00c3e6 to 0x0180d2 (0xbced bytes) 000738 1746070323.266438 trunc from 0x0180d3 to 0x00ae12 (0xd2c0 bytes) 000739 1746070323.301792 read 0x00226a thru 0x00ae12 (0x8ba9 bytes) 000742 1746070323.308149 mapwrite 0x001356 thru 0x005558 (0x4203 bytes) 000743 1746070323.373179 write 0x013344 thru 0x01dfde (0xac9b bytes) 000744 1746070323.374945 mapread 0x01d300 thru 0x01dfde (0xcdf bytes) 000746 1746070323.376064 trunc from 0x01dfdf to 0x035a33 (0x17a55 bytes) 000747 1746070323.387773 write 0x01b0bf thru 0x0229fa (0x793c bytes) 000748 1746070323.395235 mapwrite 0x020f51 thru 0x02a5af (0x965f bytes) 000749 1746070323.494957 mapwrite 0x024687 thru 0x03019f (0xbb19 bytes) 000750 1746070323.591891 mapwrite 0x00224f thru 0x005f79 (0x3d2b bytes) 000753 1746070323.692198 mapwrite 0x017ad3 thru 0x01e0b6 (0x65e4 bytes) 000755 1746070323.747897 mapwrite 0x0275e5 thru 0x02f85a (0x8276 bytes) 000756 1746070323.806173 trunc from 0x035a34 to 0x0052ec (0x30747 bytes) 000757 1746070323.831405 trunc from 0x0052ed to 0x01bfff (0x16d13 bytes) 000758 1746070323.859808 MIRROR_EXTEND to 5 mirrors 000760 1746070324.373854 read 0x00ac22 thru 0x016a7a (0xbe59 bytes) 000761 1746070324.386038 trunc from 0x01c000 to 0x01ddac (0x1dad bytes) 000762 1746070324.472738 mapwrite 0x0260e1 thru 0x0292e6 (0x3206 bytes) 000763 1746070324.559381 mapwrite 0x00e0b8 thru 0x00e4cc (0x415 bytes) 000768 1746070324.621735 read 0x0138ec thru 0x0219e7 (0xe0fc bytes) 000769 1746070324.636864 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x276:0x0]: Invalid argument (22) 000770 1746070324.791281 mapread 0x004cd8 thru 0x012c03 (0xdf2c bytes) 000771 1746070324.815847 mapwrite 0x00a6c5 thru 0x010c83 (0x65bf bytes) 000772 1746070324.883658 mapwrite 0x03ab32 thru 0x03d977 (0x2e46 bytes) 000774 1746070324.997486 MIRROR_RESYNC 4 mirrors 000775 1746070325.747083 mapwrite 0x0050a4 thru 0x00db34 (0x8a91 bytes) 000777 1746070325.933432 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x276:0x0]: Invalid argument (22) 000780 1746070326.192305 write 0x00a12d thru 0x0100f9 (0x5fcd bytes) 000782 1746070326.204474 read 0x0077ab thru 0x00bacf (0x4325 bytes) 000788 1746070326.215131 read 0x003020 thru 0x01271e (0xf6ff bytes) 000789 1746070326.226215 read 0x0196e1 thru 0x02830d (0xec2d bytes) 000790 1746070326.227078 read 0x008b37 thru 0x014438 (0xb902 bytes) 000791 1746070326.227475 read 0x006a3f thru 0x0132d8 (0xc89a bytes) 000792 1746070326.231113 trunc from 0x03d978 to 0x01060d (0x2d36a bytes) 000793 1746070326.260769 trunc from 0x01060e to 0x003f2b (0xc6e2 bytes) 000794 1746070326.288878 trunc from 0x003f2c to 0x014f0f (0x10fe4 bytes) 000797 1746070326.316096 write 0x002c70 thru 0x007c57 (0x4fe8 bytes) 000798 1746070326.325912 read 0x002658 thru 0x006455 (0x3dfe bytes) 000799 1746070326.327087 write 0x00deb1 thru 0x01b3ff (0xd54f bytes) 000802 1746070326.337529 mapread 0x00a550 thru 0x00da94 (0x3545 bytes) 000803 1746070326.338413 mapread 0x01635a thru 0x01b3ff (0x50a6 bytes) 000805 1746070326.339414 write 0x01110e thru 0x01a01d (0x8f10 bytes) 000807 1746070326.350297 MIRROR_RESYNC 3 mirrors 000809 1746070327.041261 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x276:0x0]: Invalid argument (22) 000810 1746070327.319078 trunc from 0x01b400 to 0x021f67 (0x6b68 bytes) 000811 1746070327.444769 mapwrite 0x020fb4 thru 0x02235f (0x13ac bytes) 000814 1746070327.560664 read 0x01861c thru 0x019441 (0xe26 bytes) 000815 1746070327.565751 write 0x00f1c4 thru 0x01c2ab (0xd0e8 bytes) 000817 1746070327.581739 MIRROR_RESYNC 2 mirrors 000819 1746070327.963749 mapread 0x00f6ab thru 0x01a5a8 (0xaefe bytes) 000820 1746070327.987046 MIRROR_RESYNC 2 mirrors 000821 1746070328.160108 MIRROR_RESYNC 2 mirrors 000823 1746070328.358385 write 0x022638 thru 0x0267ed (0x41b6 bytes) 000824 1746070328.455721 write 0x0240e9 thru 0x027a64 (0x397c bytes) 000825 1746070328.457141 mapread 0x025ddf thru 0x027a64 (0x1c86 bytes) 000828 1746070328.468229 MIRROR_RESYNC 2 mirrors 000829 1746070328.815258 mapwrite 0x027564 thru 0x02c5cf (0x506c bytes) 000830 1746070328.992819 write 0x007ee5 thru 0x008194 (0x2b0 bytes) 000831 1746070329.001258 write 0x019b54 thru 0x022ff5 (0x94a2 bytes) 000832 1746070329.009115 trunc from 0x02c5d0 to 0x004987 (0x27c48 bytes) 000838 1746070329.035082 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x276:0x0]: Invalid argument (22) 000839 1746070329.147767 mapread 0x000625 thru 0x004987 (0x4363 bytes) 000840 1746070329.184940 write 0x028a1c thru 0x030045 (0x762a bytes) 000843 1746070329.186715 read 0x01c386 thru 0x02b564 (0xf1df bytes) 000845 1746070329.201444 write 0x035230 thru 0x03ffff (0xadd0 bytes) 000847 1746070329.205669 mapread 0x000a72 thru 0x00e0f7 (0xd686 bytes) 000848 1746070329.225214 mapread 0x00af9a thru 0x00ba40 (0xaa7 bytes) 000849 1746070329.227006 trunc from 0x040000 to 0x00e5ae (0x31a51 bytes) 000850 1746070329.259858 read 0x0075db thru 0x00aab3 (0x34d9 bytes) 000851 1746070329.267306 MIRROR_EXTEND to 2 mirrors 000852 1746070329.808849 write 0x029f49 thru 0x03166e (0x7726 bytes) 000857 1746070329.905187 write 0x0029e0 thru 0x00a9ea (0x800b bytes) 000861 1746070329.930439 write 0x02a935 thru 0x0366c7 (0xbd93 bytes) 000862 1746070329.932564 mapread 0x00596c thru 0x011362 (0xb9f7 bytes) 000865 1746070329.946973 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x276:0x0]: Invalid argument (22) 000866 1746070330.084102 mapread 0x029f8b thru 0x0326df (0x8755 bytes) 000869 1746070330.105734 read 0x0274df thru 0x02997d (0x249f bytes) 000870 1746070330.112062 mapwrite 0x03a8f3 thru 0x03ffff (0x570d bytes) 000871 1746070330.256674 mapread 0x00ef9e thru 0x0176ad (0x8710 bytes) 000874 1746070330.271779 write 0x006758 thru 0x010537 (0x9de0 bytes) 000875 1746070330.285297 MIRROR_EXTEND to 2 mirrors 000876 1746070330.921662 read 0x00ad4a thru 0x00e4c4 (0x377b bytes) 000877 1746070330.934198 write 0x012db3 thru 0x01b9e3 (0x8c31 bytes) 000878 1746070331.042083 mapread 0x000e51 thru 0x00d0e4 (0xc294 bytes) 000879 1746070331.059287 trunc from 0x040000 to 0x00a99a (0x35665 bytes) 000880 1746070331.087215 read 0x00205b thru 0x0076cd (0x5673 bytes) 000881 1746070331.087735 trunc from 0x00a99b to 0x00ab7a (0x1e0 bytes) 000882 1746070331.104934 mapread 0x00323b thru 0x00ab7a (0x7940 bytes) 000883 1746070331.105891 write 0x02c0ff thru 0x02f546 (0x3448 bytes) 000884 1746070331.114264 MIRROR_EXTEND to 3 mirrors 000889 1746070331.501737 write 0x004848 thru 0x00ba68 (0x7221 bytes) 000890 1746070331.583804 trunc from 0x02f547 to 0x0367db (0x7295 bytes) 000891 1746070331.649143 mapread 0x004970 thru 0x0120a9 (0xd73a bytes) 000893 1746070331.666189 MIRROR_RESYNC 3 mirrors 000897 1746070332.336342 MIRROR_RESYNC 3 mirrors 000899 1746070332.509587 mapwrite 0x03ec17 thru 0x03ffff (0x13e9 bytes) 000900 1746070332.661734 read 0x034786 thru 0x03eae4 (0xa35f bytes) 000901 1746070332.675807 write 0x0324c2 thru 0x039c46 (0x7785 bytes) 000903 1746070332.688159 mapwrite 0x006991 thru 0x00a6b0 (0x3d20 bytes) 000904 1746070332.772018 write 0x02ccd6 thru 0x02eca3 (0x1fce bytes) 000905 1746070332.780973 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x276:0x0]: Invalid argument (22) 000906 1746070333.018105 trunc from 0x040000 to 0x00fc90 (0x3036f bytes) 000908 1746070333.034570 mapread 0x006127 thru 0x00d196 (0x7070 bytes) 000909 1746070333.051520 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x276:0x0]: Invalid argument (22) 000911 1746070333.172393 trunc from 0x00fc91 to 0x031423 (0x21793 bytes) 000913 1746070333.195434 write 0x01476b thru 0x01817c (0x3a12 bytes) 000914 1746070333.211473 write 0x0068d7 thru 0x01012a (0x9854 bytes) 000918 1746070333.232968 write 0x02b60a thru 0x037f27 (0xc91e bytes) 000919 1746070333.246227 mapread 0x01dab1 thru 0x027a8c (0x9fdc bytes) 000922 1746070333.282445 mapwrite 0x007eb4 thru 0x00dfd7 (0x6124 bytes) 000923 1746070333.387079 mapread 0x006f13 thru 0x00cf9c (0x608a bytes) 000924 1746070333.388184 trunc from 0x037f28 to 0x001a42 (0x364e5 bytes) 000925 1746070333.406415 write 0x0185d6 thru 0x024514 (0xbf3f bytes) 000926 1746070333.407851 read 0x020f1b thru 0x024514 (0x35fa bytes) 000928 1746070333.408514 mapwrite 0x00d094 thru 0x00d122 (0x8f bytes) 000929 1746070333.468224 read 0x01327b thru 0x01952a (0x62b0 bytes) 000933 1746070333.476211 mapwrite 0x02e350 thru 0x03a65f (0xc310 bytes) 000934 1746070333.559134 mapread 0x008a1b thru 0x017385 (0xe96b bytes) 000936 1746070333.569674 MIRROR_EXTEND to 2 mirrors 000937 1746070334.026783 trunc from 0x03a660 to 0x010d1e (0x29941 bytes) 000939 1746070334.173184 mapread 0x00328f thru 0x010d1e (0xda90 bytes) 000941 1746070334.197633 read 0x00b9c7 thru 0x010d1e (0x5358 bytes) 000945 1746070334.199239 mapwrite 0x0126b9 thru 0x01ca31 (0xa379 bytes) 000946 1746070334.278712 trunc from 0x01ca32 to 0x020f40 (0x450f bytes) 000947 1746070334.292618 write 0x00fdcd thru 0x017316 (0x754a bytes) 000951 1746070334.294407 mapwrite 0x0106ea thru 0x014d10 (0x4627 bytes) 000953 1746070334.380237 MIRROR_EXTEND to 3 mirrors 000955 1746070334.788971 mapwrite 0x004c45 thru 0x010989 (0xbd45 bytes) 000957 1746070334.967295 mapread 0x00e4ed thru 0x01d303 (0xee17 bytes) 000959 1746070334.985220 mapread 0x004b20 thru 0x010417 (0xb8f8 bytes) 000962 1746070334.986974 write 0x00eafd thru 0x014489 (0x598d bytes) 000963 1746070334.988388 mapwrite 0x0334cf thru 0x03cf36 (0x9a68 bytes) 000964 1746070335.183793 read 0x027ea7 thru 0x02f628 (0x7782 bytes) 000967 1746070335.203367 mapread 0x01d48e thru 0x023f04 (0x6a77 bytes) 000968 1746070335.215434 write 0x006758 thru 0x00a219 (0x3ac2 bytes) 000970 1746070335.217466 read 0x00df62 thru 0x013d31 (0x5dd0 bytes) 000971 1746070335.217908 trunc from 0x03cf37 to 0x00092c (0x3c60a bytes) 000972 1746070335.252143 write 0x0141f1 thru 0x02198f (0xd79f bytes) 000973 1746070335.253302 read 0x016255 thru 0x02198f (0xb73b bytes) 000974 1746070335.253780 write 0x0039dc thru 0x00688e (0x2eb3 bytes) 000977 1746070335.277641 mapwrite 0x030b0d thru 0x0382ee (0x77e2 bytes) 000978 1746070335.423635 mapread 0x025135 thru 0x02af69 (0x5e35 bytes) 000980 1746070335.437765 MIRROR_EXTEND to 4 mirrors 000982 1746070335.943717 read 0x01104e thru 0x020366 (0xf319 bytes) 000983 1746070335.971317 mapwrite 0x00df77 thru 0x00f11c (0x11a6 bytes) 000984 1746070336.231454 MIRROR_RESYNC 4 mirrors 000985 1746070336.896444 mapread 0x0095af thru 0x014125 (0xab77 bytes) 000986 1746070336.915794 mapwrite 0x008108 thru 0x00d892 (0x578b bytes) 000988 1746070337.064402 mapwrite 0x028e99 thru 0x0382de (0xf446 bytes) 000991 1746070337.124280 mapwrite 0x0321c1 thru 0x03ffff (0xde3f bytes) 000993 1746070337.226854 trunc from 0x040000 to 0x0147f7 (0x2b808 bytes) 000994 1746070337.242116 mapwrite 0x0171f7 thru 0x022118 (0xaf22 bytes) 000996 1746070337.354279 write 0x02df5b thru 0x03bced (0xdd93 bytes) 000997 1746070337.369216 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x276:0x0]: Invalid argument (22) 000998 1746070337.583849 trunc from 0x03bcee to 0x010ce3 (0x2b00a bytes) 000999 1746070337.599736 read 0x0040b4 thru 0x010ce3 (0xcc30 bytes) All operations completed A-OK! PASS 70a (48s) == sanity-flr test 71: check mirror extend parent fid ==== 23:32:20 (1746070340) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 200a: stress test ===================== 23:32:23 (1746070343) Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre2 Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7893824 .. Extending file size to 7049472 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9259488 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7981632 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8005312 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8429408 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8687936 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 25509 25510 25511 25512 25513 done 192.168.203.102@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg302-client.virtnet /mnt/lustre2 (opts:) 192.168.203.102@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg302-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (72s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:33:36 (1746070416) Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre2 Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7770272 .. Extending file size to 1907168 .. done Extending file size to 4920608 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7488352 .. Extending file size to 7804896 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7846400 .. Extending file size to 8387680 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8252512 .. done Extending file size to 9142368 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8394016 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8863232 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 27403 27404 27405 27406 27411 192.168.203.102@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg302-client.virtnet /mnt/lustre2 (opts:) 192.168.203.102@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg302-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (69s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 23:34:45 (1746070485) Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.057893 s, 36.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.309457 s, 6.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.102@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg302-client.virtnet /mnt/lustre2 (opts:) PASS 200c (10s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 23:34:56 (1746070496) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0654697 s, 32.0 MB/s PASS 202 (5s) == sanity-flr test 203: mirror file preserve mirror ID === 23:35:01 (1746070501) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2aa:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x310:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x310:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x311:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } PASS 203 (4s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 23:35:05 (1746070505) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0969353 s, 21.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x312:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (4s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 23:35:10 (1746070510) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.145435 s, 14.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x313:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x314:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x313:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x314:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } PASS 204b (4s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 23:35:14 (1746070514) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0920516 s, 22.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x315:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x315:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } PASS 204c (26s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 23:35:40 (1746070540) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg302-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0611578 s, 17.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x316:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x316:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x317:0x0] } PASS 204d (48s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 23:36:28 (1746070588) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg302-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.124933 s, 16.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x318:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x318:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 23:37:14 (1746070634) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg302-server mds-ost sync done. Creating new pool test_204f oleg302-server: Pool lustre.test_204f created Adding targets to pool oleg302-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg302-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0766826 s, 27.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x319:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x319:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg302-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg302-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg302-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (62s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 23:38:16 (1746070696) PASS 205a (5s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 23:38:21 (1746070701) PASS 205b (5s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 23:38:26 (1746070706) oleg302-server: Pool lustre.test_206 created Adding targets to pool oleg302-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg302-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg302-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg302-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg302-server: Pool lustre.test_206 destroyed PASS 206 (18s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 23:38:44 (1746070724) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0322145 s, 32.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.023463 s, 44.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x26940c67 Mirror 3: 0x26940c67 PASS 207 (6s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 23:38:50 (1746070730) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 23:38:52 (1746070732) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 23:38:54 (1746070734) Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0737926 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1a10000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xa40000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xa40000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xa40000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1d60000 Read this (4 bytes): BBBB 192.168.203.102@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg302-client.virtnet /mnt/lustre2 (opts:) PASS 209a (22s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 23:39:16 (1746070756) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.0143481 s, 571 kB/s the very first read read with warm cache PASS 209b (4s) == sanity-flr test 210a: handle broken mirrored lovea ==== 23:39:20 (1746070760) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0181136 s, 57.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x2cf lmm_fid: [0x200000404:0x2cf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x2cf lmm_fid: [0x200000404:0x2cf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x324:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834639 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-30 23:39:21.000000000 -0400 Modify: 2025-04-30 23:39:21.000000000 -0400 Change: 2025-04-30 23:39:22.000000000 -0400 Birth: - PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 23:39:24 (1746070764) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 23:39:26 (1746070766) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.491662 s, 83.3 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0547976 s, 74.7 kB/s size after second write -rw-r--r-- 1 root root 4096 Apr 30 23:39 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2d1:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Apr 30 23:39 /mnt/lustre/f211.sanity-flr PASS 211 (5s) == sanity-flr test complete, duration 1080 sec =========== 23:39:31 (1746070771) === sanity-flr: start cleanup 23:39:32 (1746070772) === === sanity-flr: finish cleanup 23:39:37 (1746070777) ===