-----============= acceptance-small: sanity-flr ============----- Tue Aug 27 11:41:50 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 6 201 44c 49a === sanity-flr: start setup 11:41:55 (1724773315) === oleg106-client.virtnet: executing check_config_client /mnt/lustre oleg106-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg106-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b66e1800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b66e1800.idle_timeout=debug disable quota as required oleg106-server: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 11:42:01 (1724773321) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f6978] == sanity-flr test 0a: lfs mirror create with -N option == 11:42:01 (1724773321) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 11:42:04 (1724773324) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 11:42:05 (1724773325) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 11:42:06 (1724773326) PASS 0d (4s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 11:42:10 (1724773330) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 11:42:11 (1724773331) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (0s) == sanity-flr test 0g: lfs mirror create flags support === 11:42:11 (1724773331) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b66e1800.stats=clear osc.lustre-OST0001-osc-ffff8800b66e1800.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 11:42:13 (1724773333) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 11:42:15 (1724773335) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 11:42:17 (1724773337) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (0s) == sanity-flr test 1: create components with setstripe options ========================================================== 11:42:18 (1724773338) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 11:42:20 (1724773340) PASS 2 (1s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 11:42:21 (1724773341) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 11:42:22 (1724773342) lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (1s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 11:42:23 (1724773343) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 11:42:25 (1724773345) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 11:42:27 (1724773347) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.245062 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0312331 s, 33.6 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 11:42:29 (1724773349) 14+0 records in 14+0 records out 14680064 bytes (15 MB) copied, 0.0871708 s, 168 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 11:42:31 (1724773351) Stopping /mnt/lustre-ost1 (opts:) on oleg106-server oleg106-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b66e1800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b66e1800.ost_server_uuid in DISCONN state after 0 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg106-server: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg106-server: oleg106-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg106-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg106-server oleg106-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b66e1800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b66e1800.ost_server_uuid in DISCONN state after 0 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg106-server: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg106-server: oleg106-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg106-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (19s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 11:42:50 (1724773370) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.0832971 s, 62.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg106-server oleg106-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b66e1800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b66e1800.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg106-server: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg106-server: oleg106-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg106-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (9s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 11:42:59 (1724773379) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg106-server oleg106-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b66e1800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b66e1800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg106-server: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg106-server: oleg106-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg106-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg106-server oleg106-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b66e1800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b66e1800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg106-server: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg106-server: oleg106-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg106-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (16s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 11:43:15 (1724773395) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.827139 s, 63.4 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 11:43 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.217s user 0m0.000s sys 0m0.184s Stopping /mnt/lustre-ost1 (opts:) on oleg106-server oleg106-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b66e1800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b66e1800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.177s user 0m0.001s sys 0m0.120s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg106-server: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg106-server: oleg106-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg106-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg106-server oleg106-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b66e1800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b66e1800.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost1)... real 0m0.202s user 0m0.000s sys 0m0.169s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg106-server: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg106-server: oleg106-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg106-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg106-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b66e1800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b66e1800.ost_server_uuid in FULL state after 0 sec PASS 33b (20s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 11:43:35 (1724773415) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 11:43:36 (1724773416) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (0s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 11:43:36 (1724773416) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 11:43:37 (1724773417) PASS 35 (1s) == sanity-flr test 36a: write to mirrored files ========== 11:43:38 (1724773418) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.1009 s, 125 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 11:43:41 (1724773421) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00850053 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00975115 s, 108 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 11:43:43 (1724773423) PASS 36c (2s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 11:43:45 (1724773425) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000616939 s, 1.7 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000775736 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00070559 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000664457 s, 1.5 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 11:43:47 (1724773427) create test files with size 15735777 .. 33d828f30f096e2e73c46e2d77a92a10 - 1ecef8a14f89a933c1cae29756fda44a - 82e34034ec5cc4fd52704535c7d6ed1d - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg106-client.virtnet /mnt/lustre (opts:) Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre PASS 37 (5s) == sanity-flr test 38: resync ============================ 11:43:52 (1724773432) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 11:43:55 (1724773435) PASS 39 (1s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 11:43:56 (1724773436) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0193493 s, 54.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0084949 s, 123 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 11:43:58 (1724773438) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0241315 s, 86.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0659388 s, 63.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0308201 s, 68.0 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) == sanity-flr test 42: lfs mirror verify ================= 11:44:00 (1724773440) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 11:44:01 (1724773441) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (0s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 11:44:02 (1724773442) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00667631 s, 157 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (1s) == sanity-flr test 44a: lfs mirror split check =========== 11:44:03 (1724773443) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 11:44:04 (1724773444) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4e:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x13:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (1s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 11:44:06 (1724773446) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0731575 s, 143 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg106-client.virtnet /mnt/lustre (opts:) Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (3s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 11:44:09 (1724773449) llite.lustre-ffff8800b3f1c800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0597587 s, 175 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg106-client.virtnet /mnt/lustre (opts:) Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800aa1c5800.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 11:44:12 (1724773452) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (7s) == sanity-flr test 46: Verify setstripe --copy option ==== 11:44:19 (1724773459) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (7s) == sanity-flr test 47: Verify mirror obj alloc =========== 11:44:26 (1724773466) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 11:44:27 (1724773467) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0514148 s, 61.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0442609 s, 71.1 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 4f7ab823a58674b451010cf57a274366 - mirror 1 checksum: 8cb167544518576f55cfb49953ee03b3 - mirror 2 checksum: 4f7ab823a58674b451010cf57a274366 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Aug 27 11:44:29 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Aug 27 11:44:29 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 4f7ab823a58674b451010cf57a274366 - mirror 1 checksum: 8cb167544518576f55cfb49953ee03b3 - mirror 2 checksum: 8cb167544518576f55cfb49953ee03b3 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Aug 27 11:44:29 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Aug 27 11:44:29 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } PASS 48 (2s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 11:44:29 (1724773469) Failing mds1 on oleg106-server Stopping /mnt/lustre-mds1 (opts:) on oleg106-server 11:44:31 (1724773471) shut down Failover mds1 to oleg106-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0000 11:44:44 (1724773484) targets are mounted 11:44:44 (1724773484) facet_failover done oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00817268 s, 128 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } PASS 50A (19s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 11:44:49 (1724773489) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 11:44:49 (1724773489) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 11:44:50 (1724773490) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00215036 s, 7.6 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 11:44:52 (1724773492) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 11:44:53 (1724773493) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (0s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 11:44:53 (1724773493) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (4s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 11:44:57 (1724773497) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 11:45:15 (1724773515) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 11:45:32 (1724773532) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (16s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 11:45:48 (1724773548) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 11:45:50 (1724773550) mirror create pid 21571 mirror split pid 21572 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4819: 21571 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4819: 21572 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 11:46:52 (1724773612) Seed set to 3612 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x27d2d 000004 1724773612.511147 trunc from 00000000 to 0x027d2c (0x27d2d bytes) 000006 1724773612.514469 write 0x0018a2 thru 0x00f606 (0xdd65 bytes) 000008 1724773612.517459 mapwrite 0x00692d thru 0x015841 (0xef15 bytes) 000009 1724773612.545995 write 0x010585 thru 0x013fbf (0x3a3b bytes) 000013 1724773612.547397 write 0x017151 thru 0x023310 (0xc1c0 bytes) 000014 1724773612.549184 read 0x00b886 thru 0x01898e (0xd109 bytes) 000015 1724773612.549504 mapread 0x01304d thru 0x01e172 (0xb126 bytes) 000017 1724773612.551334 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x3e22c 000019 1724773612.638653 trunc from 0x027d2d to 0x03e22b (0x164ff bytes) 000022 1724773612.655796 write 0x01b67d thru 0x0275eb (0xbf6f bytes) 000026 1724773612.659133 mapwrite 0x005e48 thru 0x0149d0 (0xeb89 bytes) 000028 1724773612.679239 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x719:0x0]: Invalid argument (22) 000030 1724773612.713940 mapwrite 0x024241 thru 0x0335e1 (0xf3a1 bytes) 000033 1724773612.734858 write 0x01efdd thru 0x025b84 (0x6ba8 bytes) 000035 1724773612.737353 trunc from 0x03e22c to 0x026810 (0x17a1b bytes) 000036 1724773612.740512 mapread 0x0096ca thru 0x013eef (0xa826 bytes) 000039 1724773612.744662 mapread 0x016914 thru 0x01e07a (0x7767 bytes) 000041 1724773612.745261 mapwrite 0x0356bf thru 0x03ffff (0xa941 bytes) 000042 1724773612.763274 trunc from 0x040000 to 0x009007 (0x36ff8 bytes) 000043 1724773612.766518 write 0x013da7 thru 0x01c558 (0x87b2 bytes) 000044 1724773612.767395 write 0x02081c thru 0x02854b (0x7d30 bytes) 000047 1724773612.768695 mapwrite 0x038aa1 thru 0x03ffff (0x755f bytes) 000049 1724773612.785066 mapwrite 0x0198f2 thru 0x020fb5 (0x76c4 bytes) 000050 1724773612.799127 write 0x0250f0 thru 0x02bd78 (0x6c89 bytes) 000051 1724773612.800935 mapwrite 0x009308 thru 0x012e26 (0x9b1f bytes) 000054 1724773612.815811 mapread 0x0286ab thru 0x036860 (0xe1b6 bytes) 000055 1724773612.819514 MIRROR_EXTEND to 2 mirrors 000057 1724773612.914454 trunc from 0x040000 to 0x03daa0 (0x255f bytes) 000058 1724773612.936723 MIRROR_RESYNC 2 mirrors 000059 1724773613.016693 MIRROR_EXTEND to 3 mirrors 000063 1724773613.116314 read 0x03c698 thru 0x03daa0 (0x1409 bytes) 000064 1724773613.119986 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x719:0x0]: Invalid argument (22) 000066 1724773613.169429 mapread 0x0399f7 thru 0x03daa0 (0x40aa bytes) 000068 1724773613.174008 read 0x021253 thru 0x0230e1 (0x1e8f bytes) 000069 1724773613.175409 read 0x0027f3 thru 0x012457 (0xfc65 bytes) 000070 1724773613.178330 write 0x019132 thru 0x01e608 (0x54d7 bytes) 000076 1724773613.195724 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x719:0x0]: Invalid argument (22) 000077 1724773613.223248 trunc from 0x03daa1 to 0x00faf2 (0x2dfae bytes) 000079 1724773613.230003 write 0x026fa4 thru 0x034feb (0xe048 bytes) 000083 1724773613.231911 mapwrite 0x021863 thru 0x02c96f (0xb10d bytes) 000088 1724773613.251487 trunc from 0x034fec to 0x014676 (0x20975 bytes) 000091 1724773613.255545 write 0x03bece thru 0x03ffff (0x4132 bytes) 000092 1724773613.256177 trunc from 0x040000 to 0x03a988 (0x5677 bytes) 000096 1724773613.259871 mapread 0x020cb8 thru 0x02e73f (0xda88 bytes) 000097 1724773613.264248 mapread 0x027f27 thru 0x02c816 (0x48f0 bytes) 000099 1724773613.265242 mapread 0x00091c thru 0x0023a9 (0x1a8e bytes) 000100 1724773613.268757 trunc from 0x03a989 to 0x019699 (0x212ef bytes) 000101 1724773613.272167 write 0x030fff thru 0x039ede (0x8ee0 bytes) 000107 1724773613.273941 trunc from 0x039edf to 0x00f149 (0x2ad95 bytes) 000108 1724773613.277278 mapwrite 0x01c313 thru 0x020a32 (0x4720 bytes) 000110 1724773613.297556 trunc from 0x020a33 to 0x00287e (0x1e1b4 bytes) 000112 1724773613.301589 read 0x0000bc thru 0x00287e (0x27c3 bytes) 000113 1724773613.301844 mapread 0x002516 thru 0x00287e (0x369 bytes) 000116 1724773613.302519 mapread 0x00152b thru 0x00287a (0x1350 bytes) 000117 1724773613.303013 write 0x004af7 thru 0x00f416 (0xa920 bytes) 000119 1724773613.304116 trunc from 0x00f417 to 0x02f671 (0x2025b bytes) 000120 1724773613.307735 trunc from 0x02f672 to 0x02d89e (0x1dd3 bytes) 000121 1724773613.310943 read 0x007c4f thru 0x00f214 (0x75c6 bytes) 000122 1724773613.311300 write 0x00d338 thru 0x00f148 (0x1e11 bytes) 000127 1724773613.312503 trunc from 0x02d89f to 0x035093 (0x77f5 bytes) 000128 1724773613.316439 write 0x02992e thru 0x0392b9 (0xf98c bytes) 000130 1724773613.320095 mapwrite 0x0272e0 thru 0x035619 (0xe33a bytes) 000135 1724773613.334687 read 0x02a58e thru 0x03878a (0xe1fd bytes) 000137 1724773613.335442 trunc from 0x0392ba to 0x013458 (0x25e61 bytes) 000138 1724773613.339009 read 0x00e4ff thru 0x00f9e0 (0x14e2 bytes) 000141 1724773613.339812 mapwrite 0x00c207 thru 0x015708 (0x9502 bytes) 000142 1724773613.355432 trunc from 0x015709 to 0x024928 (0xf220 bytes) 000145 1724773613.359117 read 0x0028d4 thru 0x00f124 (0xc851 bytes) 000146 1724773613.359411 trunc from 0x024929 to 0x004cfc (0x1fc2c bytes) 000150 1724773613.364013 MIRROR_EXTEND to 2 mirrors 000152 1724773613.465397 mapread 0x000092 thru 0x004cfc (0x4c6b bytes) 000153 1724773613.472340 mapread 0x000c0d thru 0x004cfc (0x40f0 bytes) 000155 1724773613.477093 read 0x000fc5 thru 0x0032b6 (0x22f2 bytes) 000156 1724773613.478661 trunc from 0x004cfd to 0x01afec (0x162f0 bytes) 000157 1724773613.493030 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x719:0x0]: Invalid argument (22) 000158 1724773613.524304 read 0x016119 thru 0x01afec (0x4ed4 bytes) 000159 1724773613.527047 mapread 0x01679f thru 0x01a8a6 (0x4108 bytes) 000162 1724773613.528126 read 0x009714 thru 0x013d1d (0xa60a bytes) 000163 1724773613.530986 read 0x0129d5 thru 0x01afec (0x8618 bytes) 000164 1724773613.531248 read 0x00e415 thru 0x01a370 (0xbf5c bytes) 000168 1724773613.532268 mapwrite 0x02be31 thru 0x032e05 (0x6fd5 bytes) 000171 1724773613.563382 mapwrite 0x0223b1 thru 0x02f173 (0xcdc3 bytes) 000173 1724773613.575754 write 0x03e21e thru 0x03ffff (0x1de2 bytes) 000174 1724773613.576375 mapwrite 0x00ad71 thru 0x01a0ec (0xf37c bytes) 000175 1724773613.587125 write 0x012b9e thru 0x019b3f (0x6fa2 bytes) 000179 1724773613.588213 write 0x002cd8 thru 0x00c569 (0x9892 bytes) 000180 1724773613.588744 read 0x033a6a thru 0x03c527 (0x8abe bytes) 000182 1724773613.591000 trunc from 0x040000 to 0x036fe3 (0x901c bytes) 000184 1724773613.594080 write 0x0229dc thru 0x02b3a2 (0x89c7 bytes) 000186 1724773613.594725 mapread 0x02052a thru 0x02d7e2 (0xd2b9 bytes) 000187 1724773613.595112 mapwrite 0x033bc5 thru 0x03f1d3 (0xb60f bytes) 000189 1724773613.622495 read 0x033b27 thru 0x03d370 (0x984a bytes) 000190 1724773613.622947 trunc from 0x03f1d4 to 0x026719 (0x18aba bytes) 000191 1724773613.627097 mapread 0x024d27 thru 0x026719 (0x19f3 bytes) 000193 1724773613.627944 mapwrite 0x00902f thru 0x00ce35 (0x3e07 bytes) 000194 1724773613.641996 MIRROR_EXTEND to 2 mirrors 000197 1724773613.726418 mapread 0x011950 thru 0x016b5b (0x520c bytes) 000198 1724773613.733742 mapwrite 0x016554 thru 0x01f2b7 (0x8d64 bytes) 000200 1724773613.776728 mapwrite 0x023eab thru 0x032d7a (0xeed0 bytes) 000202 1724773613.798253 mapread 0x005bc6 thru 0x00e21c (0x8657 bytes) 000203 1724773613.802959 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x719:0x0]: Invalid argument (22) 000204 1724773613.833669 read 0x031adc thru 0x032d7a (0x129f bytes) 000206 1724773613.836222 read 0x01c13c thru 0x01c9c7 (0x88c bytes) 000210 1724773613.840649 mapread 0x01ff95 thru 0x024f17 (0x4f83 bytes) 000211 1724773613.841245 trunc from 0x032d7b to 0x01ab1e (0x1825c bytes) 000214 1724773613.846316 mapread 0x00445e thru 0x008426 (0x3fc9 bytes) 000215 1724773613.846746 trunc from 0x01ab1f to 0x002dc4 (0x17d5a bytes) 000216 1724773613.850642 write 0x005618 thru 0x012781 (0xd16a bytes) 000219 1724773613.853405 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x3f53b 000221 1724773613.948127 trunc from 0x012782 to 0x03f53a (0x2cdb9 bytes) 000222 1724773613.966572 mapread 0x011aaa thru 0x016c72 (0x51c9 bytes) 000223 1724773613.971283 mapread 0x034052 thru 0x03f53a (0xb4e9 bytes) 000224 1724773613.977596 read 0x03b6fc thru 0x03ba79 (0x37e bytes) 000225 1724773613.978014 write 0x016ecd thru 0x025ce7 (0xee1b bytes) 000227 1724773613.980280 mapread 0x033b8a thru 0x0352e0 (0x1757 bytes) 000228 1724773613.982087 MIRROR_RESYNC 2 mirrors 000229 1724773614.078037 mapwrite 0x009a64 thru 0x010be5 (0x7182 bytes) 000231 1724773614.125286 read 0x03bd82 thru 0x03f53a (0x37b9 bytes) 000232 1724773614.127910 mapread 0x00a620 thru 0x0150d4 (0xaab5 bytes) 000235 1724773614.131143 write 0x03ba65 thru 0x03ffff (0x459b bytes) 000236 1724773614.131793 mapwrite 0x035443 thru 0x03b154 (0x5d12 bytes) 000238 1724773614.146480 mapread 0x03e3f7 thru 0x03ffff (0x1c09 bytes) 000239 1724773614.147043 write 0x037f8e thru 0x03ae99 (0x2f0c bytes) 000240 1724773614.147687 mapwrite 0x0320e5 thru 0x03caaa (0xa9c6 bytes) 000242 1724773614.160868 mapwrite 0x00b968 thru 0x00df77 (0x2610 bytes) 000243 1724773614.173103 MIRROR_RESYNC 2 mirrors 000245 1724773614.250261 write 0x020cf0 thru 0x025d4a (0x505b bytes) 000249 1724773614.265965 write 0x01b746 thru 0x0250a3 (0x995e bytes) 000251 1724773614.268588 mapwrite 0x025e71 thru 0x02783b (0x19cb bytes) 000252 1724773614.283471 trunc from 0x040000 to 0x037554 (0x8aab bytes) 000254 1724773614.291869 read 0x021e40 thru 0x0221f7 (0x3b8 bytes) 000255 1724773614.293518 MIRROR_RESYNC 2 mirrors 000256 1724773614.375805 read 0x0161d9 thru 0x01a473 (0x429b bytes) 000257 1724773614.377875 mapread 0x015e14 thru 0x024eec (0xf0d9 bytes) 000258 1724773614.383655 read 0x0341ea thru 0x037554 (0x336b bytes) 000259 1724773614.386189 write 0x00c7c3 thru 0x01a221 (0xda5f bytes) 000260 1724773614.401764 read 0x00f94f thru 0x01d791 (0xde43 bytes) 000263 1724773614.405206 mapwrite 0x020950 thru 0x0301a5 (0xf856 bytes) 000264 1724773614.420576 trunc from 0x037555 to 0x039f50 (0x29fc bytes) 000265 1724773614.428017 mapread 0x0253d7 thru 0x03060f (0xb239 bytes) 000266 1724773614.428522 mapwrite 0x00821c thru 0x011701 (0x94e6 bytes) 000268 1724773614.441325 read 0x0029ef thru 0x0048a4 (0x1eb6 bytes) 000270 1724773614.443264 MIRROR_EXTEND to 3 mirrors 000271 1724773614.527522 mapwrite 0x00b764 thru 0x00d36d (0x1c0a bytes) 000272 1724773614.566188 read 0x004c90 thru 0x0100da (0xb44b bytes) 000273 1724773614.567804 mapread 0x033bd6 thru 0x039f50 (0x637b bytes) 000275 1724773614.571117 read 0x02da6b thru 0x02ece1 (0x1277 bytes) 000276 1724773614.573516 trunc from 0x039f51 to 0x01749b (0x22ab5 bytes) 000277 1724773614.581861 mapread 0x00c73b thru 0x016313 (0x9bd9 bytes) 000280 1724773614.583039 mapread 0x0135a8 thru 0x01749b (0x3ef4 bytes) 000282 1724773614.583770 mapwrite 0x003468 thru 0x0064e5 (0x307e bytes) 000283 1724773614.596172 mapread 0x012d47 thru 0x016ae3 (0x3d9d bytes) 000284 1724773614.596725 write 0x0168fe thru 0x01d68b (0x6d8e bytes) 000285 1724773614.597339 write 0x00c607 thru 0x019d94 (0xd78e bytes) 000288 1724773614.598377 mapwrite 0x021363 thru 0x02daca (0xc768 bytes) 000289 1724773614.626071 mapread 0x012dc2 thru 0x017f0f (0x514e bytes) 000291 1724773614.627708 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x719:0x0]: Invalid argument (22) 000292 1724773614.666646 write 0x0341f4 thru 0x03ffff (0xbe0c bytes) 000294 1724773614.667775 write 0x03bda5 thru 0x03ffff (0x425b bytes) 000295 1724773614.668162 mapread 0x01a1d9 thru 0x027e97 (0xdcbf bytes) 000296 1724773614.674131 write 0x016840 thru 0x019e7a (0x363b bytes) 000297 1724773614.674596 read 0x01646e thru 0x022263 (0xbdf6 bytes) 000300 1724773614.675635 mapread 0x005d6e thru 0x00ec63 (0x8ef6 bytes) 000301 1724773614.679771 trunc from 0x040000 to 0x0354d7 (0xab28 bytes) 000302 1724773614.683186 write 0x03d173 thru 0x03dceb (0xb79 bytes) 000303 1724773614.683694 mapwrite 0x036b86 thru 0x03ffff (0x947a bytes) 000306 1724773614.701784 mapread 0x00feb8 thru 0x01cb8e (0xccd7 bytes) 000307 1724773614.702284 mapwrite 0x03661d thru 0x03ffff (0x99e3 bytes) 000308 1724773614.711241 write 0x03b6f2 thru 0x03ffff (0x490e bytes) 000309 1724773614.711762 trunc from 0x040000 to 0x02a28e (0x15d71 bytes) 000311 1724773614.715061 trunc from 0x02a28f to 0x0138b9 (0x169d5 bytes) 000312 1724773614.717879 write 0x013873 thru 0x017774 (0x3f02 bytes) 000315 1724773614.720112 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x719:0x0]: Invalid argument (22) 000317 1724773614.749252 mapread 0x00793a thru 0x00fc9b (0x8362 bytes) 000318 1724773614.755246 MIRROR_EXTEND to 2 mirrors 000319 1724773614.850617 mapread 0x007e2b thru 0x00c161 (0x4337 bytes) 000320 1724773614.855801 write 0x031f1d thru 0x036b1e (0x4c02 bytes) 000322 1724773614.868133 trunc from 0x036b1f to 0x030b15 (0x6009 bytes) 000324 1724773614.874636 read 0x015ed2 thru 0x0173c4 (0x14f3 bytes) 000327 1724773614.876700 mapread 0x02bac4 thru 0x030b15 (0x5052 bytes) 000329 1724773614.880474 MIRROR_RESYNC 2 mirrors 000330 1724773614.961710 trunc from 0x030b16 to 0x026860 (0xa2b5 bytes) 000331 1724773614.977041 mapread 0x013f15 thru 0x020a49 (0xcb35 bytes) 000333 1724773614.983432 write 0x001e8f thru 0x00b7bd (0x992f bytes) 000335 1724773614.985991 mapread 0x0009e5 thru 0x00d97e (0xcf9a bytes) 000336 1724773614.987598 write 0x028939 thru 0x02f408 (0x6ad0 bytes) 000338 1724773614.988689 mapwrite 0x01ebcc thru 0x02a326 (0xb75b bytes) 000339 1724773615.005015 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x719:0x0]: Invalid argument (22) 000344 1724773615.030028 mapwrite 0x02a254 thru 0x038e88 (0xec35 bytes) 000346 1724773615.049654 read 0x01aa97 thru 0x0293d8 (0xe942 bytes) 000348 1724773615.053150 MIRROR_EXTEND to 2 mirrors 000349 1724773615.132490 MIRROR_RESYNC 2 mirrors 000351 1724773615.168832 mapread 0x01f9d6 thru 0x02ada9 (0xb3d4 bytes) 000352 1724773615.176279 mapwrite 0x010df3 thru 0x01bf20 (0xb12e bytes) 000353 1724773615.221972 read 0x00c344 thru 0x01532d (0x8fea bytes) 000355 1724773615.222667 write 0x00dab4 thru 0x01c49a (0xe9e7 bytes) 000356 1724773615.223431 mapwrite 0x013a0b thru 0x01e213 (0xa809 bytes) 000358 1724773615.235198 mapwrite 0x00299d thru 0x00c8a8 (0x9f0c bytes) 000362 1724773615.250908 MIRROR_EXTEND to 3 mirrors 000363 1724773615.333940 trunc from 0x038e89 to 0x02e79d (0xa6eb bytes) 000364 1724773615.350188 trunc from 0x02e79e to 0x001506 (0x2d297 bytes) 000365 1724773615.354372 mapread 0x0004d2 thru 0x001506 (0x1035 bytes) 000366 1724773615.356863 read 0x0011ff thru 0x001506 (0x308 bytes) 000367 1724773615.357250 write 0x039080 thru 0x03ffff (0x6f80 bytes) 000368 1724773615.357943 mapwrite 0x03f281 thru 0x03ffff (0xd7f bytes) 000370 1724773615.370203 mapread 0x008ec0 thru 0x0136ff (0xa840 bytes) 000371 1724773615.374616 write 0x026866 thru 0x035ea1 (0xf63c bytes) 000374 1724773615.378277 trunc from 0x040000 to 0x03a6b3 (0x594c bytes) 000375 1724773615.381525 read 0x036ea6 thru 0x03a6b3 (0x380e bytes) 000377 1724773615.383902 write 0x02e533 thru 0x038c1f (0xa6ed bytes) 000378 1724773615.384593 read 0x01d79b thru 0x021ee1 (0x4747 bytes) 000379 1724773615.384907 read 0x02e74b thru 0x0343b4 (0x5c6a bytes) 000380 1724773615.385209 read 0x027810 thru 0x03195f (0xa150 bytes) 000383 1724773615.385844 read 0x012e44 thru 0x01451e (0x16db bytes) 000384 1724773615.386207 mapwrite 0x00ad4f thru 0x00aea1 (0x153 bytes) 000386 1724773615.398321 trunc from 0x03a6b4 to 0x01988a (0x20e29 bytes) 000387 1724773615.402078 mapread 0x00d1e6 thru 0x010ea9 (0x3cc4 bytes) 000389 1724773615.403983 MIRROR_RESYNC 3 mirrors 000391 1724773615.495650 mapread 0x00219d thru 0x0057ae (0x3612 bytes) 000393 1724773615.499440 write 0x02888d thru 0x0351ed (0xc961 bytes) 000397 1724773615.511889 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x719:0x0]: Invalid argument (22) 000401 1724773615.555782 trunc from 0x0351ee to 0x00153c (0x33cb1 bytes) 000402 1724773615.562735 write 0x024e41 thru 0x02d77d (0x893d bytes) 000403 1724773615.563433 write 0x0038d6 thru 0x00a7b3 (0x6ede bytes) 000404 1724773615.565528 trunc from 0x02d77e to 0x00588a (0x27ef3 bytes) 000405 1724773615.568874 write 0x030e94 thru 0x03ebff (0xdd6c bytes) 000406 1724773615.569749 trunc from 0x03ec00 to 0x020c62 (0x1df9d bytes) 000412 1724773615.574238 mapwrite 0x03ad4d thru 0x03ffff (0x52b3 bytes) 000415 1724773615.592784 trunc from 0x040000 to 0x0185ab (0x27a54 bytes) 000416 1724773615.595601 mapwrite 0x00b5b1 thru 0x019ff7 (0xea47 bytes) 000418 1724773615.613351 mapwrite 0x02764c thru 0x032fd1 (0xb986 bytes) 000420 1724773615.634123 trunc from 0x032fd2 to 0x021282 (0x11d4f bytes) 000421 1724773615.637070 mapread 0x0117b8 thru 0x01895f (0x71a8 bytes) 000423 1724773615.638587 MIRROR_RESYNC 2 mirrors 000425 1724773615.714010 trunc from 0x021283 to 0x020244 (0x103e bytes) 000426 1724773615.728259 write 0x01bc10 thru 0x02809d (0xc48e bytes) 000427 1724773615.730098 write 0x02d5ef thru 0x036e16 (0x9828 bytes) 000429 1724773615.731068 trunc from 0x036e17 to 0x025015 (0x11e01 bytes) 000431 1724773615.736528 MIRROR_RESYNC 2 mirrors 000432 1724773615.809352 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x719:0x0]: Invalid argument (22) 000433 1724773615.835850 trunc from 0x025016 to 0x0016d1 (0x23944 bytes) 000434 1724773615.845052 read 0x001214 thru 0x0016d1 (0x4be bytes) 000438 1724773615.847208 mapwrite 0x00b0b1 thru 0x012ee0 (0x7e30 bytes) 000439 1724773615.866681 read 0x006f4f thru 0x012ee0 (0xbf92 bytes) 000440 1724773615.867096 trunc from 0x012ee1 to 0x030088 (0x1d1a8 bytes) 000442 1724773615.870478 mapwrite 0x0028bc thru 0x01276e (0xfeb3 bytes) 000445 1724773615.888454 read 0x00476f thru 0x007ec3 (0x3755 bytes) 000447 1724773615.889058 mapread 0x01858d thru 0x0216f9 (0x916d bytes) 000448 1724773615.889430 read 0x02c45e thru 0x02c736 (0x2d9 bytes) 000449 1724773615.889639 read 0x0132e3 thru 0x020a60 (0xd77e bytes) 000451 1724773615.890234 mapread 0x00001a thru 0x006baf (0x6b96 bytes) 000452 1724773615.890563 mapread 0x016a30 thru 0x021f37 (0xb508 bytes) 000453 1724773615.890890 trunc from 0x030089 to 0x027004 (0x9084 bytes) 000455 1724773615.895079 mapwrite 0x0128ed thru 0x013c7b (0x138f bytes) 000457 1724773615.907498 MIRROR_EXTEND to 2 mirrors 000461 1724773615.993104 read 0x0152c5 thru 0x01e1a3 (0x8edf bytes) 000462 1724773615.996056 read 0x002d41 thru 0x00533c (0x25fc bytes) 000463 1724773615.998995 trunc from 0x027005 to 0x029931 (0x292d bytes) 000466 1724773616.016460 mapwrite 0x03df67 thru 0x03f1b3 (0x124d bytes) 000468 1724773616.046298 mapread 0x01adb0 thru 0x026aac (0xbcfd bytes) 000471 1724773616.061541 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x719:0x0]: Invalid argument (22) 000472 1724773616.087316 trunc from 0x03f1b4 to 0x02eaa8 (0x1070b bytes) 000473 1724773616.090379 read 0x02cc7e thru 0x02eaa8 (0x1e2b bytes) 000474 1724773616.091889 trunc from 0x02eaa9 to 0x02ea67 (0x41 bytes) 000475 1724773616.094629 read 0x02ccc7 thru 0x02ea67 (0x1da1 bytes) 000476 1724773616.094935 write 0x034ed0 thru 0x03ffff (0xb130 bytes) 000480 1724773616.096289 mapread 0x020c0b thru 0x02a2fd (0x96f3 bytes) 000481 1724773616.099552 mapwrite 0x02aa15 thru 0x03303b (0x8627 bytes) 000484 1724773616.112385 read 0x02d09f thru 0x039348 (0xc2aa bytes) 000485 1724773616.112694 trunc from 0x040000 to 0x0187bb (0x27844 bytes) 000486 1724773616.116335 mapwrite 0x002ae1 thru 0x00762c (0x4b4c bytes) 000487 1724773616.129165 trunc from 0x0187bc to 0x01cd01 (0x4546 bytes) 000490 1724773616.133467 read 0x005811 thru 0x00b27d (0x5a6d bytes) 000491 1724773616.133934 mapread 0x015ffc thru 0x01cd01 (0x6d06 bytes) 000492 1724773616.136135 mapwrite 0x019c2d thru 0x01ad4b (0x111f bytes) 000494 1724773616.148128 trunc from 0x01cd02 to 0x03e3ca (0x216c9 bytes) 000496 1724773616.151748 write 0x012487 thru 0x016e6e (0x49e8 bytes) 000497 1724773616.152186 trunc from 0x03e3cb to 0x034d08 (0x96c2 bytes) 000499 1724773616.154907 mapread 0x025c74 thru 0x032e40 (0xd1cd bytes) 000502 1724773616.169329 write 0x02e93e thru 0x0341c5 (0x5888 bytes) 000507 1724773616.171158 write 0x010158 thru 0x0143fc (0x42a5 bytes) 000508 1724773616.171744 mapwrite 0x013281 thru 0x022d98 (0xfb18 bytes) 000511 1724773616.187314 mapread 0x018125 thru 0x02092c (0x8808 bytes) 000514 1724773616.188369 write 0x007bf6 thru 0x015bd8 (0xdfe3 bytes) 000515 1724773616.189061 write 0x022d38 thru 0x028029 (0x52f2 bytes) 000516 1724773616.189474 read 0x0115e3 thru 0x01d7d3 (0xc1f1 bytes) 000517 1724773616.189833 mapread 0x018882 thru 0x01f4ad (0x6c2c bytes) 000518 1724773616.190281 mapread 0x022030 thru 0x02478e (0x275f bytes) 000520 1724773616.190978 mapread 0x001693 thru 0x009453 (0x7dc1 bytes) 000521 1724773616.191310 read 0x020abe thru 0x02ae39 (0xa37c bytes) 000522 1724773616.191624 read 0x030b19 thru 0x034d08 (0x41f0 bytes) 000525 1724773616.192498 read 0x033a52 thru 0x034d08 (0x12b7 bytes) 000526 1724773616.192782 trunc from 0x034d09 to 0x00c978 (0x28390 bytes) 000528 1724773616.197316 mapread 0x00b7c4 thru 0x00c978 (0x11b5 bytes) 000529 1724773616.197920 write 0x036a02 thru 0x03e7cd (0x7dcc bytes) 000530 1724773616.198721 write 0x003302 thru 0x00da72 (0xa771 bytes) 000532 1724773616.202234 write 0x03ba27 thru 0x03ffff (0x45d9 bytes) 000537 1724773616.203619 mapread 0x0282f2 thru 0x028924 (0x633 bytes) 000541 1724773616.204609 trunc from 0x040000 to 0x03cc8b (0x3374 bytes) 000546 1724773616.209464 read 0x035878 thru 0x03986d (0x3ff6 bytes) 000548 1724773616.210471 mapread 0x0127f3 thru 0x01cde0 (0xa5ee bytes) 000550 1724773616.211431 mapwrite 0x00801e thru 0x0081e1 (0x1c4 bytes) 000551 1724773616.223424 trunc from 0x03cc8c to 0x023d75 (0x18f16 bytes) 000553 1724773616.226796 write 0x016176 thru 0x025952 (0xf7dd bytes) 000555 1724773616.227763 trunc from 0x025953 to 0x02b6ef (0x5d9d bytes) 000556 1724773616.230860 read 0x002546 thru 0x00ffb8 (0xda73 bytes) 000558 1724773616.231403 read 0x0177dd thru 0x01c6e8 (0x4f0c bytes) 000559 1724773616.231747 read 0x018b23 thru 0x02808f (0xf56d bytes) 000563 1724773616.234561 trunc from 0x02b6f0 to 0x039279 (0xdb8a bytes) 000564 1724773616.237714 trunc from 0x03927a to 0x00e89f (0x2a9da bytes) 000565 1724773616.240458 write 0x034361 thru 0x03ffff (0xbc9f bytes) 000567 1724773616.241330 trunc from 0x040000 to 0x03143a (0xebc5 bytes) 000569 1724773616.245197 MIRROR_EXTEND to 2 mirrors 000571 1724773616.330743 mapread 0x026ceb thru 0x0287fb (0x1b11 bytes) 000573 1724773616.336899 read 0x02142d thru 0x02380f (0x23e3 bytes) 000574 1724773616.338595 MIRROR_EXTEND to 3 mirrors 000576 1724773616.428835 mapwrite 0x01243f thru 0x01fac2 (0xd684 bytes) 000577 1724773616.475472 write 0x0039d5 thru 0x00a253 (0x687f bytes) 000578 1724773616.478874 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x719:0x0]: Invalid argument (22) 000579 1724773616.521827 trunc from 0x03143b to 0x004956 (0x2cae4 bytes) 000584 1724773616.529880 read 0x002bf6 thru 0x004956 (0x1d61 bytes) 000586 1724773616.532076 mapread 0x003168 thru 0x004956 (0x17ef bytes) 000587 1724773616.532444 mapread 0x001c46 thru 0x004956 (0x2d11 bytes) 000588 1724773616.535228 MIRROR_EXTEND to 3 mirrors 000591 1724773616.626964 MIRROR_EXTEND to 4 mirrors 000592 1724773616.709469 read 0x002085 thru 0x003a4b (0x19c7 bytes) 000593 1724773616.711554 mapread 0x0012c6 thru 0x00160f (0x34a bytes) 000594 1724773616.714231 read 0x001d6a thru 0x004956 (0x2bed bytes) 000595 1724773616.715213 trunc from 0x004957 to 0x018c00 (0x142aa bytes) 000596 1724773616.733579 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x719:0x0]: Invalid argument (22) 000598 1724773616.774692 trunc from 0x018c01 to 0x02738a (0xe78a bytes) 000599 1724773616.777889 mapwrite 0x0016b2 thru 0x00828f (0x6bde bytes) 000601 1724773616.796676 mapread 0x00b3ad thru 0x011046 (0x5c9a bytes) 000602 1724773616.799240 read 0x022c69 thru 0x02738a (0x4722 bytes) 000604 1724773616.801849 write 0x024bae thru 0x02eac5 (0x9f18 bytes) 000606 1724773616.802608 read 0x0123f5 thru 0x012987 (0x593 bytes) 000608 1724773616.804244 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x719:0x0]: Invalid argument (22) 000610 1724773616.847779 mapwrite 0x01e84d thru 0x02981b (0xafcf bytes) 000611 1724773616.865455 read 0x0148bc thru 0x016a60 (0x21a5 bytes) 000612 1724773616.867171 read 0x0229fc thru 0x02cf74 (0xa579 bytes) 000613 1724773616.867470 write 0x026f2b thru 0x033fea (0xd0c0 bytes) 000616 1724773616.869053 read 0x01d84d thru 0x020c77 (0x342b bytes) 000617 1724773616.870631 MIRROR_RESYNC 2 mirrors 000618 1724773616.954043 trunc from 0x033feb to 0x017c94 (0x1c356 bytes) 000622 1724773616.969623 mapread 0x00408f thru 0x00c024 (0x7f96 bytes) 000623 1724773616.974738 trunc from 0x017c95 to 0x014880 (0x3414 bytes) 000624 1724773616.980766 write 0x013ed8 thru 0x01c909 (0x8a32 bytes) 000625 1724773616.983457 mapwrite 0x01d487 thru 0x02d329 (0xfea3 bytes) 000626 1724773617.005108 read 0x01aef7 thru 0x02aabc (0xfbc6 bytes) 000627 1724773617.005474 trunc from 0x02d32a to 0x01792b (0x159fe bytes) 000628 1724773617.009580 MIRROR_RESYNC 2 mirrors 000629 1724773617.102855 trunc from 0x01792c to 0x017c7b (0x350 bytes) 000631 1724773617.116651 MIRROR_EXTEND to 3 mirrors 000632 1724773617.216306 mapwrite 0x021581 thru 0x02df1d (0xc99d bytes) 000635 1724773617.250006 mapwrite 0x016b48 thru 0x021c2a (0xb0e3 bytes) 000636 1724773617.266253 mapread 0x012ab5 thru 0x01834f (0x589b bytes) 000638 1724773617.266897 mapwrite 0x00da42 thru 0x016583 (0x8b42 bytes) 000639 1724773617.283217 write 0x014086 thru 0x01bef5 (0x7e70 bytes) 000640 1724773617.283889 read 0x014435 thru 0x01b2f3 (0x6ebf bytes) 000641 1724773617.284172 trunc from 0x02df1e to 0x009f95 (0x23f88 bytes) 000642 1724773617.289020 trunc from 0x009f96 to 0x0267c2 (0x1c82d bytes) 000644 1724773617.293167 MIRROR_RESYNC 3 mirrors 000645 1724773617.398727 write 0x00f5e3 thru 0x016b41 (0x755f bytes) 000646 1724773617.414086 trunc from 0x0267c3 to 0x03998f (0x131cd bytes) 000649 1724773617.418743 write 0x0157b1 thru 0x016e59 (0x16a9 bytes) 000650 1724773617.419125 mapread 0x0156dd thru 0x0175ee (0x1f12 bytes) 000651 1724773617.421341 mapread 0x00af2c thru 0x01248c (0x7561 bytes) 000654 1724773617.423768 read 0x012cfc thru 0x016700 (0x3a05 bytes) 000655 1724773617.424025 mapwrite 0x02f2b5 thru 0x03bf6c (0xccb8 bytes) 000657 1724773617.444347 mapwrite 0x016b34 thru 0x01ef7e (0x844b bytes) 000659 1724773617.456444 trunc from 0x03bf6d to 0x010107 (0x2be65 bytes) 000663 1724773617.460997 mapwrite 0x0338ec thru 0x03839d (0x4ab2 bytes) 000664 1724773617.480138 mapwrite 0x028867 thru 0x02aec0 (0x265a bytes) 000665 1724773617.493682 trunc from 0x03839e to 0x03aee9 (0x2b4c bytes) 000666 1724773617.497886 read 0x00dd8d thru 0x017ace (0x9d42 bytes) 000668 1724773617.501139 read 0x023225 thru 0x029fc8 (0x6da4 bytes) 000670 1724773617.503058 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x719:0x0]: Invalid argument (22) 000671 1724773617.543227 trunc from 0x03aeea to 0x009cc6 (0x31223 bytes) 000673 1724773617.546836 mapwrite 0x02fd30 thru 0x032d6d (0x303e bytes) 000675 1724773617.565751 write 0x02f67f thru 0x03a0c7 (0xaa49 bytes) 000677 1724773617.566667 trunc from 0x03a0c8 to 0x0030f7 (0x36fd0 bytes) 000678 1724773617.570072 trunc from 0x0030f8 to 0x0360fa (0x33003 bytes) 000679 1724773617.574342 MIRROR_EXTEND to 3 mirrors 000680 1724773617.667204 mapread 0x0217e7 thru 0x0225ab (0xdc5 bytes) 000682 1724773617.672216 trunc from 0x0360fb to 0x021741 (0x149b9 bytes) 000686 1724773617.691326 read 0x00b183 thru 0x01b039 (0xfeb7 bytes) 000687 1724773617.694821 MIRROR_RESYNC 3 mirrors 000690 1724773617.794765 mapread 0x013138 thru 0x021741 (0xe60a bytes) 000691 1724773617.800729 mapread 0x00df7d thru 0x011404 (0x3488 bytes) 000692 1724773617.802800 mapread 0x01d62b thru 0x021741 (0x4117 bytes) 000693 1724773617.803132 trunc from 0x021742 to 0x034269 (0x12b28 bytes) 000694 1724773617.818899 write 0x02328d thru 0x02f734 (0xc4a8 bytes) 000695 1724773617.822928 trunc from 0x03426a to 0x008cf9 (0x2b570 bytes) 000696 1724773617.827759 mapwrite 0x02ae30 thru 0x038459 (0xd62a bytes) 000698 1724773617.847044 trunc from 0x03845a to 0x019b24 (0x1e935 bytes) 000699 1724773617.850210 mapread 0x008069 thru 0x0098c9 (0x1861 bytes) 000703 1724773617.853616 read 0x017ab5 thru 0x019b24 (0x2070 bytes) 000704 1724773617.855167 mapwrite 0x03edaa thru 0x03ffff (0x1256 bytes) 000707 1724773617.872677 read 0x017d1c thru 0x019b13 (0x1df8 bytes) 000709 1724773617.873489 mapread 0x0066cb thru 0x00ec3e (0x8574 bytes) 000712 1724773617.874822 mapwrite 0x013a84 thru 0x01968e (0x5c0b bytes) 000713 1724773617.889468 trunc from 0x040000 to 0x00da14 (0x325eb bytes) 000714 1724773617.893281 mapwrite 0x0248b0 thru 0x032fb9 (0xe70a bytes) 000716 1724773617.923652 mapwrite 0x0071da thru 0x008378 (0x119f bytes) 000718 1724773617.933467 MIRROR_EXTEND to 4 mirrors 000719 1724773618.019671 mapwrite 0x03a877 thru 0x03ffff (0x5789 bytes) 000722 1724773618.051760 MIRROR_RESYNC 4 mirrors 000724 1724773618.159694 write 0x03667f thru 0x03ffff (0x9981 bytes) 000727 1724773618.173703 trunc from 0x040000 to 0x034f0d (0xb0f2 bytes) 000728 1724773618.180806 read 0x007d35 thru 0x00a68c (0x2958 bytes) 000729 1724773618.182544 read 0x03337b thru 0x034f0d (0x1b93 bytes) 000730 1724773618.185465 read 0x002422 thru 0x0056ce (0x32ad bytes) 000733 1724773618.186174 trunc from 0x034f0e to 0x027bef (0xd31e bytes) 000735 1724773618.190749 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x719:0x0]: Invalid argument (22) 000736 1724773618.230152 write 0x001bf8 thru 0x00a5af (0x89b8 bytes) 000737 1724773618.232602 mapwrite 0x02e7e8 thru 0x037661 (0x8e7a bytes) 000738 1724773618.251338 trunc from 0x037662 to 0x01ca18 (0x1ac49 bytes) 000742 1724773618.255202 write 0x00c57b thru 0x01189a (0x5320 bytes) 000743 1724773618.257254 write 0x02e42f thru 0x03b90f (0xd4e1 bytes) 000747 1724773618.258708 mapwrite 0x03e39e thru 0x03ffff (0x1c62 bytes) 000748 1724773618.274029 write 0x02f518 thru 0x03a924 (0xb40d bytes) 000749 1724773618.274824 mapread 0x0036af thru 0x00655c (0x2eae bytes) 000750 1724773618.276560 MIRROR_RESYNC 3 mirrors 000754 1724773618.378917 trunc from 0x040000 to 0x0025dc (0x3da23 bytes) 000756 1724773618.396691 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x719:0x0]: Invalid argument (22) 000758 1724773618.438802 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x719:0x0]: Invalid argument (22) 000759 1724773618.465689 write 0x03c9b3 thru 0x03ffff (0x364d bytes) 000761 1724773618.466656 trunc from 0x040000 to 0x02cb54 (0x134ab bytes) 000762 1724773618.469610 trunc from 0x02cb55 to 0x0005fb (0x2c559 bytes) 000763 1724773618.472400 trunc from 0x0005fc to 0x007ee3 (0x78e8 bytes) 000765 1724773618.475594 mapread 0x007190 thru 0x007ee3 (0xd54 bytes) 000771 1724773618.478291 trunc from 0x007ee4 to 0x00fb00 (0x7c1d bytes) 000772 1724773618.481071 mapwrite 0x039cac thru 0x03ffff (0x6354 bytes) 000777 1724773618.500135 read 0x0033f4 thru 0x00eef4 (0xbb01 bytes) 000778 1724773618.502690 mapwrite 0x0031f0 thru 0x0103da (0xd1eb bytes) 000781 1724773618.517190 read 0x01fb00 thru 0x024d9c (0x529d bytes) 000782 1724773618.520067 trunc from 0x040000 to 0x018c51 (0x273ae bytes) 000783 1724773618.523849 mapread 0x000b3a thru 0x002d21 (0x21e8 bytes) 000784 1724773618.524299 read 0x006db3 thru 0x015155 (0xe3a3 bytes) 000785 1724773618.524693 mapwrite 0x028a2c thru 0x0355e6 (0xcbbb bytes) 000786 1724773618.543893 mapwrite 0x013fa2 thru 0x015ba8 (0x1c07 bytes) 000787 1724773618.554039 write 0x0154d9 thru 0x017743 (0x226b bytes) 000792 1724773618.556306 trunc from 0x0355e7 to 0x03c0b2 (0x6acc bytes) 000794 1724773618.560163 trunc from 0x03c0b3 to 0x035947 (0x676b bytes) 000795 1724773618.563270 write 0x0303ca thru 0x0329b9 (0x25f0 bytes) 000797 1724773618.564153 read 0x0189d1 thru 0x021ebf (0x94ef bytes) 000804 1724773618.566283 write 0x02f05a thru 0x03c26f (0xd216 bytes) 000806 1724773618.567425 mapwrite 0x01267a thru 0x01b5fb (0x8f82 bytes) 000807 1724773618.580528 write 0x015bb6 thru 0x020125 (0xa570 bytes) 000808 1724773618.581376 mapread 0x02407a thru 0x027c57 (0x3bde bytes) 000810 1724773618.582049 mapwrite 0x007873 thru 0x00c0a2 (0x4830 bytes) 000811 1724773618.596481 mapwrite 0x009903 thru 0x014b61 (0xb25f bytes) 000813 1724773618.609731 trunc from 0x03c270 to 0x003811 (0x38a5e bytes) 000814 1724773618.614225 read 0x000eda thru 0x00111c (0x243 bytes) 000819 1724773618.615616 write 0x01bb6f thru 0x0207f5 (0x4c87 bytes) 000822 1724773618.618567 MIRROR_EXTEND to 2 mirrors 000823 1724773618.702208 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x719:0x0]: Invalid argument (22) 000826 1724773618.727010 read 0x01c5ab thru 0x0207f5 (0x424b bytes) 000828 1724773618.729550 trunc from 0x0207f6 to 0x00b93d (0x14eb8 bytes) 000829 1724773618.737054 read 0x0041eb thru 0x006ba6 (0x29bc bytes) 000830 1724773618.739169 mapwrite 0x01a481 thru 0x027783 (0xd303 bytes) 000834 1724773618.758935 mapwrite 0x009eea thru 0x018238 (0xe34f bytes) 000835 1724773618.771424 mapread 0x015f4e thru 0x01a093 (0x4146 bytes) 000837 1724773618.772052 write 0x02f1e8 thru 0x0356c5 (0x64de bytes) 000838 1724773618.772608 mapwrite 0x007347 thru 0x01522f (0xdee9 bytes) 000841 1724773618.786220 mapwrite 0x038f40 thru 0x03ffff (0x70c0 bytes) 000844 1724773618.802965 read 0x03faa2 thru 0x03ffff (0x55e bytes) 000845 1724773618.803354 mapwrite 0x006bae thru 0x00bd5e (0x51b1 bytes) 000847 1724773618.814286 mapread 0x024a78 thru 0x02b46f (0x69f8 bytes) 000849 1724773618.815273 read 0x0084e2 thru 0x0089d8 (0x4f7 bytes) 000856 1724773618.818879 MIRROR_EXTEND to 2 mirrors 000858 1724773618.913513 trunc from 0x040000 to 0x00043f (0x3fbc0 bytes) 000859 1724773618.930470 mapwrite 0x026035 thru 0x02f7b1 (0x977d bytes) 000862 1724773618.951400 MIRROR_RESYNC 2 mirrors 000864 1724773619.034861 write 0x02da5c thru 0x032876 (0x4e1b bytes) 000865 1724773619.048125 trunc from 0x032877 to 0x019564 (0x19312 bytes) 000867 1724773619.054868 mapwrite 0x02b4e4 thru 0x02e27b (0x2d98 bytes) 000868 1724773619.070544 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x719:0x0]: Invalid argument (22) 000871 1724773619.097080 trunc from 0x02e27c to 0x02fc71 (0x19f6 bytes) 000872 1724773619.100442 mapread 0x011241 thru 0x0157d0 (0x4590 bytes) 000877 1724773619.105606 mapwrite 0x020222 thru 0x02ab3a (0xa919 bytes) 000878 1724773619.123611 write 0x025051 thru 0x032750 (0xd700 bytes) 000881 1724773619.124948 trunc from 0x032751 to 0x016b44 (0x1bc0c bytes) 000882 1724773619.128578 read 0x003ae9 thru 0x004f57 (0x146f bytes) 000885 1724773619.130587 trunc from 0x016b45 to 0x032720 (0x1bbdc bytes) 000891 1724773619.134793 trunc from 0x032721 to 0x00c5c6 (0x2615a bytes) 000894 1724773619.138517 read 0x0094e2 thru 0x00c5c6 (0x30e5 bytes) 000902 1724773619.142848 read 0x006410 thru 0x009ee3 (0x3ad4 bytes) 000908 1724773619.143920 read 0x006bef thru 0x00c5c6 (0x59d8 bytes) 000909 1724773619.144146 write 0x01eb2c thru 0x029343 (0xa818 bytes) 000912 1724773619.145122 mapread 0x0216c7 thru 0x0251e4 (0x3b1e bytes) 000916 1724773619.146220 trunc from 0x029344 to 0x029b66 (0x823 bytes) 000917 1724773619.150004 mapwrite 0x0365f2 thru 0x03bf08 (0x5917 bytes) 000918 1724773619.166809 mapread 0x011fd9 thru 0x01e4bd (0xc4e5 bytes) 000919 1724773619.167316 read 0x0130ae thru 0x020a22 (0xd975 bytes) 000921 1724773619.167738 read 0x007235 thru 0x00e056 (0x6e22 bytes) 000922 1724773619.168087 read 0x01de37 thru 0x02199d (0x3b67 bytes) 000923 1724773619.168305 read 0x020a50 thru 0x02444d (0x39fe bytes) 000924 1724773619.168531 trunc from 0x03bf09 to 0x02f60b (0xc8fd bytes) 000925 1724773619.171963 trunc from 0x02f60c to 0x037f95 (0x898a bytes) 000929 1724773619.175474 read 0x00e3a0 thru 0x00ea55 (0x6b6 bytes) 000936 1724773619.176924 read 0x01d3ce thru 0x01ff70 (0x2ba3 bytes) 000937 1724773619.178087 MIRROR_EXTEND to 2 mirrors 000938 1724773619.267959 mapread 0x01d75c thru 0x02391c (0x61c1 bytes) 000939 1724773619.272856 read 0x021f81 thru 0x02f6ea (0xd76a bytes) 000940 1724773619.275457 mapwrite 0x0333c6 thru 0x03951c (0x6157 bytes) 000941 1724773619.307553 MIRROR_EXTEND to 3 mirrors 000942 1724773619.393993 read 0x011e2f thru 0x01babf (0x9c91 bytes) 000943 1724773619.396918 trunc from 0x03951d to 0x012901 (0x26c1b bytes) 000944 1724773619.413999 mapread 0x011d67 thru 0x012901 (0xb9b bytes) 000949 1724773619.417527 read 0x00fea3 thru 0x012901 (0x2a5f bytes) 000954 1724773619.420998 trunc from 0x012902 to 0x01ee73 (0xc572 bytes) 000955 1724773619.427196 read 0x01a673 thru 0x01ee73 (0x4801 bytes) 000962 1724773619.432087 mapwrite 0x00f1ef thru 0x01db14 (0xe926 bytes) 000964 1724773619.450478 write 0x026563 thru 0x02a8e1 (0x437f bytes) 000965 1724773619.451490 read 0x0064b4 thru 0x01131a (0xae67 bytes) 000966 1724773619.453670 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x719:0x0]: Invalid argument (22) 000967 1724773619.500910 read 0x022a98 thru 0x02973e (0x6ca7 bytes) 000968 1724773619.503500 mapread 0x01fc86 thru 0x02a8e1 (0xac5c bytes) 000972 1724773619.508755 mapwrite 0x035e94 thru 0x03ffff (0xa16c bytes) 000973 1724773619.531188 trunc from 0x040000 to 0x009a57 (0x365a8 bytes) 000974 1724773619.534807 mapread 0x005e38 thru 0x009a57 (0x3c20 bytes) 000975 1724773619.537110 mapwrite 0x015284 thru 0x018294 (0x3011 bytes) 000977 1724773619.552672 write 0x006e51 thru 0x0082aa (0x145a bytes) 000980 1724773619.553677 read 0x017f99 thru 0x018294 (0x2fc bytes) 000981 1724773619.554058 mapread 0x01676f thru 0x018294 (0x1b26 bytes) 000982 1724773619.555810 MIRROR_EXTEND to 3 mirrors 000985 1724773619.636249 mapwrite 0x016234 thru 0x016774 (0x541 bytes) 000986 1724773619.675020 write 0x000964 thru 0x00c11b (0xb7b8 bytes) 000987 1724773619.677640 write 0x02da40 thru 0x035338 (0x78f9 bytes) 000988 1724773619.678234 mapwrite 0x028cc7 thru 0x02d130 (0x446a bytes) 000989 1724773619.692012 mapread 0x01f3ef thru 0x026560 (0x7172 bytes) 000990 1724773619.694508 read 0x022b5f thru 0x030393 (0xd835 bytes) 000992 1724773619.695135 trunc from 0x035339 to 0x00cf4f (0x283e9 bytes) 000993 1724773619.702623 mapread 0x003048 thru 0x00bda8 (0x8d61 bytes) 000996 1724773619.704430 MIRROR_EXTEND to 4 mirrors 000998 1724773619.783169 trunc from 0x00cf50 to 0x027f96 (0x1b047 bytes) 001000 1724773619.800308 mapwrite 0x0172f7 thru 0x01ee50 (0x7b5a bytes) All operations completed A-OK! PASS 70a (8s) == sanity-flr test 200a: stress test ===================== 11:47:00 (1724773620) Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre2 Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6381504 .. Extending file size to 1508832 .. done Extending file size to 4630304 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7037216 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7238368 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7636320 .. Extending file size to 8932736 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8960864 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8885792 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8997120 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9138688 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9180480 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 32704 32705 32707 32708 32711 192.168.201.106@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg106-client.virtnet /mnt/lustre2 (opts:) 192.168.201.106@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg106-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 11:48:04 (1724773684) Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre2 Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7626400 .. Extending file size to 9338656 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8307264 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8892896 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8919360 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8963296 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 3629 3630 3632 3634 3636 done 192.168.201.106@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg106-client.virtnet /mnt/lustre2 (opts:) 192.168.201.106@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg106-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 11:49:07 (1724773747) Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4864 2203520 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4864 2203520 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0374456 s, 56.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0838407 s, 25.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.106@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg106-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 11:49:12 (1724773752) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0219987 s, 95.3 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 11:49:14 (1724773754) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x754:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc46:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc46:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc47:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0xc41:0x0] } PASS 203 (1s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 11:49:15 (1724773755) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0251946 s, 83.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc48:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc42:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc43:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 11:49:17 (1724773757) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.02577 s, 81.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc49:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc4a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc44:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc49:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc4a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc44:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc45:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 11:49:19 (1724773759) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0220445 s, 95.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc4b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc46:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc4b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc46:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc47:0x0] } PASS 204c (22s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 11:49:41 (1724773781) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg106-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0153852 s, 68.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc4c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc49:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc48:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc4c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc49:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc48:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc4d:0x0] } PASS 204d (40s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 11:50:21 (1724773821) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg106-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc4a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc4b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0259776 s, 80.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc4a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc4b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc4e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc4a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc4b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc4e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (40s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 11:51:01 (1724773861) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg106-server mds-ost sync done. Creating new pool oleg106-server: Pool lustre.test_204f created Adding targets to pool oleg106-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg106-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0274884 s, 76.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc4c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc4f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc4c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc4f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg106-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg106-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg106-server: Pool lustre.test_204f destroyed PASS 204f (51s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 11:51:52 (1724773912) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 11:51:54 (1724773914) PASS 205b (1s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 11:51:55 (1724773915) oleg106-server: Pool lustre.test_206 created Adding targets to pool oleg106-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg106-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg106-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg106-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg106-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 11:52:08 (1724773928) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.019678 s, 53.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101272 s, 104 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x778edbc7 Mirror 3: 0x778edbc7 PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 11:52:11 (1724773931) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (0s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 11:52:11 (1724773931) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 11:52:12 (1724773932) Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0134281 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1190000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2540000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2540000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2540000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1590000 Read this (4 bytes): BBBB 192.168.201.106@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg106-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 11:52:25 (1724773945) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00244039 s, 3.4 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 11:52:27 (1724773947) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109817 s, 95.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x779 lmm_fid: [0x200000404:0x779:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x779 lmm_fid: [0x200000404:0x779:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5a:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835833 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-08-27 11:52:27.000000000 -0400 Modify: 2024-08-27 11:52:27.000000000 -0400 Change: 2024-08-27 11:52:27.000000000 -0400 Birth: - PASS 210a (1s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 11:52:28 (1724773948) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test complete, duration 638 sec ============ 11:52:29 (1724773949) === sanity-flr: start cleanup 11:52:30 (1724773950) === === sanity-flr: finish cleanup 11:52:30 (1724773950) ===