-----============= acceptance-small: sanity-flr ============----- Thu Jun 13 18:04:13 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 6 201 44c 49a === sanity-flr: start setup 18:04:17 (1718316257) === oleg342-client.virtnet: executing check_config_client /mnt/lustre oleg342-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg342-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b6138000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6138000.idle_timeout=debug disable quota as required oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 18:04:23 (1718316263) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f6955] == sanity-flr test 0a: lfs mirror create with -N option == 18:04:23 (1718316263) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 18:04:26 (1718316266) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (0s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 18:04:26 (1718316266) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 18:04:27 (1718316267) PASS 0d (4s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 18:04:31 (1718316271) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 18:04:32 (1718316272) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 18:04:33 (1718316273) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b6138000.stats=clear osc.lustre-OST0001-osc-ffff8800b6138000.stats=clear PASS 0g (1s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 18:04:34 (1718316274) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 18:04:36 (1718316276) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 18:04:38 (1718316278) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (0s) == sanity-flr test 1: create components with setstripe options ========================================================== 18:04:39 (1718316279) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x22:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (1s) == sanity-flr test 2: create components from existing files ========================================================== 18:04:40 (1718316280) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 18:04:42 (1718316282) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 18:04:43 (1718316283) lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 18:04:45 (1718316285) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (1s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 18:04:47 (1718316287) PASS 7 (1s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 18:04:48 (1718316288) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.349316 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0404455 s, 25.9 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 18:04:50 (1718316290) 14+0 records in 14+0 records out 14680064 bytes (15 MB) copied, 0.0719245 s, 204 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 18:04:52 (1718316292) Stopping /mnt/lustre-ost1 (opts:) on oleg342-server oleg342-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6138000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6138000.ost_server_uuid in DISCONN state after 0 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg342-server: oleg342-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg342-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg342-server oleg342-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6138000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6138000.ost_server_uuid in DISCONN state after 4 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg342-server: oleg342-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg342-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (22s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 18:05:14 (1718316314) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB) copied, 0.0887139 s, 70.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg342-server oleg342-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6138000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6138000.ost_server_uuid in DISCONN state after 3 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg342-server: oleg342-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg342-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (11s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 18:05:25 (1718316325) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg342-server oleg342-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6138000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6138000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg342-server: oleg342-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg342-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg342-server oleg342-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6138000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6138000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg342-server: oleg342-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg342-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (16s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 18:05:41 (1718316341) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.766685 s, 68.4 MB/s -rw-r--r-- 1 root root 52428800 Jun 13 18:05 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.186s user 0m0.000s sys 0m0.158s Stopping /mnt/lustre-ost1 (opts:) on oleg342-server oleg342-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6138000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6138000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.200s user 0m0.000s sys 0m0.168s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg342-server: oleg342-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg342-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg342-server oleg342-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6138000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6138000.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost1)... real 0m0.185s user 0m0.000s sys 0m0.118s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg342-server: oleg342-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg342-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg342-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b6138000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6138000.ost_server_uuid in FULL state after 0 sec PASS 33b (21s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 18:06:02 (1718316362) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 18:06:03 (1718316363) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (0s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 18:06:03 (1718316363) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 18:06:04 (1718316364) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 18:06:06 (1718316366) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.117354 s, 107 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (2s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 18:06:08 (1718316368) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00712526 s, 147 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102757 s, 102 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 18:06:10 (1718316370) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 18:06:13 (1718316373) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000720972 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000435183 s, 2.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000683737 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000563509 s, 1.8 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 18:06:15 (1718316375) create test files with size 15757285 .. 8a2452f672090ca3c2ddf21679ec7b88 - 363398db79214b060b39d5494cda9a58 - ac7bcfaa72384ec0e7f5aedda2fd6c7f - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg342-client.virtnet /mnt/lustre (opts:) Starting client: oleg342-client.virtnet: -o user_xattr,flock oleg342-server@tcp:/lustre /mnt/lustre PASS 37 (4s) == sanity-flr test 38: resync ============================ 18:06:19 (1718316379) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 18:06:23 (1718316383) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 18:06:24 (1718316384) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0297061 s, 35.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4c:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00896442 s, 117 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 18:06:26 (1718316386) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0343646 s, 61.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0956804 s, 43.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0402854 s, 52.1 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 18:06:29 (1718316389) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (0s) == sanity-flr test 43a: mirror pick on write ============= 18:06:29 (1718316389) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 18:06:30 (1718316390) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00746744 s, 140 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 18:06:32 (1718316392) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (0s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 18:06:33 (1718316393) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x55:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x13:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (1s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 18:06:35 (1718316395) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0626618 s, 167 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg342-client.virtnet /mnt/lustre (opts:) Starting client: oleg342-client.virtnet: -o user_xattr,flock oleg342-server@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (2s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 18:06:37 (1718316397) llite.lustre-ffff8800aac6d000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0687207 s, 153 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg342-client.virtnet /mnt/lustre (opts:) Starting client: oleg342-client.virtnet: -o user_xattr,flock oleg342-server@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800a290e800.xattr_cache=1 PASS 44e (2s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 18:06:40 (1718316400) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (6s) == sanity-flr test 46: Verify setstripe --copy option ==== 18:06:45 (1718316405) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (5s) == sanity-flr test 47: Verify mirror obj alloc =========== 18:06:50 (1718316410) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 18:06:51 (1718316411) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0568057 s, 55.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0448984 s, 70.1 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: c24e1b6e3b700dc3db27c97e1803da31 - mirror 1 checksum: 534e321bd0eec0cb2afdbcc821643d01 - mirror 2 checksum: c24e1b6e3b700dc3db27c97e1803da31 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jun 13 18:06:52 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jun 13 18:06:52 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: c24e1b6e3b700dc3db27c97e1803da31 - mirror 1 checksum: 534e321bd0eec0cb2afdbcc821643d01 - mirror 2 checksum: 534e321bd0eec0cb2afdbcc821643d01 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jun 13 18:06:52 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jun 13 18:06:52 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } PASS 48 (1s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 18:06:53 (1718316413) Failing mds1 on oleg342-server Stopping /mnt/lustre-mds1 (opts:) on oleg342-server 18:06:54 (1718316414) shut down Failover mds1 to oleg342-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-MDT0000 18:07:07 (1718316427) targets are mounted 18:07:07 (1718316427) facet_failover done oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00628597 s, 167 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x14 lmm_fid: [0x200000404:0x14:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } PASS 50A (18s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 18:07:11 (1718316431) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 18:07:12 (1718316432) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (0s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 18:07:12 (1718316432) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00179284 s, 9.1 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 18:07:14 (1718316434) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 18:07:15 (1718316435) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (0s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 18:07:15 (1718316435) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d:0x0]: Invalid argument (22) PASS 60b (4s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 18:07:19 (1718316439) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (17s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 18:07:36 (1718316456) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4e:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 18:07:53 (1718316473) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 18:08:10 (1718316490) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (1s) == sanity-flr test 70: mirror create and split race ====== 18:08:11 (1718316491) mirror create pid 20480 mirror split pid 20481 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4719: 20480 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4719: 20481 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 18:09:13 (1718316553) Seed set to 6553 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000004 1718316553.751932 MIRROR_EXTEND to 2 mirrors skipping zero size read 000007 1718316553.813593 MIRROR_RESYNC 2 mirrors 000009 1718316553.834874 write 0x0067aa thru 0x00ffc9 (0x9820 bytes) 000011 1718316553.849471 write 0x004ff6 thru 0x00cc5c (0x7c67 bytes) 000012 1718316553.854542 mapwrite 0x00454b thru 0x00c83b (0x82f1 bytes) 000014 1718316553.878130 mapwrite 0x012777 thru 0x021952 (0xf1dc bytes) 000015 1718316553.908402 mapwrite 0x001b91 thru 0x00f608 (0xda78 bytes) truncating to largest ever: 0x376da 000016 1718316553.924938 trunc from 0x021953 to 0x0376d9 (0x15d87 bytes) 000018 1718316553.929077 write 0x003b45 thru 0x004d09 (0x11c5 bytes) 000019 1718316553.929675 mapread 0x0170ef thru 0x019c03 (0x2b15 bytes) 000020 1718316553.931342 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x614:0x0]: Invalid argument (22) 000021 1718316553.964656 trunc from 0x0376da to 0x00d1e4 (0x2a4f5 bytes) 000022 1718316553.967730 write 0x026122 thru 0x02be4f (0x5d2e bytes) 000025 1718316553.968796 mapwrite 0x0024d9 thru 0x012163 (0xfc8b bytes) 000029 1718316553.990952 read 0x01d3b5 thru 0x02be4f (0xea9b bytes) 000030 1718316553.994600 trunc from 0x02be50 to 0x0032b6 (0x28b99 bytes) 000031 1718316553.998812 trunc from 0x0032b7 to 0x029f94 (0x26cde bytes) 000033 1718316554.003529 trunc from 0x029f95 to 0x023a71 (0x6523 bytes) 000036 1718316554.008201 mapread 0x0037a8 thru 0x00ef5a (0xb7b3 bytes) 000037 1718316554.013638 MIRROR_EXTEND to 2 mirrors 000038 1718316554.139244 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x614:0x0]: Invalid argument (22) 000040 1718316554.170722 mapwrite 0x0177ef thru 0x018d78 (0x158a bytes) 000041 1718316554.191299 read 0x019bf6 thru 0x01f814 (0x5c1f bytes) 000042 1718316554.193805 read 0x00f2c5 thru 0x015b17 (0x6853 bytes) truncating to largest ever: 0x3dc2c 000043 1718316554.196102 trunc from 0x023a72 to 0x03dc2b (0x1a1ba bytes) 000044 1718316554.199638 trunc from 0x03dc2c to 0x0066cd (0x3755e bytes) 000046 1718316554.203976 mapwrite 0x00d871 thru 0x013ded (0x657d bytes) 000048 1718316554.230350 read 0x011189 thru 0x013ded (0x2c65 bytes) 000049 1718316554.230790 write 0x025dae thru 0x028962 (0x2bb5 bytes) 000051 1718316554.231625 mapread 0x01b70a thru 0x01e871 (0x3168 bytes) 000052 1718316554.234357 write 0x021cec thru 0x031017 (0xf32c bytes) 000054 1718316554.235624 read 0x02c38b thru 0x031017 (0x4c8d bytes) 000055 1718316554.236021 read 0x02d3bb thru 0x031017 (0x3c5d bytes) 000056 1718316554.236386 write 0x007532 thru 0x00d971 (0x6440 bytes) 000057 1718316554.236838 trunc from 0x031018 to 0x034bc0 (0x3ba9 bytes) 000062 1718316554.241704 read 0x018c34 thru 0x023e18 (0xb1e5 bytes) 000064 1718316554.242429 mapwrite 0x02d443 thru 0x039e07 (0xc9c5 bytes) 000066 1718316554.264813 read 0x01c610 thru 0x01d908 (0x12f9 bytes) 000068 1718316554.265400 write 0x00c0b6 thru 0x0147c2 (0x870d bytes) 000069 1718316554.266025 read 0x00a236 thru 0x00bd42 (0x1b0d bytes) 000073 1718316554.267133 trunc from 0x039e08 to 0x036760 (0x36a7 bytes) 000074 1718316554.270307 read 0x0103bd thru 0x01c550 (0xc194 bytes) 000078 1718316554.271455 write 0x01e74e thru 0x024e82 (0x6735 bytes) 000079 1718316554.271882 trunc from 0x036761 to 0x03910b (0x29ab bytes) 000080 1718316554.274800 mapread 0x018b40 thru 0x024e5d (0xc31e bytes) 000081 1718316554.275174 read 0x02fef4 thru 0x034c74 (0x4d81 bytes) 000083 1718316554.275616 mapwrite 0x00484d thru 0x00798e (0x3142 bytes) 000084 1718316554.289590 read 0x00b125 thru 0x01337d (0x8259 bytes) 000088 1718316554.291943 MIRROR_EXTEND to 2 mirrors 000089 1718316554.390309 write 0x00e533 thru 0x018cf9 (0xa7c7 bytes) 000090 1718316554.408107 mapread 0x030b47 thru 0x031e4d (0x1307 bytes) 000091 1718316554.410584 trunc from 0x03910c to 0x03a2f6 (0x11eb bytes) 000092 1718316554.418363 read 0x00c208 thru 0x017bba (0xb9b3 bytes) 000093 1718316554.421079 MIRROR_EXTEND to 3 mirrors 000094 1718316554.539131 write 0x0192dc thru 0x028e9c (0xfbc1 bytes) 000095 1718316554.554439 mapwrite 0x005b58 thru 0x01016f (0xa618 bytes) 000096 1718316554.570203 trunc from 0x03a2f7 to 0x027c20 (0x126d6 bytes) 000097 1718316554.578120 write 0x031a6f thru 0x03ff6f (0xe501 bytes) 000098 1718316554.578944 read 0x00501a thru 0x00966e (0x4655 bytes) 000099 1718316554.579182 trunc from 0x03ff70 to 0x0125a2 (0x2d9cd bytes) 000100 1718316554.583227 write 0x014be1 thru 0x01e13e (0x955e bytes) 000102 1718316554.584396 mapwrite 0x01b775 thru 0x029b89 (0xe415 bytes) 000103 1718316554.606701 write 0x0351ba thru 0x0366b4 (0x14fb bytes) 000104 1718316554.607308 mapwrite 0x0018f2 thru 0x0060aa (0x47b9 bytes) 000105 1718316554.621024 read 0x02cf3b thru 0x0366b4 (0x977a bytes) 000109 1718316554.624493 read 0x0175f9 thru 0x0230c8 (0xbad0 bytes) 000110 1718316554.626181 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x614:0x0]: Invalid argument (22) 000111 1718316554.671003 read 0x027be2 thru 0x02c232 (0x4651 bytes) 000113 1718316554.673910 trunc from 0x0366b5 to 0x0257d7 (0x10edd bytes) 000114 1718316554.677913 write 0x03e81a thru 0x03ffff (0x17e6 bytes) 000118 1718316554.679215 mapread 0x0203ac thru 0x02de6d (0xdac2 bytes) 000119 1718316554.686296 MIRROR_RESYNC 2 mirrors 000122 1718316554.767382 trunc from 0x040000 to 0x0099fe (0x36601 bytes) 000123 1718316554.785023 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x614:0x0]: Invalid argument (22) 000124 1718316554.810914 write 0x0313e4 thru 0x039b5b (0x8778 bytes) 000126 1718316554.811831 write 0x0283de thru 0x02a59e (0x21c1 bytes) 000127 1718316554.815184 MIRROR_EXTEND to 2 mirrors 000130 1718316554.914167 read 0x004346 thru 0x00736d (0x3028 bytes) 000132 1718316554.915866 trunc from 0x039b5c to 0x02d006 (0xcb55 bytes) 000133 1718316554.932584 trunc from 0x02d007 to 0x01a79f (0x12867 bytes) 000135 1718316554.936362 write 0x02bcff thru 0x032222 (0x6524 bytes) 000136 1718316554.937989 MIRROR_RESYNC 2 mirrors 000137 1718316555.019397 mapwrite 0x0062f8 thru 0x0107b4 (0xa4bd bytes) 000139 1718316555.060512 trunc from 0x032223 to 0x02a80e (0x7a14 bytes) 000140 1718316555.068175 trunc from 0x02a80f to 0x0254cd (0x5341 bytes) 000141 1718316555.072419 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x614:0x0]: Invalid argument (22) 000143 1718316555.100109 mapread 0x01c95c thru 0x0254cd (0x8b72 bytes) 000147 1718316555.106838 MIRROR_EXTEND to 2 mirrors 000148 1718316555.202872 write 0x03402d thru 0x0379f9 (0x39cd bytes) 000149 1718316555.218232 mapread 0x0041b5 thru 0x004f10 (0xd5c bytes) 000154 1718316555.220464 mapread 0x029455 thru 0x02d11b (0x3cc7 bytes) 000156 1718316555.223912 mapwrite 0x0272dd thru 0x02f48b (0x81af bytes) 000157 1718316555.240237 trunc from 0x0379fa to 0x00e4c4 (0x29535 bytes) 000161 1718316555.246535 write 0x01216f thru 0x0202d4 (0xe166 bytes) 000163 1718316555.247487 read 0x0081c9 thru 0x0087c1 (0x5f9 bytes) 000164 1718316555.249122 mapwrite 0x02dd2c thru 0x03d4bc (0xf791 bytes) 000165 1718316555.273962 mapread 0x0091c8 thru 0x016820 (0xd659 bytes) 000169 1718316555.278332 read 0x014e08 thru 0x01730d (0x2506 bytes) 000172 1718316555.279275 mapread 0x026a1b thru 0x0282ec (0x18d2 bytes) 000175 1718316555.280335 read 0x023809 thru 0x025fe1 (0x27d9 bytes) 000176 1718316555.283910 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x614:0x0]: Invalid argument (22) 000178 1718316555.313548 write 0x01b611 thru 0x02a464 (0xee54 bytes) 000179 1718316555.316695 mapwrite 0x03e57b thru 0x03ffff (0x1a85 bytes) 000180 1718316555.335684 trunc from 0x040000 to 0x01190c (0x2e6f3 bytes) 000181 1718316555.339780 mapread 0x00cb20 thru 0x01190c (0x4ded bytes) 000182 1718316555.342578 mapwrite 0x014d6c thru 0x015f15 (0x11aa bytes) 000184 1718316555.362370 read 0x00dda0 thru 0x015f15 (0x8176 bytes) 000185 1718316555.362793 mapwrite 0x03737a thru 0x03c4ba (0x5141 bytes) 000188 1718316555.384802 write 0x004ac7 thru 0x004b27 (0x61 bytes) 000191 1718316555.387889 read 0x0092ce thru 0x0126c1 (0x93f4 bytes) 000194 1718316555.389183 mapwrite 0x02bd3a thru 0x03637e (0xa645 bytes) 000195 1718316555.407003 trunc from 0x03c4bb to 0x00f5d9 (0x2cee1 bytes) 000199 1718316555.411741 trunc from 0x00f5da to 0x024587 (0x14fae bytes) 000201 1718316555.417866 read 0x016ded thru 0x024587 (0xd79b bytes) 000202 1718316555.420575 mapread 0x002cdd thru 0x00a534 (0x7858 bytes) 000206 1718316555.421664 mapwrite 0x03e1eb thru 0x03fedb (0x1cf1 bytes) 000208 1718316555.440407 write 0x007fcf thru 0x01278c (0xa7be bytes) 000210 1718316555.441413 trunc from 0x03fedc to 0x0332ff (0xcbdc bytes) 000211 1718316555.445236 trunc from 0x033300 to 0x012390 (0x20f6f bytes) 000212 1718316555.449600 trunc from 0x012391 to 0x020e8f (0xeaff bytes) 000215 1718316555.453296 write 0x01f111 thru 0x027a60 (0x8950 bytes) 000219 1718316555.455859 mapwrite 0x022fb1 thru 0x026591 (0x35e1 bytes) 000221 1718316555.469788 read 0x018885 thru 0x01c886 (0x4002 bytes) 000222 1718316555.472352 trunc from 0x027a61 to 0x01b09d (0xc9c3 bytes) 000223 1718316555.475812 mapread 0x013ec5 thru 0x018044 (0x4180 bytes) 000226 1718316555.477104 trunc from 0x01b09e to 0x02e815 (0x13778 bytes) 000229 1718316555.480894 trunc from 0x02e816 to 0x00b390 (0x23485 bytes) 000234 1718316555.485307 read 0x0093fd thru 0x0097f1 (0x3f5 bytes) 000235 1718316555.485669 mapwrite 0x00f3dd thru 0x01522b (0x5e4f bytes) 000236 1718316555.506910 trunc from 0x01522c to 0x031110 (0x1bee5 bytes) 000238 1718316555.511693 mapread 0x0152d4 thru 0x016635 (0x1362 bytes) 000239 1718316555.514769 write 0x005f87 thru 0x0083a5 (0x241f bytes) 000240 1718316555.515424 write 0x02dc5d thru 0x034640 (0x69e4 bytes) 000241 1718316555.516024 read 0x023e50 thru 0x029049 (0x51fa bytes) 000243 1718316555.516547 read 0x00df1d thru 0x01dea9 (0xff8d bytes) 000245 1718316555.517306 trunc from 0x034641 to 0x01d6fd (0x16f43 bytes) 000246 1718316555.521206 mapread 0x01922b thru 0x01d6fd (0x44d3 bytes) 000247 1718316555.521676 mapwrite 0x01bbe9 thru 0x02299a (0x6db2 bytes) 000248 1718316555.541838 mapwrite 0x03108b thru 0x03ffff (0xef75 bytes) 000250 1718316555.563741 trunc from 0x040000 to 0x02a39d (0x15c62 bytes) 000255 1718316555.569240 read 0x018ad2 thru 0x0226fc (0x9c2b bytes) truncating to largest ever: 0x3de61 000256 1718316555.569692 trunc from 0x02a39e to 0x03de60 (0x13ac3 bytes) 000257 1718316555.573542 trunc from 0x03de61 to 0x00fe6d (0x2dff3 bytes) 000258 1718316555.577172 mapread 0x00ae5a thru 0x00fe6d (0x5014 bytes) 000259 1718316555.577791 write 0x01c527 thru 0x01cf65 (0xa3f bytes) 000260 1718316555.578378 mapread 0x0002bf thru 0x00be39 (0xbb7b bytes) 000261 1718316555.578871 mapread 0x0001a6 thru 0x00efcd (0xee28 bytes) 000263 1718316555.579690 mapwrite 0x02918e thru 0x034a9c (0xb90f bytes) 000264 1718316555.602405 mapread 0x019f07 thru 0x025904 (0xb9fe bytes) 000265 1718316555.603018 mapwrite 0x00791d thru 0x015d0a (0xe3ee bytes) 000266 1718316555.619241 trunc from 0x034a9d to 0x02a0a8 (0xa9f4 bytes) 000267 1718316555.622794 read 0x012917 thru 0x012d46 (0x430 bytes) 000269 1718316555.623369 read 0x00db5b thru 0x010443 (0x28e9 bytes) 000271 1718316555.623928 write 0x016574 thru 0x023c6d (0xd6fa bytes) 000272 1718316555.624562 mapwrite 0x02243d thru 0x031f5e (0xfb22 bytes) 000273 1718316555.650738 MIRROR_EXTEND to 2 mirrors 000274 1718316555.749759 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x614:0x0]: Invalid argument (22) 000277 1718316555.778721 trunc from 0x031f5f to 0x02ff13 (0x204b bytes) 000278 1718316555.788995 mapread 0x0211cf thru 0x02ab64 (0x9996 bytes) 000279 1718316555.793733 read 0x01d912 thru 0x02343b (0x5b2a bytes) 000283 1718316555.796061 write 0x03a797 thru 0x03ffff (0x5869 bytes) 000285 1718316555.796938 read 0x005840 thru 0x0122a1 (0xca62 bytes) 000289 1718316555.800589 write 0x0117f7 thru 0x01f1b7 (0xd9c1 bytes) 000290 1718316555.801436 read 0x013e17 thru 0x021d18 (0xdf02 bytes) 000291 1718316555.801818 mapwrite 0x0072e2 thru 0x00e561 (0x7280 bytes) 000293 1718316555.818064 mapread 0x0309a8 thru 0x035972 (0x4fcb bytes) 000294 1718316555.818653 mapread 0x02da70 thru 0x037829 (0x9dba bytes) 000295 1718316555.819144 trunc from 0x040000 to 0x0106ff (0x2f900 bytes) 000297 1718316555.823864 trunc from 0x010700 to 0x00271e (0xdfe1 bytes) 000298 1718316555.827571 write 0x0060d9 thru 0x012cfe (0xcc26 bytes) 000300 1718316555.828712 trunc from 0x012cff to 0x0106a8 (0x2656 bytes) 000302 1718316555.832639 mapread 0x0055d5 thru 0x0106a8 (0xb0d4 bytes) 000303 1718316555.834363 mapread 0x008455 thru 0x00aed8 (0x2a84 bytes) 000307 1718316555.835628 mapread 0x0084c9 thru 0x00c6f2 (0x422a bytes) 000308 1718316555.836053 mapread 0x0020e9 thru 0x0034bc (0x13d4 bytes) 000309 1718316555.838001 MIRROR_EXTEND to 2 mirrors 000310 1718316555.950759 trunc from 0x0106a9 to 0x029694 (0x18fec bytes) 000312 1718316555.972150 read 0x01c205 thru 0x01ce88 (0xc84 bytes) 000313 1718316555.975332 MIRROR_RESYNC 2 mirrors 000316 1718316556.087538 mapread 0x00e968 thru 0x019ed7 (0xb570 bytes) 000318 1718316556.095905 mapread 0x0222b5 thru 0x027dc1 (0x5b0d bytes) 000319 1718316556.100047 MIRROR_EXTEND to 3 mirrors 000321 1718316556.217787 mapwrite 0x03f7ed thru 0x03ffff (0x813 bytes) 000323 1718316556.252180 mapread 0x0212e7 thru 0x02a559 (0x9273 bytes) 000324 1718316556.260615 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x614:0x0]: Invalid argument (22) 000328 1718316556.312548 mapwrite 0x00de7a thru 0x017776 (0x98fd bytes) truncating to largest ever: 0x3e6d2 000329 1718316556.336181 trunc from 0x040000 to 0x03e6d1 (0x192e bytes) 000330 1718316556.341594 read 0x00c56a thru 0x00f929 (0x33c0 bytes) 000337 1718316556.343416 read 0x02977f thru 0x02bab6 (0x2338 bytes) 000339 1718316556.347083 trunc from 0x03e6d2 to 0x0397be (0x4f13 bytes) 000340 1718316556.351570 mapwrite 0x03ba88 thru 0x03d5e2 (0x1b5b bytes) 000341 1718316556.373724 read 0x01eec0 thru 0x02b523 (0xc664 bytes) 000342 1718316556.375539 MIRROR_RESYNC 2 mirrors 000343 1718316556.477384 mapwrite 0x01516e thru 0x022a1c (0xd8af bytes) 000345 1718316556.528446 mapwrite 0x009512 thru 0x00d4f0 (0x3fdf bytes) 000348 1718316556.547268 mapread 0x026a3a thru 0x032480 (0xba47 bytes) 000349 1718316556.553156 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x614:0x0]: Invalid argument (22) 000350 1718316556.581145 write 0x03df04 thru 0x03ffff (0x20fc bytes) 000351 1718316556.582744 mapwrite 0x01a364 thru 0x025e71 (0xbb0e bytes) 000355 1718316556.602222 trunc from 0x040000 to 0x0017b5 (0x3e84a bytes) 000356 1718316556.609442 mapwrite 0x02a656 thru 0x02a786 (0x131 bytes) 000357 1718316556.628780 mapwrite 0x015825 thru 0x0188b0 (0x308c bytes) 000361 1718316556.645911 mapread 0x0101d4 thru 0x012424 (0x2251 bytes) 000363 1718316556.646591 read 0x016908 thru 0x01cb8d (0x6286 bytes) 000366 1718316556.647674 read 0x02941d thru 0x02a786 (0x136a bytes) 000368 1718316556.648310 trunc from 0x02a787 to 0x03b98e (0x11208 bytes) 000371 1718316556.652809 mapwrite 0x03bc1f thru 0x03e89e (0x2c80 bytes) 000372 1718316556.673956 write 0x037559 thru 0x03ffff (0x8aa7 bytes) 000375 1718316556.675052 mapwrite 0x00d42e thru 0x013069 (0x5c3c bytes) 000376 1718316556.692907 write 0x00cf31 thru 0x00d82d (0x8fd bytes) 000378 1718316556.694038 mapwrite 0x01f400 thru 0x0212ee (0x1eef bytes) 000380 1718316556.709931 read 0x007228 thru 0x00c975 (0x574e bytes) 000385 1718316556.711745 trunc from 0x040000 to 0x00e3c8 (0x31c37 bytes) 000386 1718316556.716666 mapread 0x00a7af thru 0x00ab5c (0x3ae bytes) 000388 1718316556.717488 mapwrite 0x035da6 thru 0x03aa59 (0x4cb4 bytes) 000389 1718316556.742885 trunc from 0x03aa5a to 0x0217ec (0x1926d bytes) 000391 1718316556.747858 mapread 0x01603f thru 0x0217ec (0xb7ae bytes) 000395 1718316556.748967 mapwrite 0x037834 thru 0x03bdac (0x4579 bytes) 000397 1718316556.777001 write 0x03e99c thru 0x03ffff (0x1664 bytes) 000399 1718316556.778231 trunc from 0x040000 to 0x036d17 (0x92e8 bytes) 000400 1718316556.783822 read 0x0080f3 thru 0x016e5d (0xed6b bytes) 000401 1718316556.785952 MIRROR_EXTEND to 2 mirrors 000402 1718316556.906331 trunc from 0x036d18 to 0x028ebb (0xde5c bytes) 000403 1718316556.928822 mapwrite 0x018aee thru 0x01b815 (0x2d28 bytes) 000404 1718316556.950663 mapwrite 0x018dd0 thru 0x0279e3 (0xec14 bytes) 000407 1718316556.972790 mapwrite 0x01d877 thru 0x02b2e7 (0xda71 bytes) 000408 1718316556.996355 trunc from 0x02b2e8 to 0x01d648 (0xdc9f bytes) 000409 1718316557.000815 mapread 0x019b07 thru 0x01d648 (0x3b42 bytes) 000410 1718316557.001320 mapread 0x00ba40 thru 0x01b5f6 (0xfbb7 bytes) 000411 1718316557.004047 mapwrite 0x024bd5 thru 0x02b4d8 (0x6904 bytes) 000414 1718316557.025177 mapwrite 0x00f7a0 thru 0x0137ab (0x400c bytes) 000415 1718316557.040626 read 0x00cfa2 thru 0x013804 (0x6863 bytes) 000416 1718316557.041171 read 0x029166 thru 0x02b1c9 (0x2064 bytes) 000418 1718316557.042064 mapwrite 0x001814 thru 0x00ce8e (0xb67b bytes) 000419 1718316557.060012 write 0x002dbf thru 0x008b46 (0x5d88 bytes) 000420 1718316557.062011 MIRROR_EXTEND to 3 mirrors 000423 1718316557.170782 read 0x006df2 thru 0x014e22 (0xe031 bytes) 000424 1718316557.173347 read 0x02241c thru 0x0289bd (0x65a2 bytes) 000426 1718316557.176252 mapread 0x01278b thru 0x020ec0 (0xe736 bytes) 000428 1718316557.176986 read 0x011070 thru 0x01877f (0x7710 bytes) 000429 1718316557.177254 trunc from 0x02b4d9 to 0x03d72d (0x12255 bytes) 000432 1718316557.196196 write 0x00f7cf thru 0x01a676 (0xaea8 bytes) 000436 1718316557.202743 MIRROR_EXTEND to 4 mirrors 000437 1718316557.327880 write 0x031afb thru 0x034465 (0x296b bytes) 000438 1718316557.344270 write 0x03bbd3 thru 0x03ffff (0x442d bytes) 000441 1718316557.347365 trunc from 0x040000 to 0x008487 (0x37b78 bytes) 000442 1718316557.351949 mapread 0x003608 thru 0x007c32 (0x462b bytes) 000443 1718316557.354269 read 0x00757d thru 0x008487 (0xf0b bytes) 000444 1718316557.354614 read 0x003b73 thru 0x008487 (0x4915 bytes) 000445 1718316557.354866 read 0x005611 thru 0x008487 (0x2e77 bytes) 000446 1718316557.355105 mapwrite 0x03ffe6 thru 0x03ffff (0x1a bytes) 000448 1718316557.375751 read 0x00087d thru 0x009825 (0x8fa9 bytes) 000449 1718316557.376097 mapread 0x018835 thru 0x01deb2 (0x567e bytes) 000450 1718316557.376604 trunc from 0x040000 to 0x0195cf (0x26a30 bytes) 000451 1718316557.381290 write 0x0243ae thru 0x02c4f8 (0x814b bytes) 000454 1718316557.383846 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x614:0x0]: Invalid argument (22) 000455 1718316557.432153 write 0x00d312 thru 0x01bb3a (0xe829 bytes) 000456 1718316557.436503 MIRROR_RESYNC 3 mirrors 000457 1718316557.557615 write 0x02b9c7 thru 0x02cc62 (0x129c bytes) 000458 1718316557.572360 trunc from 0x02cc63 to 0x004095 (0x28bcd bytes) 000459 1718316557.576976 read 0x00055f thru 0x0037d5 (0x3277 bytes) 000464 1718316557.579753 read 0x002a49 thru 0x004095 (0x164d bytes) 000467 1718316557.581939 write 0x02a820 thru 0x02fc2a (0x540b bytes) 000469 1718316557.582793 write 0x019907 thru 0x025963 (0xc05d bytes) 000470 1718316557.585886 mapwrite 0x01cacc thru 0x02667b (0x9bb0 bytes) 000471 1718316557.602958 write 0x02e1ac thru 0x02ea3d (0x892 bytes) 000473 1718316557.603761 write 0x016e26 thru 0x02382a (0xca05 bytes) 000475 1718316557.604749 mapwrite 0x023830 thru 0x0304a4 (0xcc75 bytes) 000476 1718316557.624976 write 0x0175d4 thru 0x02274f (0xb17c bytes) 000479 1718316557.625997 mapwrite 0x02caff thru 0x0361b9 (0x96bb bytes) 000482 1718316557.646373 trunc from 0x0361ba to 0x009e93 (0x2c326 bytes) 000483 1718316557.649743 mapread 0x008746 thru 0x009e93 (0x174e bytes) 000484 1718316557.651318 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x614:0x0]: Invalid argument (22) 000486 1718316557.696653 write 0x006c8b thru 0x00eb27 (0x7e9d bytes) 000487 1718316557.698665 trunc from 0x00eb28 to 0x01a0ed (0xb5c6 bytes) 000488 1718316557.702130 read 0x009ba6 thru 0x00c753 (0x2bae bytes) 000489 1718316557.702459 write 0x032fcf thru 0x03ffff (0xd031 bytes) 000490 1718316557.703236 trunc from 0x040000 to 0x0255be (0x1aa41 bytes) 000491 1718316557.706885 trunc from 0x0255bf to 0x00c66e (0x18f50 bytes) 000493 1718316557.710803 trunc from 0x00c66f to 0x006d70 (0x58fe bytes) 000496 1718316557.714698 mapread 0x003bb2 thru 0x005a1b (0x1e6a bytes) 000499 1718316557.718112 read 0x001fc8 thru 0x006d70 (0x4da9 bytes) 000500 1718316557.718382 read 0x0023f1 thru 0x006d70 (0x4980 bytes) 000501 1718316557.718615 read 0x00540e thru 0x006d70 (0x1963 bytes) 000507 1718316557.719860 trunc from 0x006d71 to 0x00a6c3 (0x3953 bytes) 000508 1718316557.724764 MIRROR_RESYNC 2 mirrors 000509 1718316557.806675 read 0x007aa4 thru 0x00a6c3 (0x2c20 bytes) 000512 1718316557.808241 mapwrite 0x003254 thru 0x00d946 (0xa6f3 bytes) 000513 1718316557.844199 mapread 0x004015 thru 0x004cdd (0xcc9 bytes) 000514 1718316557.844773 mapread 0x008c06 thru 0x00d946 (0x4d41 bytes) 000515 1718316557.845267 mapwrite 0x01b591 thru 0x025741 (0xa1b1 bytes) 000516 1718316557.869145 MIRROR_RESYNC 2 mirrors 000518 1718316557.949864 mapwrite 0x00797e thru 0x01023e (0x88c1 bytes) 000519 1718316557.992317 mapwrite 0x01524a thru 0x01e552 (0x9309 bytes) 000520 1718316558.009254 MIRROR_EXTEND to 3 mirrors 000524 1718316558.119842 trunc from 0x025742 to 0x026af5 (0x13b4 bytes) 000525 1718316558.134778 write 0x017f8c thru 0x01facb (0x7b40 bytes) 000526 1718316558.137212 write 0x0157ca thru 0x024ab2 (0xf2e9 bytes) 000527 1718316558.140514 read 0x004c14 thru 0x0111c7 (0xc5b4 bytes) 000528 1718316558.142431 mapread 0x022f97 thru 0x024d47 (0x1db1 bytes) 000529 1718316558.142829 mapread 0x023a09 thru 0x026af5 (0x30ed bytes) 000531 1718316558.145826 MIRROR_EXTEND to 4 mirrors 000532 1718316558.255977 write 0x0009e4 thru 0x00845e (0x7a7b bytes) 000533 1718316558.271855 write 0x019e32 thru 0x025be4 (0xbdb3 bytes) 000539 1718316558.276277 write 0x011a45 thru 0x01ad7d (0x9339 bytes) 000540 1718316558.278582 trunc from 0x026af6 to 0x00ddb0 (0x18d45 bytes) 000542 1718316558.286939 write 0x03ec5c thru 0x03ffff (0x13a4 bytes) 000543 1718316558.287509 trunc from 0x040000 to 0x02688d (0x19772 bytes) 000544 1718316558.291364 read 0x0044cb thru 0x00797a (0x34b0 bytes) 000546 1718316558.291885 trunc from 0x02688e to 0x032459 (0xbbcc bytes) 000547 1718316558.295669 trunc from 0x03245a to 0x028738 (0x9d21 bytes) 000548 1718316558.299178 read 0x00009c thru 0x0067e9 (0x674e bytes) 000549 1718316558.299568 trunc from 0x028739 to 0x033f03 (0xb7cb bytes) 000551 1718316558.303240 trunc from 0x033f04 to 0x029407 (0xaafc bytes) 000552 1718316558.306374 read 0x00de0e thru 0x0173ff (0x95f2 bytes) 000553 1718316558.309173 mapwrite 0x01a1af thru 0x01c4ff (0x2351 bytes) 000554 1718316558.323847 write 0x0289b1 thru 0x035bca (0xd21a bytes) 000555 1718316558.324584 read 0x000a8c thru 0x0072c5 (0x683a bytes) 000556 1718316558.326359 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x614:0x0]: Invalid argument (22) 000558 1718316558.373456 MIRROR_EXTEND to 4 mirrors 000559 1718316558.466909 MIRROR_RESYNC 4 mirrors 000563 1718316558.582781 trunc from 0x035bcb to 0x01cc8d (0x18f3d bytes) 000566 1718316558.601143 write 0x016973 thru 0x023ce1 (0xd36f bytes) 000567 1718316558.603455 write 0x00c57d thru 0x019849 (0xd2cd bytes) 000570 1718316558.606068 mapwrite 0x019a1e thru 0x01d56d (0x3b50 bytes) 000575 1718316558.620942 mapwrite 0x0249e4 thru 0x026195 (0x17b2 bytes) 000577 1718316558.639646 mapwrite 0x00e57a thru 0x01515c (0x6be3 bytes) 000578 1718316558.651450 read 0x016714 thru 0x01883a (0x2127 bytes) 000579 1718316558.651831 read 0x016e66 thru 0x01a5ec (0x3787 bytes) 000581 1718316558.652422 mapread 0x00415b thru 0x00bb93 (0x7a39 bytes) 000582 1718316558.654899 read 0x01f36c thru 0x0200bc (0xd51 bytes) 000583 1718316558.655160 write 0x03a927 thru 0x03bbb9 (0x1293 bytes) 000585 1718316558.655923 mapwrite 0x011b70 thru 0x021378 (0xf809 bytes) 000589 1718316558.670552 trunc from 0x03bbba to 0x0038f3 (0x382c6 bytes) 000590 1718316558.674392 mapwrite 0x0345ff thru 0x03ffff (0xba01 bytes) 000594 1718316558.694159 read 0x021de6 thru 0x02b147 (0x9362 bytes) 000596 1718316558.694642 mapwrite 0x03f319 thru 0x03ffff (0xce7 bytes) 000597 1718316558.708381 MIRROR_RESYNC 4 mirrors 000598 1718316558.816855 mapwrite 0x033c6d thru 0x03d2cc (0x9660 bytes) 000600 1718316558.860610 mapread 0x022262 thru 0x02e927 (0xc6c6 bytes) 000604 1718316558.865878 read 0x039fb5 thru 0x03d32e (0x337a bytes) 000605 1718316558.866199 trunc from 0x040000 to 0x039055 (0x6faa bytes) 000606 1718316558.873818 mapwrite 0x03ac9b thru 0x03ffff (0x5365 bytes) 000607 1718316558.891185 MIRROR_RESYNC 4 mirrors 000609 1718316559.006288 read 0x00a0ab thru 0x015c75 (0xbbcb bytes) 000610 1718316559.008783 read 0x027a50 thru 0x02dd3b (0x62ec bytes) 000611 1718316559.011702 mapwrite 0x001953 thru 0x00676a (0x4e18 bytes) 000612 1718316559.058069 mapwrite 0x011b3e thru 0x01bd82 (0xa245 bytes) 000613 1718316559.085788 mapread 0x00baec thru 0x00d8a0 (0x1db5 bytes) 000615 1718316559.090310 trunc from 0x040000 to 0x00ff7c (0x30083 bytes) 000616 1718316559.099910 write 0x03b5f5 thru 0x03ffff (0x4a0b bytes) 000621 1718316559.101764 mapwrite 0x026645 thru 0x03524c (0xec08 bytes) 000623 1718316559.136516 mapread 0x02e125 thru 0x03ad12 (0xcbee bytes) 000626 1718316559.143379 trunc from 0x040000 to 0x01b007 (0x24ff8 bytes) 000629 1718316559.150236 MIRROR_RESYNC 4 mirrors 000630 1718316559.287877 read 0x0151ac thru 0x01b007 (0x5e5c bytes) 000633 1718316559.290379 read 0x016fa8 thru 0x01b007 (0x4060 bytes) 000634 1718316559.291465 MIRROR_EXTEND to 5 mirrors 000635 1718316559.406684 read 0x000bb1 thru 0x00b2e7 (0xa737 bytes) 000636 1718316559.408936 write 0x008172 thru 0x00d68c (0x551b bytes) 000637 1718316559.426683 mapread 0x002d28 thru 0x012480 (0xf759 bytes) 000638 1718316559.430869 read 0x0154d2 thru 0x019fa5 (0x4ad4 bytes) 000641 1718316559.431513 mapwrite 0x022136 thru 0x02f91d (0xd7e8 bytes) 000642 1718316559.454358 write 0x01d9ca thru 0x02d6e3 (0xfd1a bytes) 000645 1718316559.455570 trunc from 0x02f91e to 0x01464b (0x1b2d2 bytes) 000646 1718316559.458972 mapread 0x004fbc thru 0x00f6f2 (0xa737 bytes) 000647 1718316559.459569 write 0x019e52 thru 0x0212ca (0x7479 bytes) 000652 1718316559.460925 trunc from 0x0212cb to 0x02d7ea (0xc520 bytes) 000653 1718316559.464467 write 0x00d7df thru 0x017f6b (0xa78d bytes) 000654 1718316559.466365 write 0x01de16 thru 0x02cf46 (0xf131 bytes) 000656 1718316559.468781 trunc from 0x02d7eb to 0x028ba0 (0x4c4a bytes) 000658 1718316559.472370 mapread 0x00b753 thru 0x0158a8 (0xa156 bytes) 000659 1718316559.472817 trunc from 0x028ba1 to 0x036019 (0xd479 bytes) 000660 1718316559.476021 write 0x005f9d thru 0x01027f (0xa2e3 bytes) 000661 1718316559.476707 mapread 0x005889 thru 0x010922 (0xb09a bytes) 000662 1718316559.478074 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x614:0x0]: Invalid argument (22) 000663 1718316559.518692 MIRROR_EXTEND to 5 mirrors 000664 1718316559.612464 mapwrite 0x013b79 thru 0x01a7a8 (0x6c30 bytes) 000665 1718316559.655422 mapread 0x02920c thru 0x036019 (0xce0e bytes) 000666 1718316559.660430 MIRROR_RESYNC 5 mirrors 000667 1718316559.786039 write 0x03292f thru 0x035d18 (0x33ea bytes) 000668 1718316559.798988 mapwrite 0x03d456 thru 0x03ffff (0x2baa bytes) 000669 1718316559.819973 trunc from 0x040000 to 0x00d040 (0x32fbf bytes) 000670 1718316559.822982 read 0x007484 thru 0x00a4ba (0x3037 bytes) 000671 1718316559.824316 trunc from 0x00d041 to 0x01df17 (0x10ed7 bytes) 000672 1718316559.827111 trunc from 0x01df18 to 0x0310bf (0x131a8 bytes) 000673 1718316559.829839 read 0x0040dd thru 0x0082ea (0x420e bytes) 000674 1718316559.832222 mapread 0x02c092 thru 0x0310bf (0x502e bytes) 000675 1718316559.832654 trunc from 0x0310c0 to 0x0087ad (0x28912 bytes) 000677 1718316559.835916 trunc from 0x0087ae to 0x010cc8 (0x851b bytes) 000680 1718316559.838986 mapwrite 0x02d216 thru 0x03391d (0x6708 bytes) 000681 1718316559.860221 mapwrite 0x03b23b thru 0x03f937 (0x46fd bytes) 000682 1718316559.880022 mapread 0x034107 thru 0x03c9dd (0x88d7 bytes) 000684 1718316559.880635 read 0x00c91d thru 0x00dea9 (0x158d bytes) 000685 1718316559.884386 MIRROR_RESYNC 5 mirrors 000686 1718316560.025951 write 0x0354ae thru 0x03ffff (0xab52 bytes) 000687 1718316560.040075 read 0x00944c thru 0x010a29 (0x75de bytes) 000688 1718316560.042543 mapwrite 0x01895a thru 0x0196c8 (0xd6f bytes) 000689 1718316560.058232 trunc from 0x040000 to 0x01ff65 (0x2009a bytes) 000691 1718316560.066948 mapwrite 0x02c5a8 thru 0x03174e (0x51a7 bytes) 000692 1718316560.084247 read 0x0216a8 thru 0x02dd89 (0xc6e2 bytes) 000695 1718316560.087316 mapread 0x02d8a9 thru 0x03174e (0x3ea6 bytes) 000697 1718316560.088078 read 0x00963c thru 0x012258 (0x8c1d bytes) 000699 1718316560.088553 mapwrite 0x00aaf7 thru 0x01a90c (0xfe16 bytes) 000701 1718316560.104315 MIRROR_RESYNC 5 mirrors 000705 1718316560.250100 mapwrite 0x02bd2e thru 0x037006 (0xb2d9 bytes) 000706 1718316560.287596 write 0x024bb8 thru 0x02c9e9 (0x7e32 bytes) 000707 1718316560.290244 mapread 0x02df8d thru 0x037006 (0x907a bytes) 000712 1718316560.291510 mapwrite 0x00daad thru 0x010d40 (0x3294 bytes) 000714 1718316560.306587 MIRROR_RESYNC 5 mirrors 000716 1718316560.439417 MIRROR_EXTEND to 6 mirrors 000717 1718316560.535410 MIRROR_RESYNC 6 mirrors 000719 1718316560.576088 trunc from 0x037007 to 0x00f5d2 (0x27a34 bytes) 000720 1718316560.594226 trunc from 0x00f5d3 to 0x02b8a6 (0x1c2d4 bytes) 000721 1718316560.598536 mapwrite 0x027f33 thru 0x0379b1 (0xfa7f bytes) 000722 1718316560.623575 mapwrite 0x01cf3e thru 0x02cc7e (0xfd41 bytes) 000724 1718316560.640068 read 0x031095 thru 0x03138c (0x2f8 bytes) 000725 1718316560.641629 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x614:0x0]: Invalid argument (22) 000726 1718316560.677674 read 0x001a9e thru 0x004f5a (0x34bd bytes) 000727 1718316560.679313 write 0x003c14 thru 0x012cdb (0xf0c8 bytes) 000729 1718316560.682139 MIRROR_RESYNC 5 mirrors 000730 1718316560.816722 mapread 0x00e599 thru 0x0148c4 (0x632c bytes) 000732 1718316560.821140 write 0x00099a thru 0x00250e (0x1b75 bytes) 000734 1718316560.835245 write 0x00aa73 thru 0x00d6d7 (0x2c65 bytes) 000735 1718316560.835882 read 0x00b943 thru 0x010000 (0x46be bytes) 000738 1718316560.837774 mapread 0x007a0c thru 0x011365 (0x995a bytes) 000740 1718316560.841530 MIRROR_RESYNC 5 mirrors 000743 1718316560.979019 mapwrite 0x03fdff thru 0x03ffff (0x201 bytes) 000744 1718316561.009616 read 0x00c7bb thru 0x013595 (0x6ddb bytes) 000745 1718316561.011789 write 0x0279e2 thru 0x02ec1c (0x723b bytes) 000749 1718316561.015082 mapwrite 0x00ade2 thru 0x015828 (0xaa47 bytes) 000751 1718316561.032828 mapwrite 0x039a7c thru 0x03e42f (0x49b4 bytes) 000752 1718316561.046285 trunc from 0x040000 to 0x00f051 (0x30fae bytes) 000753 1718316561.050612 mapwrite 0x016f24 thru 0x0170a7 (0x184 bytes) 000756 1718316561.070555 read 0x0012dd thru 0x00555b (0x427f bytes) 000757 1718316561.072173 mapread 0x0134af thru 0x0170a7 (0x3bf9 bytes) 000758 1718316561.072736 write 0x020f37 thru 0x029667 (0x8731 bytes) 000759 1718316561.073351 read 0x01e435 thru 0x029667 (0xb233 bytes) 000760 1718316561.075401 read 0x025f69 thru 0x029667 (0x36ff bytes) 000761 1718316561.075764 mapwrite 0x02f6e9 thru 0x0398cb (0xa1e3 bytes) 000762 1718316561.097038 MIRROR_RESYNC 5 mirrors 000763 1718316561.235607 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x614:0x0]: Invalid argument (22) 000765 1718316561.284711 mapwrite 0x018289 thru 0x01fdf4 (0x7b6c bytes) 000768 1718316561.327433 trunc from 0x0398cc to 0x03d78e (0x3ec3 bytes) 000769 1718316561.334744 write 0x029221 thru 0x0323c9 (0x91a9 bytes) 000770 1718316561.337858 mapread 0x02976d thru 0x038133 (0xe9c7 bytes) 000771 1718316561.340674 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x614:0x0]: Invalid argument (22) 000772 1718316561.379915 trunc from 0x03d78f to 0x038af2 (0x4c9c bytes) 000773 1718316561.382975 read 0x02ed09 thru 0x037c84 (0x8f7c bytes) 000774 1718316561.384887 mapwrite 0x01432a thru 0x023904 (0xf5db bytes) 000776 1718316561.403015 read 0x0145a7 thru 0x023abf (0xf519 bytes) 000777 1718316561.403349 write 0x00d47e thru 0x01b9dd (0xe560 bytes) 000780 1718316561.406221 mapwrite 0x00e35c thru 0x017bcb (0x9870 bytes) 000782 1718316561.417707 trunc from 0x038af3 to 0x01da4b (0x1b0a7 bytes) 000784 1718316561.421146 write 0x002ca4 thru 0x00b5f7 (0x8954 bytes) 000785 1718316561.421914 read 0x016dcf thru 0x01da4b (0x6c7d bytes) 000787 1718316561.422306 write 0x03fa87 thru 0x03ffff (0x579 bytes) 000788 1718316561.422736 write 0x030394 thru 0x0389da (0x8647 bytes) 000789 1718316561.424961 read 0x02eba0 thru 0x03102a (0x248b bytes) 000791 1718316561.425404 trunc from 0x040000 to 0x034c2a (0xb3d5 bytes) 000792 1718316561.429362 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x614:0x0]: Invalid argument (22) 000794 1718316561.469361 MIRROR_RESYNC 2 mirrors 000797 1718316561.555346 trunc from 0x034c2b to 0x029ab0 (0xb17a bytes) 000798 1718316561.578717 write 0x037c4e thru 0x039cc3 (0x2076 bytes) 000799 1718316561.580978 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x614:0x0]: Invalid argument (22) 000800 1718316561.611518 write 0x007681 thru 0x013938 (0xc2b8 bytes) 000801 1718316561.614316 write 0x023390 thru 0x02d642 (0xa2b3 bytes) 000804 1718316561.618273 write 0x035a59 thru 0x037a8d (0x2035 bytes) 000805 1718316561.620337 read 0x00fbaf thru 0x01855c (0x89ae bytes) 000806 1718316561.622501 read 0x036e30 thru 0x039cc3 (0x2e94 bytes) 000808 1718316561.623052 mapwrite 0x02300a thru 0x02cafa (0x9af1 bytes) 000812 1718316561.639123 mapwrite 0x0337a4 thru 0x03e10e (0xa96b bytes) 000813 1718316561.658748 read 0x0239f5 thru 0x02aeba (0x74c6 bytes) 000814 1718316561.659112 mapwrite 0x01133d thru 0x01e240 (0xcf04 bytes) 000816 1718316561.672717 trunc from 0x03e10f to 0x03af3f (0x31cf bytes) 000817 1718316561.675908 mapread 0x02610b thru 0x0271fc (0x10f2 bytes) 000818 1718316561.676310 mapread 0x00ff6c thru 0x0111c4 (0x1259 bytes) 000819 1718316561.676672 mapwrite 0x00ef0b thru 0x00efeb (0xe1 bytes) 000820 1718316561.689735 read 0x008c95 thru 0x0112cc (0x8638 bytes) 000822 1718316561.690460 mapread 0x02ab96 thru 0x038eda (0xe345 bytes) truncating to largest ever: 0x3f5a8 000825 1718316561.691981 trunc from 0x03af40 to 0x03f5a7 (0x4668 bytes) 000826 1718316561.696043 mapread 0x00fcd8 thru 0x01db9e (0xdec7 bytes) 000829 1718316561.697268 mapread 0x01c217 thru 0x01cebe (0xca8 bytes) 000830 1718316561.697720 mapwrite 0x02ba16 thru 0x02f6a8 (0x3c93 bytes) 000831 1718316561.712099 write 0x02127e thru 0x029cc4 (0x8a47 bytes) 000832 1718316561.712774 mapwrite 0x02b015 thru 0x02d4f2 (0x24de bytes) 000833 1718316561.726096 read 0x0176c4 thru 0x021d1a (0xa657 bytes) 000834 1718316561.727904 MIRROR_EXTEND to 2 mirrors 000836 1718316561.818420 trunc from 0x03f5a8 to 0x03db33 (0x1a74 bytes) 000838 1718316561.836905 read 0x02653e thru 0x02a4be (0x3f81 bytes) 000840 1718316561.839252 read 0x0376f2 thru 0x03db33 (0x6442 bytes) 000841 1718316561.843377 MIRROR_EXTEND to 3 mirrors 000844 1718316561.941915 write 0x00fd27 thru 0x012bfb (0x2ed5 bytes) 000845 1718316561.956043 write 0x0016b3 thru 0x010f66 (0xf8b4 bytes) 000847 1718316561.958412 mapwrite 0x00f459 thru 0x019d29 (0xa8d1 bytes) 000848 1718316561.973966 write 0x03890e thru 0x03d93e (0x5031 bytes) 000849 1718316561.976102 mapwrite 0x029fd2 thru 0x032562 (0x8591 bytes) 000851 1718316561.990532 write 0x03e1ff thru 0x03ffff (0x1e01 bytes) 000852 1718316561.991077 read 0x03f65e thru 0x03ffff (0x9a2 bytes) 000853 1718316561.991298 trunc from 0x040000 to 0x01b0ab (0x24f54 bytes) 000854 1718316562.000291 MIRROR_RESYNC 3 mirrors 000857 1718316562.101579 mapwrite 0x03b61b thru 0x03ffff (0x49e5 bytes) 000858 1718316562.129671 mapread 0x01454b thru 0x01c829 (0x82df bytes) 000859 1718316562.133528 mapread 0x035a39 thru 0x037d75 (0x233d bytes) 000862 1718316562.135923 write 0x003f9f thru 0x006e5f (0x2ec1 bytes) 000863 1718316562.137769 read 0x03ec42 thru 0x03ffff (0x13be bytes) 000864 1718316562.137997 mapread 0x037c1f thru 0x03819d (0x57f bytes) 000865 1718316562.138366 trunc from 0x040000 to 0x025833 (0x1a7cc bytes) 000867 1718316562.142061 mapwrite 0x01dee6 thru 0x028b2e (0xac49 bytes) 000871 1718316562.159567 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x614:0x0]: Invalid argument (22) 000872 1718316562.196850 write 0x00906d thru 0x00a7e9 (0x177d bytes) 000873 1718316562.199124 write 0x017040 thru 0x018fc5 (0x1f86 bytes) 000874 1718316562.200967 mapread 0x00484b thru 0x00f03d (0xa7f3 bytes) 000875 1718316562.204207 mapwrite 0x026207 thru 0x02e2b1 (0x80ab bytes) 000876 1718316562.222082 read 0x00f0e1 thru 0x01b815 (0xc735 bytes) 000879 1718316562.222995 mapread 0x008ccb thru 0x00c4d0 (0x3806 bytes) 000880 1718316562.223369 mapread 0x024766 thru 0x02e2b1 (0x9b4c bytes) 000884 1718316562.224869 mapread 0x002405 thru 0x0087e3 (0x63df bytes) 000885 1718316562.225378 mapwrite 0x00e82f thru 0x01189f (0x3071 bytes) 000886 1718316562.237287 read 0x003a15 thru 0x0102b9 (0xc8a5 bytes) 000887 1718316562.237646 write 0x00d7c6 thru 0x011d6b (0x45a6 bytes) 000889 1718316562.238416 mapwrite 0x00ec7f thru 0x0164eb (0x786d bytes) 000890 1718316562.254665 MIRROR_EXTEND to 3 mirrors 000895 1718316562.355273 MIRROR_EXTEND to 4 mirrors 000896 1718316562.445431 trunc from 0x02e2b2 to 0x01ccfe (0x115b3 bytes) 000897 1718316562.459122 write 0x02e926 thru 0x03034d (0x1a28 bytes) 000898 1718316562.459639 read 0x020555 thru 0x023848 (0x32f4 bytes) 000899 1718316562.462043 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x614:0x0]: Invalid argument (22) 000900 1718316562.501553 trunc from 0x03034e to 0x02231b (0xe032 bytes) 000902 1718316562.504812 read 0x00d44d thru 0x00e163 (0xd17 bytes) 000903 1718316562.506052 read 0x015978 thru 0x0198a2 (0x3f2b bytes) 000904 1718316562.509078 MIRROR_EXTEND to 4 mirrors 000905 1718316562.589578 mapwrite 0x03bdab thru 0x03ffff (0x4255 bytes) 000906 1718316562.623112 mapwrite 0x033957 thru 0x03c7c9 (0x8e73 bytes) 000908 1718316562.636451 trunc from 0x040000 to 0x03a305 (0x5cfa bytes) 000909 1718316562.640345 read 0x038ce6 thru 0x03a305 (0x1620 bytes) 000910 1718316562.640634 mapread 0x033465 thru 0x03a305 (0x6ea1 bytes) 000913 1718316562.641503 mapwrite 0x02b6fd thru 0x030e55 (0x5759 bytes) 000916 1718316562.656115 write 0x025602 thru 0x027601 (0x2000 bytes) 000917 1718316562.656601 mapwrite 0x017539 thru 0x01b63f (0x4107 bytes) 000919 1718316562.670755 mapwrite 0x03cfec thru 0x03ffff (0x3014 bytes) 000920 1718316562.685177 write 0x01613b thru 0x01a8a0 (0x4766 bytes) 000925 1718316562.686522 write 0x0391a9 thru 0x03ffff (0x6e57 bytes) 000927 1718316562.687200 mapwrite 0x025844 thru 0x03188d (0xc04a bytes) 000929 1718316562.700296 read 0x032be8 thru 0x03dac7 (0xaee0 bytes) 000930 1718316562.700601 read 0x02baa4 thru 0x03669c (0xabf9 bytes) 000931 1718316562.700840 mapread 0x019425 thru 0x01fc59 (0x6835 bytes) 000932 1718316562.701166 trunc from 0x040000 to 0x00e216 (0x31de9 bytes) 000934 1718316562.704838 mapwrite 0x005f6b thru 0x00d60a (0x76a0 bytes) 000936 1718316562.719905 mapread 0x006adc thru 0x00e216 (0x773b bytes) 000939 1718316562.720727 write 0x02434a thru 0x02d1a8 (0x8e5f bytes) 000941 1718316562.721717 mapwrite 0x033ed8 thru 0x03d2dc (0x9405 bytes) 000945 1718316562.741564 trunc from 0x03d2dd to 0x02a36c (0x12f70 bytes) 000946 1718316562.744957 trunc from 0x02a36d to 0x030a3d (0x66d1 bytes) 000948 1718316562.748331 read 0x028383 thru 0x02f2bb (0x6f39 bytes) 000950 1718316562.751926 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x614:0x0]: Invalid argument (22) 000953 1718316562.791420 mapwrite 0x016455 thru 0x024513 (0xe0bf bytes) 000955 1718316562.810111 read 0x00941f thru 0x013e80 (0xaa62 bytes) 000956 1718316562.812353 trunc from 0x030a3e to 0x01c1cf (0x1486e bytes) 000959 1718316562.816250 write 0x03a1d7 thru 0x03ffff (0x5e29 bytes) 000965 1718316562.819180 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x614:0x0]: Invalid argument (22) 000966 1718316562.859075 write 0x004bda thru 0x013650 (0xea77 bytes) 000967 1718316562.863270 MIRROR_RESYNC 2 mirrors 000968 1718316562.941875 trunc from 0x040000 to 0x023045 (0x1cfba bytes) 000970 1718316562.958393 MIRROR_EXTEND to 3 mirrors 000971 1718316563.049565 mapread 0x018e05 thru 0x019a46 (0xc42 bytes) 000974 1718316563.052325 mapread 0x014516 thru 0x01d074 (0x8b5f bytes) 000975 1718316563.056249 write 0x033517 thru 0x03390b (0x3f5 bytes) 000976 1718316563.069030 mapread 0x02f472 thru 0x03390b (0x449a bytes) 000978 1718316563.071517 mapread 0x0098ec thru 0x016fa9 (0xd6be bytes) 000982 1718316563.076133 mapread 0x001cf2 thru 0x00c595 (0xa8a4 bytes) 000986 1718316563.079212 read 0x0119b8 thru 0x0121a1 (0x7ea bytes) 000989 1718316563.081548 MIRROR_EXTEND to 4 mirrors 000991 1718316563.169711 write 0x010961 thru 0x01b6ad (0xad4d bytes) 000992 1718316563.184392 mapwrite 0x01cc70 thru 0x01d87c (0xc0d bytes) 000994 1718316563.198393 write 0x024e6b thru 0x034d34 (0xfeca bytes) 000995 1718316563.199172 trunc from 0x034d35 to 0x0136cb (0x21669 bytes) 000998 1718316563.206730 trunc from 0x0136cc to 0x02fe87 (0x1c7bc bytes) All operations completed A-OK! PASS 70a (11s) == sanity-flr test 200a: stress test ===================== 18:09:24 (1718316564) Starting client: oleg342-client.virtnet: -o user_xattr,flock oleg342-server@tcp:/lustre /mnt/lustre2 Starting client: oleg342-client.virtnet: -o user_xattr,flock oleg342-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6350752 .. Extending file size to 9235200 .. done Extending file size to 7448896 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7949920 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8674208 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8764896 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9085600 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9288480 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 32402 32403 32405 32406 32409 192.168.203.142@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg342-client.virtnet /mnt/lustre2 (opts:) 192.168.203.142@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg342-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (63s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 18:10:27 (1718316627) Starting client: oleg342-client.virtnet: -o user_xattr,flock oleg342-server@tcp:/lustre /mnt/lustre2 Starting client: oleg342-client.virtnet: -o user_xattr,flock oleg342-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7256576 .. Extending file size to 1760000 .. done Extending file size to 8088992 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7698336 .. Extending file size to 8695200 .. Extending file size to 8605984 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9281600 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8835776 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8955520 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9056832 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9102496 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9235392 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 3401 3402 3404 3406 3408 192.168.203.142@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg342-client.virtnet /mnt/lustre2 (opts:) 192.168.203.142@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg342-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 18:11:30 (1718316690) Starting client: oleg342-client.virtnet: -o user_xattr,flock oleg342-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4864 2203648 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4864 2203648 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0322927 s, 64.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0698373 s, 30.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.142@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg342-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 18:11:35 (1718316695) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0269972 s, 77.7 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 18:11:36 (1718316696) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x651:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc13:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc13:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc14:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0xc16:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 18:11:38 (1718316698) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0320886 s, 65.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc15:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc17:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc18:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (1s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 18:11:39 (1718316699) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0292243 s, 71.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc16:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc17:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc19:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc16:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc17:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc19:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc1a:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 18:11:41 (1718316701) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0253389 s, 82.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc18:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc1b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc18:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc1b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc1c:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 18:12:04 (1718316724) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg342-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0175217 s, 59.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc19:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc1e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc1d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc19:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc1e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc1d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc1a:0x0] } PASS 204d (42s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 18:12:47 (1718316767) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg342-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc1b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc1c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.025292 s, 82.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc1b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc1c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc1f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc1b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc1c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc1f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (40s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 18:13:27 (1718316807) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg342-server mds-ost sync done. Creating new pool oleg342-server: Pool lustre.test_204f created Adding targets to pool oleg342-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg342-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0265397 s, 79.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc1d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc20:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc1d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc20:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg342-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg342-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg342-server: Pool lustre.test_204f destroyed PASS 204f (53s) == sanity-flr test 205: lfs mirror extend to set prefer and nocompr flags ========================================================== 18:14:20 (1718316860) PASS 205 (1s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 18:14:21 (1718316861) oleg342-server: Pool lustre.test_206 created Adding targets to pool oleg342-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg342-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg342-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg342-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg342-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 18:14:34 (1718316874) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0164313 s, 63.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00751988 s, 139 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xdb01b6ad Mirror 3: 0xdb01b6ad PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 18:14:36 (1718316876) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (0s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 18:14:36 (1718316876) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 18:14:37 (1718316877) Starting client: oleg342-client.virtnet: -o user_xattr,flock oleg342-server@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0135311 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x13b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x24b0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x24b0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x24b0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2390000 Read this (4 bytes): BBBB 192.168.203.142@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg342-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 18:14:50 (1718316890) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00263051 s, 3.1 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 18:14:52 (1718316892) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00809037 s, 130 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x675 lmm_fid: [0x200000404:0x675:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x675 lmm_fid: [0x200000404:0x675:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc27:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835573 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-06-13 18:14:52.000000000 -0400 Modify: 2024-06-13 18:14:52.000000000 -0400 Change: 2024-06-13 18:14:52.000000000 -0400 Birth: - PASS 210a (1s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 18:14:53 (1718316893) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test complete, duration 641 sec ============ 18:14:54 (1718316894) === sanity-flr: start cleanup 18:14:54 (1718316894) === === sanity-flr: finish cleanup 18:14:55 (1718316895) ===