-----============= acceptance-small: sanity-flr ============----- Sun Aug 4 14:09:40 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 6 201 44c 49a === sanity-flr: start setup 14:09:46 (1722794986) === oleg460-client.virtnet: executing check_config_client /mnt/lustre oleg460-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg460-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b0b62800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b0b62800.idle_timeout=debug disable quota as required oleg460-server: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 14:09:54 (1722794994) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f6921] == sanity-flr test 0a: lfs mirror create with -N option == 14:09:54 (1722794994) PASS 0a (5s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 14:09:59 (1722794999) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 14:10:00 (1722795000) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 14:10:01 (1722795001) PASS 0d (7s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 14:10:08 (1722795008) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 14:10:09 (1722795009) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 14:10:10 (1722795010) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b0b62800.stats=clear osc.lustre-OST0001-osc-ffff8800b0b62800.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 14:10:12 (1722795012) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 14:10:15 (1722795015) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 14:10:18 (1722795018) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 14:10:19 (1722795019) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x22:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 14:10:21 (1722795021) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 14:10:23 (1722795023) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 14:10:24 (1722795024) lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 14:10:26 (1722795026) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 14:10:29 (1722795029) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 14:10:31 (1722795031) 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.621088 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0355707 s, 29.5 MB/s PASS 21 (4s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 14:10:35 (1722795035) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.0473926 s, 111 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 14:10:37 (1722795037) Stopping /mnt/lustre-ost1 (opts:) on oleg460-server oleg460-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b0b62800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b0b62800.ost_server_uuid in DISCONN state after 0 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg460-server: oleg460-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg460-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg460-server oleg460-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b0b62800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b0b62800.ost_server_uuid in DISCONN state after 2 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg460-server: oleg460-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg460-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (24s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 14:11:01 (1722795061) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB) copied, 0.0912816 s, 68.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg460-server oleg460-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b0b62800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b0b62800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg460-server: oleg460-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg460-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (10s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 14:11:11 (1722795071) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg460-server oleg460-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b0b62800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b0b62800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg460-server: oleg460-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg460-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg460-server oleg460-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b0b62800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b0b62800.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg460-server: oleg460-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg460-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (21s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 14:11:32 (1722795092) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.720291 s, 72.8 MB/s -rw-r--r-- 1 root root 52428800 Aug 4 14:11 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.297s user 0m0.001s sys 0m0.149s Stopping /mnt/lustre-ost1 (opts:) on oleg460-server oleg460-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b0b62800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b0b62800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... real 0m0.231s user 0m0.000s sys 0m0.122s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg460-server: oleg460-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg460-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg460-server oleg460-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b0b62800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b0b62800.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost1)... real 0m0.206s user 0m0.001s sys 0m0.117s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg460-server: oleg460-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg460-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg460-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b0b62800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b0b62800.ost_server_uuid in FULL state after 0 sec PASS 33b (26s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 14:11:58 (1722795118) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 14:11:59 (1722795119) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 14:12:00 (1722795120) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 14:12:01 (1722795121) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 14:12:03 (1722795123) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.16455 s, 76.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 14:12:06 (1722795126) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0143938 s, 72.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0125699 s, 83.4 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 14:12:08 (1722795128) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 14:12:11 (1722795131) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000887104 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000744947 s, 1.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00078043 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00040905 s, 2.5 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 14:12:14 (1722795134) create test files with size 15731527 .. f77e8c84df144d1dfafa23b5289d554e - e9ed88a9b5235eef9199fff1e34d4ffa - 1084729ef88818995a7773cedbe0519c - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg460-client.virtnet /mnt/lustre (opts:) Starting client: oleg460-client.virtnet: -o user_xattr,flock oleg460-server@tcp:/lustre /mnt/lustre PASS 37 (6s) == sanity-flr test 38: resync ============================ 14:12:20 (1722795140) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (4s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 14:12:24 (1722795144) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 14:12:26 (1722795146) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0199754 s, 52.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00780872 s, 134 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 14:12:28 (1722795148) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0324885 s, 64.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0880572 s, 47.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0327055 s, 64.1 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 14:12:31 (1722795151) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 14:12:32 (1722795152) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 14:12:33 (1722795153) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0132482 s, 79.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 14:12:35 (1722795155) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 14:12:36 (1722795156) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x50:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x13:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 14:12:38 (1722795158) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0696747 s, 150 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg460-client.virtnet /mnt/lustre (opts:) Starting client: oleg460-client.virtnet: -o user_xattr,flock oleg460-server@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (4s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 14:12:42 (1722795162) llite.lustre-ffff8800b7c1b800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0999381 s, 105 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg460-client.virtnet /mnt/lustre (opts:) Starting client: oleg460-client.virtnet: -o user_xattr,flock oleg460-server@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800a7f2b000.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 14:12:45 (1722795165) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (9s) == sanity-flr test 46: Verify setstripe --copy option ==== 14:12:54 (1722795174) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (9s) == sanity-flr test 47: Verify mirror obj alloc =========== 14:13:03 (1722795183) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 14:13:04 (1722795184) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0522089 s, 60.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0429553 s, 73.2 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: faf7e340c0dce816e182c2b6f28b8407 - mirror 1 checksum: dd2c4cde830f9dd829663ba64165ff7d - mirror 2 checksum: faf7e340c0dce816e182c2b6f28b8407 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Aug 4 14:13:05 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Aug 4 14:13:05 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: faf7e340c0dce816e182c2b6f28b8407 - mirror 1 checksum: dd2c4cde830f9dd829663ba64165ff7d - mirror 2 checksum: dd2c4cde830f9dd829663ba64165ff7d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Aug 4 14:13:05 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Aug 4 14:13:05 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } PASS 48 (3s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 14:13:08 (1722795188) Failing mds1 on oleg460-server Stopping /mnt/lustre-mds1 (opts:) on oleg460-server 14:13:10 (1722795190) shut down Failover mds1 to oleg460-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg460-server: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-MDT0000 14:13:23 (1722795203) targets are mounted 14:13:23 (1722795203) facet_failover done oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00652168 s, 161 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 50A (20s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 14:13:28 (1722795208) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 14:13:29 (1722795209) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 14:13:30 (1722795210) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00379804 s, 4.3 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 14:13:33 (1722795213) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 14:13:34 (1722795214) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 14:13:35 (1722795215) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (5s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 14:13:41 (1722795221) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 14:13:59 (1722795239) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 14:14:16 (1722795256) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 14:14:34 (1722795274) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 14:14:36 (1722795276) mirror create pid 21469 mirror split pid 21470 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4759: 21469 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4759: 21470 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 14:15:38 (1722795338) Seed set to 5339 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x7590 000001 1722795339.104751 trunc from 00000000 to 0x00758f (0x7590 bytes) 000005 1722795339.111222 mapwrite 0x0269f8 thru 0x0283e5 (0x19ee bytes) 000008 1722795339.163445 mapwrite 0x0200f7 thru 0x02174a (0x1654 bytes) truncating to largest ever: 0x38910 000009 1722795339.187358 trunc from 0x0283e6 to 0x03890f (0x1052a bytes) 000012 1722795339.195081 trunc from 0x038910 to 0x01635c (0x225b3 bytes) 000013 1722795339.201882 trunc from 0x01635d to 0x01b6fb (0x539f bytes) 000015 1722795339.211868 MIRROR_EXTEND to 2 mirrors 000016 1722795339.396402 mapread 0x00f9c8 thru 0x01a517 (0xab50 bytes) 000017 1722795339.410811 mapread 0x004ec8 thru 0x00cba3 (0x7cdc bytes) 000018 1722795339.415972 trunc from 0x01b6fc to 0x016162 (0x5599 bytes) 000022 1722795339.460231 write 0x02dea3 thru 0x02ff91 (0x20ef bytes) 000023 1722795339.461354 mapread 0x006b32 thru 0x011d24 (0xb1f3 bytes) 000024 1722795339.471126 write 0x025631 thru 0x0320b9 (0xca89 bytes) 000025 1722795339.476473 mapwrite 0x0039cb thru 0x004782 (0xdb8 bytes) 000027 1722795339.507228 mapwrite 0x0325f3 thru 0x034b36 (0x2544 bytes) 000028 1722795339.545233 read 0x01c307 thru 0x01ef11 (0x2c0b bytes) truncating to largest ever: 0x3fb33 000029 1722795339.548648 trunc from 0x034b37 to 0x03fb32 (0xaffc bytes) 000030 1722795339.556033 write 0x00d75c thru 0x0186df (0xaf84 bytes) 000034 1722795339.561005 MIRROR_RESYNC 2 mirrors 000035 1722795339.752509 MIRROR_EXTEND to 3 mirrors 000037 1722795339.959420 read 0x0222b1 thru 0x02a3a1 (0x80f1 bytes) 000038 1722795339.964213 write 0x032546 thru 0x03a163 (0x7c1e bytes) 000039 1722795339.996288 mapwrite 0x038c89 thru 0x03ffff (0x7377 bytes) 000040 1722795340.051972 mapwrite 0x002759 thru 0x008a48 (0x62f0 bytes) 000041 1722795340.086977 mapread 0x015952 thru 0x01c692 (0x6d41 bytes) 000043 1722795340.093271 mapwrite 0x01068f thru 0x01b548 (0xaeba bytes) 000044 1722795340.126255 read 0x023a27 thru 0x02d52d (0x9b07 bytes) 000045 1722795340.133073 MIRROR_EXTEND to 4 mirrors 000047 1722795340.341396 read 0x0202ae thru 0x02e242 (0xdf95 bytes) 000048 1722795340.347525 mapread 0x00194c thru 0x0098cc (0x7f81 bytes) 000050 1722795340.361968 mapwrite 0x032ff3 thru 0x0378ca (0x48d8 bytes) 000051 1722795340.447925 mapwrite 0x01edda thru 0x02d4bf (0xe6e6 bytes) 000053 1722795340.490101 trunc from 0x040000 to 0x031809 (0xe7f6 bytes) 000058 1722795340.508706 read 0x02bc6d thru 0x03179b (0x5b2f bytes) 000060 1722795340.509758 trunc from 0x03180a to 0x034388 (0x2b7f bytes) 000061 1722795340.517003 trunc from 0x034389 to 0x0052e8 (0x2f0a0 bytes) 000062 1722795340.524727 trunc from 0x0052e9 to 0x00aadc (0x57f4 bytes) 000063 1722795340.531722 trunc from 0x00aadd to 0x02c915 (0x21e39 bytes) 000064 1722795340.539293 write 0x0157d7 thru 0x020fe7 (0xb811 bytes) 000065 1722795340.547377 read 0x018e7f thru 0x026236 (0xd3b8 bytes) 000066 1722795340.547979 mapread 0x0159d6 thru 0x01e096 (0x86c1 bytes) 000067 1722795340.548794 mapread 0x01b504 thru 0x028a3d (0xd53a bytes) 000068 1722795340.549551 write 0x01e8b6 thru 0x02c609 (0xdd54 bytes) 000071 1722795340.553985 MIRROR_RESYNC 4 mirrors 000073 1722795340.818380 write 0x03234d thru 0x03eb1f (0xc7d3 bytes) 000075 1722795340.841236 write 0x01b71c thru 0x0270f1 (0xb9d6 bytes) 000077 1722795340.849214 MIRROR_RESYNC 4 mirrors 000078 1722795341.144732 MIRROR_RESYNC 4 mirrors 000079 1722795341.219837 mapread 0x03ab32 thru 0x03eb1f (0x3fee bytes) 000080 1722795341.227329 read 0x02e382 thru 0x036819 (0x8498 bytes) 000081 1722795341.230184 trunc from 0x03eb20 to 0x00ee68 (0x2fcb7 bytes) 000082 1722795341.251076 read 0x002c23 thru 0x00bd05 (0x90e3 bytes) 000083 1722795341.256127 MIRROR_RESYNC 4 mirrors 000084 1722795341.375428 write 0x027688 thru 0x032daa (0xb723 bytes) 000085 1722795341.388537 mapread 0x031c4d thru 0x032daa (0x115e bytes) 000087 1722795341.390409 MIRROR_RESYNC 4 mirrors 000089 1722795341.522996 read 0x02ad30 thru 0x032daa (0x807b bytes) 000090 1722795341.525258 mapwrite 0x0014db thru 0x00d1b7 (0xbcdd bytes) 000091 1722795341.580167 write 0x0081a4 thru 0x010e17 (0x8c74 bytes) 000092 1722795341.580746 mapread 0x00cfcf thru 0x016916 (0x9948 bytes) 000093 1722795341.584295 mapwrite 0x01f246 thru 0x022d9a (0x3b55 bytes) 000096 1722795341.602480 write 0x007df8 thru 0x00b8b0 (0x3ab9 bytes) 000099 1722795341.603329 mapwrite 0x02bb81 thru 0x039127 (0xd5a7 bytes) 000101 1722795341.633246 mapread 0x0060c8 thru 0x00e360 (0x8299 bytes) 000102 1722795341.633880 mapread 0x021bfb thru 0x026653 (0x4a59 bytes) 000104 1722795341.634600 mapwrite 0x01e116 thru 0x021c7e (0x3b69 bytes) 000106 1722795341.646501 read 0x00b9c7 thru 0x0129a8 (0x6fe2 bytes) 000107 1722795341.646834 read 0x00040e thru 0x00333c (0x2f2f bytes) 000108 1722795341.648346 mapread 0x025838 thru 0x030f5f (0xb728 bytes) 000109 1722795341.648767 read 0x02505a thru 0x02518d (0x134 bytes) 000114 1722795341.649864 mapread 0x01ca44 thru 0x027ec1 (0xb47e bytes) 000116 1722795341.650451 trunc from 0x039128 to 0x02ecec (0xa43b bytes) 000117 1722795341.654568 mapread 0x00cd91 thru 0x01c6a3 (0xf913 bytes) 000118 1722795341.655152 mapread 0x008d51 thru 0x012741 (0x99f1 bytes) 000121 1722795341.656430 read 0x008655 thru 0x013d26 (0xb6d2 bytes) 000123 1722795341.658411 MIRROR_RESYNC 4 mirrors 000127 1722795341.790322 read 0x0258d7 thru 0x02ecec (0x9416 bytes) 000128 1722795341.792333 read 0x02a8a4 thru 0x02bb87 (0x12e4 bytes) 000129 1722795341.792489 trunc from 0x02eced to 0x02a553 (0x4799 bytes) 000133 1722795341.811533 MIRROR_EXTEND to 5 mirrors 000134 1722795341.903226 write 0x019893 thru 0x01c1fe (0x296c bytes) 000135 1722795341.918161 trunc from 0x02a554 to 0x035192 (0xac3f bytes) 000137 1722795341.926350 trunc from 0x035193 to 0x002560 (0x32c32 bytes) 000139 1722795341.930413 mapwrite 0x0335f7 thru 0x03ffff (0xca09 bytes) 000140 1722795341.953845 mapread 0x019b2e thru 0x01a9a8 (0xe7b bytes) 000141 1722795341.956800 write 0x002855 thru 0x0119eb (0xf197 bytes) 000142 1722795341.961114 trunc from 0x040000 to 0x02b69e (0x14961 bytes) 000143 1722795341.966551 mapread 0x01e1d1 thru 0x022117 (0x3f47 bytes) 000144 1722795341.968256 MIRROR_RESYNC 5 mirrors 000146 1722795342.205459 trunc from 0x02b69f to 0x03df8b (0x128ed bytes) 000147 1722795342.228784 mapread 0x035e59 thru 0x03df8b (0x8133 bytes) 000148 1722795342.236832 mapwrite 0x01ce60 thru 0x02ca70 (0xfc11 bytes) 000153 1722795342.275595 trunc from 0x03df8c to 0x02e59b (0xf9f0 bytes) 000154 1722795342.285252 mapread 0x02ce51 thru 0x02e59b (0x174b bytes) 000156 1722795342.291072 MIRROR_EXTEND to 6 mirrors 000158 1722795342.481014 write 0x019811 thru 0x0250f4 (0xb8e4 bytes) 000159 1722795342.518518 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x615:0x0]: Invalid argument (22) 000160 1722795342.610541 write 0x02e534 thru 0x03c591 (0xe05e bytes) 000162 1722795342.614938 trunc from 0x03c592 to 0x004ad2 (0x37abf bytes) 000163 1722795342.630344 write 0x00eba3 thru 0x014c0a (0x6068 bytes) 000164 1722795342.631500 trunc from 0x014c0b to 0x01f7b9 (0xabaf bytes) 000166 1722795342.641722 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x615:0x0]: Invalid argument (22) 000167 1722795342.728788 read 0x0141af thru 0x01f0b8 (0xaf0a bytes) 000168 1722795342.733843 read 0x00f273 thru 0x01cbd1 (0xd95f bytes) 000169 1722795342.738208 read 0x00e11e thru 0x018034 (0x9f17 bytes) 000170 1722795342.738520 write 0x02a7a2 thru 0x031160 (0x69bf bytes) 000172 1722795342.741436 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x615:0x0]: Invalid argument (22) 000174 1722795342.828578 write 0x002f44 thru 0x00b61c (0x86d9 bytes) 000175 1722795342.834413 mapwrite 0x02a0c1 thru 0x035d08 (0xbc48 bytes) 000176 1722795342.881213 mapread 0x0278a2 thru 0x02e6ec (0x6e4b bytes) 000178 1722795342.882506 mapread 0x02da9d thru 0x035d08 (0x826c bytes) 000180 1722795342.883688 read 0x0219ad thru 0x026e2a (0x547e bytes) 000181 1722795342.886907 mapread 0x028fd4 thru 0x035d08 (0xcd35 bytes) 000184 1722795342.889244 trunc from 0x035d09 to 0x0149bd (0x2134b bytes) 000186 1722795342.904846 MIRROR_RESYNC 3 mirrors 000190 1722795343.120058 mapread 0x011d67 thru 0x01428b (0x2525 bytes) 000191 1722795343.127594 read 0x01352f thru 0x0149bd (0x148f bytes) 000192 1722795343.127877 mapread 0x0005dd thru 0x00908a (0x8aae bytes) 000197 1722795343.135642 trunc from 0x0149be to 0x011914 (0x30a9 bytes) 000198 1722795343.163421 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x615:0x0]: Invalid argument (22) 000200 1722795343.227270 read 0x00d8c8 thru 0x00f081 (0x17ba bytes) 000201 1722795343.229771 write 0x00be83 thru 0x018956 (0xcad4 bytes) 000203 1722795343.232400 mapwrite 0x001b17 thru 0x00bf9a (0xa484 bytes) 000205 1722795343.266624 trunc from 0x018957 to 0x027df3 (0xf49d bytes) 000207 1722795343.273555 mapwrite 0x0338a8 thru 0x036fc0 (0x3719 bytes) 000211 1722795343.310148 read 0x036394 thru 0x036fc0 (0xc2d bytes) 000212 1722795343.310615 mapread 0x01f1bf thru 0x022c44 (0x3a86 bytes) 000213 1722795343.315622 write 0x002076 thru 0x0108dd (0xe868 bytes) 000215 1722795343.319064 MIRROR_EXTEND to 3 mirrors 000217 1722795343.540336 read 0x02416f thru 0x0335ff (0xf491 bytes) 000221 1722795343.546384 write 0x005f96 thru 0x015dae (0xfe19 bytes) 000222 1722795343.581121 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x615:0x0]: Invalid argument (22) 000223 1722795343.671429 mapwrite 0x034e48 thru 0x0382b2 (0x346b bytes) 000224 1722795343.727131 mapread 0x01f2df thru 0x026327 (0x7049 bytes) 000225 1722795343.733147 mapwrite 0x02450d thru 0x028bf6 (0x46ea bytes) 000228 1722795343.759704 trunc from 0x0382b3 to 0x03ecfa (0x6a48 bytes) 000229 1722795343.767623 mapwrite 0x03b869 thru 0x03c1f0 (0x988 bytes) 000231 1722795343.796561 read 0x02d8ab thru 0x03653c (0x8c92 bytes) 000234 1722795343.798423 mapread 0x034a94 thru 0x037a96 (0x3003 bytes) 000237 1722795343.800978 write 0x0092ac thru 0x012bcb (0x9920 bytes) 000238 1722795343.809167 read 0x032e20 thru 0x03b3bd (0x859e bytes) 000239 1722795343.809862 read 0x008fc9 thru 0x017b8a (0xebc2 bytes) 000240 1722795343.815175 read 0x03397b thru 0x037ef6 (0x457c bytes) 000241 1722795343.815818 mapread 0x0320ba thru 0x03b2a4 (0x91eb bytes) 000246 1722795343.818242 read 0x01d9c7 thru 0x0264d2 (0x8b0c bytes) 000248 1722795343.821865 MIRROR_RESYNC 2 mirrors 000249 1722795344.001108 read 0x01d516 thru 0x028e5b (0xb946 bytes) 000256 1722795344.007007 read 0x03ab78 thru 0x03ecfa (0x4183 bytes) 000259 1722795344.013492 read 0x0312e2 thru 0x032fa5 (0x1cc4 bytes) 000260 1722795344.013849 trunc from 0x03ecfb to 0x003d49 (0x3afb1 bytes) 000262 1722795344.050620 trunc from 0x003d4a to 0x00eb79 (0xae30 bytes) 000263 1722795344.059916 read 0x003361 thru 0x00eb79 (0xb819 bytes) 000264 1722795344.065704 trunc from 0x00eb7a to 0x021b93 (0x1301a bytes) 000266 1722795344.073633 mapread 0x0073ac thru 0x014312 (0xcf67 bytes) 000267 1722795344.079375 mapread 0x0026f6 thru 0x0070be (0x49c9 bytes) 000268 1722795344.080157 trunc from 0x021b94 to 0x01ce1d (0x4d76 bytes) 000269 1722795344.090091 MIRROR_RESYNC 2 mirrors 000272 1722795344.228080 trunc from 0x01ce1e to 0x001c00 (0x1b21d bytes) 000273 1722795344.256527 write 0x0073d8 thru 0x007e8c (0xab5 bytes) 000274 1722795344.257581 trunc from 0x007e8d to 0x004fbe (0x2ece bytes) 000275 1722795344.266479 mapread 0x0028c2 thru 0x004fbe (0x26fd bytes) 000276 1722795344.277206 MIRROR_EXTEND to 3 mirrors 000279 1722795344.446961 write 0x0363c2 thru 0x03ffff (0x9c3e bytes) 000282 1722795344.478117 write 0x00f7da thru 0x010899 (0x10c0 bytes) 000284 1722795344.484474 read 0x0142aa thru 0x01d897 (0x95ee bytes) 000287 1722795344.491455 read 0x03c16c thru 0x03e2c9 (0x215e bytes) 000288 1722795344.492376 mapread 0x004f70 thru 0x00ef26 (0x9fb7 bytes) 000292 1722795344.502953 MIRROR_RESYNC 3 mirrors 000295 1722795344.699657 mapread 0x009749 thru 0x00b699 (0x1f51 bytes) 000299 1722795344.710150 mapread 0x0234a3 thru 0x0248e0 (0x143e bytes) 000300 1722795344.717134 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x615:0x0]: Invalid argument (22) 000302 1722795344.827066 write 0x013965 thru 0x01b8b8 (0x7f54 bytes) 000304 1722795344.861511 mapread 0x02faf2 thru 0x03b1ef (0xb6fe bytes) 000306 1722795344.871416 mapwrite 0x02b998 thru 0x034c10 (0x9279 bytes) 000307 1722795344.907296 trunc from 0x040000 to 0x03a50e (0x5af1 bytes) 000310 1722795344.925527 write 0x001503 thru 0x007278 (0x5d76 bytes) 000312 1722795344.931434 write 0x01bf7c thru 0x025199 (0x921e bytes) 000314 1722795344.933455 mapread 0x0190f9 thru 0x022de7 (0x9cef bytes) 000315 1722795344.934583 trunc from 0x03a50f to 0x001dcb (0x38743 bytes) 000318 1722795344.946176 MIRROR_RESYNC 2 mirrors 000321 1722795345.111326 write 0x01bcd3 thru 0x0231e9 (0x7517 bytes) 000322 1722795345.138178 mapread 0x009263 thru 0x013b5b (0xa8f9 bytes) 000323 1722795345.152163 read 0x013611 thru 0x01e88a (0xb27a bytes) 000324 1722795345.155242 mapwrite 0x00e0c7 thru 0x01adfe (0xcd38 bytes) 000327 1722795345.187620 trunc from 0x0231ea to 0x0312ec (0xe103 bytes) 000329 1722795345.198091 write 0x00a5c9 thru 0x017ac0 (0xd4f8 bytes) 000331 1722795345.200735 write 0x031cb5 thru 0x03ff62 (0xe2ae bytes) 000332 1722795345.207021 mapwrite 0x02d640 thru 0x030e1e (0x37df bytes) 000333 1722795345.240113 write 0x01352e thru 0x0161cd (0x2ca0 bytes) 000334 1722795345.241642 mapwrite 0x037fdb thru 0x03ffff (0x8025 bytes) 000335 1722795345.278626 trunc from 0x040000 to 0x001381 (0x3ec7e bytes) 000338 1722795345.286878 trunc from 0x001382 to 0x01d5c6 (0x1c245 bytes) 000341 1722795345.295143 read 0x019ed6 thru 0x01d5c6 (0x36f1 bytes) 000342 1722795345.301778 trunc from 0x01d5c7 to 0x02c2dd (0xed17 bytes) 000343 1722795345.309400 trunc from 0x02c2de to 0x01f8da (0xca03 bytes) 000345 1722795345.317454 trunc from 0x01f8db to 0x03a2d2 (0x1a9f8 bytes) 000347 1722795345.325018 mapread 0x00c43f thru 0x014c66 (0x8828 bytes) 000349 1722795345.326402 trunc from 0x03a2d3 to 0x022ed1 (0x17401 bytes) 000350 1722795345.333304 mapwrite 0x0034c0 thru 0x008244 (0x4d85 bytes) 000351 1722795345.363903 read 0x019108 thru 0x01aeea (0x1de3 bytes) 000352 1722795345.364449 mapread 0x000142 thru 0x008ba4 (0x8a63 bytes) 000354 1722795345.367949 MIRROR_RESYNC 2 mirrors 000355 1722795345.534374 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x615:0x0]: Invalid argument (22) 000356 1722795345.589970 mapwrite 0x0227b3 thru 0x031d01 (0xf54f bytes) 000362 1722795345.656841 mapread 0x0031b0 thru 0x00fa53 (0xc8a4 bytes) 000363 1722795345.665555 write 0x03e71a thru 0x03ffff (0x18e6 bytes) 000365 1722795345.666650 read 0x00fa71 thru 0x011d3c (0x22cc bytes) 000366 1722795345.667124 mapread 0x035734 thru 0x036cab (0x1578 bytes) 000368 1722795345.675863 MIRROR_EXTEND to 2 mirrors 000370 1722795345.867353 MIRROR_RESYNC 2 mirrors 000371 1722795345.939638 write 0x02b74f thru 0x03a6f3 (0xefa5 bytes) 000372 1722795345.970363 write 0x029691 thru 0x037dc4 (0xe734 bytes) 000373 1722795345.977213 MIRROR_RESYNC 2 mirrors 000376 1722795346.156103 mapread 0x03f5ca thru 0x03ffff (0xa36 bytes) 000377 1722795346.159386 mapread 0x035944 thru 0x03ffff (0xa6bc bytes) 000378 1722795346.169491 write 0x005b7d thru 0x01515c (0xf5e0 bytes) 000379 1722795346.203393 trunc from 0x040000 to 0x000354 (0x3fcab bytes) 000381 1722795346.214293 mapread 0x0001fb thru 0x000354 (0x15a bytes) 000382 1722795346.217824 write 0x0328c4 thru 0x03bfe4 (0x9721 bytes) 000383 1722795346.219264 write 0x00fb88 thru 0x01011f (0x598 bytes) 000384 1722795346.226591 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x615:0x0]: Invalid argument (22) 000386 1722795346.285627 trunc from 0x03bfe5 to 0x03a41f (0x1bc5 bytes) 000387 1722795346.292610 mapread 0x00ae12 thru 0x01aa6e (0xfc5d bytes) 000389 1722795346.306453 write 0x00d12a thru 0x019acc (0xc9a3 bytes) 000391 1722795346.307888 trunc from 0x03a420 to 0x002348 (0x380d7 bytes) 000393 1722795346.315345 mapread 0x000a5e thru 0x002348 (0x18eb bytes) 000394 1722795346.321310 trunc from 0x002349 to 0x01eb96 (0x1c84e bytes) 000396 1722795346.329474 trunc from 0x01eb97 to 0x014fd0 (0x9bc6 bytes) 000397 1722795346.336513 mapread 0x00f0aa thru 0x014fd0 (0x5f27 bytes) 000400 1722795346.342541 mapwrite 0x005afa thru 0x00d2ad (0x77b4 bytes) 000402 1722795346.375255 write 0x0020b7 thru 0x0117a3 (0xf6ed bytes) 000403 1722795346.376468 write 0x035d01 thru 0x03ffff (0xa2ff bytes) 000405 1722795346.378134 mapread 0x0005ae thru 0x0098d4 (0x9327 bytes) 000406 1722795346.378894 trunc from 0x040000 to 0x03e2a1 (0x1d5e bytes) 000407 1722795346.386856 read 0x02b4d9 thru 0x02e9c9 (0x34f1 bytes) 000416 1722795346.393663 mapread 0x000ee8 thru 0x006919 (0x5a32 bytes) 000417 1722795346.394434 write 0x0108ba thru 0x0146f6 (0x3e3d bytes) 000419 1722795346.395686 mapread 0x0100f4 thru 0x01cb57 (0xca64 bytes) 000420 1722795346.400943 mapread 0x010dbb thru 0x01d284 (0xc4ca bytes) 000422 1722795346.402253 trunc from 0x03e2a2 to 0x01928f (0x25012 bytes) 000427 1722795346.412269 trunc from 0x019290 to 0x02d362 (0x140d3 bytes) 000431 1722795346.420758 mapwrite 0x0358c6 thru 0x03ffff (0xa73a bytes) 000433 1722795346.464835 write 0x036764 thru 0x038372 (0x1c0f bytes) 000434 1722795346.465726 read 0x007ff5 thru 0x0172c0 (0xf2cc bytes) 000435 1722795346.466309 read 0x02337a thru 0x0271b9 (0x3e40 bytes) 000436 1722795346.471563 read 0x021a6d thru 0x0221a9 (0x73d bytes) 000438 1722795346.472726 write 0x009e66 thru 0x00b44b (0x15e6 bytes) 000440 1722795346.474091 read 0x0050a5 thru 0x0134db (0xe437 bytes) 000441 1722795346.474563 mapwrite 0x0228f2 thru 0x031ea8 (0xf5b7 bytes) 000444 1722795346.507402 MIRROR_EXTEND to 2 mirrors 000445 1722795346.686084 trunc from 0x040000 to 0x024934 (0x1b6cb bytes) 000446 1722795346.723396 trunc from 0x024935 to 0x02b2e1 (0x69ad bytes) 000448 1722795346.733212 mapwrite 0x011859 thru 0x015226 (0x39ce bytes) 000449 1722795346.771822 write 0x02b4fa thru 0x031026 (0x5b2d bytes) 000450 1722795346.776500 write 0x0085a4 thru 0x0127b3 (0xa210 bytes) 000451 1722795346.784218 MIRROR_RESYNC 2 mirrors 000453 1722795346.949530 write 0x03c436 thru 0x03e656 (0x2221 bytes) 000454 1722795346.974506 write 0x0034a8 thru 0x00e9e3 (0xb53c bytes) 000455 1722795346.979872 mapwrite 0x02caf1 thru 0x032085 (0x5595 bytes) 000456 1722795347.018585 MIRROR_RESYNC 2 mirrors 000457 1722795347.193889 MIRROR_EXTEND to 3 mirrors 000459 1722795347.416848 MIRROR_RESYNC 3 mirrors 000460 1722795347.490147 write 0x0290e4 thru 0x03301b (0x9f38 bytes) 000464 1722795347.522933 mapwrite 0x03d880 thru 0x03ffff (0x2780 bytes) 000465 1722795347.566388 read 0x019871 thru 0x0257cf (0xbf5f bytes) 000469 1722795347.575617 MIRROR_EXTEND to 4 mirrors 000470 1722795347.790545 mapread 0x006b95 thru 0x012a97 (0xbf03 bytes) 000472 1722795347.808524 mapread 0x020014 thru 0x02fdc5 (0xfdb2 bytes) 000474 1722795347.824298 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x615:0x0]: Invalid argument (22) 000476 1722795347.932219 mapwrite 0x0059f1 thru 0x00c9c6 (0x6fd6 bytes) 000477 1722795348.005429 write 0x017541 thru 0x025148 (0xdc08 bytes) 000478 1722795348.013569 mapread 0x032ab9 thru 0x039898 (0x6de0 bytes) 000480 1722795348.019337 mapread 0x027c92 thru 0x029200 (0x156f bytes) 000481 1722795348.024145 trunc from 0x040000 to 0x005df6 (0x3a209 bytes) 000484 1722795348.042262 write 0x018ba2 thru 0x01e539 (0x5998 bytes) 000485 1722795348.043763 write 0x02747c thru 0x027b9b (0x720 bytes) 000490 1722795348.047064 mapread 0x018152 thru 0x0195b4 (0x1463 bytes) 000491 1722795348.047891 mapread 0x02356e thru 0x027b9b (0x462e bytes) 000492 1722795348.052697 mapwrite 0x003a81 thru 0x006afe (0x307e bytes) 000493 1722795348.086355 mapread 0x0083cf thru 0x014622 (0xc254 bytes) 000495 1722795348.092202 mapread 0x016da9 thru 0x021440 (0xa698 bytes) 000498 1722795348.097312 MIRROR_RESYNC 3 mirrors 000500 1722795348.302258 write 0x01faa1 thru 0x01fb9b (0xfb bytes) 000503 1722795348.329611 trunc from 0x027b9c to 0x01547f (0x1271c bytes) 000504 1722795348.346534 write 0x030102 thru 0x034221 (0x4120 bytes) 000505 1722795348.347607 mapread 0x02cf32 thru 0x034221 (0x72f0 bytes) 000506 1722795348.354621 mapwrite 0x022a7f thru 0x02b68c (0x8c0e bytes) 000507 1722795348.388768 mapread 0x0209f2 thru 0x0284bb (0x7aca bytes) 000508 1722795348.389746 mapwrite 0x002c71 thru 0x0075d4 (0x4964 bytes) 000509 1722795348.422212 MIRROR_RESYNC 3 mirrors 000510 1722795348.624413 mapwrite 0x0257b7 thru 0x02db12 (0x835c bytes) 000512 1722795348.701178 MIRROR_RESYNC 3 mirrors 000516 1722795348.909606 read 0x016b09 thru 0x01fd80 (0x9278 bytes) 000520 1722795348.917507 write 0x00196a thru 0x005672 (0x3d09 bytes) 000523 1722795348.947697 mapread 0x017a16 thru 0x0208c3 (0x8eae bytes) 000525 1722795348.957600 read 0x030327 thru 0x034221 (0x3efb bytes) 000528 1722795348.966307 MIRROR_RESYNC 3 mirrors 000529 1722795349.195419 read 0x01721e thru 0x01aa94 (0x3877 bytes) 000530 1722795349.198881 mapwrite 0x01f142 thru 0x02e067 (0xef26 bytes) 000531 1722795349.281247 mapwrite 0x03f8c5 thru 0x03ffff (0x73b bytes) 000532 1722795349.321562 write 0x026de1 thru 0x034e72 (0xe092 bytes) 000534 1722795349.325587 mapread 0x001af8 thru 0x0047a0 (0x2ca9 bytes) 000535 1722795349.333833 MIRROR_RESYNC 3 mirrors 000536 1722795349.537971 MIRROR_RESYNC 3 mirrors 000537 1722795349.612860 mapwrite 0x030d3b thru 0x03b06e (0xa334 bytes) 000539 1722795349.698874 mapwrite 0x03f7cc thru 0x03ffff (0x834 bytes) 000540 1722795349.725794 read 0x023fd6 thru 0x02d663 (0x968e bytes) 000541 1722795349.730391 mapwrite 0x010d75 thru 0x0151dd (0x4469 bytes) 000542 1722795349.761486 mapwrite 0x039512 thru 0x03ffff (0x6aee bytes) 000543 1722795349.789722 write 0x039f77 thru 0x03d11f (0x31a9 bytes) 000545 1722795349.791161 mapwrite 0x01981a thru 0x023bf2 (0xa3d9 bytes) 000546 1722795349.824274 mapread 0x0102f4 thru 0x013da8 (0x3ab5 bytes) 000548 1722795349.825461 mapread 0x0057ec thru 0x00a03d (0x4852 bytes) 000549 1722795349.830259 mapread 0x00be26 thru 0x011a29 (0x5c04 bytes) 000550 1722795349.831055 write 0x03a782 thru 0x03ffff (0x587e bytes) 000551 1722795349.831982 write 0x030d92 thru 0x03c156 (0xb3c5 bytes) 000552 1722795349.833085 trunc from 0x040000 to 0x0086a5 (0x3795a bytes) 000553 1722795349.851425 trunc from 0x0086a6 to 0x02d8b3 (0x2520e bytes) 000554 1722795349.858551 mapread 0x026f65 thru 0x02d8b3 (0x694f bytes) 000555 1722795349.863728 trunc from 0x02d8b4 to 0x03b3a7 (0xdaf4 bytes) 000558 1722795349.873234 mapwrite 0x03d187 thru 0x03ffff (0x2e79 bytes) 000560 1722795349.915647 read 0x02ab2e thru 0x02e13e (0x3611 bytes) 000561 1722795349.925833 MIRROR_EXTEND to 4 mirrors 000567 1722795350.185705 mapwrite 0x01a2d4 thru 0x0291ce (0xeefb bytes) 000568 1722795350.272257 mapread 0x02d70b thru 0x03685f (0x9155 bytes) 000569 1722795350.280888 write 0x022d90 thru 0x026620 (0x3891 bytes) 000570 1722795350.284429 MIRROR_EXTEND to 5 mirrors 000571 1722795350.514023 write 0x02d883 thru 0x0376d0 (0x9e4e bytes) 000574 1722795350.545540 trunc from 0x040000 to 0x01a23c (0x25dc3 bytes) 000576 1722795350.566071 MIRROR_RESYNC 5 mirrors 000577 1722795350.867481 mapread 0x0167f5 thru 0x01a23c (0x3a48 bytes) 000578 1722795350.876379 mapwrite 0x03c12d thru 0x03ffff (0x3ed3 bytes) 000584 1722795350.951620 mapread 0x019b25 thru 0x01a4cc (0x9a8 bytes) 000587 1722795350.958074 read 0x0262a2 thru 0x02cfbf (0x6d1e bytes) 000588 1722795350.964338 MIRROR_EXTEND to 6 mirrors 000589 1722795351.193648 trunc from 0x040000 to 0x00c7e8 (0x33817 bytes) 000593 1722795351.225166 read 0x0087a1 thru 0x00c7e8 (0x4048 bytes) 000594 1722795351.230511 read 0x008ddf thru 0x00c7e8 (0x3a0a bytes) 000597 1722795351.232051 mapwrite 0x0210fa thru 0x029a8c (0x8993 bytes) 000598 1722795351.282316 trunc from 0x029a8d to 0x039829 (0xfd9d bytes) 000599 1722795351.289773 write 0x0264ff thru 0x029041 (0x2b43 bytes) 000600 1722795351.293078 MIRROR_RESYNC 6 mirrors 000601 1722795351.638170 mapwrite 0x026a01 thru 0x0343d6 (0xd9d6 bytes) 000602 1722795351.722982 mapread 0x026a1a thru 0x0364ff (0xfae6 bytes) 000603 1722795351.726598 MIRROR_EXTEND to 7 mirrors 000604 1722795351.921788 mapread 0x019ccc thru 0x01e7cb (0x4b00 bytes) 000606 1722795351.937159 mapwrite 0x0148a9 thru 0x01f235 (0xa98d bytes) 000607 1722795352.024463 read 0x000dcb thru 0x007955 (0x6b8b bytes) 000610 1722795352.029820 mapread 0x00cf40 thru 0x01155b (0x461c bytes) 000613 1722795352.036785 mapread 0x019f1e thru 0x01f8cd (0x59b0 bytes) 000615 1722795352.038112 trunc from 0x03982a to 0x02da72 (0xbdb7 bytes) 000616 1722795352.057658 trunc from 0x02da73 to 0x020caa (0xcdc8 bytes) 000617 1722795352.066817 MIRROR_RESYNC 7 mirrors 000618 1722795352.442530 mapread 0x01cc8a thru 0x020caa (0x4021 bytes) 000619 1722795352.451880 trunc from 0x020cab to 0x02463a (0x3990 bytes) 000620 1722795352.490874 mapwrite 0x03b770 thru 0x03ce2a (0x16bb bytes) 000624 1722795352.533464 write 0x0353b4 thru 0x03541f (0x6c bytes) 000627 1722795352.535169 write 0x005385 thru 0x008afc (0x3778 bytes) 000629 1722795352.542541 read 0x0284b3 thru 0x0359cd (0xd51b bytes) 000632 1722795352.549573 write 0x004d2c thru 0x009cca (0x4f9f bytes) 000633 1722795352.550790 write 0x01242e thru 0x01fc27 (0xd7fa bytes) 000634 1722795352.559143 mapread 0x02ac1c thru 0x036bb0 (0xbf95 bytes) 000635 1722795352.560539 trunc from 0x03ce2b to 0x0260ed (0x16d3d bytes) 000636 1722795352.571612 MIRROR_RESYNC 7 mirrors 000637 1722795352.961731 MIRROR_RESYNC 7 mirrors 000638 1722795353.050426 mapwrite 0x00edf2 thru 0x01e794 (0xf9a3 bytes) 000639 1722795353.142556 read 0x020020 thru 0x0260ed (0x60ce bytes) 000640 1722795353.147477 read 0x018308 thru 0x01faaa (0x77a3 bytes) 000643 1722795353.149041 trunc from 0x0260ee to 0x00fffd (0x160f0 bytes) 000644 1722795353.166311 read 0x00fcc8 thru 0x00fffd (0x336 bytes) 000647 1722795353.167578 trunc from 0x00fffe to 0x00dd81 (0x227c bytes) 000650 1722795353.174648 read 0x0037e4 thru 0x00b94e (0x816b bytes) 000651 1722795353.179262 read 0x00988d thru 0x00dd81 (0x44f5 bytes) 000652 1722795353.179819 mapread 0x0034fb thru 0x0096a4 (0x61aa bytes) 000654 1722795353.181089 write 0x005b4e thru 0x00f2c1 (0x9774 bytes) 000655 1722795353.182202 trunc from 0x00f2c2 to 0x02d0c3 (0x1de02 bytes) 000657 1722795353.192499 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x615:0x0]: Invalid argument (22) 000661 1722795353.294988 MIRROR_EXTEND to 7 mirrors 000662 1722795353.505608 MIRROR_RESYNC 7 mirrors 000664 1722795353.840906 trunc from 0x02d0c4 to 0x02b20d (0x1eb6 bytes) 000665 1722795353.880017 write 0x03869a thru 0x03ffff (0x7966 bytes) 000669 1722795353.882983 mapwrite 0x006e8d thru 0x007577 (0x6eb bytes) 000670 1722795353.918188 trunc from 0x040000 to 0x0252db (0x1ad24 bytes) 000673 1722795353.929439 write 0x018570 thru 0x022908 (0xa399 bytes) 000675 1722795353.938294 mapwrite 0x0019bc thru 0x00914e (0x7793 bytes) 000676 1722795353.975165 write 0x02804d thru 0x034656 (0xc60a bytes) 000679 1722795353.977472 mapwrite 0x01d278 thru 0x026057 (0x8de0 bytes) 000680 1722795354.010603 read 0x004b6c thru 0x0089bb (0x3e50 bytes) 000682 1722795354.011598 mapread 0x01ef8d thru 0x0267dd (0x7851 bytes) 000683 1722795354.012378 read 0x01de0d thru 0x02431e (0x6512 bytes) 000684 1722795354.012928 write 0x02ec59 thru 0x03919a (0xa542 bytes) 000685 1722795354.014078 trunc from 0x03919b to 0x00f4fd (0x29c9d bytes) 000689 1722795354.024140 mapwrite 0x0357f6 thru 0x03c1f1 (0x69fc bytes) 000692 1722795354.068498 write 0x01342b thru 0x01669e (0x3274 bytes) 000693 1722795354.069469 read 0x0220ab thru 0x02d7da (0xb730 bytes) 000694 1722795354.072434 MIRROR_RESYNC 7 mirrors 000695 1722795354.472761 MIRROR_EXTEND to 8 mirrors 000700 1722795354.711421 trunc from 0x03c1f2 to 0x022c9e (0x19553 bytes) 000702 1722795354.751388 read 0x0196c8 thru 0x0220be (0x89f7 bytes) 000703 1722795354.756636 trunc from 0x022c9f to 0x02e606 (0xb968 bytes) 000704 1722795354.768823 MIRROR_RESYNC 8 mirrors 000705 1722795355.208321 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x615:0x0]: Invalid argument (22) 000706 1722795355.341948 write 0x01d1db thru 0x026800 (0x9626 bytes) 000709 1722795355.376420 read 0x01c637 thru 0x022fdc (0x69a6 bytes) 000710 1722795355.379552 write 0x031334 thru 0x03ffff (0xeccc bytes) 000712 1722795355.381661 mapread 0x008139 thru 0x010b2b (0x89f3 bytes) 000716 1722795355.394557 MIRROR_EXTEND to 8 mirrors 000717 1722795355.620266 write 0x007d43 thru 0x00f2aa (0x7568 bytes) 000718 1722795355.655079 mapread 0x020905 thru 0x028b21 (0x821d bytes) 000719 1722795355.664771 read 0x01181f thru 0x012eac (0x168e bytes) 000720 1722795355.670236 MIRROR_RESYNC 8 mirrors 000721 1722795356.045171 mapwrite 0x01012d thru 0x018828 (0x86fc bytes) 000722 1722795356.131258 trunc from 0x040000 to 0x00074d (0x3f8b2 bytes) 000723 1722795356.150434 mapread 0x0003f0 thru 0x00074d (0x35e bytes) 000725 1722795356.157634 MIRROR_EXTEND to 9 mirrors 000726 1722795356.328753 mapread 0x0000a2 thru 0x00074d (0x6ac bytes) 000727 1722795356.329510 trunc from 0x00074e to 0x032cf7 (0x325aa bytes) 000728 1722795356.366817 mapread 0x023c02 thru 0x02cfb6 (0x93b5 bytes) 000731 1722795356.381052 MIRROR_EXTEND to 10 mirrors 000732 1722795356.579950 MIRROR_EXTEND to 11 mirrors 000733 1722795356.788275 mapread 0x02baaa thru 0x02bf4c (0x4a3 bytes) 000734 1722795356.794953 trunc from 0x032cf8 to 0x0365d0 (0x38d9 bytes) 000735 1722795356.832544 mapread 0x0133c0 thru 0x017878 (0x44b9 bytes) 000737 1722795356.841154 read 0x019be0 thru 0x0223bc (0x87dd bytes) 000738 1722795356.844616 trunc from 0x0365d1 to 0x0146ae (0x21f22 bytes) 000739 1722795356.854247 mapread 0x01087b thru 0x0146ae (0x3e34 bytes) 000740 1722795356.855140 read 0x0021fa thru 0x007036 (0x4e3d bytes) 000741 1722795356.860332 write 0x03e2ae thru 0x03ffff (0x1d52 bytes) 000743 1722795356.861794 read 0x01bf52 thru 0x01c082 (0x131 bytes) 000744 1722795356.868133 mapwrite 0x016a6c thru 0x01fea2 (0x9437 bytes) 000745 1722795356.903180 MIRROR_RESYNC 11 mirrors 000746 1722795357.443006 mapread 0x01e22d thru 0x02524e (0x7022 bytes) 000748 1722795357.457260 mapwrite 0x038aba thru 0x03ffff (0x7546 bytes) 000749 1722795357.542148 write 0x0232d1 thru 0x027370 (0x40a0 bytes) 000751 1722795357.548381 write 0x028091 thru 0x032bad (0xab1d bytes) 000757 1722795357.552224 write 0x03e29b thru 0x03ffff (0x1d65 bytes) 000759 1722795357.553513 mapread 0x0004a9 thru 0x008b57 (0x86af bytes) 000760 1722795357.561831 trunc from 0x040000 to 0x00b018 (0x34fe7 bytes) 000761 1722795357.572337 write 0x010b4f thru 0x01aa93 (0x9f45 bytes) 000762 1722795357.573562 mapread 0x017bc7 thru 0x01aa93 (0x2ecd bytes) 000763 1722795357.574422 read 0x012dd8 thru 0x01aa93 (0x7cbc bytes) 000765 1722795357.575388 trunc from 0x01aa94 to 0x036118 (0x1b685 bytes) 000767 1722795357.585914 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x615:0x0]: Invalid argument (22) 000774 1722795357.686694 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x615:0x0]: Invalid argument (22) 000776 1722795357.779336 MIRROR_EXTEND to 10 mirrors 000779 1722795358.038100 MIRROR_EXTEND to 11 mirrors 000781 1722795358.249917 write 0x021115 thru 0x021de2 (0xcce bytes) 000782 1722795358.281656 read 0x001271 thru 0x007860 (0x65f0 bytes) 000784 1722795358.286885 trunc from 0x036119 to 0x03e566 (0x844e bytes) 000786 1722795358.296903 mapwrite 0x000f3e thru 0x00c2de (0xb3a1 bytes) 000789 1722795358.337106 trunc from 0x03e567 to 0x01f92f (0x1ec37 bytes) 000790 1722795358.344260 write 0x008582 thru 0x00cace (0x454d bytes) 000792 1722795358.345579 write 0x03fa4c thru 0x03ffff (0x5b4 bytes) 000793 1722795358.346456 mapread 0x01030b thru 0x0143d6 (0x40cc bytes) 000797 1722795358.353178 mapwrite 0x02cf1d thru 0x036dbf (0x9ea3 bytes) 000798 1722795358.397146 MIRROR_RESYNC 11 mirrors 000799 1722795358.928129 read 0x0223cd thru 0x03033f (0xdf73 bytes) 000802 1722795358.940536 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x615:0x0]: Invalid argument (22) 000803 1722795359.175577 mapread 0x022175 thru 0x02ac73 (0x8aff bytes) 000804 1722795359.191290 read 0x02b0f9 thru 0x0334a1 (0x83a9 bytes) 000806 1722795359.203831 mapwrite 0x03222d thru 0x0336ae (0x1482 bytes) 000809 1722795359.278507 read 0x036b0a thru 0x03cee6 (0x63dd bytes) 000811 1722795359.286617 mapread 0x02d60d thru 0x02d617 (0xb bytes) 000813 1722795359.288413 trunc from 0x040000 to 0x0038f7 (0x3c708 bytes) 000816 1722795359.308484 mapwrite 0x03605b thru 0x03dc18 (0x7bbe bytes) 000818 1722795359.365359 write 0x006e50 thru 0x008de5 (0x1f96 bytes) 000819 1722795359.370575 mapread 0x003f99 thru 0x013cd8 (0xfd40 bytes) 000820 1722795359.399915 write 0x0329f9 thru 0x0344e5 (0x1aed bytes) 000821 1722795359.407189 MIRROR_EXTEND to 11 mirrors 000823 1722795359.648426 mapread 0x005810 thru 0x0116dc (0xbecd bytes) 000824 1722795359.665057 mapread 0x02e70c thru 0x030db8 (0x26ad bytes) 000826 1722795359.674791 mapread 0x030b8e thru 0x03bf28 (0xb39b bytes) 000828 1722795359.684109 mapwrite 0x0245e3 thru 0x030a41 (0xc45f bytes) 000834 1722795359.779411 trunc from 0x03dc19 to 0x03f77e (0x1b66 bytes) 000835 1722795359.797213 write 0x03a688 thru 0x03ffff (0x5978 bytes) 000840 1722795359.804612 mapread 0x005ba2 thru 0x00a575 (0x49d4 bytes) 000841 1722795359.810738 trunc from 0x040000 to 0x01a437 (0x25bc8 bytes) 000842 1722795359.821633 MIRROR_EXTEND to 12 mirrors 000843 1722795360.015417 read 0x019535 thru 0x01a437 (0xf03 bytes) 000845 1722795360.025063 MIRROR_SPLIT mirror 6 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x615:0x0]: Invalid argument (22) 000846 1722795360.140248 mapwrite 0x01b0c5 thru 0x01ecba (0x3bf6 bytes) 000847 1722795360.218224 read 0x001b10 thru 0x0069c2 (0x4eb3 bytes) 000848 1722795360.223049 trunc from 0x01ecbb to 0x00627c (0x18a3e bytes) 000849 1722795360.232754 write 0x029072 thru 0x03680b (0xd79a bytes) 000851 1722795360.234880 read 0x016e7c thru 0x023319 (0xc49e bytes) 000852 1722795360.241470 read 0x020be6 thru 0x03080e (0xfc29 bytes) 000853 1722795360.242117 mapwrite 0x021042 thru 0x0230f5 (0x20b4 bytes) 000854 1722795360.280362 mapwrite 0x009650 thru 0x00a7e8 (0x1199 bytes) 000855 1722795360.302123 trunc from 0x03680c to 0x0218b9 (0x14f52 bytes) 000857 1722795360.309975 trunc from 0x0218ba to 0x00f734 (0x12185 bytes) 000859 1722795360.319718 MIRROR_RESYNC 11 mirrors 000860 1722795360.869413 trunc from 0x00f735 to 0x02811b (0x189e7 bytes) 000861 1722795360.909038 trunc from 0x02811c to 0x024f77 (0x31a4 bytes) 000863 1722795360.919294 read 0x00c621 thru 0x010af8 (0x44d8 bytes) 000864 1722795360.926619 MIRROR_RESYNC 11 mirrors 000865 1722795361.498485 write 0x03ca90 thru 0x03ffff (0x3570 bytes) 000866 1722795361.529699 write 0x0023a7 thru 0x00a4cf (0x8129 bytes) 000867 1722795361.537322 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x615:0x0]: Invalid argument (22) 000868 1722795361.637760 MIRROR_EXTEND to 11 mirrors 000869 1722795361.910194 MIRROR_EXTEND to 12 mirrors 000870 1722795362.133132 trunc from 0x040000 to 0x008015 (0x37fea bytes) 000871 1722795362.163882 mapread 0x00234e thru 0x008015 (0x5cc8 bytes) 000873 1722795362.172933 read 0x00639a thru 0x008015 (0x1c7c bytes) 000874 1722795362.173618 read 0x005834 thru 0x008015 (0x27e2 bytes) 000877 1722795362.176871 MIRROR_SPLIT mirror 4 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x615:0x0]: Invalid argument (22) 000879 1722795362.260462 MIRROR_RESYNC 11 mirrors 000880 1722795362.810488 write 0x03b277 thru 0x03ffff (0x4d89 bytes) 000881 1722795362.839602 read 0x021f9d thru 0x029d1c (0x7d80 bytes) 000882 1722795362.844872 mapread 0x02f596 thru 0x0325bf (0x302a bytes) 000883 1722795362.858040 MIRROR_RESYNC 11 mirrors 000885 1722795363.192764 mapwrite 0x01c675 thru 0x022560 (0x5eec bytes) 000886 1722795363.242494 mapread 0x01c77f thru 0x02c2fc (0xfb7e bytes) 000888 1722795363.245374 mapread 0x0106d1 thru 0x01ff00 (0xf830 bytes) 000890 1722795363.249920 read 0x02417e thru 0x024d43 (0xbc6 bytes) 000891 1722795363.251406 MIRROR_EXTEND to 12 mirrors 000894 1722795363.415046 trunc from 0x040000 to 0x01eb47 (0x214b8 bytes) 000895 1722795363.455870 write 0x009066 thru 0x00b276 (0x2211 bytes) 000896 1722795363.461148 read 0x00e8d8 thru 0x00ef43 (0x66c bytes) 000898 1722795363.467062 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x615:0x0]: Invalid argument (22) 000899 1722795363.559885 read 0x0084e1 thru 0x01613a (0xdc5a bytes) 000901 1722795363.566067 mapread 0x01a492 thru 0x01ad82 (0x8f1 bytes) 000902 1722795363.569319 mapread 0x0191eb thru 0x01cfa0 (0x3db6 bytes) 000903 1722795363.576254 mapwrite 0x01ba9e thru 0x02a6f7 (0xec5a bytes) 000905 1722795363.625518 mapread 0x004b0b thru 0x014a70 (0xff66 bytes) 000906 1722795363.630809 mapread 0x00b634 thru 0x0173bb (0xbd88 bytes) 000908 1722795363.632100 mapread 0x015765 thru 0x01771b (0x1fb7 bytes) 000909 1722795363.633251 trunc from 0x02a6f8 to 0x0222b3 (0x8444 bytes) 000910 1722795363.640858 mapwrite 0x01571c thru 0x015efd (0x7e2 bytes) 000911 1722795363.669509 mapread 0x016bf9 thru 0x01a932 (0x3d3a bytes) 000912 1722795363.670325 trunc from 0x0222b4 to 0x03db4d (0x1b89a bytes) 000913 1722795363.680557 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x615:0x0]: Invalid argument (22) 000914 1722795363.773420 mapread 0x02576a thru 0x02ed57 (0x95ee bytes) 000915 1722795363.784208 mapread 0x01d87c thru 0x02352a (0x5caf bytes) 000916 1722795363.791945 MIRROR_EXTEND to 11 mirrors 000917 1722795364.000058 trunc from 0x03db4e to 0x01e546 (0x1f607 bytes) 000919 1722795364.044344 read 0x008ab2 thru 0x00b757 (0x2ca6 bytes) 000922 1722795364.048813 write 0x039e95 thru 0x03ffff (0x616b bytes) 000925 1722795364.050972 read 0x006060 thru 0x00b9e7 (0x5988 bytes) 000926 1722795364.058052 write 0x01c4d3 thru 0x01fd44 (0x3872 bytes) 000928 1722795364.059614 mapread 0x0292fa thru 0x038783 (0xf48a bytes) 000930 1722795364.061329 mapread 0x0123e8 thru 0x01c545 (0xa15e bytes) 000932 1722795364.065520 MIRROR_RESYNC 11 mirrors 000933 1722795364.552214 write 0x001806 thru 0x009221 (0x7a1c bytes) 000934 1722795364.585768 write 0x01560d thru 0x01720a (0x1bfe bytes) 000937 1722795364.591909 write 0x016c4e thru 0x01b26f (0x4622 bytes) 000941 1722795364.594056 trunc from 0x040000 to 0x013319 (0x2cce6 bytes) 000944 1722795364.612362 write 0x000943 thru 0x004914 (0x3fd2 bytes) 000945 1722795364.617282 mapread 0x00e852 thru 0x011200 (0x29af bytes) 000947 1722795364.618587 mapread 0x002eee thru 0x00bc87 (0x8d9a bytes) 000949 1722795364.620009 read 0x004707 thru 0x013300 (0xebfa bytes) 000950 1722795364.620586 read 0x00a6f6 thru 0x00e8b3 (0x41be bytes) 000951 1722795364.621117 mapread 0x004d31 thru 0x00b061 (0x6331 bytes) 000952 1722795364.621882 trunc from 0x01331a to 0x008941 (0xa9d8 bytes) 000953 1722795364.632101 MIRROR_RESYNC 11 mirrors 000954 1722795365.194305 read 0x0044e4 thru 0x0052dd (0xdfa bytes) 000956 1722795365.197431 read 0x008858 thru 0x008941 (0xea bytes) 000957 1722795365.200017 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x615:0x0]: Invalid argument (22) 000958 1722795365.339632 mapwrite 0x00c528 thru 0x00f0a2 (0x2b7b bytes) 000961 1722795365.420893 read 0x0091b3 thru 0x00f0a2 (0x5ef0 bytes) 000963 1722795365.421947 read 0x0026a6 thru 0x00c1d6 (0x9b31 bytes) 000964 1722795365.429043 MIRROR_EXTEND to 11 mirrors 000965 1722795365.628458 MIRROR_RESYNC 11 mirrors 000967 1722795366.108507 mapwrite 0x00bb22 thru 0x017715 (0xbbf4 bytes) 000968 1722795366.199050 read 0x002924 thru 0x0083d1 (0x5aae bytes) 000969 1722795366.202323 trunc from 0x017716 to 0x00ef05 (0x8810 bytes) 000973 1722795366.214293 read 0x004c23 thru 0x00d39c (0x877a bytes) 000974 1722795366.214942 trunc from 0x00ef06 to 0x00c2d1 (0x2c34 bytes) 000975 1722795366.222083 read 0x006343 thru 0x00c2d1 (0x5f8f bytes) 000976 1722795366.222731 trunc from 0x00c2d2 to 0x011c7c (0x59ab bytes) 000977 1722795366.230026 trunc from 0x011c7d to 0x0352ae (0x23632 bytes) 000978 1722795366.237959 write 0x0210ae thru 0x02f453 (0xe3a6 bytes) 000980 1722795366.247440 write 0x0129a4 thru 0x014242 (0x189f bytes) 000982 1722795366.248722 trunc from 0x0352af to 0x002423 (0x32e8b bytes) 000983 1722795366.256555 read 0x0015be thru 0x002423 (0xe66 bytes) 000985 1722795366.257627 write 0x031d6e thru 0x03ffff (0xe292 bytes) 000987 1722795366.259552 trunc from 0x040000 to 0x012639 (0x2d9c6 bytes) 000988 1722795366.267612 mapwrite 0x017578 thru 0x01ee41 (0x78ca bytes) 000990 1722795366.311586 read 0x004bed thru 0x010a17 (0xbe2b bytes) 000995 1722795366.313987 mapwrite 0x020ec2 thru 0x02e8dd (0xda1c bytes) 000998 1722795366.360622 mapread 0x015f86 thru 0x017dec (0x1e67 bytes) 000999 1722795366.361462 read 0x00aecf thru 0x01666c (0xb79e bytes) All operations completed A-OK! PASS 70a (30s) == sanity-flr test 200a: stress test ===================== 14:16:08 (1722795368) Starting client: oleg460-client.virtnet: -o user_xattr,flock oleg460-server@tcp:/lustre /mnt/lustre2 Starting client: oleg460-client.virtnet: -o user_xattr,flock oleg460-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1964832 .. Extending file size to 3769472 .. done Extending file size to 4945632 .. Extending file size to 4498080 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7340512 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9148608 .. Extending file size to 7446048 .. Extending file size to 7454848 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7573344 .. done Extending file size to 8830592 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8892128 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9046400 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 29098 29099 29101 29102 29105 failed 192.168.204.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg460-client.virtnet /mnt/lustre2 (opts:) 192.168.204.160@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg460-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 14:17:12 (1722795432) Starting client: oleg460-client.virtnet: -o user_xattr,flock oleg460-server@tcp:/lustre /mnt/lustre2 Starting client: oleg460-client.virtnet: -o user_xattr,flock oleg460-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1376480 .. Extending file size to 1772768 .. Extending file size to 7988672 .. Extending file size to 6506944 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7219488 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8777728 .. Extending file size to 7846912 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8880832 .. Extending file size to 8147616 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8769632 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9284544 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9258720 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 31644 31645 31646 31649 31651 192.168.204.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg460-client.virtnet /mnt/lustre2 (opts:) 192.168.204.160@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg460-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (66s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 14:18:18 (1722795498) Starting client: oleg460-client.virtnet: -o user_xattr,flock oleg460-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4608 2203904 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4608 2203904 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0534391 s, 39.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.132072 s, 15.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg460-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 14:18:25 (1722795505) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0468794 s, 44.7 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 14:18:27 (1722795507) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x655:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7dc:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7dc:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7dd:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x7df:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 14:18:30 (1722795510) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0396155 s, 52.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7de:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7e0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7e1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 14:18:32 (1722795512) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0488906 s, 42.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7e0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7e2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7e0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7e2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7e3:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 14:18:35 (1722795515) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0422483 s, 49.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7e4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7e4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7e5:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 14:18:58 (1722795538) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg460-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0306462 s, 34.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7e2:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7e6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7e2:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7e6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7e3:0x0] } PASS 204d (42s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 14:19:40 (1722795580) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg460-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7e5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0434418 s, 48.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7e5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7e8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7e5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7e8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (44s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 14:20:24 (1722795624) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg460-server mds-ost sync done. Creating new pool oleg460-server: Pool lustre.test_204f created Adding targets to pool oleg460-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg460-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0410804 s, 51.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7e9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7e9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg460-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg460-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg460-server: Pool lustre.test_204f destroyed PASS 204f (53s) == sanity-flr test 205: lfs mirror extend to set prefer and nocompr flags ========================================================== 14:21:17 (1722795677) PASS 205 (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 14:21:20 (1722795680) oleg460-server: Pool lustre.test_206 created Adding targets to pool oleg460-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg460-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg460-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg460-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg460-server: Pool lustre.test_206 destroyed PASS 206 (15s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 14:21:35 (1722795695) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.031713 s, 33.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0139166 s, 75.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xf27cecaa Mirror 3: 0xf27cecaa PASS 207 (4s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 14:21:39 (1722795699) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 14:21:40 (1722795700) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 14:21:41 (1722795701) Starting client: oleg460-client.virtnet: -o user_xattr,flock oleg460-server@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0298588 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x25d0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1610000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1610000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1610000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xb10000 Read this (4 bytes): BBBB 192.168.204.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg460-client.virtnet /mnt/lustre2 (opts:) PASS 209a (16s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 14:21:58 (1722795718) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00452743 s, 1.8 MB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 14:22:00 (1722795720) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0165089 s, 63.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x679 lmm_fid: [0x200000404:0x679:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x679 lmm_fid: [0x200000404:0x679:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7ef:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835577 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-08-04 14:22:00.000000000 -0400 Modify: 2024-08-04 14:22:00.000000000 -0400 Change: 2024-08-04 14:22:01.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 14:22:03 (1722795723) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test complete, duration 743 sec ============ 14:22:04 (1722795724) === sanity-flr: start cleanup 14:22:04 (1722795724) === === sanity-flr: finish cleanup 14:22:05 (1722795725) ===