-----============= acceptance-small: sanity-flr ============----- Thu Feb 27 11:20:39 EST 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg303-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 11:20:43 (1740673243) === oleg303-client.virtnet: executing check_config_client /mnt/lustre oleg303-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg303-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b643b000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b643b000.idle_timeout=debug disable quota as required oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 11:20:53 (1740673253) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f6987] == sanity-flr test 0a: lfs mirror create with -N option == 11:20:53 (1740673253) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 11:20:56 (1740673256) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 11:20:57 (1740673257) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 11:20:58 (1740673258) PASS 0d (4s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 11:21:02 (1740673262) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (0s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 11:21:02 (1740673262) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 11:21:03 (1740673263) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b643b000.stats=clear osc.lustre-OST0001-osc-ffff8800b643b000.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 11:21:05 (1740673265) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 11:21:07 (1740673267) PASS 0j (1s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 11:21:08 (1740673268) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 11:21:09 (1740673269) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 11:21:11 (1740673271) PASS 2 (1s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 11:21:12 (1740673272) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 11:21:13 (1740673273) lfs extend: cannot get the last init comp error: lfs extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (1s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 11:21:14 (1740673274) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 11:21:16 (1740673276) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 11:21:18 (1740673278) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.185119 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0319006 s, 32.9 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 11:21:20 (1740673280) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0103959 s, 101 MB/s PASS 22 (1s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 11:21:21 (1740673281) Stopping /mnt/lustre-ost1 (opts:) on oleg303-server oleg303-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b643b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b643b000.ost_server_uuid in DISCONN state after 2 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg303-server: oleg303-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg303-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg303-server oleg303-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b643b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b643b000.ost_server_uuid in DISCONN state after 3 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg303-server: oleg303-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg303-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (24s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 11:21:45 (1740673305) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB) copied, 0.14769 s, 63.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg303-server oleg303-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b643b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b643b000.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg303-server: oleg303-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg303-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (10s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 11:21:55 (1740673315) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg303-server oleg303-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b643b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b643b000.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg303-server: oleg303-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg303-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg303-server oleg303-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b643b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b643b000.ost_server_uuid in DISCONN state after 4 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg303-server: oleg303-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg303-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (20s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 11:22:16 (1740673336) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.686745 s, 76.3 MB/s -rw-r--r-- 1 root root 52428800 Feb 27 11:22 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.209s user 0m0.000s sys 0m0.100s Stopping /mnt/lustre-ost1 (opts:) on oleg303-server oleg303-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b643b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b643b000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.229s user 0m0.003s sys 0m0.100s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg303-server: oleg303-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg303-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg303-server oleg303-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b643b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b643b000.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost1)... real 0m0.248s user 0m0.001s sys 0m0.105s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg303-server: oleg303-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg303-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg303-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b643b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b643b000.ost_server_uuid in FULL state after 0 sec PASS 33b (21s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 11:22:37 (1740673357) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 11:22:38 (1740673358) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (0s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 11:22:38 (1740673358) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 11:22:39 (1740673359) PASS 35 (1s) == sanity-flr test 36a: write to mirrored files ========== 11:22:40 (1740673360) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.0994248 s, 127 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 11:22:43 (1740673363) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00734729 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00952068 s, 110 MB/s PASS 36b (1s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 11:22:44 (1740673364) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 11:22:47 (1740673367) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000658407 s, 1.6 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000635267 s, 1.6 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000562515 s, 1.8 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000544043 s, 1.9 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 11:22:49 (1740673369) create test files with size 15743997 .. 9196e682fe402c0b919a3efd0875ac09 - 3bd66bc0345d9ab8572d3303b5a786c7 - b1a2f35f11ec7398ab34c14c622c251c - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg303-client.virtnet /mnt/lustre (opts:) Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre PASS 37 (4s) == sanity-flr test 38: resync ============================ 11:22:53 (1740673373) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 11:22:56 (1740673376) PASS 39 (1s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 11:22:57 (1740673377) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0196504 s, 53.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00904938 s, 116 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 11:22:59 (1740673379) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0242478 s, 86.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0690741 s, 60.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0341996 s, 61.3 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) == sanity-flr test 42: lfs mirror verify ================= 11:23:01 (1740673381) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 11:23:02 (1740673382) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (0s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 11:23:02 (1740673382) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00737651 s, 142 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 11:23:04 (1740673384) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 11:23:05 (1740673385) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (1s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 11:23:07 (1740673387) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0521867 s, 201 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg303-client.virtnet /mnt/lustre (opts:) Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (24s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 11:23:31 (1740673411) llite.lustre-ffff8800aa961000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0465061 s, 225 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg303-client.virtnet /mnt/lustre (opts:) Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800a326f000.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 11:23:34 (1740673414) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (6s) == sanity-flr test 46: Verify setstripe --copy option ==== 11:23:40 (1740673420) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (6s) == sanity-flr test 47: Verify mirror obj alloc =========== 11:23:46 (1740673426) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (0s) == sanity-flr test 48: Verify snapshot mirror ============ 11:23:46 (1740673426) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.05536 s, 56.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0392901 s, 80.1 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 862cd2b6705e2b5cf3f36ece5a1bc003 - mirror 1 checksum: beda7b24399cfdefb075e2536c70f2ce - mirror 2 checksum: 862cd2b6705e2b5cf3f36ece5a1bc003 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Feb 27 11:23:47 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Feb 27 11:23:47 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 862cd2b6705e2b5cf3f36ece5a1bc003 - mirror 1 checksum: beda7b24399cfdefb075e2536c70f2ce - mirror 2 checksum: beda7b24399cfdefb075e2536c70f2ce - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Feb 27 11:23:47 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Feb 27 11:23:47 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 48 (2s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 11:23:49 (1740673429) Failing mds1 on oleg303-server Stopping /mnt/lustre-mds1 (opts:) on oleg303-server 11:23:50 (1740673430) shut down facet: mds1 facet_host: oleg303-server facet_failover_host: oleg303-server Failover mds1 to oleg303-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-MDT0000 11:24:03 (1740673443) targets are mounted 11:24:03 (1740673443) facet_failover done oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00691722 s, 152 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } PASS 50A (19s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 11:24:08 (1740673448) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 11:24:09 (1740673449) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (0s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 11:24:09 (1740673449) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00205284 s, 8.0 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 11:24:11 (1740673451) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 11:24:12 (1740673452) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (0s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 11:24:12 (1740673452) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (4s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 11:24:16 (1740673456) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (17s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 11:24:33 (1740673473) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 11:24:50 (1740673490) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 11:25:07 (1740673507) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (1s) == sanity-flr test 70: mirror create and split race ====== 11:25:08 (1740673508) mirror create pid 22495 mirror split pid 22496 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4866: 22495 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4866: 22496 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 11:26:10 (1740673570) Seed set to 3570 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x97a4 000001 1740673570.538089 trunc from 00000000 to 0x0097a3 (0x97a4 bytes) 000002 1740673570.541262 write 0x01ae6a thru 0x024cce (0x9e65 bytes) 000004 1740673570.542447 mapread 0x01f671 thru 0x01f801 (0x191 bytes) 000006 1740673570.543088 mapread 0x00ec13 thru 0x016ba1 (0x7f8f bytes) 000007 1740673570.547189 read 0x01de5a thru 0x024cce (0x6e75 bytes) 000008 1740673570.547543 read 0x01bb7b thru 0x024cce (0x9154 bytes) 000009 1740673570.547800 mapwrite 0x00817a thru 0x011263 (0x90ea bytes) 000010 1740673570.574426 mapwrite 0x038717 thru 0x03f5b3 (0x6e9d bytes) 000011 1740673570.593583 read 0x02d7a5 thru 0x036fbb (0x9817 bytes) 000012 1740673570.594069 read 0x0088b1 thru 0x016f0a (0xe65a bytes) 000016 1740673570.594933 mapread 0x010cd1 thru 0x01a8da (0x9c0a bytes) 000017 1740673570.595495 read 0x00e9a6 thru 0x0158ab (0x6f06 bytes) 000018 1740673570.595879 mapwrite 0x03621a thru 0x03db5e (0x7945 bytes) truncating to largest ever: 0x28f1e 000019 1740673570.608668 trunc from 0x03f5b4 to 0x028f1d (0x16696 bytes) 000020 1740673570.612408 write 0x039ea7 thru 0x03ffff (0x6159 bytes) 000021 1740673570.612966 read 0x018c6e thru 0x021e56 (0x91e9 bytes) 000022 1740673570.613268 mapread 0x02a1cf thru 0x02eebf (0x4cf1 bytes) 000023 1740673570.615749 write 0x03238e thru 0x03bba7 (0x981a bytes) 000024 1740673570.616368 read 0x00a590 thru 0x017699 (0xd10a bytes) 000027 1740673570.617218 mapread 0x025b88 thru 0x032cd6 (0xd14f bytes) 000035 1740673570.619686 mapread 0x02a05e thru 0x02e5ef (0x4592 bytes) 000037 1740673570.620372 write 0x0101b6 thru 0x012db2 (0x2bfd bytes) 000039 1740673570.621130 read 0x03fb73 thru 0x03ffff (0x48d bytes) 000040 1740673570.621477 mapread 0x024688 thru 0x03289b (0xe214 bytes) 000042 1740673570.622040 write 0x000340 thru 0x0075e0 (0x72a1 bytes) 000043 1740673570.622525 read 0x0272e9 thru 0x02a189 (0x2ea1 bytes) 000044 1740673570.622766 read 0x02bcb8 thru 0x02f687 (0x39d0 bytes) 000046 1740673570.623310 mapread 0x03ebff thru 0x03ffff (0x1401 bytes) 000047 1740673570.623716 trunc from 0x040000 to 0x003b2d (0x3c4d2 bytes) 000048 1740673570.627638 mapread 0x000d8f thru 0x003b2d (0x2d9f bytes) 000049 1740673570.628096 trunc from 0x003b2e to 0x006c2a (0x30fd bytes) 000052 1740673570.632046 write 0x02f9ff thru 0x03ae2b (0xb42d bytes) 000053 1740673570.632970 write 0x0169ce thru 0x019fe5 (0x3618 bytes) 000054 1740673570.635684 mapread 0x01e5cf thru 0x02975c (0xb18e bytes) 000056 1740673570.636334 read 0x014d08 thru 0x019708 (0x4a01 bytes) 000057 1740673570.636590 trunc from 0x03ae2c to 0x0091ba (0x31c71 bytes) 000058 1740673570.640104 trunc from 0x0091bb to 0x00b243 (0x2089 bytes) 000059 1740673570.643324 read 0x00acd4 thru 0x00b243 (0x570 bytes) 000060 1740673570.644882 read 0x000147 thru 0x00b0cd (0xaf87 bytes) 000061 1740673570.645316 mapread 0x005797 thru 0x00b243 (0x5aad bytes) 000062 1740673570.645698 read 0x000fec thru 0x00a00f (0x9024 bytes) 000063 1740673570.645948 read 0x00559c thru 0x00b243 (0x5ca8 bytes) 000064 1740673570.646304 mapwrite 0x009c9d thru 0x0173fa (0xd75e bytes) 000066 1740673570.665605 trunc from 0x0173fb to 0x01a2b8 (0x2ebe bytes) 000070 1740673570.669638 mapread 0x017b52 thru 0x01a2b8 (0x2767 bytes) truncating to largest ever: 0x3cb8d 000071 1740673570.671560 trunc from 0x01a2b9 to 0x03cb8c (0x228d4 bytes) 000073 1740673570.676430 MIRROR_EXTEND to 2 mirrors 000074 1740673570.763338 read 0x02059a thru 0x028836 (0x829d bytes) 000076 1740673570.765529 mapwrite 0x0377e1 thru 0x03f2ed (0x7b0d bytes) 000077 1740673570.799033 read 0x023ac9 thru 0x0311a4 (0xd6dc bytes) 000079 1740673570.803638 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b7:0x0]: Invalid argument (22) 000081 1740673570.833243 trunc from 0x03f2ee to 0x02c404 (0x12ee9 bytes) 000084 1740673570.837190 mapwrite 0x004eee thru 0x00a7dd (0x58f0 bytes) 000085 1740673570.854369 write 0x02ac8c thru 0x037cdd (0xd052 bytes) 000086 1740673570.857572 write 0x0032ce thru 0x00e906 (0xb639 bytes) 000090 1740673570.859312 read 0x000393 thru 0x00a2df (0x9f4d bytes) 000095 1740673570.861806 mapwrite 0x00c6c4 thru 0x019153 (0xca90 bytes) 000096 1740673570.877271 mapwrite 0x00301e thru 0x0097d0 (0x67b3 bytes) 000098 1740673570.887402 trunc from 0x037cde to 0x026f8b (0x10d52 bytes) 000101 1740673570.891775 write 0x023359 thru 0x02df4a (0xabf2 bytes) 000102 1740673570.892476 read 0x01bcd6 thru 0x0287ab (0xcad6 bytes) 000104 1740673570.894885 read 0x00aee7 thru 0x01660c (0xb726 bytes) 000105 1740673570.895278 trunc from 0x02df4b to 0x01810b (0x15e3f bytes) 000107 1740673570.898919 trunc from 0x01810c to 0x01792d (0x7de bytes) 000108 1740673570.902108 trunc from 0x01792e to 0x01d247 (0x591a bytes) 000111 1740673570.906186 mapread 0x0176b4 thru 0x018626 (0xf73 bytes) 000113 1740673570.908867 mapwrite 0x03de65 thru 0x03ffff (0x219b bytes) 000115 1740673570.927859 read 0x0350db thru 0x0358a0 (0x7c6 bytes) 000116 1740673570.929303 MIRROR_EXTEND to 2 mirrors 000119 1740673571.024619 write 0x026062 thru 0x030288 (0xa227 bytes) 000121 1740673571.039542 read 0x004a74 thru 0x004f2c (0x4b9 bytes) 000123 1740673571.041125 trunc from 0x040000 to 0x0070ad (0x38f52 bytes) 000124 1740673571.048358 mapwrite 0x033035 thru 0x03cfb1 (0x9f7d bytes) 000125 1740673571.068549 write 0x027f12 thru 0x02e072 (0x6161 bytes) 000127 1740673571.071385 trunc from 0x03cfb2 to 0x0206dd (0x1c8d4 bytes) 000130 1740673571.075962 mapread 0x00ba7a thru 0x014cc3 (0x924a bytes) 000131 1740673571.080211 write 0x00fbaf thru 0x013f37 (0x4389 bytes) 000134 1740673571.082349 MIRROR_RESYNC 2 mirrors truncating to largest ever: 0x3f13c 000135 1740673571.162660 trunc from 0x0206de to 0x03f13b (0x1ea5e bytes) 000136 1740673571.179426 mapwrite 0x01d712 thru 0x02c376 (0xec65 bytes) 000138 1740673571.198895 mapwrite 0x0320e9 thru 0x03b651 (0x9569 bytes) 000139 1740673571.215210 write 0x03f07a thru 0x03ffff (0xf86 bytes) 000140 1740673571.215657 mapwrite 0x01d7e1 thru 0x025c9a (0x84ba bytes) 000141 1740673571.227200 trunc from 0x040000 to 0x01fad7 (0x20528 bytes) 000142 1740673571.232042 mapread 0x014853 thru 0x0186cc (0x3e7a bytes) 000143 1740673571.234369 read 0x0023ab thru 0x011b37 (0xf78d bytes) 000147 1740673571.237639 write 0x03bc78 thru 0x03ffff (0x4388 bytes) 000149 1740673571.238421 write 0x019bd2 thru 0x02890e (0xed3d bytes) 000151 1740673571.241594 MIRROR_EXTEND to 3 mirrors 000152 1740673571.332202 write 0x023215 thru 0x0295c8 (0x63b4 bytes) 000153 1740673571.348111 MIRROR_RESYNC 3 mirrors 000155 1740673571.451804 trunc from 0x040000 to 0x02641a (0x19be5 bytes) 000156 1740673571.469007 MIRROR_EXTEND to 4 mirrors 000158 1740673571.552660 mapwrite 0x0015a0 thru 0x00c062 (0xaac3 bytes) 000160 1740673571.597989 read 0x01384e thru 0x021b87 (0xe33a bytes) 000161 1740673571.600486 trunc from 0x02641b to 0x01d11b (0x92ff bytes) 000162 1740673571.607489 trunc from 0x01d11c to 0x017f71 (0x51aa bytes) 000164 1740673571.610779 mapwrite 0x006172 thru 0x012895 (0xc724 bytes) 000165 1740673571.626434 MIRROR_RESYNC 4 mirrors 000166 1740673571.741510 mapread 0x016049 thru 0x017f71 (0x1f29 bytes) 000168 1740673571.742888 mapread 0x00d415 thru 0x014a74 (0x7660 bytes) 000169 1740673571.745393 mapread 0x012c32 thru 0x017f71 (0x5340 bytes) 000170 1740673571.745643 write 0x004953 thru 0x0102ca (0xb978 bytes) 000172 1740673571.760798 write 0x03db2b thru 0x03fcca (0x21a0 bytes) 000174 1740673571.761603 trunc from 0x03fccb to 0x033ce8 (0xbfe2 bytes) 000176 1740673571.769330 write 0x0224e5 thru 0x02e8af (0xc3cb bytes) 000178 1740673571.772976 mapwrite 0x033aff thru 0x039d03 (0x6205 bytes) 000179 1740673571.793744 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b7:0x0]: Invalid argument (22) truncating to largest ever: 0x3f4c7 000181 1740673571.836853 trunc from 0x039d04 to 0x03f4c6 (0x57c3 bytes) 000182 1740673571.840457 mapread 0x02b543 thru 0x02ebb9 (0x3677 bytes) 000185 1740673571.844936 trunc from 0x03f4c7 to 0x01e975 (0x20b51 bytes) 000186 1740673571.850558 trunc from 0x01e976 to 0x020447 (0x1ad2 bytes) 000188 1740673571.854178 mapwrite 0x01c49f thru 0x02346a (0x6fcc bytes) 000191 1740673571.872902 MIRROR_EXTEND to 4 mirrors 000194 1740673571.957410 trunc from 0x02346b to 0x031df5 (0xe98b bytes) 000196 1740673571.975173 mapread 0x001613 thru 0x00e25c (0xcc4a bytes) 000198 1740673571.980582 write 0x0250cb thru 0x027455 (0x238b bytes) 000199 1740673571.982860 write 0x00c206 thru 0x01b0b5 (0xeeb0 bytes) 000200 1740673571.984480 mapwrite 0x02c201 thru 0x03afd0 (0xedd0 bytes) 000202 1740673572.007057 write 0x01a2eb thru 0x020dee (0x6b04 bytes) 000203 1740673572.007695 mapread 0x01507f thru 0x01bf9c (0x6f1e bytes) 000205 1740673572.008349 write 0x03c33a thru 0x03ffff (0x3cc6 bytes) 000206 1740673572.008866 write 0x029d7e thru 0x02a85c (0xadf bytes) 000207 1740673572.009332 trunc from 0x040000 to 0x0061ac (0x39e53 bytes) 000208 1740673572.013368 mapread 0x000106 thru 0x0061ac (0x60a7 bytes) 000210 1740673572.015173 mapwrite 0x02dee3 thru 0x02e837 (0x955 bytes) 000212 1740673572.031824 trunc from 0x02e838 to 0x039117 (0xa8e0 bytes) 000213 1740673572.035493 write 0x030744 thru 0x03b9bb (0xb278 bytes) 000216 1740673572.040381 MIRROR_RESYNC 4 mirrors 000217 1740673572.174333 mapread 0x006034 thru 0x00bcba (0x5c87 bytes) 000222 1740673572.178735 write 0x0109ec thru 0x010e73 (0x488 bytes) 000224 1740673572.194038 mapwrite 0x026d5c thru 0x027061 (0x306 bytes) 000225 1740673572.210311 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b7:0x0]: Invalid argument (22) 000226 1740673572.254190 mapwrite 0x0212b1 thru 0x02f3a8 (0xe0f8 bytes) 000230 1740673572.276237 MIRROR_EXTEND to 4 mirrors 000231 1740673572.369337 mapwrite 0x004bf9 thru 0x010601 (0xba09 bytes) 000232 1740673572.414455 trunc from 0x03b9bc to 0x027125 (0x14896 bytes) 000234 1740673572.422389 write 0x0028e6 thru 0x00f5ed (0xcd08 bytes) 000235 1740673572.424018 MIRROR_RESYNC 4 mirrors 000236 1740673572.547373 write 0x015543 thru 0x01f4c5 (0x9f83 bytes) 000240 1740673572.562419 write 0x0216b4 thru 0x02ba66 (0xa3b3 bytes) 000244 1740673572.565812 write 0x03e3e4 thru 0x03ffff (0x1c1c bytes) 000250 1740673572.568216 mapwrite 0x0164b0 thru 0x02619b (0xfcec bytes) 000251 1740673572.583285 read 0x013943 thru 0x017bd2 (0x4290 bytes) 000252 1740673572.584985 mapwrite 0x03207f thru 0x03fcdd (0xdc5f bytes) 000253 1740673572.602322 trunc from 0x040000 to 0x017dae (0x28251 bytes) 000254 1740673572.610879 mapread 0x00b56d thru 0x017dae (0xc842 bytes) 000256 1740673572.614322 MIRROR_EXTEND to 5 mirrors 000258 1740673572.699322 write 0x024029 thru 0x032119 (0xe0f1 bytes) 000259 1740673572.713585 mapread 0x009ca1 thru 0x0197cc (0xfb2c bytes) 000262 1740673572.720710 read 0x029330 thru 0x032119 (0x8dea bytes) 000263 1740673572.721162 trunc from 0x03211a to 0x007175 (0x2afa4 bytes) 000264 1740673572.729529 mapwrite 0x033b59 thru 0x03d1ad (0x9655 bytes) 000268 1740673572.751173 read 0x00a77d thru 0x014cca (0xa54e bytes) 000270 1740673572.754749 read 0x024498 thru 0x024674 (0x1dd bytes) 000272 1740673572.756560 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b7:0x0]: Invalid argument (22) 000274 1740673572.799936 write 0x01a44b thru 0x022c37 (0x87ed bytes) 000276 1740673572.802893 trunc from 0x03d1ae to 0x005b37 (0x37676 bytes) 000277 1740673572.806363 trunc from 0x005b38 to 0x014d17 (0xf1e0 bytes) 000280 1740673572.812224 MIRROR_EXTEND to 5 mirrors 000281 1740673572.900510 read 0x00dbae thru 0x014d17 (0x716a bytes) 000283 1740673572.904398 read 0x005ef7 thru 0x01443c (0xe546 bytes) 000284 1740673572.907302 write 0x01c4b5 thru 0x02b86d (0xf3b9 bytes) 000285 1740673572.921549 write 0x03393a thru 0x035ae2 (0x21a9 bytes) 000289 1740673572.922872 trunc from 0x035ae3 to 0x00f43c (0x266a6 bytes) 000290 1740673572.930481 write 0x015eb4 thru 0x01b328 (0x5475 bytes) 000291 1740673572.932136 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b7:0x0]: Invalid argument (22) 000293 1740673572.976754 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b7:0x0]: Invalid argument (22) 000294 1740673573.019842 mapwrite 0x0101a1 thru 0x01b662 (0xb4c2 bytes) 000295 1740673573.044956 mapread 0x014307 thru 0x01b662 (0x735c bytes) 000297 1740673573.047449 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b7:0x0]: Invalid argument (22) 000298 1740673573.090645 mapread 0x018302 thru 0x018890 (0x58f bytes) 000299 1740673573.093490 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b7:0x0]: Invalid argument (22) 000300 1740673573.121540 read 0x0152be thru 0x016a79 (0x17bc bytes) 000301 1740673573.123369 mapwrite 0x007a76 thru 0x01714a (0xf6d5 bytes) 000304 1740673573.143110 mapread 0x003f35 thru 0x013d24 (0xfdf0 bytes) 000305 1740673573.145160 mapwrite 0x039c16 thru 0x03b6ad (0x1a98 bytes) 000306 1740673573.163269 read 0x03a04c thru 0x03b6ad (0x1662 bytes) 000308 1740673573.163745 read 0x01a7c7 thru 0x01b0e5 (0x91f bytes) 000309 1740673573.163995 mapwrite 0x01d8aa thru 0x023c45 (0x639c bytes) 000310 1740673573.178844 read 0x027434 thru 0x0290cc (0x1c99 bytes) 000313 1740673573.179582 write 0x02ba57 thru 0x03a2bc (0xe866 bytes) 000316 1740673573.181013 write 0x025149 thru 0x027488 (0x2340 bytes) 000317 1740673573.181427 trunc from 0x03b6ae to 0x01488a (0x26e23 bytes) 000318 1740673573.185720 mapread 0x010ccb thru 0x01488a (0x3bc0 bytes) 000319 1740673573.186286 mapread 0x00020e thru 0x0039d8 (0x37cb bytes) 000321 1740673573.187122 trunc from 0x01488b to 0x021383 (0xcaf9 bytes) 000323 1740673573.190983 mapread 0x0070e6 thru 0x00e3a7 (0x72c2 bytes) 000325 1740673573.191754 read 0x00b951 thru 0x01206f (0x671f bytes) 000326 1740673573.192105 mapread 0x01143f thru 0x01ecbd (0xd87f bytes) 000327 1740673573.194897 mapwrite 0x026edf thru 0x02c404 (0x5526 bytes) 000328 1740673573.214011 read 0x0188ac thru 0x024098 (0xb7ed bytes) 000329 1740673573.214391 read 0x019a22 thru 0x022330 (0x890f bytes) 000330 1740673573.214651 read 0x0117a1 thru 0x011ad0 (0x330 bytes) 000331 1740673573.214933 write 0x011fad thru 0x0197c4 (0x7818 bytes) 000333 1740673573.215707 trunc from 0x02c405 to 0x020848 (0xbbbc bytes) 000335 1740673573.219542 trunc from 0x020849 to 0x00415d (0x1c6eb bytes) 000336 1740673573.223132 mapwrite 0x00bc55 thru 0x00d868 (0x1c14 bytes) 000339 1740673573.241332 mapwrite 0x016128 thru 0x01f7e3 (0x96bc bytes) 000344 1740673573.260926 write 0x0356c7 thru 0x03ffff (0xa939 bytes) 000345 1740673573.261743 write 0x01e6e2 thru 0x028be5 (0xa504 bytes) 000346 1740673573.263634 trunc from 0x040000 to 0x03225a (0xdda5 bytes) 000347 1740673573.267429 mapread 0x02a6fb thru 0x03225a (0x7b60 bytes) 000348 1740673573.269828 mapwrite 0x0347a3 thru 0x0380aa (0x3908 bytes) 000349 1740673573.288821 read 0x01c54f thru 0x022efb (0x69ad bytes) 000350 1740673573.289265 trunc from 0x0380ab to 0x007c7f (0x3042b bytes) 000351 1740673573.293279 trunc from 0x007c80 to 0x0380c4 (0x30445 bytes) 000352 1740673573.296790 write 0x02e73b thru 0x03c506 (0xddcc bytes) 000353 1740673573.300283 trunc from 0x03c507 to 0x02bfd1 (0x10535 bytes) 000354 1740673573.303718 read 0x025d81 thru 0x02bfd1 (0x6251 bytes) 000360 1740673573.304983 mapwrite 0x010bad thru 0x01fb94 (0xefe8 bytes) 000362 1740673573.319804 read 0x022877 thru 0x02bfd1 (0x975b bytes) 000363 1740673573.320209 trunc from 0x02bfd2 to 0x02d992 (0x19c1 bytes) 000364 1740673573.323585 write 0x013f9b thru 0x0146b8 (0x71e bytes) 000365 1740673573.324187 trunc from 0x02d993 to 0x00140c (0x2c586 bytes) 000366 1740673573.327707 mapread 0x000751 thru 0x00140c (0xcbc bytes) 000367 1740673573.328123 trunc from 0x00140d to 0x004e56 (0x3a4a bytes) 000373 1740673573.332461 read 0x003a0b thru 0x004e56 (0x144c bytes) 000374 1740673573.334022 mapread 0x00165b thru 0x004e56 (0x37fc bytes) 000376 1740673573.334605 trunc from 0x004e57 to 0x000132 (0x4d24 bytes) 000377 1740673573.339087 MIRROR_EXTEND to 2 mirrors 000378 1740673573.416023 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b7:0x0]: Invalid argument (22) 000381 1740673573.444710 trunc from 0x000133 to 0x00a06a (0x9f38 bytes) 000382 1740673573.453191 write 0x005e7a thru 0x00b5fd (0x5784 bytes) 000385 1740673573.455206 write 0x01eae1 thru 0x026eeb (0x840b bytes) 000386 1740673573.455989 read 0x0061fa thru 0x01538a (0xf191 bytes) 000387 1740673573.458226 write 0x014be1 thru 0x01c6ba (0x7ada bytes) 000390 1740673573.460447 write 0x03580e thru 0x03ffff (0xa7f2 bytes) 000393 1740673573.461906 write 0x017b1e thru 0x023d41 (0xc224 bytes) 000395 1740673573.462739 write 0x01f257 thru 0x02d31f (0xe0c9 bytes) 000396 1740673573.464498 mapread 0x033feb thru 0x0382d7 (0x42ed bytes) 000397 1740673573.466853 mapread 0x0347cc thru 0x03ffff (0xb834 bytes) 000398 1740673573.467511 mapwrite 0x01b49b thru 0x01d454 (0x1fba bytes) 000399 1740673573.481353 read 0x02e0d5 thru 0x034dd6 (0x6d02 bytes) 000401 1740673573.481928 trunc from 0x040000 to 0x016306 (0x29cf9 bytes) 000404 1740673573.486385 read 0x00a150 thru 0x016306 (0xc1b7 bytes) 000405 1740673573.486721 read 0x0055db thru 0x0065f4 (0x101a bytes) 000407 1740673573.487315 mapwrite 0x000250 thru 0x005c6a (0x5a1b bytes) 000408 1740673573.502926 MIRROR_EXTEND to 2 mirrors 000410 1740673573.589384 mapread 0x001fc2 thru 0x004d09 (0x2d48 bytes) 000411 1740673573.593469 mapwrite 0x02ff98 thru 0x038191 (0x81fa bytes) 000412 1740673573.628615 trunc from 0x038192 to 0x02a128 (0xe069 bytes) 000414 1740673573.633574 mapread 0x022c97 thru 0x0299b0 (0x6d1a bytes) 000416 1740673573.636697 mapread 0x021a5d thru 0x0278f8 (0x5e9c bytes) 000418 1740673573.637586 trunc from 0x02a129 to 0x018073 (0x120b5 bytes) 000419 1740673573.640905 trunc from 0x018074 to 0x02fb0a (0x17a97 bytes) 000420 1740673573.644006 trunc from 0x02fb0b to 0x02299b (0xd16f bytes) 000422 1740673573.647815 mapread 0x016e20 thru 0x01d79f (0x6980 bytes) 000425 1740673573.650923 mapread 0x009de6 thru 0x00ba4e (0x1c69 bytes) 000426 1740673573.654358 MIRROR_RESYNC 2 mirrors 000427 1740673573.737439 trunc from 0x02299c to 0x008081 (0x1a91a bytes) 000429 1740673573.755675 mapread 0x00101e thru 0x008081 (0x7064 bytes) 000433 1740673573.762281 MIRROR_EXTEND to 3 mirrors 000435 1740673573.843751 MIRROR_EXTEND to 4 mirrors 000436 1740673573.926456 trunc from 0x008082 to 0x0309f4 (0x28973 bytes) 000437 1740673573.945473 trunc from 0x0309f5 to 0x01d7d2 (0x13222 bytes) 000438 1740673573.950279 trunc from 0x01d7d3 to 0x01bdb5 (0x1a1d bytes) 000441 1740673573.954168 mapwrite 0x03b85b thru 0x03ffff (0x47a5 bytes) 000443 1740673573.976021 mapwrite 0x03113c thru 0x03fed1 (0xed96 bytes) 000444 1740673573.994355 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b7:0x0]: Invalid argument (22) 000445 1740673574.038935 mapwrite 0x01d6e1 thru 0x020d25 (0x3645 bytes) 000447 1740673574.055204 mapread 0x001f34 thru 0x00a6c0 (0x878d bytes) 000448 1740673574.058824 write 0x018833 thru 0x0228ff (0xa0cd bytes) 000450 1740673574.059636 trunc from 0x040000 to 0x0087fb (0x37804 bytes) 000452 1740673574.064058 trunc from 0x0087fc to 0x022e80 (0x1a685 bytes) 000453 1740673574.067390 trunc from 0x022e81 to 0x0016ca (0x217b6 bytes) 000454 1740673574.070653 read 0x000223 thru 0x0016ca (0x14a8 bytes) 000455 1740673574.072165 mapread 0x000d7e thru 0x0016ca (0x94d bytes) 000457 1740673574.073885 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b7:0x0]: Invalid argument (22) 000459 1740673574.116895 mapread 0x000ffc thru 0x0016ca (0x6cf bytes) 000461 1740673574.119776 trunc from 0x0016cb to 0x024d14 (0x2364a bytes) 000462 1740673574.123379 write 0x03e4a0 thru 0x03ffff (0x1b60 bytes) 000463 1740673574.124009 mapwrite 0x01c9b4 thru 0x0237c6 (0x6e13 bytes) 000464 1740673574.139745 write 0x03aa7e thru 0x03ffff (0x5582 bytes) 000467 1740673574.143111 mapwrite 0x0249b4 thru 0x028e59 (0x44a6 bytes) 000469 1740673574.156245 mapread 0x02ec22 thru 0x034ee7 (0x62c6 bytes) 000470 1740673574.159694 MIRROR_EXTEND to 3 mirrors 000471 1740673574.246186 write 0x0234f6 thru 0x0316cf (0xe1da bytes) 000475 1740673574.261851 write 0x02924f thru 0x02e317 (0x50c9 bytes) 000477 1740673574.262840 trunc from 0x040000 to 0x02cefb (0x13104 bytes) 000478 1740673574.270771 write 0x035a4d thru 0x03a0a9 (0x465d bytes) 000479 1740673574.271475 mapwrite 0x02284b thru 0x02d7c0 (0xaf76 bytes) 000480 1740673574.288802 mapread 0x0160ca thru 0x01a326 (0x425d bytes) 000482 1740673574.291608 trunc from 0x03a0aa to 0x01d964 (0x1c745 bytes) 000483 1740673574.295390 read 0x00bb21 thru 0x00ca0e (0xeee bytes) 000484 1740673574.296870 read 0x005e8e thru 0x00f5b5 (0x9728 bytes) 000488 1740673574.300436 trunc from 0x01d965 to 0x012d3e (0xac26 bytes) 000490 1740673574.305998 MIRROR_RESYNC 3 mirrors 000494 1740673574.405073 write 0x01e6f9 thru 0x0204c9 (0x1dd1 bytes) 000495 1740673574.417673 mapread 0x0021bc thru 0x00a49f (0x82e4 bytes) 000497 1740673574.422387 mapread 0x017a2b thru 0x01dd7b (0x6351 bytes) 000500 1740673574.426409 MIRROR_RESYNC 3 mirrors 000502 1740673574.522558 write 0x00ad81 thru 0x0116b0 (0x6930 bytes) 000503 1740673574.536075 write 0x003320 thru 0x008c8d (0x596e bytes) 000504 1740673574.538444 mapwrite 0x031631 thru 0x0358ca (0x429a bytes) 000505 1740673574.560883 read 0x004af4 thru 0x005809 (0xd16 bytes) 000506 1740673574.561226 mapread 0x018e74 thru 0x01d61c (0x47a9 bytes) 000507 1740673574.563665 mapwrite 0x012c22 thru 0x019a99 (0x6e78 bytes) 000508 1740673574.574840 write 0x008ff0 thru 0x00dda3 (0x4db4 bytes) 000509 1740673574.575383 write 0x00e295 thru 0x01168c (0x33f8 bytes) 000514 1740673574.577719 MIRROR_EXTEND to 4 mirrors 000523 1740673574.688786 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b7:0x0]: Invalid argument (22) 000524 1740673574.737468 trunc from 0x0358cb to 0x039d33 (0x4469 bytes) 000526 1740673574.754232 MIRROR_EXTEND to 4 mirrors 000528 1740673574.841944 read 0x02b095 thru 0x02cd0e (0x1c7a bytes) 000529 1740673574.845110 MIRROR_EXTEND to 5 mirrors 000531 1740673574.937861 write 0x01bf7a thru 0x022458 (0x64df bytes) 000534 1740673574.953796 mapread 0x028039 thru 0x029c32 (0x1bfa bytes) 000535 1740673574.956253 trunc from 0x039d34 to 0x00bdd0 (0x2df63 bytes) 000538 1740673574.964244 trunc from 0x00bdd1 to 0x035850 (0x29a80 bytes) 000540 1740673574.967815 mapwrite 0x00ffee thru 0x0188e1 (0x88f4 bytes) 000541 1740673574.985319 trunc from 0x035851 to 0x02499b (0x10eb5 bytes) 000542 1740673574.989091 trunc from 0x02499c to 0x02af2e (0x6593 bytes) 000547 1740673574.993432 mapwrite 0x034c35 thru 0x03982d (0x4bf9 bytes) 000548 1740673575.010948 mapwrite 0x01da02 thru 0x028e5e (0xb45d bytes) 000551 1740673575.026333 read 0x02fc1d thru 0x0316ed (0x1ad1 bytes) 000552 1740673575.026635 read 0x032c9a thru 0x036e62 (0x41c9 bytes) 000553 1740673575.026904 mapwrite 0x008692 thru 0x0113e4 (0x8d53 bytes) 000554 1740673575.038921 read 0x0280d3 thru 0x0357f5 (0xd723 bytes) 000555 1740673575.041330 trunc from 0x03982e to 0x020dba (0x18a73 bytes) 000556 1740673575.046566 MIRROR_RESYNC 5 mirrors 000557 1740673575.198333 mapread 0x003b83 thru 0x0117f3 (0xdc71 bytes) 000560 1740673575.204686 write 0x014f53 thru 0x021fcf (0xd07d bytes) 000562 1740673575.221608 MIRROR_RESYNC 5 mirrors 000566 1740673575.369494 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b7:0x0]: Invalid argument (22) 000568 1740673575.427099 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b7:0x0]: Invalid argument (22) 000569 1740673575.480293 read 0x011632 thru 0x012fec (0x19bb bytes) 000571 1740673575.482120 write 0x001426 thru 0x00f42d (0xe008 bytes) 000572 1740673575.497977 mapread 0x0191e9 thru 0x01b567 (0x237f bytes) 000576 1740673575.501396 write 0x02a103 thru 0x02c1d1 (0x20cf bytes) 000580 1740673575.502711 read 0x02a03a thru 0x02c1d1 (0x2198 bytes) 000581 1740673575.503035 trunc from 0x02c1d2 to 0x038444 (0xc273 bytes) 000582 1740673575.510818 mapwrite 0x00931e thru 0x00a156 (0xe39 bytes) 000584 1740673575.524370 read 0x00c7a5 thru 0x01b424 (0xec80 bytes) 000585 1740673575.527394 write 0x02d76e thru 0x02d8e4 (0x177 bytes) 000587 1740673575.528255 trunc from 0x038445 to 0x00a533 (0x2df11 bytes) 000589 1740673575.532309 trunc from 0x00a534 to 0x0272f5 (0x1cdc2 bytes) 000590 1740673575.536574 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b7:0x0]: Invalid argument (22) 000595 1740673575.582052 mapwrite 0x008276 thru 0x014e60 (0xcbeb bytes) 000596 1740673575.604239 trunc from 0x0272f6 to 0x031853 (0xa55e bytes) 000597 1740673575.608830 mapwrite 0x020e9c thru 0x02aaef (0x9c54 bytes) 000598 1740673575.628206 mapwrite 0x001ee9 thru 0x007db7 (0x5ecf bytes) 000601 1740673575.643418 MIRROR_RESYNC 2 mirrors 000602 1740673575.735319 mapwrite 0x026bc6 thru 0x02d457 (0x6892 bytes) 000603 1740673575.777583 trunc from 0x031854 to 0x017670 (0x1a1e3 bytes) 000605 1740673575.786548 trunc from 0x017671 to 0x03485e (0x1d1ee bytes) 000606 1740673575.790340 mapread 0x01c0e6 thru 0x0203b9 (0x42d4 bytes) 000607 1740673575.793253 trunc from 0x03485f to 0x011cd3 (0x22b8b bytes) 000609 1740673575.798001 MIRROR_EXTEND to 3 mirrors 000612 1740673575.878261 write 0x0285fd thru 0x031d65 (0x9769 bytes) 000613 1740673575.891798 read 0x021aeb thru 0x02e7f4 (0xcd0a bytes) 000614 1740673575.895163 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b7:0x0]: Invalid argument (22) 000617 1740673575.942161 write 0x002a4b thru 0x00e2ab (0xb861 bytes) 000620 1740673575.945360 trunc from 0x031d66 to 0x007ef8 (0x29e6d bytes) 000621 1740673575.952415 write 0x0168cf thru 0x01f2cf (0x8a01 bytes) 000624 1740673575.953742 trunc from 0x01f2d0 to 0x00a28e (0x15041 bytes) 000625 1740673575.957356 trunc from 0x00a28f to 0x001ceb (0x85a3 bytes) 000626 1740673575.960736 trunc from 0x001cec to 0x03c15a (0x3a46f bytes) 000627 1740673575.964168 write 0x00db34 thru 0x01d18e (0xf65b bytes) 000630 1740673575.968421 mapread 0x015013 thru 0x01b12a (0x6118 bytes) 000631 1740673575.968846 read 0x014b03 thru 0x015f69 (0x1467 bytes) 000632 1740673575.969173 mapwrite 0x00cc2a thru 0x0152c8 (0x869f bytes) 000634 1740673575.985122 MIRROR_RESYNC 2 mirrors 000635 1740673576.073863 mapwrite 0x021e90 thru 0x029a87 (0x7bf8 bytes) 000636 1740673576.116743 mapread 0x00ea86 thru 0x01b564 (0xcadf bytes) 000637 1740673576.120707 mapread 0x02f247 thru 0x03c15a (0xcf14 bytes) 000639 1740673576.125111 trunc from 0x03c15b to 0x0164f4 (0x25c66 bytes) 000641 1740673576.130878 MIRROR_RESYNC 2 mirrors 000642 1740673576.219560 write 0x013fcf thru 0x01bfa8 (0x7fda bytes) 000643 1740673576.232300 mapread 0x010e8b thru 0x013c46 (0x2dbc bytes) 000644 1740673576.235633 mapread 0x018697 thru 0x01bef0 (0x385a bytes) 000647 1740673576.237745 MIRROR_RESYNC 2 mirrors 000648 1740673576.316654 write 0x03f3be thru 0x03ffff (0xc42 bytes) 000650 1740673576.330169 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b7:0x0]: Invalid argument (22) 000651 1740673576.361182 read 0x0173ed thru 0x01a506 (0x311a bytes) 000652 1740673576.363979 MIRROR_EXTEND to 2 mirrors 000653 1740673576.453692 write 0x03be22 thru 0x03f5d7 (0x37b6 bytes) 000656 1740673576.469049 mapwrite 0x0212dc thru 0x02c371 (0xb096 bytes) 000657 1740673576.488154 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b7:0x0]: Invalid argument (22) 000658 1740673576.515994 mapread 0x03bf24 thru 0x03ffff (0x40dc bytes) 000659 1740673576.520549 trunc from 0x040000 to 0x025a4a (0x1a5b5 bytes) 000660 1740673576.527828 trunc from 0x025a4b to 0x024fa1 (0xaa9 bytes) 000662 1740673576.531714 write 0x03f442 thru 0x03ffff (0xbbe bytes) 000664 1740673576.532660 mapwrite 0x01f64d thru 0x026c3b (0x75ef bytes) 000666 1740673576.548060 read 0x03d518 thru 0x03ffff (0x2ae8 bytes) 000669 1740673576.549961 write 0x00aab9 thru 0x013f75 (0x94bd bytes) 000670 1740673576.553347 mapwrite 0x034e90 thru 0x03972c (0x489d bytes) 000671 1740673576.568458 mapread 0x014cef thru 0x0154af (0x7c1 bytes) 000672 1740673576.570237 write 0x010046 thru 0x016771 (0x672c bytes) 000673 1740673576.572136 MIRROR_EXTEND to 2 mirrors 000675 1740673576.661174 read 0x02ac85 thru 0x03214e (0x74ca bytes) 000677 1740673576.663541 read 0x00be1c thru 0x011c3a (0x5e1f bytes) 000678 1740673576.667166 MIRROR_RESYNC 2 mirrors 000679 1740673576.701632 write 0x022ecb thru 0x028fa0 (0x60d6 bytes) 000680 1740673576.717586 mapread 0x03b547 thru 0x03ffff (0x4ab9 bytes) 000681 1740673576.719764 read 0x015e77 thru 0x024179 (0xe303 bytes) 000685 1740673576.722780 write 0x013f23 thru 0x01cf44 (0x9022 bytes) 000687 1740673576.725491 mapread 0x00bc54 thru 0x014239 (0x85e6 bytes) 000691 1740673576.728203 read 0x01d6b2 thru 0x02a931 (0xd280 bytes) 000692 1740673576.731796 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b7:0x0]: Invalid argument (22) 000693 1740673576.763339 write 0x00292b thru 0x012398 (0xfa6e bytes) 000694 1740673576.766029 trunc from 0x040000 to 0x01a895 (0x2576a bytes) 000695 1740673576.772994 trunc from 0x01a896 to 0x006b77 (0x13d1e bytes) 000696 1740673576.776474 read 0x0053dd thru 0x006b77 (0x179b bytes) 000698 1740673576.778257 MIRROR_EXTEND to 2 mirrors 000699 1740673576.867223 trunc from 0x006b78 to 0x02eb93 (0x2801c bytes) 000700 1740673576.885412 trunc from 0x02eb94 to 0x0213e5 (0xd7ae bytes) 000705 1740673576.891782 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b7:0x0]: Invalid argument (22) 000707 1740673576.919833 trunc from 0x0213e6 to 0x014bbf (0xc826 bytes) 000711 1740673576.924661 mapwrite 0x00c344 thru 0x01a683 (0xe340 bytes) 000712 1740673576.946628 trunc from 0x01a684 to 0x03b6d7 (0x21054 bytes) 000714 1740673576.949915 mapread 0x009ad9 thru 0x00d22f (0x3757 bytes) 000715 1740673576.950465 write 0x03cef9 thru 0x03ffff (0x3107 bytes) 000719 1740673576.951956 mapread 0x01bfe8 thru 0x027832 (0xb84b bytes) 000720 1740673576.957264 MIRROR_EXTEND to 2 mirrors 000721 1740673577.055161 trunc from 0x040000 to 0x0220b0 (0x1df4f bytes) 000722 1740673577.070477 mapread 0x00dbcb thru 0x00df22 (0x358 bytes) 000724 1740673577.072457 mapread 0x00f987 thru 0x01dccf (0xe349 bytes) 000726 1740673577.078596 trunc from 0x0220b1 to 0x02a84b (0x879b bytes) 000727 1740673577.083052 write 0x0350d7 thru 0x03988b (0x47b5 bytes) 000728 1740673577.083727 write 0x006b66 thru 0x013bbb (0xd056 bytes) 000729 1740673577.086171 write 0x01621a thru 0x01a94b (0x4732 bytes) 000731 1740673577.086842 mapread 0x002e2a thru 0x007898 (0x4a6f bytes) 000733 1740673577.087517 read 0x01daa6 thru 0x027f6b (0xa4c6 bytes) 000734 1740673577.089936 trunc from 0x03988c to 0x029ad7 (0xfdb4 bytes) 000735 1740673577.093741 read 0x0087c3 thru 0x008c09 (0x447 bytes) 000736 1740673577.094254 mapwrite 0x039798 thru 0x03c9d7 (0x3240 bytes) 000737 1740673577.116174 trunc from 0x03c9d8 to 0x03d2fe (0x927 bytes) 000738 1740673577.119936 trunc from 0x03d2ff to 0x003a72 (0x3988c bytes) 000739 1740673577.123562 read 0x001ddc thru 0x003a72 (0x1c97 bytes) 000740 1740673577.123847 mapwrite 0x038a4d thru 0x03ffff (0x75b3 bytes) 000741 1740673577.144702 trunc from 0x040000 to 0x037d94 (0x826b bytes) 000742 1740673577.148522 trunc from 0x037d95 to 0x022d60 (0x15034 bytes) 000743 1740673577.151792 mapread 0x002e6f thru 0x010e70 (0xe002 bytes) 000744 1740673577.152616 mapread 0x014444 thru 0x01a770 (0x632d bytes) 000746 1740673577.154797 MIRROR_RESYNC 2 mirrors 000747 1740673577.242979 read 0x017a92 thru 0x01c907 (0x4e76 bytes) 000749 1740673577.246555 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b7:0x0]: Invalid argument (22) 000751 1740673577.275241 trunc from 0x022d61 to 0x00185b (0x21505 bytes) 000752 1740673577.284371 trunc from 0x00185c to 0x03322b (0x319d0 bytes) 000753 1740673577.287584 read 0x014865 thru 0x023467 (0xec03 bytes) 000756 1740673577.290895 write 0x0122eb thru 0x01e636 (0xc34c bytes) 000757 1740673577.292475 mapwrite 0x006abe thru 0x0102a7 (0x97ea bytes) 000758 1740673577.311021 write 0x0340b8 thru 0x03fe0b (0xbd54 bytes) 000759 1740673577.311730 trunc from 0x03fe0c to 0x03980e (0x65fd bytes) 000761 1740673577.316177 mapread 0x011ebc thru 0x0151c2 (0x3307 bytes) 000764 1740673577.318664 write 0x012bd9 thru 0x01f9f2 (0xce1a bytes) 000766 1740673577.320938 MIRROR_EXTEND to 2 mirrors 000769 1740673577.426225 mapread 0x024954 thru 0x02c70b (0x7db8 bytes) 000770 1740673577.431089 trunc from 0x03980f to 0x023399 (0x16475 bytes) 000771 1740673577.448638 mapread 0x00b475 thru 0x00da9a (0x2626 bytes) 000772 1740673577.451241 read 0x01039e thru 0x017098 (0x6cfb bytes) 000773 1740673577.452876 trunc from 0x02339a to 0x01cdb7 (0x65e2 bytes) 000775 1740673577.457908 write 0x025180 thru 0x030689 (0xb50a bytes) 000780 1740673577.461307 MIRROR_EXTEND to 3 mirrors 000781 1740673577.550726 read 0x00a431 thru 0x0196c6 (0xf296 bytes) 000784 1740673577.553490 trunc from 0x03068a to 0x0116bd (0x1efcc bytes) 000785 1740673577.576024 trunc from 0x0116be to 0x002279 (0xf444 bytes) 000788 1740673577.581238 trunc from 0x00227a to 0x03ef28 (0x3ccaf bytes) 000789 1740673577.584605 mapwrite 0x02250b thru 0x023f2f (0x1a25 bytes) 000791 1740673577.600242 mapwrite 0x031a04 thru 0x033733 (0x1d30 bytes) 000792 1740673577.613761 read 0x013352 thru 0x01ec4b (0xb8fa bytes) 000793 1740673577.617810 read 0x0147c3 thru 0x020d88 (0xc5c6 bytes) 000794 1740673577.618280 mapread 0x01367a thru 0x01bc41 (0x85c8 bytes) 000795 1740673577.618963 mapwrite 0x03f662 thru 0x03ffff (0x99e bytes) 000797 1740673577.636574 trunc from 0x040000 to 0x030e5a (0xf1a5 bytes) 000799 1740673577.641042 trunc from 0x030e5b to 0x0159b2 (0x1b4a8 bytes) 000800 1740673577.644935 read 0x000ab1 thru 0x00ed0e (0xe25e bytes) 000802 1740673577.645621 mapwrite 0x0196b9 thru 0x022b3d (0x9485 bytes) 000803 1740673577.665385 trunc from 0x022b3e to 0x031b05 (0xefc8 bytes) 000804 1740673577.668776 read 0x02ee68 thru 0x031b05 (0x2c9e bytes) 000805 1740673577.671504 trunc from 0x031b06 to 0x0185ab (0x1955a bytes) 000810 1740673577.677518 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b7:0x0]: Invalid argument (22) 000811 1740673577.723830 mapread 0x009b40 thru 0x017435 (0xd8f6 bytes) 000812 1740673577.730521 trunc from 0x0185ac to 0x00aed0 (0xd6db bytes) 000813 1740673577.734382 mapwrite 0x0293b3 thru 0x02f494 (0x60e2 bytes) 000814 1740673577.752662 mapwrite 0x02a726 thru 0x033eca (0x97a5 bytes) 000815 1740673577.771129 read 0x01f168 thru 0x027acd (0x8966 bytes) 000816 1740673577.774216 MIRROR_EXTEND to 3 mirrors 000819 1740673577.870407 mapwrite 0x025a54 thru 0x02cb9e (0x714b bytes) 000823 1740673577.913563 read 0x02870c thru 0x028ce4 (0x5d9 bytes) 000824 1740673577.914886 MIRROR_RESYNC 3 mirrors 000826 1740673578.014860 mapread 0x0323df thru 0x033eca (0x1aec bytes) 000829 1740673578.016506 mapread 0x012e16 thru 0x01f005 (0xc1f0 bytes) 000830 1740673578.020942 mapread 0x029de7 thru 0x02a580 (0x79a bytes) 000831 1740673578.022709 write 0x03d49e thru 0x03ffff (0x2b62 bytes) 000832 1740673578.035994 MIRROR_EXTEND to 4 mirrors 000835 1740673578.143198 MIRROR_EXTEND to 5 mirrors 000841 1740673578.244250 read 0x02cf60 thru 0x033092 (0x6133 bytes) 000842 1740673578.246424 read 0x023163 thru 0x024687 (0x1525 bytes) 000843 1740673578.249423 mapwrite 0x0173eb thru 0x022d85 (0xb99b bytes) 000845 1740673578.302891 mapread 0x0117ab thru 0x014a16 (0x326c bytes) 000847 1740673578.307589 mapwrite 0x0294ff thru 0x037f7b (0xea7d bytes) 000848 1740673578.336180 trunc from 0x040000 to 0x01db36 (0x224c9 bytes) 000849 1740673578.341283 mapwrite 0x024153 thru 0x024973 (0x821 bytes) 000850 1740673578.358545 trunc from 0x024974 to 0x006b1c (0x1de57 bytes) 000851 1740673578.361895 write 0x01671c thru 0x01bdd2 (0x56b7 bytes) 000852 1740673578.362528 read 0x00cde8 thru 0x016982 (0x9b9b bytes) 000853 1740673578.365083 trunc from 0x01bdd3 to 0x03e3c9 (0x225f7 bytes) 000855 1740673578.368697 mapwrite 0x0218e8 thru 0x028ed8 (0x75f1 bytes) 000856 1740673578.384834 mapwrite 0x03b353 thru 0x03ffff (0x4cad bytes) 000857 1740673578.401897 read 0x0328b5 thru 0x039a3f (0x718b bytes) 000858 1740673578.402317 mapwrite 0x02fea8 thru 0x032e1a (0x2f73 bytes) 000859 1740673578.412211 mapread 0x027736 thru 0x029e65 (0x2730 bytes) 000862 1740673578.413337 mapwrite 0x024eb6 thru 0x02e24a (0x9395 bytes) 000865 1740673578.426445 write 0x03bfc0 thru 0x03d10b (0x114c bytes) 000869 1740673578.427695 write 0x034568 thru 0x03513a (0xbd3 bytes) 000872 1740673578.428828 read 0x02c42b thru 0x03b59f (0xf175 bytes) 000873 1740673578.429274 trunc from 0x040000 to 0x003d94 (0x3c26b bytes) 000874 1740673578.434963 MIRROR_RESYNC 5 mirrors 000875 1740673578.570432 MIRROR_EXTEND to 6 mirrors 000876 1740673578.660462 MIRROR_EXTEND to 7 mirrors 000878 1740673578.746236 mapread 0x000b4e thru 0x003d94 (0x3247 bytes) 000880 1740673578.751826 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b7:0x0]: Invalid argument (22) 000881 1740673578.807615 mapwrite 0x020a21 thru 0x02697c (0x5f5c bytes) 000884 1740673578.846529 MIRROR_RESYNC 6 mirrors 000886 1740673579.007267 trunc from 0x02697d to 0x03f268 (0x188ec bytes) 000889 1740673579.024743 write 0x00a806 thru 0x013090 (0x888b bytes) 000890 1740673579.027543 mapread 0x003a86 thru 0x0123c8 (0xe943 bytes) 000891 1740673579.029966 read 0x004415 thru 0x012d03 (0xe8ef bytes) 000892 1740673579.030456 trunc from 0x03f269 to 0x02608d (0x191db bytes) 000897 1740673579.036200 write 0x036667 thru 0x03bb34 (0x54ce bytes) 000898 1740673579.037859 MIRROR_RESYNC 6 mirrors 000899 1740673579.203420 trunc from 0x03bb35 to 0x02a66f (0x114c5 bytes) 000900 1740673579.218073 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b7:0x0]: Invalid argument (22) 000901 1740673579.261505 write 0x018baf thru 0x026a3f (0xde91 bytes) 000903 1740673579.264589 trunc from 0x02a670 to 0x019d13 (0x1095c bytes) 000905 1740673579.268350 write 0x02b6f7 thru 0x02f324 (0x3c2e bytes) 000906 1740673579.268838 mapread 0x010708 thru 0x017b9e (0x7497 bytes) 000907 1740673579.271239 mapread 0x012002 thru 0x0217ef (0xf7ee bytes) 000910 1740673579.273693 trunc from 0x02f325 to 0x009955 (0x259cf bytes) 000911 1740673579.277305 read 0x007807 thru 0x009955 (0x214f bytes) 000914 1740673579.279189 write 0x03c307 thru 0x03ffff (0x3cf9 bytes) 000915 1740673579.279692 write 0x033db3 thru 0x03ffff (0xc24d bytes) 000917 1740673579.282173 trunc from 0x040000 to 0x0373df (0x8c20 bytes) 000920 1740673579.285942 read 0x019f09 thru 0x027223 (0xd31b bytes) 000921 1740673579.289292 write 0x03a844 thru 0x03ffff (0x57bc bytes) 000922 1740673579.289791 mapwrite 0x0337f9 thru 0x03ffff (0xc807 bytes) 000923 1740673579.304535 trunc from 0x040000 to 0x0396eb (0x6914 bytes) 000924 1740673579.308061 mapwrite 0x00fc70 thru 0x0120be (0x244f bytes) 000925 1740673579.319663 trunc from 0x0396ec to 0x036e95 (0x2856 bytes) 000926 1740673579.323310 mapread 0x0270d2 thru 0x035223 (0xe152 bytes) 000928 1740673579.325186 MIRROR_EXTEND to 6 mirrors 000933 1740673579.414818 write 0x036736 thru 0x03ffff (0x98ca bytes) 000934 1740673579.429683 mapread 0x037c47 thru 0x03f933 (0x7ced bytes) 000939 1740673579.431459 trunc from 0x040000 to 0x00c654 (0x339ab bytes) 000940 1740673579.439032 trunc from 0x00c655 to 0x029294 (0x1cc40 bytes) 000941 1740673579.442310 write 0x038640 thru 0x03da32 (0x53f3 bytes) 000942 1740673579.442941 mapwrite 0x012438 thru 0x01d8cf (0xb498 bytes) 000944 1740673579.462443 write 0x02ad92 thru 0x02bfbe (0x122d bytes) 000948 1740673579.465980 mapwrite 0x00af48 thru 0x01264f (0x7708 bytes) 000949 1740673579.479802 mapread 0x00bc3e thru 0x01b76d (0xfb30 bytes) 000950 1740673579.480305 mapwrite 0x034eef thru 0x03fb6b (0xac7d bytes) 000951 1740673579.499593 trunc from 0x03fb6c to 0x0261ca (0x199a1 bytes) 000952 1740673579.503007 trunc from 0x0261cb to 0x01510b (0x110bf bytes) 000953 1740673579.507294 MIRROR_EXTEND to 7 mirrors 000955 1740673579.595060 mapwrite 0x021d57 thru 0x02c2b0 (0xa55a bytes) 000956 1740673579.635130 mapwrite 0x037211 thru 0x03ffff (0x8def bytes) 000957 1740673579.654515 read 0x018359 thru 0x02108a (0x8d32 bytes) 000958 1740673579.656292 mapwrite 0x01e49f thru 0x024fdb (0x6b3d bytes) 000959 1740673579.669765 MIRROR_EXTEND to 8 mirrors 000960 1740673579.772535 write 0x003a82 thru 0x007ba7 (0x4126 bytes) 000963 1740673579.790534 trunc from 0x040000 to 0x021388 (0x1ec77 bytes) 000964 1740673579.798684 read 0x00e37d thru 0x00f187 (0xe0b bytes) 000967 1740673579.802063 MIRROR_EXTEND to 9 mirrors 000968 1740673579.903324 mapread 0x01f6c2 thru 0x021388 (0x1cc7 bytes) 000970 1740673579.908299 mapwrite 0x0322e4 thru 0x037c07 (0x5924 bytes) 000971 1740673579.943096 mapread 0x02b2ec thru 0x03635c (0xb071 bytes) 000972 1740673579.944768 MIRROR_EXTEND to 10 mirrors 000973 1740673580.037127 read 0x025f74 thru 0x030874 (0xa901 bytes) 000974 1740673580.040060 trunc from 0x037c08 to 0x031d34 (0x5ed3 bytes) 000977 1740673580.059661 write 0x001926 thru 0x00da3d (0xc118 bytes) 000978 1740673580.062780 mapread 0x01add9 thru 0x01d5d2 (0x27fa bytes) 000979 1740673580.065506 read 0x01077d thru 0x018c32 (0x84b6 bytes) 000980 1740673580.068180 mapwrite 0x03ab05 thru 0x03ffff (0x54fb bytes) 000982 1740673580.088717 trunc from 0x040000 to 0x01e906 (0x216f9 bytes) 000986 1740673580.092715 mapread 0x00e71f thru 0x010e26 (0x2708 bytes) 000987 1740673580.093310 trunc from 0x01e907 to 0x023af5 (0x51ef bytes) 000988 1740673580.096233 mapwrite 0x016275 thru 0x0185c4 (0x2350 bytes) 000989 1740673580.108079 mapread 0x001632 thru 0x009b7d (0x854c bytes) 000991 1740673580.109036 read 0x011a7c thru 0x012f83 (0x1508 bytes) 000992 1740673580.109349 write 0x003da1 thru 0x009030 (0x5290 bytes) 000993 1740673580.109845 mapread 0x0093e9 thru 0x016d83 (0xd99b bytes) 000995 1740673580.110506 mapread 0x00956a thru 0x00bdbf (0x2856 bytes) 000996 1740673580.110882 trunc from 0x023af6 to 0x01c1a1 (0x7954 bytes) 000999 1740673580.114915 trunc from 0x01c1a2 to 0x02842b (0xc28a bytes) 001000 1740673580.118416 mapwrite 0x01e131 thru 0x022a0c (0x48dc bytes) All operations completed A-OK! PASS 70a (11s) == sanity-flr test 71: check mirror extend parent fid ==== 11:26:21 (1740673581) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (0s) == sanity-flr test 200a: stress test ===================== 11:26:21 (1740673581) Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre2 Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4444832 .. Extending file size to 6430464 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6476704 .. Extending file size to 8310912 .. done Extending file size to 7030432 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7116064 .. Extending file size to 8551744 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7137984 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8711616 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8726112 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9031424 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9135616 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8963968 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9034624 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 9306016 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9249984 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 30449 30450 30451 30453 30456 done 192.168.203.103@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg303-client.virtnet /mnt/lustre2 (opts:) 192.168.203.103@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg303-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 11:27:25 (1740673645) Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre2 Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 2782784 .. Extending file size to 6964416 .. failed Extending file size to 3452576 .. Extending file size to 6876128 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 6978720 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7406784 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7838848 .. Extending file size to 8564928 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8490880 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8517856 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8821664 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9073632 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8569152 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9100384 .. Extending file size to 8648544 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8790240 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9149184 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9141728 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 2405 2406 2407 2409 2412 done 192.168.203.103@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg303-client.virtnet /mnt/lustre2 (opts:) 192.168.203.103@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg303-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 11:28:28 (1740673708) Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 5504 2202624 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7492608 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 5504 2202624 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 22528 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7492608 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0314655 s, 66.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.072894 s, 28.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.103@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg303-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 11:28:33 (1740673713) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0222869 s, 94.1 MB/s PASS 202 (1s) == sanity-flr test 203: mirror file preserve mirror ID === 11:28:34 (1740673714) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9fe:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xddb:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xddb:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xddc:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0xdde:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 11:28:36 (1740673716) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0244866 s, 85.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xddd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xddf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (1s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 11:28:37 (1740673717) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0241724 s, 86.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdde:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xddf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdde:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xddf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde2:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 11:28:39 (1740673719) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0219061 s, 95.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xde0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xde0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde4:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 11:29:02 (1740673742) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg303-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0173867 s, 60.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xde1:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xde1:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xde2:0x0] } PASS 204d (41s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 11:29:43 (1740673783) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg303-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xde3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xde4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0215569 s, 97.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xde3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xde4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xde3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xde4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (40s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 11:30:23 (1740673823) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg303-server mds-ost sync done. Creating new pool test_204f oleg303-server: Pool lustre.test_204f created Adding targets to pool oleg303-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg303-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0244517 s, 85.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xde5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xde5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg303-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg303-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg303-server: Pool lustre.test_204f destroyed PASS 204f (52s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 11:31:16 (1740673876) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 11:31:17 (1740673877) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 11:31:19 (1740673879) oleg303-server: Pool lustre.test_206 created Adding targets to pool oleg303-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg303-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg303-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg303-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg303-server: Pool lustre.test_206 destroyed PASS 206 (11s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 11:31:30 (1740673890) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0169461 s, 61.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00741618 s, 141 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x74ddfd27 Mirror 3: 0x74ddfd27 PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 11:31:33 (1740673893) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 11:31:33 (1740673893) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 11:31:34 (1740673894) Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.020601 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xb20000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xb50000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xb50000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xb50000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x11c0000 Read this (4 bytes): BBBB 192.168.203.103@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg303-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 11:31:47 (1740673907) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00294118 s, 2.8 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 11:31:49 (1740673909) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0103699 s, 101 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0xa23 lmm_fid: [0x200000404:0xa23:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0xa23 lmm_fid: [0x200000404:0xa23:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf2:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322836515 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-02-27 11:31:49.000000000 -0500 Modify: 2025-02-27 11:31:49.000000000 -0500 Change: 2025-02-27 11:31:50.000000000 -0500 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 11:31:51 (1740673911) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 11:31:52 (1740673912) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.122957 s, 333 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0139997 s, 293 kB/s size after second write -rw-r--r-- 1 root root 4096 Feb 27 11:31 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa25:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Feb 27 11:31 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 675 sec ============ 11:31:54 (1740673914) === sanity-flr: start cleanup 11:31:54 (1740673914) === === sanity-flr: finish cleanup 11:31:55 (1740673915) ===