-----============= acceptance-small: sanity-flr ============----- Tue Dec 17 05:10:24 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg436-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 05:10:32 (1734430232) === oleg436-client.virtnet: executing check_config_client /mnt/lustre oleg436-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg436-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012aad0800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012aad0800.idle_timeout=debug disable quota as required oleg436-server: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 05:10:44 (1734430244) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7587] == sanity-flr test 0a: lfs mirror create with -N option == 05:10:44 (1734430244) PASS 0a (5s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 05:10:49 (1734430249) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 05:10:50 (1734430250) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 05:10:51 (1734430251) PASS 0d (8s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 05:10:59 (1734430259) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 05:11:01 (1734430261) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 05:11:02 (1734430262) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88012aad0800.stats=clear osc.lustre-OST0001-osc-ffff88012aad0800.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 05:11:05 (1734430265) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 05:11:10 (1734430270) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 05:11:12 (1734430272) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 05:11:14 (1734430274) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 05:11:17 (1734430277) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 05:11:20 (1734430280) PASS 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 05:11:22 (1734430282) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 05:11:25 (1734430285) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 05:11:28 (1734430288) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 05:11:30 (1734430290) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB) copied, 0.415819 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0545077 s, 19.2 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 05:11:33 (1734430293) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0376553 s, 111 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 05:11:35 (1734430295) Stopping /mnt/lustre-ost1 (opts:) on oleg436-server oleg436-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012aad0800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012aad0800.ost_server_uuid in DISCONN state after 3 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg436-server: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg436-server: oleg436-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg436-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg436-server oleg436-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012aad0800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012aad0800.ost_server_uuid in DISCONN state after 2 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg436-server: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg436-server: oleg436-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg436-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (31s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 05:12:06 (1734430326) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB) copied, 0.124397 s, 59.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg436-server oleg436-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012aad0800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012aad0800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg436-server: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg436-server: oleg436-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg436-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (13s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 05:12:19 (1734430339) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg436-server oleg436-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012aad0800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012aad0800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg436-server: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg436-server: oleg436-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg436-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg436-server oleg436-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012aad0800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012aad0800.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg436-server: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg436-server: oleg436-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg436-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (26s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 05:12:45 (1734430365) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.99379 s, 52.8 MB/s -rw-r--r-- 1 root root 52428800 Dec 17 05:12 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.244s user 0m0.000s sys 0m0.106s Stopping /mnt/lustre-ost1 (opts:) on oleg436-server oleg436-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012aad0800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012aad0800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... real 0m0.256s user 0m0.000s sys 0m0.101s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg436-server: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg436-server: oleg436-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg436-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg436-server oleg436-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012aad0800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012aad0800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.365s user 0m0.002s sys 0m0.114s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg436-server: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg436-server: oleg436-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg436-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg436-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012aad0800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012aad0800.ost_server_uuid in FULL state after 0 sec PASS 33b (28s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 05:13:13 (1734430393) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 05:13:14 (1734430394) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 05:13:15 (1734430395) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 05:13:16 (1734430396) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 05:13:18 (1734430398) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.158731 s, 79.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 05:13:21 (1734430401) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0105418 s, 99.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102985 s, 102 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 05:13:23 (1734430403) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 05:13:26 (1734430406) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.0007856 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000764545 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00068968 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00075354 s, 1.4 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 05:13:29 (1734430409) create test files with size 15760787 .. 5a59911997fa02bb0fea2b44639aea2a - 215711437c5ea2d40620d5fdd3f34967 - 29b716e1eb1f75ccd6d31bcfd1cadd5c - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg436-client.virtnet /mnt/lustre (opts:) Starting client: oleg436-client.virtnet: -o user_xattr,flock 192.168.204.136@tcp:/lustre /mnt/lustre PASS 37 (7s) == sanity-flr test 38: resync ============================ 05:13:36 (1734430416) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (4s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 05:13:40 (1734430420) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 05:13:42 (1734430422) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0249867 s, 42.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x39:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3a:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x35:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0145246 s, 72.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x36:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x37:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 05:13:44 (1734430424) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0333828 s, 62.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.139476 s, 30.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0428778 s, 48.9 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 05:13:48 (1734430428) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (0s) == sanity-flr test 43a: mirror pick on write ============= 05:13:49 (1734430429) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 05:13:50 (1734430430) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0147881 s, 70.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 05:13:52 (1734430432) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0987354 s, 31.9 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x12:0x0]: Invalid argument (22) PASS 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 05:13:55 (1734430435) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x42:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x46:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x17:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 05:13:58 (1734430438) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0827713 s, 127 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg436-client.virtnet /mnt/lustre (opts:) Starting client: oleg436-client.virtnet: -o user_xattr,flock 192.168.204.136@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (20s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 05:14:18 (1734430458) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800a2e64800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.127312 s, 82.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg436-client.virtnet /mnt/lustre (opts:) Starting client: oleg436-client.virtnet: -o user_xattr,flock 192.168.204.136@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800aaff4000.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 05:14:23 (1734430463) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (12s) == sanity-flr test 46: Verify setstripe --copy option ==== 05:14:35 (1734430475) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (12s) == sanity-flr test 47: Verify mirror obj alloc =========== 05:14:47 (1734430487) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 05:14:48 (1734430488) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.102195 s, 30.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0801996 s, 39.2 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 20892042807882545dc255e15e0fc38b - mirror 1 checksum: 4ede115b869da383390db5dc2394c5ab - mirror 2 checksum: 20892042807882545dc255e15e0fc38b - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Dec 17 05:14:49 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Dec 17 05:14:49 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 20892042807882545dc255e15e0fc38b - mirror 1 checksum: 4ede115b869da383390db5dc2394c5ab - mirror 2 checksum: 4ede115b869da383390db5dc2394c5ab - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Dec 17 05:14:49 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Dec 17 05:14:49 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } PASS 48 (4s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 05:14:52 (1734430492) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0155248 s, 67.5 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 05:14:54 (1734430494) Failing mds1 on oleg436-server Stopping /mnt/lustre-mds1 (opts:) on oleg436-server 05:15:02 (1734430502) shut down facet: mds1 facet_host: oleg436-server facet_failover_host: oleg436-server Failover mds1 to oleg436-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg436-server: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-MDT0000 05:15:17 (1734430517) targets are mounted 05:15:17 (1734430517) facet_failover done oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x11:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0163497 s, 64.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x11 lmm_fid: [0x200000405:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } PASS 50A (30s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 05:15:24 (1734430524) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0140971 s, 1.5 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00719519 s, 2.8 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 05:15:27 (1734430527) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.021053 s, 973 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00603848 s, 3.4 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 05:15:31 (1734430531) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00613704 s, 2.7 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 05:15:34 (1734430534) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0276868 s, 740 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 05:15:37 (1734430537) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00255188 s, 1.6 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0211877 s, 193 kB/s PASS 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 05:15:41 (1734430541) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) PASS 60b (8s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 05:15:49 (1734430549) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (21s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 05:16:10 (1734430570) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x24:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 05:16:28 (1734430588) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 05:16:46 (1734430606) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 05:16:49 (1734430609) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 26341 mirror split pid 26342 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7547: 26341 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4860: 26342 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 05:17:51 (1734430671) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 672 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fallocating to largest ever: 0x3bbd0 000001 1734430672.948089 fallocate 0x03145c thru 0x03bbcf (0xa774 bytes) 000002 1734430672.953459 mapread 0x028007 thru 0x034496 (0xc490 bytes) 000004 1734430672.969325 trunc from 0x03bbd0 to 0x026c24 (0x14fab bytes) 000006 1734430672.979366 write 0x01a41f thru 0x020a13 (0x65f5 bytes) 000007 1734430672.987858 punch from 0x004496 to 0x00dc14 (0x977f bytes) 000009 1734430672.991214 mapwrite 0x01deb7 thru 0x026694 (0x87de bytes) 000012 1734430673.019205 MIRROR_EXTEND to 2 mirrors 000013 1734430673.393770 MIRROR_RESYNC 2 mirrors 000014 1734430673.511199 write 0x01f2ef thru 0x025305 (0x6017 bytes) 000016 1734430673.552837 write 0x004e35 thru 0x00a9ae (0x5b7a bytes) 000017 1734430673.558224 mapwrite 0x02d474 thru 0x035257 (0x7de4 bytes) 000018 1734430673.617765 mapwrite 0x02ef42 thru 0x03e815 (0xf8d4 bytes) 000019 1734430673.652987 read 0x030a6d thru 0x036864 (0x5df8 bytes) 000021 1734430673.657835 MIRROR_EXTEND to 3 mirrors 000022 1734430673.910634 punch from 0x0209d5 to 0x02d19c (0xc7c8 bytes) fallocating to largest ever: 0x40000 000023 1734430673.933494 fallocate 0x037e5f thru 0x03ffff (0x81a1 bytes) 000024 1734430673.937357 write 0x0086e0 thru 0x00f546 (0x6e67 bytes) 000025 1734430673.946376 write 0x03c472 thru 0x03ffff (0x3b8e bytes) 000027 1734430673.952018 trunc from 0x040000 to 0x02111b (0x1eee4 bytes) 000028 1734430673.977361 write 0x0115da thru 0x020595 (0xefbc bytes) 000029 1734430673.984630 fallocate 0x01ecca thru 0x026522 (0x7859 bytes) 000030 1734430673.988433 write 0x02b4f8 thru 0x037065 (0xbb6e bytes) 000032 1734430673.992264 MIRROR_RESYNC 3 mirrors 000033 1734430674.410312 write 0x02af84 thru 0x031726 (0x67a3 bytes) 000034 1734430674.432829 write 0x0077da thru 0x01292b (0xb152 bytes) 000035 1734430674.441102 mapread 0x00f4d3 thru 0x00f9fa (0x528 bytes) 000036 1734430674.442256 read 0x01655e thru 0x019406 (0x2ea9 bytes) 000037 1734430674.447225 mapread 0x00f704 thru 0x01a782 (0xb07f bytes) 000038 1734430674.450574 trunc from 0x037066 to 0x019799 (0x1d8cc bytes) 000039 1734430674.465871 MIRROR_EXTEND to 4 mirrors 000040 1734430674.696230 mapwrite 0x00f0a9 thru 0x015377 (0x62cf bytes) 000041 1734430674.754641 punch from 0x015f97 to 0x019799 (0x3803 bytes) 000042 1734430674.758263 mapread 0x00ac50 thru 0x00d913 (0x2cc4 bytes) 000043 1734430674.759329 write 0x036c24 thru 0x03ffff (0x93dc bytes) 000044 1734430674.763448 punch from 0x00eaba to 0x012c21 (0x4168 bytes) 000045 1734430674.767200 read 0x00226a thru 0x00f8c5 (0xd65c bytes) 000046 1734430674.773125 fallocate 0x033758 thru 0x0391b9 (0x5a62 bytes) 000047 1734430674.776051 mapwrite 0x02b19e thru 0x030c56 (0x5ab9 bytes) 000048 1734430674.802299 punch from 0x016d1e to 0x01718e (0x471 bytes) 000049 1734430674.809490 write 0x01a5a2 thru 0x024c9e (0xa6fd bytes) 000050 1734430674.817315 mapwrite 0x00666c thru 0x00c8a0 (0x6235 bytes) 000051 1734430674.835123 trunc from 0x040000 to 0x039cad (0x6352 bytes) 000052 1734430674.864064 read 0x029334 thru 0x02ac1b (0x18e8 bytes) 000053 1734430674.864451 fallocate 0x028d23 thru 0x036104 (0xd3e2 bytes) 000054 1734430674.869302 punch from 0x021016 to 0x026d69 (0x5d54 bytes) 000055 1734430674.872810 punch from 0x015f78 to 0x01d909 (0x7992 bytes) 000056 1734430674.876751 mapread 0x015de3 thru 0x017ed8 (0x20f6 bytes) 000057 1734430674.882249 write 0x011862 thru 0x015d85 (0x4524 bytes) 000059 1734430674.883193 mapwrite 0x01aa9b thru 0x01ecea (0x4250 bytes) 000060 1734430674.895772 mapwrite 0x03749c thru 0x03ffff (0x8b64 bytes) 000061 1734430674.916192 punch from 0x025b07 to 0x03304d (0xd547 bytes) 000062 1734430674.921617 trunc from 0x040000 to 0x0036f8 (0x3c907 bytes) 000063 1734430674.930456 punch from 0x00079f to 0x0036f8 (0x2f5a bytes) 000064 1734430674.934498 fallocate 0x03ef04 thru 0x03ffff (0x10fc bytes) 000065 1734430674.939352 punch from 0x024726 to 0x02aad6 (0x63b1 bytes) 000066 1734430674.943004 fallocate 0x0238e6 thru 0x027f0d (0x4628 bytes) 000067 1734430674.946824 trunc from 0x040000 to 0x03897b (0x7684 bytes) 000068 1734430674.956252 read 0x020e26 thru 0x02d2ae (0xc489 bytes) 000069 1734430674.965177 write 0x03b4e9 thru 0x03ffff (0x4b17 bytes) 000071 1734430674.972543 MIRROR_RESYNC 4 mirrors 000072 1734430675.435656 mapwrite 0x02b7ec thru 0x02f57a (0x3d8f bytes) 000073 1734430675.488827 write 0x02d2dd thru 0x03ccac (0xf9d0 bytes) 000074 1734430675.493408 write 0x03b317 thru 0x03ffff (0x4ce9 bytes) 000075 1734430675.494914 read 0x033e60 thru 0x03b1af (0x7350 bytes) 000076 1734430675.495527 mapread 0x00f2a6 thru 0x01d324 (0xe07f bytes) 000077 1734430675.506582 MIRROR_EXTEND to 5 mirrors 000078 1734430675.774533 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000079 1734430675.886220 MIRROR_RESYNC 4 mirrors 000081 1734430676.303672 trunc from 0x040000 to 0x01c6f6 (0x23909 bytes) 000082 1734430676.329277 read 0x00cfb3 thru 0x01998d (0xc9db bytes) 000083 1734430676.336903 MIRROR_RESYNC 4 mirrors 000085 1734430676.678471 fallocate 0x036813 thru 0x03ffff (0x97ed bytes) 000086 1734430676.702299 punch from 0x036006 to 0x03aa38 (0x4a33 bytes) 000087 1734430676.705362 fallocate 0x03ae5c thru 0x03b726 (0x8cb bytes) 000088 1734430676.708858 punch from 0x01cec2 to 0x0293a7 (0xc4e6 bytes) 000090 1734430676.714193 write 0x004692 thru 0x011819 (0xd188 bytes) 000092 1734430676.725955 fallocate 0x031a48 thru 0x03d458 (0xba11 bytes) 000093 1734430676.729650 punch from 0x010b5b to 0x01ca50 (0xbef6 bytes) 000094 1734430676.742851 fallocate 0x010bd5 thru 0x01a547 (0x9973 bytes) 000095 1734430676.745243 mapread 0x02e829 thru 0x03b44a (0xcc22 bytes) 000096 1734430676.754376 trunc from 0x040000 to 0x01194b (0x2e6b4 bytes) 000097 1734430676.770259 fallocate 0x00cefa thru 0x01c7a8 (0xf8af bytes) 000099 1734430676.778592 MIRROR_RESYNC 4 mirrors 000100 1734430677.178214 mapwrite 0x014a3b thru 0x017f2e (0x34f4 bytes) 000102 1734430677.221205 read 0x002fdc thru 0x00655e (0x3583 bytes) 000103 1734430677.224981 trunc from 0x017f2f to 0x033130 (0x1b202 bytes) 000104 1734430677.235709 fallocate 0x03b5d9 thru 0x03ee9a (0x38c2 bytes) 000105 1734430677.239814 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000106 1734430677.349399 mapread 0x037e39 thru 0x03ee9a (0x7062 bytes) 000107 1734430677.358180 write 0x03b464 thru 0x03ffff (0x4b9c bytes) 000109 1734430677.363773 write 0x0390e9 thru 0x03ad41 (0x1c59 bytes) 000110 1734430677.364735 read 0x008eb8 thru 0x008edf (0x28 bytes) 000111 1734430677.373387 MIRROR_EXTEND to 4 mirrors 000113 1734430677.697582 trunc from 0x040000 to 0x01738e (0x28c71 bytes) 000114 1734430677.739442 read 0x00998b thru 0x0155a8 (0xbc1e bytes) 000115 1734430677.744806 trunc from 0x01738f to 0x02341b (0xc08d bytes) 000116 1734430677.758324 write 0x03b450 thru 0x03fe72 (0x4a23 bytes) 000117 1734430677.759341 trunc from 0x03fe73 to 0x00e34d (0x31b25 bytes) 000118 1734430677.767638 punch from 0x00a79d to 0x00e34d (0x3bb1 bytes) 000119 1734430677.770282 trunc from 0x00e34e to 0x03d025 (0x2ecd8 bytes) 000120 1734430677.783096 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000121 1734430677.886963 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000122 1734430678.012964 read 0x022753 thru 0x026f60 (0x480e bytes) 000123 1734430678.018213 mapread 0x0213de thru 0x023bdf (0x2802 bytes) 000124 1734430678.022221 mapread 0x02281e thru 0x02d19d (0xa980 bytes) 000125 1734430678.028348 mapread 0x02f63e thru 0x03c016 (0xc9d9 bytes) 000126 1734430678.036528 MIRROR_EXTEND to 3 mirrors 000127 1734430678.373073 mapread 0x01318f thru 0x01cd9e (0x9c10 bytes) 000128 1734430678.396136 fallocate 0x01b4e8 thru 0x024e2c (0x9945 bytes) 000129 1734430678.423259 mapwrite 0x00cf28 thru 0x012ccc (0x5da5 bytes) 000130 1734430678.440820 punch from 0x027505 to 0x02b535 (0x4031 bytes) 000131 1734430678.443264 read 0x032db6 thru 0x03d025 (0xa270 bytes) 000132 1734430678.449187 fallocate 0x038026 thru 0x03f465 (0x7440 bytes) 000133 1734430678.452747 mapwrite 0x002286 thru 0x0037e9 (0x1564 bytes) 000134 1734430678.467419 mapwrite 0x03c695 thru 0x03ffff (0x396b bytes) 000135 1734430678.505008 fallocate 0x02e2ae thru 0x0351d1 (0x6f24 bytes) 000136 1734430678.507831 read 0x00a1a3 thru 0x00d9e8 (0x3846 bytes) 000137 1734430678.508250 mapread 0x0126a7 thru 0x020400 (0xdd5a bytes) 000138 1734430678.516187 punch from 0x017f81 to 0x01d479 (0x54f9 bytes) 000139 1734430678.519576 read 0x00506f thru 0x00a487 (0x5419 bytes) 000142 1734430678.520743 read 0x028aee thru 0x02d422 (0x4935 bytes) 000143 1734430678.521752 trunc from 0x040000 to 0x021add (0x1e522 bytes) 000144 1734430678.534784 mapwrite 0x0335f7 thru 0x03ac39 (0x7643 bytes) 000145 1734430678.578605 trunc from 0x03ac3a to 0x02c375 (0xe8c4 bytes) 000146 1734430678.592092 fallocate 0x0344dd thru 0x036fc8 (0x2aec bytes) 000147 1734430678.597201 mapread 0x000158 thru 0x00260c (0x24b5 bytes) 000148 1734430678.599096 trunc from 0x036fc9 to 0x01d02e (0x19f9a bytes) 000149 1734430678.608888 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000150 1734430678.713769 mapwrite 0x002e07 thru 0x008188 (0x5382 bytes) 000151 1734430678.748745 mapread 0x0069c4 thru 0x00fb8c (0x91c9 bytes) 000152 1734430678.749495 read 0x015d63 thru 0x01d02e (0x72cc bytes) 000153 1734430678.758992 read 0x002567 thru 0x005510 (0x2faa bytes) 000154 1734430678.759432 mapread 0x0087df thru 0x015aaf (0xd2d1 bytes) 000155 1734430678.765715 mapread 0x00872e thru 0x01410c (0xb9df bytes) 000156 1734430678.766278 mapwrite 0x02f444 thru 0x03a8fb (0xb4b8 bytes) 000157 1734430678.809038 trunc from 0x03a8fc to 0x010604 (0x2a2f7 bytes) 000158 1734430678.818579 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000159 1734430678.892892 mapwrite 0x009100 thru 0x00b4d8 (0x23d9 bytes) 000160 1734430678.915710 mapwrite 0x03eb89 thru 0x03f696 (0xb0e bytes) 000161 1734430678.953208 read 0x022438 thru 0x02ff1a (0xdae3 bytes) 000162 1734430678.958709 mapwrite 0x014caf thru 0x01b190 (0x64e2 bytes) 000163 1734430678.977164 read 0x0200f3 thru 0x022c8f (0x2b9d bytes) 000164 1734430678.985346 write 0x035c11 thru 0x03be40 (0x6230 bytes) 000165 1734430678.986930 mapread 0x00a21b thru 0x014ab3 (0xa899 bytes) 000166 1734430678.988660 fallocate 0x000b98 thru 0x000d78 (0x1e1 bytes) 000169 1734430678.992694 mapwrite 0x007707 thru 0x00ffe8 (0x88e2 bytes) 000170 1734430679.004823 write 0x027737 thru 0x027b4d (0x417 bytes) 000171 1734430679.005474 mapread 0x001df0 thru 0x003b50 (0x1d61 bytes) 000172 1734430679.006037 punch from 0x033296 to 0x03f696 (0xc401 bytes) 000173 1734430679.009861 write 0x031af7 thru 0x035dbb (0x42c5 bytes) 000174 1734430679.014093 trunc from 0x03f697 to 0x00b39c (0x342fa bytes) 000175 1734430679.021063 read 0x0057d9 thru 0x0084e9 (0x2d11 bytes) 000176 1734430679.021467 fallocate 0x0331ea thru 0x03ce94 (0x9cab bytes) 000177 1734430679.024539 mapread 0x000d5b thru 0x00591f (0x4bc5 bytes) 000178 1734430679.029873 MIRROR_EXTEND to 2 mirrors 000179 1734430679.254239 mapwrite 0x033567 thru 0x03ef3c (0xb9d6 bytes) 000180 1734430679.324557 read 0x000110 thru 0x0026f2 (0x25e3 bytes) 000182 1734430679.331473 read 0x013b5c thru 0x018aa9 (0x4f4e bytes) 000183 1734430679.337683 write 0x039500 thru 0x03ffff (0x6b00 bytes) 000184 1734430679.339516 mapread 0x03657c thru 0x03ffff (0x9a84 bytes) 000185 1734430679.343228 MIRROR_EXTEND to 3 mirrors 000186 1734430679.604665 mapread 0x016d66 thru 0x01733d (0x5d8 bytes) 000187 1734430679.617761 trunc from 0x040000 to 0x001297 (0x3ed68 bytes) 000188 1734430679.662514 mapread 0x0006ef thru 0x001297 (0xba9 bytes) 000190 1734430679.672254 write 0x03f839 thru 0x03ffff (0x7c7 bytes) 000192 1734430679.673309 trunc from 0x040000 to 0x02f546 (0x10ab9 bytes) 000195 1734430679.689750 fallocate 0x00076e thru 0x003c92 (0x3525 bytes) 000197 1734430679.693635 mapwrite 0x03bd32 thru 0x03ffff (0x42ce bytes) 000198 1734430679.722239 punch from 0x008188 to 0x016f2a (0xeda3 bytes) 000199 1734430679.725484 mapwrite 0x00acab thru 0x019101 (0xe457 bytes) 000200 1734430679.754785 MIRROR_EXTEND to 4 mirrors 000201 1734430680.018179 punch from 0x001f03 to 0x00e8e9 (0xc9e7 bytes) 000202 1734430680.042550 punch from 0x0138d7 to 0x01c261 (0x898b bytes) 000203 1734430680.047533 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000204 1734430680.154045 fallocate 0x01f152 thru 0x029791 (0xa640 bytes) 000205 1734430680.157703 mapwrite 0x031266 thru 0x03479c (0x3537 bytes) 000207 1734430680.177178 read 0x00da8d thru 0x01289b (0x4e0f bytes) 000208 1734430680.181248 punch from 0x009398 to 0x0142e3 (0xaf4c bytes) 000209 1734430680.183855 write 0x03f297 thru 0x03ffff (0xd69 bytes) 000210 1734430680.188920 mapread 0x02af78 thru 0x033062 (0x80eb bytes) 000211 1734430680.199265 punch from 0x020fab to 0x023b53 (0x2ba9 bytes) 000213 1734430680.203153 trunc from 0x040000 to 0x019c2b (0x263d4 bytes) 000215 1734430680.220892 trunc from 0x019c2c to 0x032718 (0x18aed bytes) 000216 1734430680.230046 trunc from 0x032719 to 0x03ab6d (0x8455 bytes) 000217 1734430680.237882 write 0x02834b thru 0x0345b5 (0xc26b bytes) 000218 1734430680.244226 fallocate 0x03ea1f thru 0x03ffff (0x15e1 bytes) 000219 1734430680.247575 trunc from 0x040000 to 0x014b39 (0x2b4c6 bytes) 000221 1734430680.255250 trunc from 0x014b3a to 0x032785 (0x1dc4c bytes) 000222 1734430680.264303 read 0x02532b thru 0x0278cc (0x25a2 bytes) 000223 1734430680.271085 write 0x03300c thru 0x036c57 (0x3c4c bytes) 000224 1734430680.272461 punch from 0x01903f to 0x028e01 (0xfdc3 bytes) 000225 1734430680.275234 mapread 0x02b95a thru 0x032de9 (0x7490 bytes) 000226 1734430680.276432 trunc from 0x036c58 to 0x006349 (0x3090e bytes) 000229 1734430680.289010 trunc from 0x00634a to 0x02252c (0x1c1e3 bytes) 000230 1734430680.295656 mapread 0x01aa67 thru 0x02252c (0x7ac6 bytes) 000235 1734430680.316811 write 0x03d6b2 thru 0x03ffff (0x294e bytes) 000236 1734430680.317593 write 0x00778c thru 0x014590 (0xce05 bytes) 000239 1734430680.325541 fallocate 0x0237b2 thru 0x029666 (0x5eb5 bytes) 000241 1734430680.331357 mapread 0x01e686 thru 0x02cba5 (0xe520 bytes) 000242 1734430680.355466 write 0x03b33e thru 0x03ffff (0x4cc2 bytes) 000243 1734430680.356261 read 0x03e43a thru 0x03ffff (0x1bc6 bytes) 000244 1734430680.356639 mapread 0x004959 thru 0x009d3f (0x53e7 bytes) 000246 1734430680.357792 write 0x03a1ad thru 0x03ffff (0x5e53 bytes) 000247 1734430680.358541 read 0x0277bb thru 0x0329d8 (0xb21e bytes) 000248 1734430680.365839 MIRROR_RESYNC 3 mirrors 000249 1734430680.687454 MIRROR_EXTEND to 4 mirrors 000250 1734430680.965085 write 0x014a50 thru 0x01c842 (0x7df3 bytes) 000251 1734430680.994212 fallocate 0x006503 thru 0x00bb6a (0x5668 bytes) 000253 1734430681.000256 mapread 0x017645 thru 0x01c8d5 (0x5291 bytes) 000254 1734430681.002378 mapwrite 0x02bf90 thru 0x02f12d (0x319e bytes) 000255 1734430681.036571 punch from 0x001913 to 0x004fe6 (0x36d4 bytes) 000256 1734430681.040450 mapwrite 0x03d60c thru 0x03ffff (0x29f4 bytes) 000257 1734430681.062624 MIRROR_RESYNC 4 mirrors 000258 1734430681.516862 mapwrite 0x0218cf thru 0x0263bf (0x4af1 bytes) 000259 1734430681.580885 punch from 0x0086d6 to 0x00af22 (0x284d bytes) 000260 1734430681.585238 read 0x03a700 thru 0x03ffff (0x5900 bytes) 000262 1734430681.594283 read 0x00b596 thru 0x00f304 (0x3d6f bytes) 000263 1734430681.600211 write 0x0090a0 thru 0x013b91 (0xaaf2 bytes) 000264 1734430681.602102 write 0x03cf24 thru 0x03ffff (0x30dc bytes) 000266 1734430681.603224 write 0x034a38 thru 0x03d297 (0x8860 bytes) 000268 1734430681.605336 mapwrite 0x03f99f thru 0x03fe36 (0x498 bytes) 000269 1734430681.616791 read 0x01b1d7 thru 0x020fc7 (0x5df1 bytes) 000273 1734430681.619805 fallocate 0x03074b thru 0x034622 (0x3ed8 bytes) 000274 1734430681.625395 mapread 0x02919f thru 0x029510 (0x372 bytes) 000275 1734430681.626170 fallocate 0x006e55 thru 0x00f598 (0x8744 bytes) 000276 1734430681.630095 write 0x019c52 thru 0x01ff05 (0x62b4 bytes) 000277 1734430681.632909 MIRROR_RESYNC 4 mirrors 000278 1734430682.071421 write 0x035590 thru 0x03a942 (0x53b3 bytes) 000279 1734430682.105675 mapread 0x000056 thru 0x006658 (0x6603 bytes) 000280 1734430682.116552 trunc from 0x040000 to 0x01c8ce (0x23731 bytes) 000281 1734430682.142585 punch from 0x009a80 to 0x016f30 (0xd4b1 bytes) 000282 1734430682.150358 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000283 1734430682.269377 fallocate 0x00c3bf thru 0x011f81 (0x5bc3 bytes) 000284 1734430682.272970 mapread 0x006c8b thru 0x00c6cc (0x5a42 bytes) 000285 1734430682.285481 read 0x015ef7 thru 0x01c8ce (0x69d8 bytes) 000286 1734430682.292945 mapwrite 0x028773 thru 0x036bbe (0xe44c bytes) 000289 1734430682.330933 read 0x036273 thru 0x036bbe (0x94c bytes) 000291 1734430682.331562 fallocate 0x00674d thru 0x00df17 (0x77cb bytes) 000292 1734430682.336416 write 0x03702e thru 0x03ffff (0x8fd2 bytes) 000294 1734430682.341666 MIRROR_RESYNC 3 mirrors 000296 1734430682.716520 read 0x03e2fe thru 0x03ffff (0x1d02 bytes) 000297 1734430682.720347 write 0x00a4be thru 0x00c8e1 (0x2424 bytes) 000298 1734430682.754052 punch from 0x03c4a6 to 0x03ffff (0x3b5a bytes) 000299 1734430682.757308 punch from 0x003cc2 to 0x011b76 (0xdeb5 bytes) 000300 1734430682.769178 write 0x02a53d thru 0x039a8f (0xf553 bytes) 000302 1734430682.780245 trunc from 0x040000 to 0x00ac1c (0x353e3 bytes) 000303 1734430682.804431 MIRROR_EXTEND to 4 mirrors 000305 1734430683.050341 read 0x0090cb thru 0x00ac1c (0x1b52 bytes) 000306 1734430683.059369 mapwrite 0x03758d thru 0x03c669 (0x50dd bytes) 000308 1734430683.122081 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000309 1734430683.226694 write 0x02c8ea thru 0x031d3f (0x5456 bytes) 000311 1734430683.236384 write 0x03a2e6 thru 0x03ffff (0x5d1a bytes) 000312 1734430683.242523 read 0x0384d4 thru 0x03d487 (0x4fb4 bytes) 000313 1734430683.243000 trunc from 0x040000 to 0x02b9b4 (0x1464b bytes) 000314 1734430683.252547 fallocate 0x0332be thru 0x037e7f (0x4bc2 bytes) 000315 1734430683.256416 mapwrite 0x0164bf thru 0x01750c (0x104e bytes) 000316 1734430683.277749 punch from 0x018796 to 0x01d428 (0x4c93 bytes) 000317 1734430683.281969 mapread 0x01835d thru 0x0215d4 (0x9278 bytes) 000318 1734430683.290703 write 0x0268bf thru 0x02a8bc (0x3ffe bytes) 000320 1734430683.293776 mapread 0x02ddce thru 0x034a56 (0x6c89 bytes) 000321 1734430683.299398 trunc from 0x037e80 to 0x00e2c8 (0x29bb7 bytes) 000322 1734430683.310687 read 0x000056 thru 0x00c2f8 (0xc2a3 bytes) 000323 1734430683.315903 fallocate 0x0062d9 thru 0x006df5 (0xb1d bytes) 000324 1734430683.320672 mapread 0x0076af thru 0x0094cd (0x1e1f bytes) 000326 1734430683.321764 mapread 0x003778 thru 0x00e2c8 (0xab51 bytes) 000327 1734430683.322376 mapread 0x0046af thru 0x00d4d6 (0x8e28 bytes) 000328 1734430683.322912 mapwrite 0x03b6e5 thru 0x03ffff (0x491b bytes) 000329 1734430683.353416 write 0x0176b5 thru 0x017ad2 (0x41e bytes) 000331 1734430683.354402 read 0x0382c8 thru 0x03ffff (0x7d38 bytes) 000332 1734430683.357272 MIRROR_RESYNC 3 mirrors 000334 1734430683.673543 mapread 0x01b5d8 thru 0x02a314 (0xed3d bytes) 000335 1734430683.685698 mapread 0x01fe9f thru 0x023d4d (0x3eaf bytes) 000336 1734430683.686686 trunc from 0x040000 to 0x00d755 (0x328aa bytes) 000337 1734430683.726792 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000338 1734430683.815494 MIRROR_EXTEND to 3 mirrors 000340 1734430684.050099 mapread 0x0053d0 thru 0x007042 (0x1c73 bytes) 000341 1734430684.066192 fallocate 0x013811 thru 0x01d50b (0x9cfb bytes) 000342 1734430684.083173 mapread 0x003a9a thru 0x00d604 (0x9b6b bytes) 000343 1734430684.094519 mapwrite 0x03582e thru 0x03a97f (0x5152 bytes) 000345 1734430684.147072 MIRROR_RESYNC 3 mirrors 000346 1734430684.403230 trunc from 0x03a980 to 0x0290ce (0x118b1 bytes) 000347 1734430684.442461 punch from 0x01d62b to 0x02522c (0x7c02 bytes) 000348 1734430684.447042 fallocate 0x01b0cd thru 0x02041a (0x534e bytes) 000350 1734430684.449563 write 0x020874 thru 0x027c8a (0x7417 bytes) 000351 1734430684.454787 mapread 0x0211f0 thru 0x0290ce (0x7edf bytes) 000352 1734430684.459867 mapwrite 0x00aea1 thru 0x00be9c (0xffc bytes) 000353 1734430684.472371 write 0x021679 thru 0x02f96d (0xe2f5 bytes) 000354 1734430684.474305 mapread 0x004508 thru 0x013c72 (0xf76b bytes) 000355 1734430684.480607 write 0x01a2b3 thru 0x01e687 (0x43d5 bytes) 000356 1734430684.484075 trunc from 0x02f96e to 0x0387d0 (0x8e63 bytes) 000357 1734430684.497561 read 0x00fc4e thru 0x0168a3 (0x6c56 bytes) 000358 1734430684.498140 write 0x01afa4 thru 0x026984 (0xb9e1 bytes) 000359 1734430684.502847 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000360 1734430684.641468 write 0x0071d9 thru 0x0074aa (0x2d2 bytes) 000362 1734430684.646637 fallocate 0x01db23 thru 0x01e0d9 (0x5b7 bytes) 000363 1734430684.650072 trunc from 0x0387d1 to 0x03c812 (0x4042 bytes) 000364 1734430684.662955 trunc from 0x03c813 to 0x00f0df (0x2d733 bytes) 000365 1734430684.673813 mapwrite 0x025b98 thru 0x031475 (0xb8de bytes) 000366 1734430684.716269 MIRROR_EXTEND to 3 mirrors 000367 1734430684.992272 trunc from 0x031476 to 0x01716d (0x1a308 bytes) 000368 1734430685.041438 mapwrite 0x03b2aa thru 0x03ee83 (0x3bda bytes) 000369 1734430685.080019 write 0x02a4f8 thru 0x037241 (0xcd4a bytes) 000370 1734430685.085985 mapwrite 0x008806 thru 0x00a2df (0x1ada bytes) 000371 1734430685.111369 fallocate 0x0117ba thru 0x01f45a (0xdca1 bytes) 000372 1734430685.116336 fallocate 0x009532 thru 0x012621 (0x90f0 bytes) 000373 1734430685.123430 mapwrite 0x002a65 thru 0x009c60 (0x71fc bytes) 000374 1734430685.134942 mapread 0x02f81e thru 0x03ee83 (0xf666 bytes) 000375 1734430685.136635 write 0x028a0d thru 0x02be2c (0x3420 bytes) 000376 1734430685.139487 mapread 0x027fb8 thru 0x028517 (0x560 bytes) 000377 1734430685.140938 read 0x03e732 thru 0x03ee83 (0x752 bytes) 000378 1734430685.141309 fallocate 0x0178f9 thru 0x0232d6 (0xb9de bytes) 000379 1734430685.150459 MIRROR_RESYNC 3 mirrors 000380 1734430685.479436 trunc from 0x03ee84 to 0x03538c (0x9af7 bytes) 000381 1734430685.519197 mapwrite 0x03dc9f thru 0x03ffff (0x2361 bytes) 000383 1734430685.556799 mapread 0x02c8ce thru 0x032f8b (0x66be bytes) 000384 1734430685.561694 mapread 0x03ec2d thru 0x03ffff (0x13d3 bytes) 000385 1734430685.562287 read 0x036d74 thru 0x03befc (0x5189 bytes) 000386 1734430685.562601 write 0x022e88 thru 0x024c9e (0x1e17 bytes) 000387 1734430685.567926 punch from 0x0208bf to 0x02835f (0x7aa1 bytes) 000388 1734430685.574940 MIRROR_RESYNC 3 mirrors 000389 1734430685.952164 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000390 1734430686.091239 trunc from 0x040000 to 0x036918 (0x96e7 bytes) 000391 1734430686.128943 fallocate 0x003b55 thru 0x00b24b (0x76f7 bytes) 000392 1734430686.134236 read 0x005a80 thru 0x006817 (0xd98 bytes) 000393 1734430686.141984 MIRROR_EXTEND to 3 mirrors 000394 1734430686.444771 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000395 1734430686.576941 read 0x01d645 thru 0x02452f (0x6eeb bytes) 000396 1734430686.591372 read 0x00cc98 thru 0x013a8d (0x6df6 bytes) 000398 1734430686.619620 mapread 0x00b3bc thru 0x013478 (0x80bd bytes) 000399 1734430686.633752 read 0x01f2e5 thru 0x026591 (0x72ad bytes) 000400 1734430686.645146 write 0x02dedc thru 0x039890 (0xb9b5 bytes) 000401 1734430686.676929 write 0x02af3a thru 0x03106c (0x6133 bytes) 000402 1734430686.677952 trunc from 0x039891 to 0x014b09 (0x24d87 bytes) 000405 1734430686.704244 mapwrite 0x036729 thru 0x03ffff (0x98d7 bytes) 000406 1734430686.741241 trunc from 0x040000 to 0x01024c (0x2fdb3 bytes) 000407 1734430686.754065 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000409 1734430686.811303 write 0x016ed9 thru 0x020b9b (0x9cc3 bytes) 000410 1734430686.814411 read 0x015844 thru 0x01d7f0 (0x7fad bytes) 000411 1734430686.818029 mapwrite 0x01e8cd thru 0x01f32e (0xa62 bytes) 000412 1734430686.835212 trunc from 0x020b9c to 0x00e37e (0x1281d bytes) 000414 1734430686.847438 write 0x03fa58 thru 0x03ffff (0x5a8 bytes) 000415 1734430686.848469 fallocate 0x003bad thru 0x00dccf (0xa123 bytes) 000416 1734430686.852593 punch from 0x000194 to 0x00c910 (0xc77d bytes) 000417 1734430686.855157 fallocate 0x01c01e thru 0x024ac4 (0x8aa7 bytes) 000418 1734430686.860782 fallocate 0x02d11a thru 0x03076e (0x3655 bytes) 000419 1734430686.864323 fallocate 0x0065cf thru 0x00e2b8 (0x7cea bytes) 000420 1734430686.868808 fallocate 0x03cb47 thru 0x03ffff (0x34b9 bytes) 000421 1734430686.873200 mapwrite 0x03f917 thru 0x03ffff (0x6e9 bytes) 000422 1734430686.891707 trunc from 0x040000 to 0x00a1d5 (0x35e2a bytes) 000423 1734430686.907712 fallocate 0x025d85 thru 0x029ef4 (0x4170 bytes) 000425 1734430686.912509 mapwrite 0x033d0f thru 0x03f831 (0xbb23 bytes) 000426 1734430686.942104 fallocate 0x010d19 thru 0x013817 (0x2aff bytes) 000427 1734430686.950588 mapread 0x014e93 thru 0x01e972 (0x9ae0 bytes) 000428 1734430686.964678 fallocate 0x025190 thru 0x027a91 (0x2902 bytes) 000429 1734430686.967678 mapwrite 0x02eeb5 thru 0x030be9 (0x1d35 bytes) 000431 1734430686.978988 write 0x00ef1b thru 0x01a065 (0xb14b bytes) 000432 1734430686.985411 read 0x029384 thru 0x02a8e3 (0x1560 bytes) 000433 1734430686.989978 read 0x0349db thru 0x035cc0 (0x12e6 bytes) 000434 1734430686.990371 mapwrite 0x013a1b thru 0x02295a (0xef40 bytes) 000435 1734430687.006347 write 0x01364c thru 0x020b6b (0xd520 bytes) 000436 1734430687.007348 mapread 0x03198b thru 0x03f831 (0xdea7 bytes) 000437 1734430687.008843 mapwrite 0x00fd8b thru 0x017d5e (0x7fd4 bytes) 000438 1734430687.019166 trunc from 0x03f832 to 0x03c087 (0x37aa bytes) 000440 1734430687.029064 read 0x00659d thru 0x01016a (0x9bce bytes) 000441 1734430687.029509 punch from 0x013a45 to 0x018aaf (0x506b bytes) 000442 1734430687.033543 punch from 0x03b199 to 0x03c087 (0xeef bytes) 000443 1734430687.037180 punch from 0x015436 to 0x01b86c (0x6437 bytes) 000444 1734430687.040674 write 0x00ca75 thru 0x0145f8 (0x7b84 bytes) 000445 1734430687.044086 write 0x0015dd thru 0x004997 (0x33bb bytes) 000446 1734430687.044659 fallocate 0x02110f thru 0x0306e8 (0xf5da bytes) 000447 1734430687.048353 fallocate 0x016ccc thru 0x02573b (0xea70 bytes) 000448 1734430687.051764 mapread 0x038601 thru 0x03c087 (0x3a87 bytes) 000450 1734430687.058062 trunc from 0x03c088 to 0x036ce2 (0x53a5 bytes) 000452 1734430687.068136 fallocate 0x008393 thru 0x015969 (0xd5d7 bytes) 000453 1734430687.071858 write 0x025686 thru 0x0285f0 (0x2f6b bytes) 000455 1734430687.072898 trunc from 0x036ce3 to 0x0035a5 (0x3373d bytes) 000456 1734430687.080657 trunc from 0x0035a6 to 0x029ca9 (0x26704 bytes) 000457 1734430687.089598 mapread 0x00647d thru 0x011a3a (0xb5be bytes) 000458 1734430687.111473 write 0x00cf97 thru 0x00ed05 (0x1d6f bytes) 000459 1734430687.112144 write 0x035255 thru 0x039e4f (0x4bfb bytes) 000460 1734430687.112789 mapwrite 0x033300 thru 0x0388d4 (0x55d5 bytes) 000461 1734430687.130584 fallocate 0x00fe3d thru 0x015562 (0x5726 bytes) 000462 1734430687.134380 mapread 0x0297a4 thru 0x0367c3 (0xd020 bytes) 000464 1734430687.157045 fallocate 0x028c1f thru 0x037c53 (0xf035 bytes) 000465 1734430687.161429 read 0x039aa4 thru 0x039e4f (0x3ac bytes) 000468 1734430687.162352 punch from 0x00555a to 0x015204 (0xfcab bytes) 000469 1734430687.165965 trunc from 0x039e50 to 0x03f489 (0x563a bytes) 000470 1734430687.175668 fallocate 0x01a939 thru 0x01e214 (0x38dc bytes) 000471 1734430687.180100 mapwrite 0x036ecb thru 0x036fc5 (0xfb bytes) 000472 1734430687.193993 write 0x008d9e thru 0x0182ee (0xf551 bytes) 000473 1734430687.200092 mapread 0x02df63 thru 0x02e760 (0x7fe bytes) 000474 1734430687.201170 fallocate 0x0093a2 thru 0x00a8b7 (0x1516 bytes) 000477 1734430687.205425 read 0x003297 thru 0x00cd82 (0x9aec bytes) 000479 1734430687.205988 mapwrite 0x00c3ee thru 0x01ab30 (0xe743 bytes) 000481 1734430687.220191 MIRROR_EXTEND to 2 mirrors 000482 1734430687.482507 mapread 0x026cfb thru 0x02b3bc (0x46c2 bytes) 000486 1734430687.518149 MIRROR_RESYNC 2 mirrors 000489 1734430687.608133 write 0x021aa1 thru 0x029ddb (0x833b bytes) 000490 1734430687.634409 fallocate 0x023073 thru 0x028d49 (0x5cd7 bytes) 000491 1734430687.645052 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000492 1734430687.710383 write 0x013a64 thru 0x014499 (0xa36 bytes) 000494 1734430687.718805 mapread 0x02e317 thru 0x032d55 (0x4a3f bytes) 000496 1734430687.725834 punch from 0x005cb4 to 0x011236 (0xb583 bytes) 000497 1734430687.730973 write 0x03fbd9 thru 0x03ffff (0x427 bytes) 000498 1734430687.734932 read 0x00a358 thru 0x00def6 (0x3b9f bytes) 000499 1734430687.737442 fallocate 0x03e5c1 thru 0x03ffff (0x1a3f bytes) 000500 1734430687.739743 read 0x03dcd8 thru 0x03ffff (0x2328 bytes) 000501 1734430687.740132 read 0x01042b thru 0x01ac67 (0xa83d bytes) 000502 1734430687.746954 fallocate 0x02aea3 thru 0x02b603 (0x761 bytes) 000503 1734430687.749139 read 0x036f12 thru 0x037344 (0x433 bytes) 000504 1734430687.749512 punch from 0x034000 to 0x03b193 (0x7194 bytes) 000505 1734430687.752387 write 0x0364ab thru 0x03ffff (0x9b55 bytes) 000506 1734430687.758561 write 0x006450 thru 0x015299 (0xee4a bytes) 000507 1734430687.760004 punch from 0x02f7cf to 0x03a493 (0xacc5 bytes) 000509 1734430687.770402 read 0x028799 thru 0x037252 (0xeaba bytes) 000510 1734430687.775819 punch from 0x00e889 to 0x01ba34 (0xd1ac bytes) 000511 1734430687.779820 mapwrite 0x00cdf5 thru 0x012ac7 (0x5cd3 bytes) 000513 1734430687.802467 mapread 0x00ea3a thru 0x017dd6 (0x939d bytes) 000515 1734430687.813904 write 0x020aaf thru 0x03091b (0xfe6d bytes) 000517 1734430687.815420 trunc from 0x040000 to 0x01febe (0x20141 bytes) 000519 1734430687.832645 trunc from 0x01febf to 0x015c49 (0xa275 bytes) 000522 1734430687.842911 mapread 0x012c92 thru 0x014b5f (0x1ece bytes) 000523 1734430687.844987 trunc from 0x015c4a to 0x012086 (0x3bc3 bytes) 000526 1734430687.853689 punch from 0x00cfb3 to 0x010902 (0x3950 bytes) 000527 1734430687.857933 mapread 0x00d4cc thru 0x012086 (0x4bbb bytes) 000528 1734430687.869410 punch from 0x002189 to 0x004e26 (0x2c9e bytes) 000529 1734430687.871759 punch from 0x01171f to 0x012086 (0x968 bytes) 000530 1734430687.874545 punch from 0x011716 to 0x012086 (0x971 bytes) 000531 1734430687.877894 fallocate 0x02ab7e thru 0x031340 (0x67c3 bytes) 000533 1734430687.881307 mapwrite 0x02ee52 thru 0x02f20b (0x3ba bytes) 000534 1734430687.895114 fallocate 0x0078b5 thru 0x00978e (0x1eda bytes) 000535 1734430687.897410 mapwrite 0x018551 thru 0x01f462 (0x6f12 bytes) 000536 1734430687.923864 mapwrite 0x03641c thru 0x039c25 (0x380a bytes) 000537 1734430687.954115 fallocate 0x015749 thru 0x02345d (0xdd15 bytes) 000538 1734430687.956590 trunc from 0x039c26 to 0x02902e (0x10bf7 bytes) 000540 1734430687.967491 MIRROR_EXTEND to 2 mirrors 000541 1734430688.283104 fallocate 0x01a06f thru 0x023322 (0x92b4 bytes) 000542 1734430688.308589 MIRROR_EXTEND to 3 mirrors 000544 1734430688.603506 trunc from 0x02902f to 0x0222da (0x6d54 bytes) 000545 1734430688.633916 MIRROR_EXTEND to 4 mirrors 000546 1734430688.926261 read 0x008032 thru 0x0146b2 (0xc681 bytes) 000547 1734430688.938131 fallocate 0x02c2db thru 0x0364dd (0xa203 bytes) 000548 1734430688.960896 read 0x01f604 thru 0x02be6a (0xc867 bytes) 000550 1734430688.970066 trunc from 0x0364de to 0x00fe09 (0x266d4 bytes) 000551 1734430688.985665 read 0x000630 thru 0x00074b (0x11c bytes) 000552 1734430688.986124 write 0x031535 thru 0x034b94 (0x3660 bytes) 000554 1734430688.987194 fallocate 0x01b9f8 thru 0x01f013 (0x361c bytes) 000555 1734430688.991987 fallocate 0x01d9f0 thru 0x021c3c (0x424d bytes) 000556 1734430688.996567 read 0x0088bc thru 0x00d2b8 (0x49fd bytes) 000557 1734430689.000763 MIRROR_RESYNC 4 mirrors 000559 1734430689.350175 mapwrite 0x014cc1 thru 0x0174fb (0x283b bytes) 000560 1734430689.412485 punch from 0x017d76 to 0x018a52 (0xcdd bytes) 000561 1734430689.417716 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000562 1734430689.528339 trunc from 0x034b95 to 0x03ee97 (0xa303 bytes) 000563 1734430689.540761 write 0x00cc04 thru 0x00f168 (0x2565 bytes) 000564 1734430689.547344 fallocate 0x00653b thru 0x00ef8c (0x8a52 bytes) 000565 1734430689.551064 write 0x017f7a thru 0x01a520 (0x25a7 bytes) 000566 1734430689.558974 MIRROR_RESYNC 3 mirrors 000568 1734430689.945302 read 0x03c330 thru 0x03ee97 (0x2b68 bytes) 000569 1734430689.947889 punch from 0x03708a to 0x03ee97 (0x7e0e bytes) 000570 1734430689.967253 fallocate 0x01a912 thru 0x022688 (0x7d77 bytes) 000571 1734430689.970194 mapwrite 0x027ead thru 0x02f600 (0x7754 bytes) 000572 1734430690.004203 fallocate 0x02ea19 thru 0x031326 (0x290e bytes) 000573 1734430690.007171 punch from 0x02543c to 0x027bbd (0x2782 bytes) 000574 1734430690.014335 write 0x03d9d7 thru 0x03ffff (0x2629 bytes) 000575 1734430690.023695 punch from 0x004d67 to 0x005db7 (0x1051 bytes) 000576 1734430690.026455 write 0x03d271 thru 0x03dcc4 (0xa54 bytes) 000578 1734430690.027207 write 0x032f01 thru 0x03f703 (0xc803 bytes) 000579 1734430690.033686 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000581 1734430690.147581 trunc from 0x040000 to 0x03287a (0xd785 bytes) 000582 1734430690.155218 write 0x008f58 thru 0x0134be (0xa567 bytes) 000583 1734430690.161675 punch from 0x02f7c7 to 0x03287a (0x30b4 bytes) 000584 1734430690.165336 punch from 0x00510c to 0x0119f8 (0xc8ed bytes) 000585 1734430690.172751 mapread 0x01b99e thru 0x020a46 (0x50a9 bytes) 000586 1734430690.180379 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000587 1734430690.242868 write 0x0211cd thru 0x028895 (0x76c9 bytes) 000588 1734430690.251768 fallocate 0x0065ba thru 0x00a92f (0x4376 bytes) 000589 1734430690.254481 punch from 0x03034c to 0x03287a (0x252f bytes) 000590 1734430690.257046 punch from 0x026eed to 0x029496 (0x25aa bytes) 000591 1734430690.264332 fallocate 0x01c6b0 thru 0x01ee56 (0x27a7 bytes) 000593 1734430690.267515 trunc from 0x03287b to 0x027e29 (0xaa51 bytes) 000594 1734430690.275301 trunc from 0x027e2a to 0x018ee5 (0xef44 bytes) 000596 1734430690.282343 punch from 0x00b1ac to 0x00d573 (0x23c8 bytes) 000597 1734430690.285445 fallocate 0x0363d0 thru 0x03ffff (0x9c30 bytes) 000598 1734430690.290457 trunc from 0x040000 to 0x030aa1 (0xf55e bytes) 000599 1734430690.304989 trunc from 0x030aa2 to 0x033f6b (0x34ca bytes) 000600 1734430690.314889 mapread 0x01de84 thru 0x0224df (0x465c bytes) 000601 1734430690.322294 mapread 0x02266c thru 0x0293bb (0x6d50 bytes) 000603 1734430690.323279 write 0x037a62 thru 0x037bd5 (0x174 bytes) 000604 1734430690.323898 trunc from 0x037bd6 to 0x0258f0 (0x122e5 bytes) 000606 1734430690.332546 punch from 0x002d78 to 0x00fc87 (0xcf10 bytes) 000607 1734430690.339148 MIRROR_EXTEND to 2 mirrors 000608 1734430690.582496 fallocate 0x023db3 thru 0x02a1f5 (0x6443 bytes) 000612 1734430690.617298 mapwrite 0x00eeb1 thru 0x01d725 (0xe875 bytes) 000613 1734430690.651105 mapread 0x008996 thru 0x010a88 (0x80f3 bytes) 000614 1734430690.656139 trunc from 0x02a1f6 to 0x009bb2 (0x20643 bytes) 000615 1734430690.667304 write 0x02f85b thru 0x034a11 (0x51b7 bytes) 000616 1734430690.672880 MIRROR_RESYNC 2 mirrors 000617 1734430690.868346 MIRROR_EXTEND to 3 mirrors 000618 1734430691.103893 MIRROR_EXTEND to 4 mirrors 000619 1734430691.327819 write 0x030d7b thru 0x037288 (0x650e bytes) 000623 1734430691.355198 punch from 0x0295f3 to 0x02e99d (0x53ab bytes) 000624 1734430691.358409 punch from 0x01d9ab to 0x027dbc (0xa412 bytes) 000625 1734430691.362161 mapwrite 0x01d89f thru 0x0246cf (0x6e31 bytes) 000626 1734430691.378288 mapread 0x033ea2 thru 0x03594e (0x1aad bytes) 000627 1734430691.378980 fallocate 0x030320 thru 0x030405 (0xe6 bytes) 000630 1734430691.382894 punch from 0x00dcd1 to 0x01dcc7 (0xfff7 bytes) 000631 1734430691.386171 mapread 0x002227 thru 0x007f6d (0x5d47 bytes) 000632 1734430691.392199 MIRROR_RESYNC 4 mirrors 000633 1734430691.799134 mapwrite 0x020dc0 thru 0x028545 (0x7786 bytes) 000634 1734430691.865233 MIRROR_RESYNC 4 mirrors 000635 1734430692.277844 mapread 0x02a898 thru 0x032d31 (0x849a bytes) 000636 1734430692.295542 trunc from 0x037289 to 0x0190cb (0x1e1bd bytes) 000638 1734430692.324957 mapread 0x00be7b thru 0x012d78 (0x6efe bytes) 000639 1734430692.330105 read 0x00ef16 thru 0x012d62 (0x3e4d bytes) 000640 1734430692.330644 mapwrite 0x02343d thru 0x0328bf (0xf483 bytes) 000641 1734430692.379138 mapread 0x00b5ef thru 0x013b03 (0x8515 bytes) 000642 1734430692.381980 write 0x0355f4 thru 0x03ffff (0xaa0c bytes) 000643 1734430692.383509 write 0x01318f thru 0x019a67 (0x68d9 bytes) 000645 1734430692.393716 punch from 0x023545 to 0x028f20 (0x59dc bytes) 000646 1734430692.400027 write 0x02dfa8 thru 0x02e2a7 (0x300 bytes) 000647 1734430692.402270 mapread 0x0160b5 thru 0x021732 (0xb67e bytes) 000648 1734430692.407751 mapwrite 0x017375 thru 0x022077 (0xad03 bytes) 000649 1734430692.443421 mapread 0x02477f thru 0x02fd27 (0xb5a9 bytes) 000651 1734430692.450859 punch from 0x02818b to 0x034807 (0xc67d bytes) 000652 1734430692.465041 MIRROR_EXTEND to 5 mirrors 000653 1734430692.802334 write 0x02fe47 thru 0x039911 (0x9acb bytes) 000654 1734430692.831859 mapread 0x01f23e thru 0x029e6c (0xac2f bytes) 000655 1734430692.843504 mapwrite 0x035736 thru 0x035e9c (0x767 bytes) 000656 1734430692.862186 mapwrite 0x02465f thru 0x02f1e5 (0xab87 bytes) 000657 1734430692.875344 punch from 0x03700a to 0x03ffff (0x8ff6 bytes) 000658 1734430692.879019 read 0x003bf5 thru 0x00cd36 (0x9142 bytes) 000659 1734430692.886053 trunc from 0x040000 to 0x03804b (0x7fb4 bytes) 000660 1734430692.918338 punch from 0x016b18 to 0x01a089 (0x3572 bytes) 000663 1734430692.925380 write 0x0083ce thru 0x00d3e1 (0x5014 bytes) 000664 1734430692.932959 mapwrite 0x006e05 thru 0x00c7b2 (0x59ae bytes) 000666 1734430692.955262 read 0x022237 thru 0x02aa31 (0x87fb bytes) 000667 1734430692.955635 trunc from 0x03804c to 0x007249 (0x30e02 bytes) 000669 1734430692.971362 punch from 0x0058c7 to 0x007249 (0x1983 bytes) 000670 1734430692.974506 punch from 0x007129 to 0x007249 (0x121 bytes) 000671 1734430692.980950 mapread 0x00320a thru 0x005999 (0x2790 bytes) 000672 1734430692.986156 fallocate 0x010c90 thru 0x014aa7 (0x3e18 bytes) 000674 1734430692.992034 MIRROR_EXTEND to 6 mirrors 000675 1734430693.259131 write 0x033c1a thru 0x03d9ba (0x9da1 bytes) 000677 1734430693.280391 fallocate 0x0084c0 thru 0x00f1a7 (0x6ce8 bytes) 000678 1734430693.282861 trunc from 0x03d9bb to 0x02c23a (0x11780 bytes) 000679 1734430693.298003 mapread 0x0005f5 thru 0x00a78e (0xa19a bytes) 000681 1734430693.314124 mapread 0x028fe2 thru 0x02c23a (0x3259 bytes) 000682 1734430693.321584 mapread 0x01fd2f thru 0x02c23a (0xc50c bytes) 000683 1734430693.329777 punch from 0x006788 to 0x007e33 (0x16ac bytes) 000685 1734430693.334070 read 0x021363 thru 0x02bf3c (0xabda bytes) 000687 1734430693.337245 MIRROR_RESYNC 6 mirrors 000688 1734430693.826685 MIRROR_RESYNC 6 mirrors 000689 1734430693.934067 fallocate 0x039aee thru 0x03b671 (0x1b84 bytes) 000690 1734430693.961310 mapwrite 0x014203 thru 0x0191cf (0x4fcd bytes) 000691 1734430693.987060 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000692 1734430694.092353 mapread 0x016dc1 thru 0x023eac (0xd0ec bytes) 000694 1734430694.116100 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000695 1734430694.212561 trunc from 0x02c23b to 0x028faa (0x3290 bytes) 000696 1734430694.232417 fallocate 0x004706 thru 0x011527 (0xce22 bytes) 000697 1734430694.238473 punch from 0x0266ac to 0x028faa (0x28ff bytes) 000698 1734430694.246118 MIRROR_EXTEND to 5 mirrors 000699 1734430694.564662 fallocate 0x01fb2b thru 0x0251a1 (0x5677 bytes) 000700 1734430694.585670 read 0x0052d3 thru 0x0104e7 (0xb215 bytes) 000701 1734430694.592813 write 0x003fc9 thru 0x008ebe (0x4ef6 bytes) 000702 1734430694.598867 mapwrite 0x00ba95 thru 0x01134d (0x58b9 bytes) 000703 1734430694.615683 punch from 0x01c983 to 0x0240b6 (0x7734 bytes) 000704 1734430694.619212 write 0x03d16e thru 0x03d3f9 (0x28c bytes) 000705 1734430694.622102 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000706 1734430694.718791 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000707 1734430694.821379 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000708 1734430694.933708 read 0x027f43 thru 0x033f3f (0xbffd bytes) 000709 1734430694.939568 mapread 0x026a0f thru 0x0326b0 (0xbca2 bytes) 000710 1734430694.944121 punch from 0x025010 to 0x028a6d (0x3a5e bytes) 000711 1734430694.946440 fallocate 0x01d2ac thru 0x020a7b (0x37d0 bytes) 000712 1734430694.948906 fallocate 0x03f0be thru 0x03ffff (0xf42 bytes) 000713 1734430694.951573 fallocate 0x020a47 thru 0x02e0be (0xd678 bytes) 000714 1734430694.953922 read 0x00801d thru 0x01488c (0xc870 bytes) 000716 1734430694.960549 fallocate 0x0070e2 thru 0x010825 (0x9744 bytes) 000717 1734430694.963430 trunc from 0x040000 to 0x01500a (0x2aff5 bytes) 000719 1734430694.974756 mapread 0x00bea9 thru 0x01075e (0x48b6 bytes) 000720 1734430694.975271 mapwrite 0x033de7 thru 0x039c4c (0x5e66 bytes) 000721 1734430695.011927 punch from 0x014e65 to 0x020dc8 (0xbf64 bytes) 000723 1734430695.014938 mapread 0x01a614 thru 0x0248b5 (0xa2a2 bytes) 000724 1734430695.045307 mapwrite 0x019147 thru 0x02657d (0xd437 bytes) 000725 1734430695.075944 punch from 0x032f98 to 0x036af6 (0x3b5f bytes) 000726 1734430695.080944 mapwrite 0x033a91 thru 0x03766a (0x3bda bytes) 000727 1734430695.104960 mapread 0x000266 thru 0x003906 (0x36a1 bytes) 000728 1734430695.105743 punch from 0x01de34 to 0x0277d8 (0x99a5 bytes) 000729 1734430695.108973 trunc from 0x039c4d to 0x0018b8 (0x38394 bytes) 000730 1734430695.117562 write 0x0378c1 thru 0x03ffff (0x873f bytes) 000731 1734430695.118710 punch from 0x021679 to 0x030f66 (0xf8ee bytes) 000732 1734430695.121298 read 0x00ba6f thru 0x00ec17 (0x31a9 bytes) 000733 1734430695.126602 fallocate 0x01be9a thru 0x01e299 (0x2400 bytes) 000734 1734430695.130978 trunc from 0x040000 to 0x032379 (0xdc86 bytes) 000735 1734430695.145225 mapwrite 0x02cdf2 thru 0x02d4aa (0x6b9 bytes) 000736 1734430695.155938 read 0x022d98 thru 0x02624d (0x34b6 bytes) 000737 1734430695.156474 mapread 0x007fb7 thru 0x0144a8 (0xc4f2 bytes) 000739 1734430695.157386 trunc from 0x03237a to 0x028f95 (0x93e4 bytes) 000740 1734430695.172948 MIRROR_EXTEND to 3 mirrors 000741 1734430695.377500 mapread 0x027d1e thru 0x028f95 (0x1278 bytes) 000742 1734430695.386832 punch from 0x018b4e to 0x023d05 (0xb1b8 bytes) 000743 1734430695.408305 read 0x027399 thru 0x02805f (0xcc7 bytes) 000744 1734430695.413289 fallocate 0x000744 thru 0x006ffd (0x68ba bytes) 000745 1734430695.416403 punch from 0x00b035 to 0x010e39 (0x5e05 bytes) 000746 1734430695.418329 fallocate 0x01bcea thru 0x026c5b (0xaf72 bytes) 000748 1734430695.422852 mapwrite 0x031345 thru 0x031d87 (0xa43 bytes) 000749 1734430695.473084 mapwrite 0x004ad4 thru 0x0127c1 (0xdcee bytes) 000751 1734430695.503703 read 0x024a55 thru 0x027b73 (0x311f bytes) 000752 1734430695.509408 fallocate 0x017b3b thru 0x01f8b5 (0x7d7b bytes) 000753 1734430695.513821 MIRROR_EXTEND to 4 mirrors 000754 1734430695.754353 fallocate 0x00df8e thru 0x01cc70 (0xece3 bytes) 000757 1734430695.776899 trunc from 0x031d88 to 0x020afc (0x1128b bytes) 000758 1734430695.813519 MIRROR_RESYNC 4 mirrors 000759 1734430696.174199 MIRROR_RESYNC 4 mirrors 000760 1734430696.282979 write 0x0201ff thru 0x0220c6 (0x1ec8 bytes) 000761 1734430696.312897 mapwrite 0x001450 thru 0x0044d8 (0x3089 bytes) 000762 1734430696.338286 mapread 0x003800 thru 0x00fa25 (0xc226 bytes) 000763 1734430696.340103 read 0x010330 thru 0x01c41f (0xc0f0 bytes) 000764 1734430696.349761 write 0x02f7c2 thru 0x0345d8 (0x4e17 bytes) 000766 1734430696.352186 trunc from 0x0345d9 to 0x0118f1 (0x22ce7 bytes) 000767 1734430696.370266 fallocate 0x035835 thru 0x03ffff (0xa7cb bytes) 000768 1734430696.373472 mapread 0x005bae thru 0x00d9e1 (0x7e34 bytes) 000769 1734430696.374627 punch from 0x000938 to 0x0041a1 (0x386a bytes) 000770 1734430696.378450 mapread 0x0021cf thru 0x00bee3 (0x9d15 bytes) 000771 1734430696.382645 write 0x03b582 thru 0x03ffff (0x4a7e bytes) 000772 1734430696.384272 mapread 0x010e8e thru 0x01c495 (0xb608 bytes) 000773 1734430696.392782 mapread 0x00dd7e thru 0x01c6ba (0xe93d bytes) 000774 1734430696.393365 mapwrite 0x035964 thru 0x039430 (0x3acd bytes) 000775 1734430696.420671 write 0x03ed68 thru 0x03ffff (0x1298 bytes) 000776 1734430696.421355 read 0x029ffe thru 0x0326dd (0x86e0 bytes) 000777 1734430696.434583 punch from 0x00585f to 0x012341 (0xcae3 bytes) 000778 1734430696.440735 trunc from 0x040000 to 0x03e067 (0x1f98 bytes) 000779 1734430696.477320 mapwrite 0x02ca75 thru 0x02fb48 (0x30d4 bytes) 000780 1734430696.505854 mapread 0x01297f thru 0x01cefe (0xa580 bytes) 000781 1734430696.511192 trunc from 0x03e068 to 0x00a06f (0x33ff8 bytes) 000782 1734430696.520357 mapread 0x006118 thru 0x00a06f (0x3f58 bytes) 000783 1734430696.534155 write 0x02c008 thru 0x02c707 (0x700 bytes) 000784 1734430696.534920 punch from 0x0272ce to 0x02c707 (0x543a bytes) 000785 1734430696.540790 punch from 0x00688c to 0x00c8c8 (0x603d bytes) 000786 1734430696.543030 read 0x010dbb thru 0x01803f (0x7285 bytes) 000787 1734430696.547049 trunc from 0x02c708 to 0x00a417 (0x222f0 bytes) 000788 1734430696.556895 read 0x0061aa thru 0x00a417 (0x426e bytes) 000790 1734430696.557704 read 0x0053ba thru 0x007b0d (0x2754 bytes) 000791 1734430696.558344 punch from 0x005f8c to 0x00a417 (0x448c bytes) 000792 1734430696.561375 fallocate 0x02fe83 thru 0x03cd32 (0xceb0 bytes) 000793 1734430696.564464 mapwrite 0x02c228 thru 0x0300b9 (0x3e92 bytes) 000794 1734430696.603472 punch from 0x007994 to 0x00e9a4 (0x7011 bytes) 000797 1734430696.607539 mapwrite 0x003736 thru 0x00864d (0x4f18 bytes) 000799 1734430696.620963 write 0x008823 thru 0x00fddd (0x75bb bytes) 000801 1734430696.622480 fallocate 0x00a29b thru 0x010e08 (0x6b6e bytes) 000802 1734430696.627832 mapread 0x009a50 thru 0x00b28b (0x183c bytes) 000803 1734430696.632525 MIRROR_RESYNC 4 mirrors 000804 1734430697.078788 MIRROR_EXTEND to 5 mirrors 000805 1734430697.359173 trunc from 0x0300ba to 0x024c22 (0xb497 bytes) 000806 1734430697.402755 fallocate 0x02fd50 thru 0x03b044 (0xb2f5 bytes) 000808 1734430697.410620 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000810 1734430697.522639 write 0x00fecc thru 0x01b953 (0xba88 bytes) 000811 1734430697.535825 punch from 0x02271f to 0x024c22 (0x2504 bytes) 000812 1734430697.538845 mapwrite 0x030ee7 thru 0x03e2e9 (0xd403 bytes) 000813 1734430697.581802 mapread 0x039ef4 thru 0x03e2e9 (0x43f6 bytes) 000814 1734430697.582426 mapwrite 0x009fb2 thru 0x00c6fd (0x274c bytes) 000816 1734430697.603037 trunc from 0x03e2ea to 0x02537c (0x18f6d bytes) 000817 1734430697.612144 mapwrite 0x00a765 thru 0x015704 (0xafa0 bytes) 000818 1734430697.640697 trunc from 0x02537d to 0x030218 (0xae9c bytes) 000819 1734430697.653003 read 0x0196c1 thru 0x01a8c2 (0x1202 bytes) 000820 1734430697.653683 punch from 0x002e07 to 0x00d37c (0xa576 bytes) 000821 1734430697.658438 fallocate 0x01bf51 thru 0x02174c (0x57fc bytes) 000822 1734430697.663688 read 0x02affe thru 0x030218 (0x521b bytes) 000823 1734430697.670919 read 0x007894 thru 0x00a9cd (0x313a bytes) 000824 1734430697.672240 read 0x010622 thru 0x01b441 (0xae20 bytes) 000825 1734430697.672760 trunc from 0x030219 to 0x00f763 (0x20ab5 bytes) 000827 1734430697.687225 fallocate 0x02ea81 thru 0x032b38 (0x40b8 bytes) 000828 1734430697.689438 mapread 0x00c9a8 thru 0x00f763 (0x2dbc bytes) 000829 1734430697.690636 trunc from 0x00f764 to 0x00eedd (0x886 bytes) 000831 1734430697.702967 MIRROR_RESYNC 4 mirrors 000832 1734430698.057033 fallocate 0x009298 thru 0x015e7f (0xcbe8 bytes) 000833 1734430698.081500 trunc from 0x015e80 to 0x036b7d (0x20cfe bytes) 000835 1734430698.113152 mapread 0x028f85 thru 0x036b7d (0xdbf9 bytes) 000836 1734430698.145295 mapread 0x034f5b thru 0x036b7d (0x1c23 bytes) 000837 1734430698.147062 punch from 0x02ca01 to 0x036b7d (0xa17d bytes) 000838 1734430698.153571 mapread 0x035ff6 thru 0x036b7d (0xb88 bytes) 000839 1734430698.161224 punch from 0x012c03 to 0x01453b (0x1939 bytes) 000840 1734430698.164787 read 0x008cbb thru 0x00b4e5 (0x282b bytes) 000841 1734430698.170686 MIRROR_EXTEND to 5 mirrors 000842 1734430698.542013 write 0x02f57e thru 0x03a62b (0xb0ae bytes) 000843 1734430698.568777 trunc from 0x03a62c to 0x0197ca (0x20e61 bytes) 000844 1734430698.598792 fallocate 0x03b631 thru 0x03ffff (0x49cf bytes) 000845 1734430698.607511 MIRROR_RESYNC 5 mirrors 000846 1734430699.286233 write 0x035abb thru 0x03ffff (0xa545 bytes) 000848 1734430699.317255 mapwrite 0x022217 thru 0x0319ed (0xf7d7 bytes) 000849 1734430699.347439 fallocate 0x02e045 thru 0x0343d2 (0x638e bytes) 000850 1734430699.350751 mapread 0x013d13 thru 0x01cba2 (0x8e90 bytes) 000851 1734430699.365814 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000852 1734430699.462344 punch from 0x03b20a to 0x03f4c5 (0x42bc bytes) 000853 1734430699.465363 punch from 0x0095e2 to 0x00b70e (0x212d bytes) 000854 1734430699.468709 punch from 0x030b5c to 0x036e28 (0x62cd bytes) 000855 1734430699.471492 mapread 0x016eaa thru 0x024fe4 (0xe13b bytes) 000856 1734430699.488485 punch from 0x02db3c to 0x035ef1 (0x83b6 bytes) 000857 1734430699.492208 trunc from 0x040000 to 0x030cab (0xf354 bytes) 000858 1734430699.508647 MIRROR_EXTEND to 5 mirrors 000860 1734430699.841498 fallocate 0x0108aa thru 0x019606 (0x8d5d bytes) 000862 1734430699.867546 mapwrite 0x000eb0 thru 0x00956e (0x86bf bytes) 000863 1734430699.903620 MIRROR_RESYNC 5 mirrors 000864 1734430700.515673 MIRROR_RESYNC 5 mirrors 000865 1734430700.627955 read 0x00d9d1 thru 0x01241f (0x4a4f bytes) 000866 1734430700.634485 mapread 0x02c7df thru 0x030cab (0x44cd bytes) 000867 1734430700.645089 punch from 0x01a73a to 0x01c745 (0x200c bytes) 000868 1734430700.667233 read 0x00f980 thru 0x010fd5 (0x1656 bytes) 000869 1734430700.674170 mapread 0x01eac1 thru 0x024973 (0x5eb3 bytes) 000870 1734430700.674907 punch from 0x02ded1 to 0x030cab (0x2ddb bytes) 000871 1734430700.677755 punch from 0x019317 to 0x020274 (0x6f5e bytes) 000872 1734430700.681243 read 0x030010 thru 0x030cab (0xc9c bytes) 000873 1734430700.688304 mapwrite 0x020795 thru 0x020b95 (0x401 bytes) 000874 1734430700.700482 mapwrite 0x00591b thru 0x01097e (0xb064 bytes) 000875 1734430700.714581 trunc from 0x030cac to 0x006068 (0x2ac43 bytes) 000876 1734430700.725949 punch from 0x001033 to 0x006068 (0x5036 bytes) 000878 1734430700.735794 write 0x02ae91 thru 0x0311d4 (0x6344 bytes) 000879 1734430700.737392 write 0x0037ba thru 0x00d923 (0xa16a bytes) 000880 1734430700.748407 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000881 1734430700.861223 trunc from 0x0311d5 to 0x002c43 (0x2e591 bytes) 000882 1734430700.873289 write 0x006018 thru 0x00eb59 (0x8b42 bytes) 000883 1734430700.875052 write 0x0305f4 thru 0x0380c1 (0x7ace bytes) 000885 1734430700.876639 fallocate 0x00b8f5 thru 0x010efd (0x5609 bytes) 000886 1734430700.883456 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000887 1734430701.032161 write 0x0362ea thru 0x03cc11 (0x6928 bytes) 000888 1734430701.053622 fallocate 0x0391c1 thru 0x03e2e6 (0x5126 bytes) 000889 1734430701.059429 mapread 0x0215b1 thru 0x024361 (0x2db1 bytes) 000890 1734430701.074835 MIRROR_RESYNC 3 mirrors 000891 1734430701.454336 read 0x0111d4 thru 0x015d37 (0x4b64 bytes) 000892 1734430701.460223 fallocate 0x029716 thru 0x029724 (0xf bytes) 000893 1734430701.481839 write 0x01d068 thru 0x02bb27 (0xeac0 bytes) 000894 1734430701.489861 write 0x031548 thru 0x03cac6 (0xb57f bytes) 000895 1734430701.498926 mapwrite 0x005ed3 thru 0x00b43d (0x556b bytes) 000896 1734430701.521207 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000897 1734430701.671081 trunc from 0x03e2e7 to 0x009722 (0x34bc4 bytes) 000898 1734430701.705156 mapread 0x001bac thru 0x009722 (0x7b77 bytes) 000899 1734430701.712901 mapread 0x0089e2 thru 0x009722 (0xd41 bytes) 000901 1734430701.713745 write 0x01a623 thru 0x01b55a (0xf38 bytes) 000902 1734430701.714392 punch from 0x00ffcb to 0x01b55a (0xb590 bytes) 000903 1734430701.720285 fallocate 0x02e9c0 thru 0x032b3a (0x417b bytes) 000904 1734430701.722951 fallocate 0x013d35 thru 0x01b9c3 (0x7c8f bytes) 000905 1734430701.729418 read 0x021cb6 thru 0x024933 (0x2c7e bytes) 000908 1734430701.735377 punch from 0x0102a6 to 0x019799 (0x94f4 bytes) 000909 1734430701.741440 punch from 0x001bbf to 0x00c13a (0xa57c bytes) 000910 1734430701.746860 punch from 0x0202ef to 0x02b2d7 (0xafe9 bytes) 000911 1734430701.752578 fallocate 0x009095 thru 0x014268 (0xb1d4 bytes) 000912 1734430701.759076 MIRROR_EXTEND to 3 mirrors 000914 1734430702.087646 trunc from 0x032b3b to 0x0305e8 (0x2552 bytes) 000915 1734430702.135017 write 0x01ef82 thru 0x022b98 (0x3c17 bytes) 000917 1734430702.143142 mapread 0x026f5d thru 0x02bf53 (0x4ff7 bytes) 000919 1734430702.153316 MIRROR_RESYNC 3 mirrors 000920 1734430702.563594 mapread 0x005f43 thru 0x00619c (0x25a bytes) 000921 1734430702.575751 write 0x00161c thru 0x00fb02 (0xe4e7 bytes) 000922 1734430702.608904 mapread 0x023e34 thru 0x028b7f (0x4d4c bytes) 000923 1734430702.615150 mapwrite 0x020eff thru 0x023af3 (0x2bf5 bytes) 000924 1734430702.632986 punch from 0x022755 to 0x0305e8 (0xde94 bytes) 000925 1734430702.637927 read 0x00f495 thru 0x0117d5 (0x2341 bytes) 000926 1734430702.642121 write 0x00ff17 thru 0x01fc65 (0xfd4f bytes) 000927 1734430702.643273 mapwrite 0x02dbc3 thru 0x034d7d (0x71bb bytes) 000928 1734430702.687121 write 0x026684 thru 0x034932 (0xe2af bytes) 000929 1734430702.688217 write 0x008165 thru 0x00eee4 (0x6d80 bytes) 000930 1734430702.688988 mapwrite 0x00a471 thru 0x010876 (0x6406 bytes) 000931 1734430702.719067 fallocate 0x001df5 thru 0x002912 (0xb1e bytes) 000932 1734430702.721540 write 0x014dac thru 0x022ff0 (0xe245 bytes) 000933 1734430702.726831 write 0x01786d thru 0x01cafe (0x5292 bytes) 000934 1734430702.728225 write 0x01b543 thru 0x01fabb (0x4579 bytes) 000935 1734430702.728818 mapread 0x023f0f thru 0x0335c7 (0xf6b9 bytes) 000937 1734430702.735782 mapwrite 0x0196fb thru 0x01b818 (0x211e bytes) 000938 1734430702.749043 punch from 0x0132b4 to 0x015932 (0x267f bytes) 000939 1734430702.753536 mapwrite 0x0034cc thru 0x006b61 (0x3696 bytes) 000940 1734430702.766362 trunc from 0x034d7e to 0x02d1f0 (0x7b8d bytes) 000941 1734430702.780471 MIRROR_EXTEND to 4 mirrors 000943 1734430703.083955 mapread 0x027fad thru 0x02831f (0x373 bytes) 000944 1734430703.103544 write 0x03d57b thru 0x03ffff (0x2a85 bytes) 000946 1734430703.134135 write 0x025b3e thru 0x02a687 (0x4b4a bytes) 000947 1734430703.142042 trunc from 0x040000 to 0x010c5b (0x2f3a4 bytes) 000948 1734430703.169890 write 0x02cc96 thru 0x033c1d (0x6f88 bytes) 000949 1734430703.170789 mapread 0x024835 thru 0x02c79c (0x7f68 bytes) 000950 1734430703.176580 mapwrite 0x0017b1 thru 0x0112d4 (0xfb24 bytes) 000951 1734430703.200675 trunc from 0x033c1e to 0x0168b5 (0x1d368 bytes) 000952 1734430703.213639 punch from 0x0158f9 to 0x0168b5 (0xfbd bytes) 000955 1734430703.217179 mapwrite 0x026e0d thru 0x036dae (0xffa2 bytes) 000957 1734430703.252332 mapread 0x011886 thru 0x01bca7 (0xa422 bytes) 000958 1734430703.258559 punch from 0x02c924 to 0x036dae (0xa48b bytes) 000959 1734430703.262163 mapwrite 0x02e62a thru 0x03064b (0x2022 bytes) 000960 1734430703.278370 punch from 0x0107fe to 0x016ccf (0x64d2 bytes) 000961 1734430703.282192 trunc from 0x036daf to 0x022d36 (0x14078 bytes) 000962 1734430703.295121 punch from 0x010cba to 0x016fbe (0x6305 bytes) 000963 1734430703.299910 punch from 0x022d1a to 0x022d36 (0x1d bytes) 000964 1734430703.302325 fallocate 0x02b5cf thru 0x039fe7 (0xea19 bytes) 000965 1734430703.306065 read 0x016a21 thru 0x022d36 (0xc316 bytes) 000966 1734430703.310638 write 0x02fa95 thru 0x03c2e8 (0xc854 bytes) 000967 1734430703.311750 fallocate 0x00bc4a thru 0x0169da (0xad91 bytes) 000968 1734430703.321539 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000969 1734430703.450551 fallocate 0x01eb88 thru 0x01fdc1 (0x123a bytes) 000970 1734430703.456925 write 0x02d266 thru 0x03ae77 (0xdc12 bytes) 000971 1734430703.464376 mapwrite 0x01ee36 thru 0x01ee44 (0xf bytes) 000972 1734430703.481618 fallocate 0x03926d thru 0x03fe54 (0x6be8 bytes) 000973 1734430703.485646 mapwrite 0x011423 thru 0x01d0c8 (0xbca6 bytes) 000975 1734430703.505115 read 0x0226b6 thru 0x02cfc0 (0xa90b bytes) 000976 1734430703.509630 punch from 0x02ac43 to 0x034cc4 (0xa082 bytes) 000977 1734430703.513014 mapwrite 0x01dd0c thru 0x023cbb (0x5fb0 bytes) 000978 1734430703.524911 mapread 0x0150f7 thru 0x01d33b (0x8245 bytes) 000979 1734430703.529248 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x4a5:0x0]: Invalid argument (22) 000980 1734430703.684513 fallocate 0x025183 thru 0x030a9d (0xb91b bytes) 000981 1734430703.689894 punch from 0x016325 to 0x025e9f (0xfb7b bytes) 000982 1734430703.693299 write 0x00d980 thru 0x01580b (0x7e8c bytes) 000983 1734430703.702782 read 0x0232bf thru 0x02ec61 (0xb9a3 bytes) 000984 1734430703.707886 mapread 0x03677a thru 0x0373f5 (0xc7c bytes) 000985 1734430703.713095 read 0x029f42 thru 0x030e9b (0x6f5a bytes) 000986 1734430703.713880 write 0x00a121 thru 0x0121c6 (0x80a6 bytes) 000987 1734430703.720770 mapwrite 0x03d12d thru 0x03ffff (0x2ed3 bytes) 000988 1734430703.743904 mapwrite 0x0183de thru 0x02761a (0xf23d bytes) 000990 1734430703.762139 punch from 0x029ab7 to 0x032324 (0x886e bytes) 000991 1734430703.765301 mapwrite 0x02b9a0 thru 0x03511a (0x977b bytes) 000995 1734430703.811074 mapwrite 0x03c378 thru 0x03ffff (0x3c88 bytes) 000998 1734430703.821594 mapread 0x028b6d thru 0x035449 (0xc8dd bytes) 000999 1734430703.836340 punch from 0x039c59 to 0x03c4d1 (0x2879 bytes) All operations completed A-OK! PASS 70a (35s) == sanity-flr test 200a: stress test ===================== 05:18:26 (1734430706) Starting client: oleg436-client.virtnet: -o user_xattr,flock 192.168.204.136@tcp:/lustre /mnt/lustre2 Starting client: oleg436-client.virtnet: -o user_xattr,flock 192.168.204.136@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 6392736 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7678560 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8397984 .. Extending file size to 8017248 .. Extending file size to 8807488 .. Extending file size to 9057440 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9287936 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8921536 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 31868 31869 31870 31871 31874 done 192.168.204.136@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg436-client.virtnet /mnt/lustre2 (opts:) 192.168.204.136@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg436-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (66s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 05:19:32 (1734430772) Starting client: oleg436-client.virtnet: -o user_xattr,flock 192.168.204.136@tcp:/lustre /mnt/lustre2 Starting client: oleg436-client.virtnet: -o user_xattr,flock 192.168.204.136@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 1274848 .. Extending file size to 1597376 .. failed Extending file size to 7331456 .. Extending file size to 2514432 .. Extending file size to 2977888 .. Extending file size to 7498208 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7447072 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8217856 .. Extending file size to 8241280 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8399872 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9023520 .. Extending file size to 8468160 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8794784 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9081472 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 1387 1388 1390 1391 1392 192.168.204.136@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg436-client.virtnet /mnt/lustre2 (opts:) 192.168.204.136@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg436-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (65s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 05:20:37 (1734430837) Starting client: oleg436-client.virtnet: -o user_xattr,flock 192.168.204.136@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2336 1285352 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 22420 3584600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 14572 3592448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 36992 7177048 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2336 1285352 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2440 1285248 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 22420 3584600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 14572 3592448 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 36992 7177048 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0468732 s, 44.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.091716 s, 22.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.136@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg436-client.virtnet /mnt/lustre2 (opts:) PASS 200c (7s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 05:20:45 (1734430845) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0580088 s, 36.2 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 05:20:48 (1734430848) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x39:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x81:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x81:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x82:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x9e:0x0] } PASS 203 (4s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 05:20:52 (1734430852) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0402831 s, 52.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x506:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x507:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x508:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 05:20:55 (1734430855) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0462322 s, 45.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x83:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x84:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x9f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x83:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x84:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x9f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa0:0x0] } PASS 204b (4s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 05:20:59 (1734430859) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0355451 s, 59.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x85:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x85:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa2:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 05:21:24 (1734430884) Waiting for MDT destroys to complete oleg436-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 oleg436-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg436-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0344534 s, 30.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x507:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x50a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x509:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x507:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x50a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x509:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x508:0x0] } PASS 204d (35s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 05:21:59 (1734430919) Waiting for MDT destroys to complete oleg436-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 oleg436-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg436-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0558074 s, 37.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x86:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x86:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (35s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 05:22:34 (1734430954) Waiting for MDT destroys to complete oleg436-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 oleg436-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg436-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg436-server: Pool lustre.test_204f created Adding targets to pool oleg436-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg436-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0392709 s, 53.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x87:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x87:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg436-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg436-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg436-server: Pool lustre.test_204f destroyed PASS 204f (49s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 05:23:23 (1734431003) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 05:23:26 (1734431006) PASS 205b (4s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 05:23:30 (1734431010) oleg436-server: Pool lustre.test_206 created Adding targets to pool oleg436-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg436-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg436-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg436-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg436-server: Pool lustre.test_206 destroyed PASS 206 (18s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 05:23:48 (1734431028) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0423544 s, 24.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0206229 s, 50.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xc752aae5 Mirror 3: 0xc752aae5 PASS 207 (4s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 05:23:52 (1734431032) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 05:23:54 (1734431034) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 05:23:56 (1734431036) Starting client: oleg436-client.virtnet: -o user_xattr,flock 192.168.204.136@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0126769 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1b20000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1850000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1850000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1850000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x24f0000 Read this (4 bytes): BBBB 192.168.204.136@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg436-client.virtnet /mnt/lustre2 (opts:) PASS 209a (19s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 05:24:14 (1734431054) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00578195 s, 1.4 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 05:24:16 (1734431056) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0131505 s, 79.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5a lmm_fid: [0x200000405:0x5a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5a lmm_fid: [0x200000405:0x5a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x92:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611226 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-12-17 05:24:17.000000000 -0500 Modify: 2024-12-17 05:24:17.000000000 -0500 Change: 2024-12-17 05:24:17.000000000 -0500 Birth: - PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 05:24:20 (1734431060) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0137298 s, 76.4 MB/s oleg436-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg436-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg436-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg436-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 oleg436-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg436-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg436-server: O/280000401/d15/af: File not found by ext2_lookup PASS 210b (14s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 05:24:34 (1734431074) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0918321 s, 446 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0101895 s, 402 kB/s size after second write -rw-r--r-- 1 root root 4096 Dec 17 05:24 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5f:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Dec 17 05:24 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test complete, duration 853 sec ============ 05:24:38 (1734431078) === sanity-flr: start cleanup 05:24:38 (1734431078) === === sanity-flr: finish cleanup 05:24:42 (1734431082) ===