-----============= acceptance-small: sanity-flr ============----- Fri Dec 20 01:14:07 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg203-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 01:14:15 (1734675255) === oleg203-client.virtnet: executing check_config_client /mnt/lustre oleg203-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg203-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012a6dc800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012a6dc800.idle_timeout=debug disable quota as required oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 01:14:32 (1734675272) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7608] == sanity-flr test 0a: lfs mirror create with -N option == 01:14:33 (1734675273) PASS 0a (7s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:14:40 (1734675280) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:14:42 (1734675282) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:14:43 (1734675283) PASS 0d (10s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:14:53 (1734675293) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:14:55 (1734675295) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 01:14:57 (1734675297) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88012a6dc800.stats=clear osc.lustre-OST0001-osc-ffff88012a6dc800.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:15:00 (1734675300) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (5s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:15:06 (1734675306) PASS 0j (5s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:15:10 (1734675310) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:15:12 (1734675312) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (5s) == sanity-flr test 2: create components from existing files ========================================================== 01:15:16 (1734675316) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:15:20 (1734675320) PASS 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:15:24 (1734675324) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:15:28 (1734675328) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (4s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:15:34 (1734675334) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:15:37 (1734675337) 11+0 records in 11+0 records out 11534336 bytes (12 MB) copied, 0.512647 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0469797 s, 22.3 MB/s PASS 21 (4s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:15:41 (1734675341) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0191922 s, 109 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:15:44 (1734675344) Stopping /mnt/lustre-ost1 (opts:) on oleg203-server oleg203-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a6dc800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a6dc800.ost_server_uuid in DISCONN state after 1 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg203-server: oleg203-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg203-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg203-server oleg203-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a6dc800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a6dc800.ost_server_uuid in DISCONN state after 0 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg203-server: oleg203-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg203-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (34s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:16:18 (1734675378) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.098202 s, 42.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg203-server oleg203-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a6dc800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a6dc800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg203-server: oleg203-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg203-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (20s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:16:38 (1734675398) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg203-server oleg203-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a6dc800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a6dc800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg203-server: oleg203-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg203-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg203-server oleg203-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a6dc800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a6dc800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg203-server: oleg203-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg203-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (32s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:17:10 (1734675430) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 1.64858 s, 31.8 MB/s -rw-r--r-- 1 root root 52428800 Dec 20 01:17 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.572s user 0m0.000s sys 0m0.156s Stopping /mnt/lustre-ost1 (opts:) on oleg203-server oleg203-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a6dc800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a6dc800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.373s user 0m0.000s sys 0m0.146s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg203-server: oleg203-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg203-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg203-server oleg203-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a6dc800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a6dc800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.448s user 0m0.000s sys 0m0.161s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg203-server: oleg203-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg203-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg203-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012a6dc800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a6dc800.ost_server_uuid in FULL state after 0 sec PASS 33b (39s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:17:50 (1734675470) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:17:51 (1734675471) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:17:53 (1734675473) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 01:17:54 (1734675474) PASS 35 (4s) == sanity-flr test 36a: write to mirrored files ========== 01:17:58 (1734675478) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.359881 s, 35.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (6s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:18:04 (1734675484) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0190911 s, 54.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0157213 s, 66.7 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:18:07 (1734675487) PASS 36c (5s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:18:12 (1734675492) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00365534 s, 280 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00128344 s, 798 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00105457 s, 971 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000724463 s, 1.4 MB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 01:18:16 (1734675496) create test files with size 15749858 .. b0ad5fe3c9a54be1d77c0cd692272c17 - b48ce19c105ad5cd6b089c6544597613 - 46b99c33baed639759a975b449e170e0 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg203-client.virtnet /mnt/lustre (opts:) Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre PASS 37 (10s) == sanity-flr test 38: resync ============================ 01:18:26 (1734675506) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (5s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:18:32 (1734675512) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:18:34 (1734675514) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0403541 s, 26.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x37:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x38:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0214857 s, 48.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3d:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 01:18:38 (1734675518) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0396722 s, 52.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.138026 s, 30.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0645954 s, 32.5 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (5s) == sanity-flr test 42: lfs mirror verify ================= 01:18:43 (1734675523) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 01:18:44 (1734675524) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:18:45 (1734675525) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0141762 s, 74.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 01:18:48 (1734675528) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.146773 s, 21.4 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x10:0x0]: Invalid argument (22) PASS 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:18:53 (1734675533) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x42:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x47:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x15:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:18:56 (1734675536) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0942228 s, 111 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg203-client.virtnet /mnt/lustre (opts:) Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (16s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:19:12 (1734675552) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800b60b6800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.116296 s, 90.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg203-client.virtnet /mnt/lustre (opts:) Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff88012e166000.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:19:17 (1734675557) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (11s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:19:28 (1734675568) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (10s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:19:38 (1734675578) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 01:19:39 (1734675579) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0883813 s, 35.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0629221 s, 50.0 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 43b05cbad62a9b51b77e7b44c950e2d6 - mirror 1 checksum: f7cdaf19d59867a3e8bd9355f9bbf330 - mirror 2 checksum: 43b05cbad62a9b51b77e7b44c950e2d6 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Dec 20 01:19:40 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4c:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Dec 20 01:19:40 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 43b05cbad62a9b51b77e7b44c950e2d6 - mirror 1 checksum: f7cdaf19d59867a3e8bd9355f9bbf330 - mirror 2 checksum: f7cdaf19d59867a3e8bd9355f9bbf330 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Dec 20 01:19:40 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4c:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Dec 20 01:19:40 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } PASS 48 (4s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 01:19:43 (1734675583) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0121278 s, 86.5 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:19:45 (1734675585) Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 01:19:53 (1734675593) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 01:20:08 (1734675608) targets are mounted 01:20:08 (1734675608) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0178172 s, 58.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } PASS 50A (31s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:20:16 (1734675616) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0190848 s, 1.1 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0100987 s, 2.0 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (5s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:20:21 (1734675621) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0282719 s, 724 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00624575 s, 3.3 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:20:25 (1734675625) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00451403 s, 3.6 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:20:28 (1734675628) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.038011 s, 539 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:20:31 (1734675631) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0032203 s, 1.3 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0225826 s, 181 kB/s PASS 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:20:35 (1734675635) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) PASS 60b (7s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:20:42 (1734675642) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:21:02 (1734675662) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:21:21 (1734675681) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:21:39 (1734675699) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 01:21:42 (1734675702) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 26351 mirror split pid 26352 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4860: 26351 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4860: 26352 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 01:22:44 (1734675764) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 5764 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported 000001 1734675764.751033 write 0x025e48 thru 0x02823d (0x23f6 bytes) 000002 1734675764.756035 read 0x012bc9 thru 0x020836 (0xdc6e bytes) 000003 1734675764.760884 punch from 0x0035e9 to 0x00909f (0x5ab7 bytes) truncating to largest ever: 0x24501 000004 1734675764.765061 trunc from 0x02823e to 0x024500 (0x3d3d bytes) 000005 1734675764.772063 write 0x03f2ff thru 0x03ffff (0xd01 bytes) 000006 1734675764.772738 fallocate 0x008716 thru 0x014028 (0xb913 bytes) fallocating to largest ever: 0x288cb 000007 1734675764.777682 fallocate 0x0202a7 thru 0x0288ca (0x8624 bytes) 000008 1734675764.783954 mapwrite 0x02867e thru 0x034043 (0xb9c6 bytes) 000011 1734675764.830956 write 0x009a8d thru 0x00ee88 (0x53fc bytes) 000012 1734675764.835278 mapread 0x00747c thru 0x010763 (0x92e8 bytes) 000013 1734675764.836136 punch from 0x01e80e to 0x02a380 (0xbb73 bytes) 000014 1734675764.842775 MIRROR_EXTEND to 2 mirrors 000015 1734675765.113415 mapwrite 0x0284a1 thru 0x033127 (0xac87 bytes) fallocating to largest ever: 0x38120 000016 1734675765.183192 fallocate 0x032f77 thru 0x03811f (0x51a9 bytes) 000017 1734675765.185973 punch from 0x03113a to 0x03ffff (0xeec6 bytes) 000018 1734675765.188375 mapwrite 0x02dbc1 thru 0x03d979 (0xfdb9 bytes) 000019 1734675765.211757 punch from 0x031039 to 0x039f8b (0x8f53 bytes) 000020 1734675765.215650 write 0x02ab86 thru 0x03a867 (0xfce2 bytes) 000021 1734675765.219617 mapwrite 0x0144db thru 0x014579 (0x9f bytes) 000022 1734675765.235442 mapwrite 0x022838 thru 0x030d85 (0xe54e bytes) 000023 1734675765.254162 MIRROR_RESYNC 2 mirrors 000024 1734675765.477451 trunc from 0x040000 to 0x015bc0 (0x2a43f bytes) 000025 1734675765.497596 mapwrite 0x03dbdc thru 0x03ffff (0x2424 bytes) 000026 1734675765.522156 mapread 0x028773 thru 0x02c02a (0x38b8 bytes) 000028 1734675765.527912 mapwrite 0x0065e5 thru 0x00cf17 (0x6933 bytes) 000029 1734675765.543562 mapwrite 0x03daa8 thru 0x03ffff (0x2558 bytes) 000031 1734675765.552902 write 0x03b196 thru 0x03ffff (0x4e6a bytes) 000032 1734675765.562557 MIRROR_EXTEND to 3 mirrors 000033 1734675765.815767 mapwrite 0x03d6a4 thru 0x03ffff (0x295c bytes) 000034 1734675765.876402 fallocate 0x018e7b thru 0x026e33 (0xdfb9 bytes) 000037 1734675765.880613 fallocate 0x00c172 thru 0x00cdab (0xc3a bytes) 000039 1734675765.889813 MIRROR_EXTEND to 4 mirrors 000041 1734675766.135863 fallocate 0x01171a thru 0x01e308 (0xcbef bytes) 000042 1734675766.150792 read 0x0091c0 thru 0x00ae7d (0x1cbe bytes) 000043 1734675766.153647 punch from 0x0035ae to 0x005cd7 (0x272a bytes) 000044 1734675766.156029 mapwrite 0x01cbad thru 0x0217c4 (0x4c18 bytes) 000045 1734675766.176108 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d1:0x0]: Invalid argument (22) 000046 1734675766.260481 trunc from 0x040000 to 0x0268d5 (0x1972a bytes) 000048 1734675766.270687 mapread 0x0014ad thru 0x002b2e (0x1682 bytes) 000050 1734675766.276399 trunc from 0x0268d6 to 0x022318 (0x45bd bytes) 000051 1734675766.285265 write 0x02d4b2 thru 0x0307c6 (0x3315 bytes) 000055 1734675766.287018 read 0x003fee thru 0x00bc9a (0x7cad bytes) 000056 1734675766.291172 read 0x01c0c3 thru 0x01ddf3 (0x1d31 bytes) 000057 1734675766.296439 mapread 0x0139a9 thru 0x0161a6 (0x27fe bytes) 000058 1734675766.297167 fallocate 0x011282 thru 0x0204d9 (0xf258 bytes) 000059 1734675766.300143 read 0x0066ca thru 0x0131f1 (0xcb28 bytes) 000060 1734675766.300599 read 0x02a0af thru 0x0307c6 (0x6718 bytes) 000063 1734675766.301718 write 0x00a6d8 thru 0x01a177 (0xfaa0 bytes) 000064 1734675766.302864 punch from 0x028a2c to 0x0307c6 (0x7d9b bytes) 000066 1734675766.310832 fallocate 0x007dad thru 0x00fb43 (0x7d97 bytes) 000067 1734675766.313086 read 0x0214d2 thru 0x02d56c (0xc09b bytes) 000068 1734675766.317215 fallocate 0x000dd8 thru 0x009f76 (0x919f bytes) 000069 1734675766.319830 punch from 0x00cb6e to 0x0119c6 (0x4e59 bytes) 000070 1734675766.322355 write 0x027aaa thru 0x02a198 (0x26ef bytes) 000071 1734675766.323000 read 0x01a89b thru 0x02a7b5 (0xff1b bytes) 000072 1734675766.323466 punch from 0x02a162 to 0x0307c6 (0x6665 bytes) 000073 1734675766.329178 write 0x02968d thru 0x02c941 (0x32b5 bytes) 000075 1734675766.333359 fallocate 0x0276b1 thru 0x0302f1 (0x8c41 bytes) 000076 1734675766.336840 read 0x02a761 thru 0x0307c6 (0x6066 bytes) 000077 1734675766.340568 punch from 0x013b99 to 0x01bbfe (0x8066 bytes) 000078 1734675766.343994 read 0x01868f thru 0x01dadb (0x544d bytes) 000079 1734675766.349784 write 0x031eb0 thru 0x034ffb (0x314c bytes) 000080 1734675766.350652 mapread 0x00485c thru 0x01086c (0xc011 bytes) 000081 1734675766.351368 trunc from 0x034ffc to 0x00e786 (0x26875 bytes) 000082 1734675766.359138 punch from 0x003040 to 0x00e56c (0xb52d bytes) 000083 1734675766.362744 read 0x00a0d2 thru 0x00e786 (0x46b5 bytes) 000084 1734675766.368389 read 0x00b3d9 thru 0x00be5d (0xa85 bytes) 000085 1734675766.368835 fallocate 0x006123 thru 0x0118d7 (0xb7b5 bytes) 000086 1734675766.372121 mapwrite 0x000156 thru 0x0087ad (0x8658 bytes) 000087 1734675766.383533 read 0x00031d thru 0x008a87 (0x876b bytes) 000088 1734675766.384104 write 0x01908f thru 0x01b2b2 (0x2224 bytes) 000089 1734675766.387348 MIRROR_EXTEND to 4 mirrors 000090 1734675766.569719 punch from 0x006e8a to 0x00db74 (0x6ceb bytes) 000091 1734675766.586218 write 0x0009ff thru 0x004fa8 (0x45aa bytes) 000093 1734675766.593094 write 0x020750 thru 0x02cc46 (0xc4f7 bytes) 000094 1734675766.593910 mapwrite 0x00e58c thru 0x00f67a (0x10ef bytes) 000095 1734675766.615184 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d1:0x0]: Invalid argument (22) 000097 1734675766.713755 mapread 0x00c382 thru 0x014c9f (0x891e bytes) 000098 1734675766.725528 mapwrite 0x00d78c thru 0x018d21 (0xb596 bytes) 000099 1734675766.737395 trunc from 0x02cc47 to 0x00f7c6 (0x1d480 bytes) 000100 1734675766.759987 write 0x030c01 thru 0x0357df (0x4bdf bytes) 000102 1734675766.761074 write 0x0186c2 thru 0x01dbe5 (0x5524 bytes) 000103 1734675766.765999 punch from 0x010e85 to 0x0159e0 (0x4b5c bytes) 000105 1734675766.770291 trunc from 0x0357e0 to 0x02fd62 (0x5a7d bytes) 000108 1734675766.779824 trunc from 0x02fd63 to 0x0320b8 (0x2356 bytes) 000110 1734675766.786236 fallocate 0x029d62 thru 0x033bbc (0x9e5b bytes) 000111 1734675766.788763 mapwrite 0x001d9a thru 0x005d09 (0x3f70 bytes) 000112 1734675766.802646 mapwrite 0x010ebb thru 0x01a58f (0x96d5 bytes) 000113 1734675766.817123 fallocate 0x0325b9 thru 0x0351c9 (0x2c11 bytes) 000115 1734675766.820768 trunc from 0x033bbd to 0x00cd48 (0x26e74 bytes) 000116 1734675766.828704 fallocate 0x00bfbd thru 0x010d57 (0x4d9b bytes) 000117 1734675766.831489 read 0x005529 thru 0x006c13 (0x16eb bytes) 000118 1734675766.831955 read 0x0063cd thru 0x00c582 (0x61b6 bytes) 000119 1734675766.832360 write 0x02a0fe thru 0x033285 (0x9188 bytes) 000120 1734675766.833614 mapread 0x002e55 thru 0x009d6f (0x6f1b bytes) 000122 1734675766.834577 punch from 0x016567 to 0x01a79d (0x4237 bytes) 000123 1734675766.837042 punch from 0x031f4d to 0x033285 (0x1339 bytes) 000124 1734675766.845887 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d1:0x0]: Invalid argument (22) 000125 1734675766.933522 trunc from 0x033286 to 0x01264e (0x20c37 bytes) 000126 1734675766.957088 mapread 0x00c023 thru 0x00e895 (0x2873 bytes) 000127 1734675766.980159 fallocate 0x01b7bc thru 0x0267ba (0xafff bytes) 000129 1734675766.983979 mapwrite 0x00c0e4 thru 0x01556d (0x948a bytes) 000130 1734675767.003089 mapwrite 0x004557 thru 0x01238b (0xde35 bytes) 000133 1734675767.020420 mapread 0x007174 thru 0x00aca2 (0x3b2f bytes) 000134 1734675767.021041 mapread 0x01f14e thru 0x0267ba (0x766d bytes) 000135 1734675767.029841 MIRROR_EXTEND to 3 mirrors 000136 1734675767.294226 mapwrite 0x030d02 thru 0x03d312 (0xc611 bytes) 000137 1734675767.367014 mapwrite 0x0119b2 thru 0x01c8ee (0xaf3d bytes) 000139 1734675767.387698 trunc from 0x03d313 to 0x01555d (0x27db5 bytes) 000140 1734675767.398864 trunc from 0x01555e to 0x012777 (0x2de6 bytes) 000141 1734675767.407114 punch from 0x00999e to 0x012777 (0x8dda bytes) fallocating to largest ever: 0x40000 000142 1734675767.410556 fallocate 0x03a2ee thru 0x03ffff (0x5d12 bytes) 000143 1734675767.420812 punch from 0x001678 to 0x009a23 (0x83ac bytes) 000144 1734675767.423410 trunc from 0x012778 to 0x0216f7 (0xef80 bytes) 000146 1734675767.434220 trunc from 0x0216f8 to 0x032150 (0x10a59 bytes) 000147 1734675767.441863 mapwrite 0x01dd4d thru 0x026d9c (0x9050 bytes) 000148 1734675767.465558 mapread 0x024237 thru 0x02ace2 (0x6aac bytes) 000149 1734675767.467036 mapwrite 0x00f763 thru 0x01dea1 (0xe73f bytes) 000150 1734675767.482509 read 0x02a993 thru 0x0309ea (0x6058 bytes) 000151 1734675767.485670 trunc from 0x032151 to 0x02555e (0xcbf2 bytes) 000152 1734675767.491863 fallocate 0x0187d7 thru 0x02226a (0x9a94 bytes) 000153 1734675767.495032 fallocate 0x0221b4 thru 0x030444 (0xe291 bytes) 000155 1734675767.501330 read 0x00c0ec thru 0x01bee7 (0xfdfc bytes) 000160 1734675767.503650 write 0x00c2b6 thru 0x010248 (0x3f93 bytes) 000161 1734675767.504801 mapread 0x02187d thru 0x024941 (0x30c5 bytes) 000162 1734675767.505582 trunc from 0x030445 to 0x02daff (0x2945 bytes) 000164 1734675767.519123 fallocate 0x01b412 thru 0x0208ce (0x54bd bytes) 000165 1734675767.523844 mapwrite 0x034060 thru 0x039d02 (0x5ca3 bytes) 000166 1734675767.554882 fallocate 0x038d87 thru 0x03ffff (0x7279 bytes) 000167 1734675767.557505 mapwrite 0x01f508 thru 0x0245f1 (0x50ea bytes) 000168 1734675767.569241 mapwrite 0x005266 thru 0x006db9 (0x1b54 bytes) 000174 1734675767.579465 mapwrite 0x033e64 thru 0x038c62 (0x4dff bytes) 000175 1734675767.590817 punch from 0x0032d1 to 0x00b73b (0x846b bytes) 000176 1734675767.596448 write 0x03d4eb thru 0x03ffff (0x2b15 bytes) 000177 1734675767.599173 fallocate 0x027056 thru 0x030241 (0x91ec bytes) 000178 1734675767.603491 write 0x03c906 thru 0x03ffff (0x36fa bytes) 000179 1734675767.609870 mapread 0x033a32 thru 0x03ffff (0xc5ce bytes) 000180 1734675767.613699 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d1:0x0]: Invalid argument (22) 000181 1734675767.725153 mapwrite 0x035101 thru 0x03978c (0x468c bytes) 000182 1734675767.743606 fallocate 0x01634b thru 0x01c842 (0x64f8 bytes) 000184 1734675767.749152 write 0x034dc0 thru 0x0370d1 (0x2312 bytes) 000185 1734675767.753492 mapwrite 0x01205a thru 0x01502e (0x2fd5 bytes) 000186 1734675767.772567 fallocate 0x0327c4 thru 0x03f532 (0xcd6f bytes) 000188 1734675767.776109 trunc from 0x040000 to 0x020e81 (0x1f17e bytes) 000189 1734675767.789760 write 0x029b38 thru 0x030619 (0x6ae2 bytes) 000190 1734675767.790732 fallocate 0x0209e4 thru 0x02fba0 (0xf1bd bytes) 000191 1734675767.795179 mapwrite 0x00351f thru 0x00575f (0x2241 bytes) 000193 1734675767.819120 write 0x02cd5f thru 0x02ec20 (0x1ec2 bytes) 000195 1734675767.820611 mapwrite 0x008bb1 thru 0x00eae4 (0x5f34 bytes) 000197 1734675767.834425 punch from 0x02fe16 to 0x030619 (0x804 bytes) 000198 1734675767.839419 read 0x00b34a thru 0x00e7ad (0x3464 bytes) 000199 1734675767.839957 write 0x002cc5 thru 0x00972f (0x6a6b bytes) 000200 1734675767.840787 read 0x01bb29 thru 0x02887d (0xcd55 bytes) 000202 1734675767.848025 mapread 0x022dd2 thru 0x02ffb2 (0xd1e1 bytes) 000203 1734675767.849302 mapread 0x01643b thru 0x01c556 (0x611c bytes) 000204 1734675767.850108 read 0x020b62 thru 0x02e0d5 (0xd574 bytes) 000205 1734675767.850541 mapread 0x00f158 thru 0x01e556 (0xf3ff bytes) 000206 1734675767.851187 mapwrite 0x03450e thru 0x037236 (0x2d29 bytes) 000207 1734675767.878348 trunc from 0x037237 to 0x038b58 (0x1922 bytes) 000208 1734675767.886090 mapread 0x02a857 thru 0x03145c (0x6c06 bytes) 000211 1734675767.887437 write 0x01cfe5 thru 0x0267ef (0x980b bytes) 000212 1734675767.888631 write 0x0090f6 thru 0x00f253 (0x615e bytes) 000213 1734675767.889587 fallocate 0x03945a thru 0x03ffff (0x6ba6 bytes) 000215 1734675767.893515 read 0x03387b thru 0x03748a (0x3c10 bytes) 000216 1734675767.894240 read 0x03487f thru 0x036af4 (0x2276 bytes) 000217 1734675767.897222 MIRROR_EXTEND to 3 mirrors 000219 1734675768.188135 mapread 0x016802 thru 0x01e45f (0x7c5e bytes) 000220 1734675768.209709 write 0x030038 thru 0x03d651 (0xd61a bytes) 000221 1734675768.234803 write 0x00b589 thru 0x0151cb (0x9c43 bytes) 000222 1734675768.248228 read 0x003726 thru 0x012127 (0xea02 bytes) 000223 1734675768.252320 mapread 0x0286ca thru 0x028ac5 (0x3fc bytes) 000225 1734675768.257126 read 0x033946 thru 0x03d651 (0x9d0c bytes) 000227 1734675768.257910 mapread 0x000d70 thru 0x0088d3 (0x7b64 bytes) 000228 1734675768.263695 fallocate 0x01cfb1 thru 0x026520 (0x9570 bytes) 000230 1734675768.267948 punch from 0x014c27 to 0x01db32 (0x8f0c bytes) 000232 1734675768.280729 fallocate 0x03ada5 thru 0x03ffff (0x525b bytes) 000233 1734675768.286241 mapwrite 0x021a93 thru 0x028070 (0x65de bytes) 000234 1734675768.304546 read 0x0017c5 thru 0x003451 (0x1c8d bytes) 000236 1734675768.305342 fallocate 0x019ce0 thru 0x01aa03 (0xd24 bytes) 000238 1734675768.312810 MIRROR_RESYNC 3 mirrors 000239 1734675768.558236 mapread 0x01353b thru 0x0181f3 (0x4cb9 bytes) 000241 1734675768.570166 punch from 0x011068 to 0x017b2e (0x6ac7 bytes) 000242 1734675768.587623 read 0x021a0e thru 0x0312c2 (0xf8b5 bytes) 000243 1734675768.593025 fallocate 0x01560d thru 0x01d149 (0x7b3d bytes) 000245 1734675768.595452 punch from 0x01a1d0 to 0x023b0f (0x9940 bytes) 000246 1734675768.598569 punch from 0x001bac to 0x003c50 (0x20a5 bytes) 000247 1734675768.601342 fallocate 0x03e126 thru 0x03ffff (0x1eda bytes) 000248 1734675768.603461 mapwrite 0x003ed2 thru 0x00d693 (0x97c2 bytes) 000249 1734675768.619124 punch from 0x01aac0 to 0x01b6be (0xbff bytes) 000250 1734675768.622767 write 0x009018 thru 0x014468 (0xb451 bytes) 000251 1734675768.628967 mapread 0x03eb77 thru 0x03ffff (0x1489 bytes) 000252 1734675768.634668 mapread 0x024e05 thru 0x026a58 (0x1c54 bytes) 000254 1734675768.635771 read 0x03ba16 thru 0x03d7a8 (0x1d93 bytes) 000255 1734675768.636481 mapread 0x024c8b thru 0x02b5db (0x6951 bytes) 000256 1734675768.639777 MIRROR_RESYNC 3 mirrors 000257 1734675769.073197 fallocate 0x033773 thru 0x03ffff (0xc88d bytes) 000258 1734675769.096796 fallocate 0x0045df thru 0x00edce (0xa7f0 bytes) 000260 1734675769.100240 fallocate 0x01b713 thru 0x01bedc (0x7ca bytes) 000261 1734675769.105566 write 0x02df0a thru 0x03ba1d (0xdb14 bytes) 000262 1734675769.119308 trunc from 0x040000 to 0x03a343 (0x5cbc bytes) 000263 1734675769.146637 write 0x010d52 thru 0x015c8f (0x4f3e bytes) 000264 1734675769.150583 mapread 0x011bc2 thru 0x013d9d (0x21dc bytes) 000265 1734675769.151465 mapread 0x0193c6 thru 0x0249ed (0xb628 bytes) 000267 1734675769.162188 mapwrite 0x001f1c thru 0x005672 (0x3757 bytes) 000268 1734675769.189981 trunc from 0x03a344 to 0x0165fc (0x23d47 bytes) 000269 1734675769.201779 mapread 0x00af48 thru 0x014145 (0x91fe bytes) 000270 1734675769.202410 mapread 0x00cedf thru 0x0165fc (0x971e bytes) 000271 1734675769.204321 mapread 0x013d0c thru 0x0165fc (0x28f1 bytes) 000273 1734675769.206626 write 0x00ef9f thru 0x00f943 (0x9a5 bytes) 000274 1734675769.207749 write 0x031aff thru 0x033d83 (0x2285 bytes) 000276 1734675769.209333 write 0x0348e0 thru 0x034d6c (0x48d bytes) 000278 1734675769.217434 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d1:0x0]: Invalid argument (22) 000279 1734675769.366898 mapread 0x01ef64 thru 0x028963 (0x9a00 bytes) 000283 1734675769.385852 read 0x000b74 thru 0x008f81 (0x840e bytes) 000284 1734675769.391456 read 0x005bdf thru 0x00ecb4 (0x90d6 bytes) 000285 1734675769.399206 read 0x00909f thru 0x014854 (0xb7b6 bytes) 000286 1734675769.399650 mapread 0x00eda8 thru 0x01b86b (0xcac4 bytes) 000287 1734675769.400524 fallocate 0x021b33 thru 0x029076 (0x7544 bytes) 000289 1734675769.409330 mapread 0x032352 thru 0x034d6c (0x2a1b bytes) 000290 1734675769.409866 write 0x008ae3 thru 0x013f86 (0xb4a4 bytes) 000291 1734675769.413827 fallocate 0x01f5f2 thru 0x023799 (0x41a8 bytes) 000292 1734675769.420501 write 0x01b532 thru 0x0242e6 (0x8db5 bytes) 000294 1734675769.421359 mapwrite 0x01e521 thru 0x021f73 (0x3a53 bytes) 000295 1734675769.438118 punch from 0x031389 to 0x034d6c (0x39e4 bytes) 000296 1734675769.444581 write 0x027367 thru 0x030bb1 (0x984b bytes) 000300 1734675769.449780 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d1:0x0]: Invalid argument (22) 000302 1734675769.536371 write 0x03376b thru 0x035bbd (0x2453 bytes) 000303 1734675769.542191 read 0x003cbc thru 0x01004b (0xc390 bytes) 000305 1734675769.547881 write 0x03bfdb thru 0x03ffff (0x4025 bytes) 000306 1734675769.548521 write 0x0307db thru 0x03ec4c (0xe472 bytes) 000307 1734675769.552299 write 0x019848 thru 0x027447 (0xdc00 bytes) 000310 1734675769.562351 fallocate 0x003772 thru 0x011c47 (0xe4d6 bytes) 000311 1734675769.568611 punch from 0x0055d9 to 0x012a17 (0xd43f bytes) 000312 1734675769.574090 fallocate 0x03b1ef thru 0x03ffff (0x4e11 bytes) 000313 1734675769.577101 mapread 0x000b56 thru 0x0106db (0xfb86 bytes) 000316 1734675769.592034 trunc from 0x040000 to 0x0070dd (0x38f22 bytes) 000317 1734675769.604030 trunc from 0x0070de to 0x02d1bc (0x260df bytes) 000318 1734675769.612639 mapread 0x017e51 thru 0x025eda (0xe08a bytes) 000319 1734675769.629920 mapread 0x02793e thru 0x02d1bc (0x587f bytes) 000320 1734675769.630494 mapwrite 0x0362d7 thru 0x03a1c3 (0x3eed bytes) 000321 1734675769.665217 write 0x016dd4 thru 0x0261ff (0xf42c bytes) 000323 1734675769.667426 trunc from 0x03a1c4 to 0x030ee1 (0x92e2 bytes) 000324 1734675769.682151 MIRROR_EXTEND to 2 mirrors 000325 1734675769.938396 mapread 0x016c8c thru 0x025c42 (0xefb7 bytes) 000326 1734675769.971994 read 0x00e6db thru 0x014f7f (0x68a5 bytes) 000327 1734675769.986396 punch from 0x01b2c2 to 0x01ddd8 (0x2b17 bytes) 000328 1734675770.007741 trunc from 0x030ee2 to 0x037d02 (0x6e21 bytes) 000330 1734675770.039426 fallocate 0x01771c thru 0x026c62 (0xf547 bytes) 000331 1734675770.043238 trunc from 0x037d03 to 0x011ec4 (0x25e3e bytes) 000334 1734675770.055247 MIRROR_RESYNC 2 mirrors 000335 1734675770.305843 punch from 0x00da5b to 0x011ec4 (0x446a bytes) 000336 1734675770.326052 trunc from 0x011ec5 to 0x03948a (0x275c6 bytes) 000337 1734675770.345117 trunc from 0x03948b to 0x0264cd (0x12fbd bytes) 000338 1734675770.365428 mapwrite 0x0221fd thru 0x02a4e4 (0x82e8 bytes) 000339 1734675770.402232 write 0x000db4 thru 0x00fe2b (0xf078 bytes) 000340 1734675770.410156 trunc from 0x02a4e5 to 0x01f6ed (0xadf7 bytes) 000341 1734675770.422605 mapread 0x003922 thru 0x011db9 (0xe498 bytes) 000343 1734675770.428581 write 0x0028d0 thru 0x005d46 (0x3477 bytes) 000344 1734675770.433338 MIRROR_EXTEND to 3 mirrors 000345 1734675770.688574 trunc from 0x01f6ee to 0x002709 (0x1cfe4 bytes) 000347 1734675770.725746 read 0x0011d2 thru 0x002709 (0x1538 bytes) 000348 1734675770.729196 mapread 0x002698 thru 0x002709 (0x72 bytes) 000350 1734675770.730180 trunc from 0x00270a to 0x02ae3b (0x28732 bytes) 000351 1734675770.743729 write 0x01ea41 thru 0x02830b (0x98cb bytes) 000352 1734675770.753481 MIRROR_RESYNC 3 mirrors 000353 1734675771.028291 read 0x005d10 thru 0x007b40 (0x1e31 bytes) 000354 1734675771.032137 punch from 0x003034 to 0x00cda7 (0x9d74 bytes) 000355 1734675771.051439 read 0x0067ce thru 0x008844 (0x2077 bytes) 000356 1734675771.057308 write 0x03dc4d thru 0x03ffff (0x23b3 bytes) 000357 1734675771.062910 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d1:0x0]: Invalid argument (22) 000358 1734675771.152462 write 0x03d911 thru 0x03efa8 (0x1698 bytes) 000359 1734675771.161030 punch from 0x00dd94 to 0x019e07 (0xc074 bytes) 000360 1734675771.165560 MIRROR_RESYNC 2 mirrors 000361 1734675771.384449 read 0x013df1 thru 0x02300f (0xf21f bytes) 000362 1734675771.390488 write 0x00c03e thru 0x013430 (0x73f3 bytes) 000363 1734675771.412090 fallocate 0x02b02c thru 0x036ab0 (0xba85 bytes) 000364 1734675771.415186 mapwrite 0x038485 thru 0x03ffff (0x7b7b bytes) 000365 1734675771.435290 write 0x01407e thru 0x014631 (0x5b4 bytes) 000367 1734675771.439830 write 0x010f57 thru 0x017ab0 (0x6b5a bytes) 000368 1734675771.440721 write 0x006cc7 thru 0x008ec3 (0x21fd bytes) 000369 1734675771.446336 punch from 0x005761 to 0x00e106 (0x89a6 bytes) 000370 1734675771.455676 read 0x009eef thru 0x00c38c (0x249e bytes) 000371 1734675771.460968 read 0x02ce97 thru 0x02fb5e (0x2cc8 bytes) 000372 1734675771.461646 write 0x035296 thru 0x036e64 (0x1bcf bytes) 000373 1734675771.462598 read 0x03ab24 thru 0x03bb5f (0x103c bytes) 000374 1734675771.463204 trunc from 0x040000 to 0x020b4b (0x1f4b4 bytes) 000375 1734675771.480835 mapwrite 0x027f9e thru 0x036dcb (0xee2e bytes) 000376 1734675771.504558 punch from 0x0323eb to 0x036dcb (0x49e1 bytes) 000377 1734675771.507596 trunc from 0x036dcc to 0x038d1c (0x1f51 bytes) 000378 1734675771.523685 punch from 0x0334e9 to 0x036842 (0x335a bytes) 000379 1734675771.527604 punch from 0x005a78 to 0x007e0b (0x2394 bytes) 000380 1734675771.530197 write 0x002391 thru 0x00651e (0x418e bytes) 000381 1734675771.535038 read 0x02b12e thru 0x0304b6 (0x5389 bytes) 000382 1734675771.538186 MIRROR_RESYNC 2 mirrors 000383 1734675771.787716 trunc from 0x038d1d to 0x03b21c (0x2500 bytes) 000385 1734675771.819514 punch from 0x01415a to 0x016ee7 (0x2d8e bytes) 000386 1734675771.822153 mapread 0x01b53e thru 0x023fad (0x8a70 bytes) 000387 1734675771.833091 trunc from 0x03b21d to 0x00642d (0x34def bytes) 000388 1734675771.844205 mapread 0x003d37 thru 0x00642d (0x26f7 bytes) 000390 1734675771.849476 mapread 0x005898 thru 0x00642d (0xb96 bytes) 000392 1734675771.853142 MIRROR_EXTEND to 3 mirrors 000393 1734675772.023692 mapread 0x002ba7 thru 0x00642d (0x3887 bytes) 000394 1734675772.041816 mapwrite 0x027580 thru 0x035e05 (0xe886 bytes) 000395 1734675772.115052 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d1:0x0]: Invalid argument (22) 000396 1734675772.222383 read 0x01d6ee thru 0x01f903 (0x2216 bytes) 000397 1734675772.226440 trunc from 0x035e06 to 0x00c82a (0x295db bytes) 000398 1734675772.241884 trunc from 0x00c82b to 0x017f78 (0xb74e bytes) 000399 1734675772.252146 punch from 0x01656a to 0x017f78 (0x1a0f bytes) 000400 1734675772.254353 punch from 0x012615 to 0x016d33 (0x471f bytes) 000401 1734675772.256672 punch from 0x005fe3 to 0x01362f (0xd64d bytes) 000403 1734675772.260323 mapread 0x004545 thru 0x00b646 (0x7102 bytes) 000405 1734675772.275468 mapread 0x00325b thru 0x00974b (0x64f1 bytes) 000406 1734675772.276122 write 0x003cc5 thru 0x01077d (0xcab9 bytes) 000407 1734675772.277105 fallocate 0x023563 thru 0x02d81f (0xa2bd bytes) 000408 1734675772.282261 fallocate 0x0004e5 thru 0x008f3e (0x8a5a bytes) 000409 1734675772.285448 trunc from 0x017f79 to 0x00103a (0x16f3e bytes) 000410 1734675772.293598 mapwrite 0x01e064 thru 0x0239b3 (0x5950 bytes) 000411 1734675772.322685 write 0x03fae5 thru 0x03ffff (0x51b bytes) 000412 1734675772.323527 mapread 0x0184dc thru 0x01d3c7 (0x4eec bytes) 000413 1734675772.324049 mapread 0x02fb6e thru 0x030aa3 (0xf36 bytes) 000414 1734675772.329093 fallocate 0x0241be thru 0x024f65 (0xda8 bytes) 000415 1734675772.333377 trunc from 0x040000 to 0x00417c (0x3be83 bytes) 000416 1734675772.342017 mapread 0x001f72 thru 0x00417c (0x220b bytes) 000417 1734675772.346041 mapwrite 0x034edb thru 0x038d61 (0x3e87 bytes) 000418 1734675772.373494 mapread 0x037782 thru 0x038d61 (0x15e0 bytes) 000419 1734675772.376956 MIRROR_EXTEND to 3 mirrors 000420 1734675772.626089 punch from 0x01c3e8 to 0x028edc (0xcaf5 bytes) 000421 1734675772.655428 fallocate 0x03ef7a thru 0x03ffff (0x1086 bytes) 000422 1734675772.659308 mapwrite 0x01d4d1 thru 0x0224b3 (0x4fe3 bytes) 000423 1734675772.685669 write 0x020e55 thru 0x02bb1f (0xaccb bytes) 000424 1734675772.696918 MIRROR_RESYNC 3 mirrors 000425 1734675773.152911 punch from 0x0080bf to 0x012e92 (0xadd4 bytes) 000426 1734675773.176702 write 0x030a98 thru 0x03cbdc (0xc145 bytes) 000427 1734675773.184828 trunc from 0x040000 to 0x03ceae (0x3151 bytes) 000428 1734675773.201146 mapwrite 0x03a54f thru 0x03ffff (0x5ab1 bytes) 000429 1734675773.250656 write 0x00ce28 thru 0x01424a (0x7423 bytes) 000430 1734675773.265348 mapwrite 0x0295b6 thru 0x02a0db (0xb26 bytes) 000432 1734675773.283519 fallocate 0x02f678 thru 0x03d53f (0xdec8 bytes) 000433 1734675773.289329 trunc from 0x040000 to 0x030069 (0xff96 bytes) 000434 1734675773.312208 MIRROR_RESYNC 3 mirrors 000435 1734675773.730382 MIRROR_RESYNC 3 mirrors 000436 1734675773.820142 write 0x01210d thru 0x020d10 (0xec04 bytes) 000437 1734675773.849719 trunc from 0x03006a to 0x03a5f5 (0xa58c bytes) 000439 1734675773.862109 read 0x00c63f thru 0x01153a (0x4efc bytes) 000440 1734675773.869435 mapread 0x0391c3 thru 0x03a5f5 (0x1433 bytes) 000441 1734675773.877614 mapwrite 0x02d834 thru 0x03608d (0x885a bytes) 000442 1734675773.901496 mapread 0x0366ba thru 0x03a5f5 (0x3f3c bytes) 000443 1734675773.902353 punch from 0x02893c to 0x02f466 (0x6b2b bytes) 000445 1734675773.907827 punch from 0x02f286 to 0x03a5f5 (0xb370 bytes) 000446 1734675773.913385 write 0x01b02b thru 0x01d41b (0x23f1 bytes) 000447 1734675773.914480 read 0x02a6bf thru 0x035d47 (0xb689 bytes) 000448 1734675773.921079 punch from 0x031d32 to 0x034df0 (0x30bf bytes) 000450 1734675773.927843 mapwrite 0x0066cb thru 0x00e6bd (0x7ff3 bytes) 000451 1734675773.948196 fallocate 0x00ba78 thru 0x016818 (0xada1 bytes) 000452 1734675773.952014 mapread 0x00c9ee thru 0x0114b7 (0x4aca bytes) 000453 1734675773.953430 mapread 0x011418 thru 0x01571f (0x4308 bytes) 000454 1734675773.954015 mapread 0x02725a thru 0x02cef3 (0x5c9a bytes) 000456 1734675773.954884 fallocate 0x00330a thru 0x00467b (0x1372 bytes) 000457 1734675773.963420 mapwrite 0x00b143 thru 0x00bbf2 (0xab0 bytes) 000458 1734675773.980339 punch from 0x00ab52 to 0x00ca65 (0x1f14 bytes) 000459 1734675773.985888 mapwrite 0x0043dd thru 0x005005 (0xc29 bytes) 000460 1734675773.993833 punch from 0x036b49 to 0x03a1e9 (0x36a1 bytes) 000461 1734675773.998177 read 0x00a4e6 thru 0x015193 (0xacae bytes) 000462 1734675774.003402 punch from 0x013a24 to 0x0169ae (0x2f8b bytes) 000463 1734675774.010074 punch from 0x00fdd4 to 0x010435 (0x662 bytes) 000464 1734675774.015245 write 0x021929 thru 0x025664 (0x3d3c bytes) 000466 1734675774.016285 write 0x02db04 thru 0x031f93 (0x4490 bytes) 000468 1734675774.017654 mapwrite 0x030458 thru 0x037da8 (0x7951 bytes) 000469 1734675774.029531 read 0x018140 thru 0x01f3f3 (0x72b4 bytes) 000470 1734675774.032619 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d1:0x0]: Invalid argument (22) 000471 1734675774.145458 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d1:0x0]: Invalid argument (22) 000472 1734675774.232883 mapread 0x026b64 thru 0x02ce96 (0x6333 bytes) 000473 1734675774.250779 write 0x025c8f thru 0x0348ec (0xec5e bytes) 000474 1734675774.258084 punch from 0x001550 to 0x008946 (0x73f7 bytes) 000475 1734675774.262841 trunc from 0x03a5f6 to 0x007faf (0x32646 bytes) 000476 1734675774.271651 write 0x021d4a thru 0x02b460 (0x9717 bytes) 000477 1734675774.273250 write 0x00e993 thru 0x01c0a2 (0xd710 bytes) 000479 1734675774.280632 mapwrite 0x019c80 thru 0x01d4e0 (0x3861 bytes) 000481 1734675774.299105 read 0x0013a2 thru 0x00413a (0x2d99 bytes) 000482 1734675774.307075 fallocate 0x01204f thru 0x017076 (0x5028 bytes) 000483 1734675774.311821 fallocate 0x005987 thru 0x0100c2 (0xa73c bytes) 000484 1734675774.315061 read 0x000d65 thru 0x00a873 (0x9b0f bytes) 000485 1734675774.320440 mapread 0x00a8fb thru 0x00de33 (0x3539 bytes) 000486 1734675774.321970 mapwrite 0x031d04 thru 0x0320a5 (0x3a2 bytes) 000487 1734675774.351340 mapread 0x01fcae thru 0x02f29f (0xf5f2 bytes) 000488 1734675774.352030 punch from 0x0215d3 to 0x023c40 (0x266e bytes) 000489 1734675774.357092 fallocate 0x013f77 thru 0x01d83e (0x98c8 bytes) 000491 1734675774.366535 MIRROR_EXTEND to 2 mirrors 000492 1734675774.672726 fallocate 0x02ba87 thru 0x030f86 (0x5500 bytes) 000493 1734675774.695917 write 0x00006d thru 0x00da95 (0xda29 bytes) 000494 1734675774.708530 read 0x01e6b2 thru 0x029e9c (0xb7eb bytes) 000495 1734675774.713830 write 0x03abf9 thru 0x03ffff (0x5407 bytes) 000496 1734675774.717891 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d1:0x0]: Invalid argument (22) 000497 1734675774.806350 trunc from 0x040000 to 0x0306f1 (0xf90e bytes) 000498 1734675774.827457 fallocate 0x005de7 thru 0x0066ec (0x906 bytes) 000499 1734675774.830577 write 0x01f58c thru 0x028479 (0x8eee bytes) 000501 1734675774.839709 write 0x02af32 thru 0x02bd6d (0xe3c bytes) 000504 1734675774.845050 read 0x014247 thru 0x017b71 (0x392b bytes) 000505 1734675774.849605 mapread 0x00b222 thru 0x0112e3 (0x60c2 bytes) 000506 1734675774.857023 fallocate 0x001a7b thru 0x00243e (0x9c4 bytes) 000507 1734675774.860267 punch from 0x0175db to 0x01b713 (0x4139 bytes) 000510 1734675774.867205 MIRROR_EXTEND to 2 mirrors 000514 1734675775.217414 mapread 0x00d08d thru 0x0132e7 (0x625b bytes) 000515 1734675775.249219 trunc from 0x0306f2 to 0x0363ca (0x5cd9 bytes) 000516 1734675775.288181 punch from 0x015b8f to 0x0202d8 (0xa74a bytes) 000517 1734675775.292557 mapwrite 0x0169f7 thru 0x01fd19 (0x9323 bytes) 000519 1734675775.339185 write 0x020509 thru 0x027746 (0x723e bytes) 000520 1734675775.341049 write 0x015691 thru 0x01b4eb (0x5e5b bytes) 000521 1734675775.341877 read 0x036009 thru 0x0363ca (0x3c2 bytes) 000523 1734675775.349900 fallocate 0x039990 thru 0x03cf22 (0x3593 bytes) 000524 1734675775.355061 read 0x005998 thru 0x00e786 (0x8def bytes) 000525 1734675775.363452 trunc from 0x0363cb to 0x0253bf (0x1100b bytes) 000526 1734675775.378146 fallocate 0x035cfe thru 0x03ffff (0xa302 bytes) 000527 1734675775.384527 mapread 0x02e00f thru 0x039a57 (0xba49 bytes) 000528 1734675775.417878 read 0x019625 thru 0x021a8a (0x8466 bytes) 000529 1734675775.418362 punch from 0x00d7b0 to 0x01662b (0x8e7c bytes) 000530 1734675775.425737 mapwrite 0x0276f5 thru 0x034c0f (0xd51b bytes) 000532 1734675775.451430 mapread 0x01faab thru 0x02a453 (0xa9a9 bytes) 000533 1734675775.452230 read 0x02cfb1 thru 0x03930c (0xc35c bytes) 000535 1734675775.453063 fallocate 0x013a6f thru 0x015cfd (0x228f bytes) 000538 1734675775.461579 MIRROR_EXTEND to 3 mirrors 000540 1734675775.811876 mapread 0x02e1b8 thru 0x035c44 (0x7a8d bytes) 000541 1734675775.833464 trunc from 0x040000 to 0x01f858 (0x207a7 bytes) 000543 1734675775.877115 punch from 0x006498 to 0x010c71 (0xa7da bytes) 000544 1734675775.882435 fallocate 0x014153 thru 0x016738 (0x25e6 bytes) 000545 1734675775.887769 MIRROR_RESYNC 3 mirrors 000546 1734675776.189128 read 0x01ca4f thru 0x01f858 (0x2e0a bytes) 000547 1734675776.192715 mapread 0x009c53 thru 0x00df69 (0x4317 bytes) 000548 1734675776.207661 punch from 0x0137d2 to 0x01aa28 (0x7257 bytes) 000549 1734675776.236010 mapwrite 0x00fa2a thru 0x0132dd (0x38b4 bytes) 000550 1734675776.254349 punch from 0x008bda to 0x016524 (0xd94b bytes) 000551 1734675776.261341 MIRROR_EXTEND to 4 mirrors 000552 1734675776.502372 mapread 0x01ead7 thru 0x01f858 (0xd82 bytes) 000554 1734675776.520488 trunc from 0x01f859 to 0x02ac2c (0xb3d4 bytes) 000556 1734675776.569133 trunc from 0x02ac2d to 0x01557f (0x156ad bytes) 000557 1734675776.589985 mapwrite 0x036f57 thru 0x038de5 (0x1e8f bytes) 000558 1734675776.623540 mapwrite 0x014603 thru 0x015e8c (0x188a bytes) 000559 1734675776.634742 read 0x00cd1f thru 0x014dc5 (0x80a7 bytes) 000560 1734675776.641225 MIRROR_RESYNC 4 mirrors 000562 1734675777.019448 fallocate 0x00bbd1 thru 0x00d6e6 (0x1b16 bytes) 000563 1734675777.040236 mapread 0x026397 thru 0x02fcd9 (0x9943 bytes) 000564 1734675777.055580 fallocate 0x033da3 thru 0x03ffff (0xc25d bytes) 000565 1734675777.062438 MIRROR_EXTEND to 5 mirrors 000566 1734675777.365776 fallocate 0x03cada thru 0x03ea04 (0x1f2b bytes) 000567 1734675777.397414 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d1:0x0]: Invalid argument (22) 000568 1734675777.504464 trunc from 0x03ea05 to 0x039e88 (0x4b7c bytes) 000569 1734675777.516161 punch from 0x035e2b to 0x039e88 (0x405e bytes) 000571 1734675777.520373 mapread 0x021954 thru 0x02c0db (0xa788 bytes) 000572 1734675777.536107 mapread 0x03154f thru 0x035168 (0x3c1a bytes) 000575 1734675777.545213 mapread 0x0041b6 thru 0x0139fc (0xf847 bytes) 000576 1734675777.554047 read 0x021a7d thru 0x0229ad (0xf31 bytes) 000577 1734675777.556953 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d1:0x0]: Invalid argument (22) 000578 1734675777.667769 mapwrite 0x02231a thru 0x02b4b5 (0x919c bytes) 000579 1734675777.699091 mapwrite 0x011d86 thru 0x013ed2 (0x214d bytes) 000580 1734675777.710595 punch from 0x0044c4 to 0x0110e8 (0xcc25 bytes) 000581 1734675777.714730 read 0x0383ce thru 0x039e88 (0x1abb bytes) 000582 1734675777.717680 mapwrite 0x01abd4 thru 0x0249ee (0x9e1b bytes) 000583 1734675777.735425 mapwrite 0x00f3ae thru 0x01585a (0x64ad bytes) 000584 1734675777.752666 mapread 0x009ea4 thru 0x01978e (0xf8eb bytes) 000585 1734675777.753775 fallocate 0x032bd8 thru 0x033033 (0x45c bytes) 000586 1734675777.758213 trunc from 0x039e89 to 0x024627 (0x15861 bytes) 000587 1734675777.769571 mapread 0x0024fd thru 0x006997 (0x449b bytes) 000588 1734675777.773912 punch from 0x02337d to 0x024627 (0x12ab bytes) 000591 1734675777.777631 fallocate 0x025018 thru 0x0313f9 (0xc3e2 bytes) 000592 1734675777.782090 mapread 0x00edd7 thru 0x00ff8b (0x11b5 bytes) 000594 1734675777.782953 trunc from 0x024628 to 0x0218e1 (0x2d46 bytes) 000595 1734675777.792884 mapread 0x00d538 thru 0x012633 (0x50fc bytes) 000596 1734675777.793852 read 0x021446 thru 0x0218e1 (0x49c bytes) 000597 1734675777.794847 write 0x024be9 thru 0x028b68 (0x3f80 bytes) 000598 1734675777.796527 write 0x02ad92 thru 0x038ab4 (0xdd23 bytes) 000599 1734675777.798855 mapread 0x00f204 thru 0x01f1ce (0xffcb bytes) 000600 1734675777.800805 punch from 0x027518 to 0x030f15 (0x99fe bytes) 000601 1734675777.810339 trunc from 0x038ab5 to 0x02933e (0xf776 bytes) 000603 1734675777.823102 mapwrite 0x02bd85 thru 0x0393ae (0xd62a bytes) 000604 1734675777.858606 fallocate 0x01a037 thru 0x0261fd (0xc1c7 bytes) 000605 1734675777.864076 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d1:0x0]: Invalid argument (22) 000606 1734675777.993113 write 0x014278 thru 0x019eb1 (0x5c3a bytes) 000607 1734675778.007044 punch from 0x0371ec to 0x0393ae (0x21c3 bytes) 000608 1734675778.014426 trunc from 0x0393af to 0x001754 (0x37c5a bytes) 000609 1734675778.030759 punch from 0x000037 to 0x001754 (0x171e bytes) 000610 1734675778.040124 MIRROR_EXTEND to 3 mirrors 000611 1734675778.324023 read 0x0007bc thru 0x001754 (0xf99 bytes) 000612 1734675778.336828 write 0x02a010 thru 0x02c449 (0x243a bytes) 000613 1734675778.371425 read 0x016c73 thru 0x0183a4 (0x1732 bytes) 000614 1734675778.378527 punch from 0x02555e to 0x02c449 (0x6eec bytes) 000615 1734675778.387239 punch from 0x006b90 to 0x00f51d (0x898e bytes) 000616 1734675778.391680 mapwrite 0x011892 thru 0x015fff (0x476e bytes) 000617 1734675778.404214 trunc from 0x02c44a to 0x026214 (0x6235 bytes) 000618 1734675778.421731 punch from 0x01daa0 to 0x026214 (0x8775 bytes) 000619 1734675778.425717 write 0x0386e9 thru 0x03ffff (0x7917 bytes) 000620 1734675778.427084 mapread 0x00d353 thru 0x011b06 (0x47b4 bytes) 000621 1734675778.445903 trunc from 0x040000 to 0x005d7a (0x3a285 bytes) 000622 1734675778.461640 punch from 0x002639 to 0x005d7a (0x3742 bytes) 000623 1734675778.466601 fallocate 0x0201fa thru 0x02b454 (0xb25b bytes) 000624 1734675778.470459 trunc from 0x005d7b to 0x02091f (0x1aba5 bytes) 000626 1734675778.478365 mapwrite 0x00b7ff thru 0x01084c (0x504e bytes) 000627 1734675778.505875 punch from 0x01e05f to 0x02091f (0x28c1 bytes) 000630 1734675778.509806 mapread 0x0111bf thru 0x011b89 (0x9cb bytes) 000631 1734675778.513642 mapread 0x01658b thru 0x0184ec (0x1f62 bytes) 000632 1734675778.524338 MIRROR_EXTEND to 4 mirrors 000633 1734675778.838484 punch from 0x01ff65 to 0x02091f (0x9bb bytes) 000636 1734675778.865585 mapread 0x019628 thru 0x01d2b8 (0x3c91 bytes) 000637 1734675778.880164 write 0x03d80f thru 0x03ffff (0x27f1 bytes) 000638 1734675778.885830 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d1:0x0]: Invalid argument (22) 000639 1734675779.010631 fallocate 0x01065a thru 0x010aa2 (0x449 bytes) 000640 1734675779.015518 mapwrite 0x02ca03 thru 0x02ec45 (0x2243 bytes) 000641 1734675779.038824 read 0x0398f9 thru 0x03b083 (0x178b bytes) 000642 1734675779.044089 mapwrite 0x026694 thru 0x02c723 (0x6090 bytes) 000643 1734675779.063739 trunc from 0x040000 to 0x024926 (0x1b6d9 bytes) 000644 1734675779.093422 MIRROR_EXTEND to 4 mirrors 000645 1734675779.390024 read 0x01ca09 thru 0x024926 (0x7f1e bytes) 000646 1734675779.404904 mapwrite 0x0319e4 thru 0x0389fc (0x7019 bytes) 000647 1734675779.482843 read 0x02241f thru 0x024dfe (0x29e0 bytes) 000649 1734675779.494780 mapwrite 0x037400 thru 0x03ffff (0x8c00 bytes) 000650 1734675779.547082 trunc from 0x040000 to 0x006619 (0x399e6 bytes) 000651 1734675779.570753 MIRROR_RESYNC 4 mirrors 000653 1734675779.908333 mapwrite 0x008e8d thru 0x017c32 (0xeda6 bytes) 000654 1734675779.970455 MIRROR_EXTEND to 5 mirrors 000655 1734675780.226363 mapread 0x008333 thru 0x00cc9d (0x496b bytes) 000656 1734675780.249418 trunc from 0x017c33 to 0x03f85b (0x27c29 bytes) 000657 1734675780.281489 read 0x014450 thru 0x018cda (0x488b bytes) 000660 1734675780.289273 MIRROR_RESYNC 5 mirrors 000663 1734675780.715575 punch from 0x015fd3 to 0x023196 (0xd1c4 bytes) 000666 1734675780.741574 mapwrite 0x02649d thru 0x028e26 (0x298a bytes) 000667 1734675780.762111 trunc from 0x03f85c to 0x021f0c (0x1d94f bytes) 000668 1734675780.781841 punch from 0x00c4ae to 0x01095e (0x44b1 bytes) 000669 1734675780.784568 mapread 0x011e88 thru 0x01669a (0x4813 bytes) 000670 1734675780.789113 fallocate 0x017aa0 thru 0x02320c (0xb76d bytes) 000671 1734675780.795489 MIRROR_RESYNC 5 mirrors 000672 1734675781.420598 mapwrite 0x0360b4 thru 0x03ffff (0x9f4c bytes) 000673 1734675781.476382 write 0x0127f1 thru 0x01d62c (0xae3c bytes) 000674 1734675781.488127 fallocate 0x02b441 thru 0x02df7b (0x2b3b bytes) 000675 1734675781.494882 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d1:0x0]: Invalid argument (22) 000676 1734675781.625028 write 0x02ed49 thru 0x03152b (0x27e3 bytes) 000677 1734675781.632495 punch from 0x004225 to 0x0053bf (0x119b bytes) 000678 1734675781.635484 mapread 0x00b9f2 thru 0x019fc9 (0xe5d8 bytes) 000679 1734675781.648230 trunc from 0x040000 to 0x031777 (0xe888 bytes) 000680 1734675781.660717 MIRROR_EXTEND to 5 mirrors 000681 1734675781.994308 write 0x00e57d thru 0x0198c5 (0xb349 bytes) 000683 1734675782.025659 MIRROR_EXTEND to 6 mirrors 000685 1734675782.386771 mapwrite 0x0291fd thru 0x036c67 (0xda6b bytes) 000686 1734675782.464673 mapwrite 0x027214 thru 0x0334cf (0xc2bc bytes) 000687 1734675782.481514 read 0x006b2a thru 0x008b28 (0x1fff bytes) 000688 1734675782.493353 MIRROR_RESYNC 6 mirrors 000690 1734675783.288502 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d1:0x0]: Invalid argument (22) 000691 1734675783.448996 MIRROR_EXTEND to 6 mirrors 000692 1734675783.845088 fallocate 0x01d956 thru 0x026b24 (0x91cf bytes) 000693 1734675783.863265 read 0x01e710 thru 0x02042b (0x1d1c bytes) 000694 1734675783.866918 trunc from 0x036c68 to 0x02c889 (0xa3de bytes) 000695 1734675783.887235 read 0x004940 thru 0x00a012 (0x56d3 bytes) 000696 1734675783.895613 trunc from 0x02c88a to 0x030071 (0x37e8 bytes) 000697 1734675783.903772 trunc from 0x030072 to 0x003bdb (0x2c496 bytes) 000698 1734675783.913669 punch from 0x002392 to 0x003bdb (0x184a bytes) 000699 1734675783.921342 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d1:0x0]: Invalid argument (22) 000700 1734675784.029431 fallocate 0x02f439 thru 0x038fdd (0x9ba5 bytes) 000703 1734675784.034496 read 0x00cda2 thru 0x019cfb (0xcf5a bytes) 000704 1734675784.041035 fallocate 0x03adab thru 0x03ffff (0x5255 bytes) 000705 1734675784.045079 punch from 0x030aac to 0x03ffff (0xf554 bytes) 000706 1734675784.048017 fallocate 0x03aab4 thru 0x03ffff (0x554c bytes) 000707 1734675784.052037 read 0x021912 thru 0x03130d (0xf9fc bytes) 000708 1734675784.058747 fallocate 0x0078cb thru 0x01546e (0xdba4 bytes) 000709 1734675784.064606 punch from 0x0392b2 to 0x03ffff (0x6d4e bytes) 000711 1734675784.068488 read 0x026ce8 thru 0x02ad42 (0x405b bytes) 000713 1734675784.069981 mapwrite 0x005aec thru 0x008587 (0x2a9c bytes) 000714 1734675784.083449 read 0x026ab0 thru 0x0321ee (0xb73f bytes) 000716 1734675784.084218 punch from 0x016391 to 0x018d09 (0x2979 bytes) 000717 1734675784.087062 trunc from 0x040000 to 0x02b281 (0x14d7e bytes) 000719 1734675784.099833 fallocate 0x029718 thru 0x0314ef (0x7dd8 bytes) 000720 1734675784.103725 write 0x00d4da thru 0x01bb12 (0xe639 bytes) 000723 1734675784.109433 MIRROR_EXTEND to 6 mirrors 000724 1734675784.394041 mapread 0x007e7c thru 0x00fc96 (0x7e1b bytes) 000725 1734675784.415664 MIRROR_EXTEND to 7 mirrors 000726 1734675784.687803 MIRROR_RESYNC 7 mirrors 000727 1734675785.181036 fallocate 0x002b43 thru 0x00ec9d (0xc15b bytes) 000728 1734675785.204704 read 0x0166a9 thru 0x023732 (0xd08a bytes) 000730 1734675785.214906 read 0x0254c2 thru 0x02b281 (0x5dc0 bytes) 000733 1734675785.225199 mapread 0x0243ac thru 0x026e06 (0x2a5b bytes) 000734 1734675785.226290 mapwrite 0x0332ba thru 0x038afa (0x5841 bytes) 000735 1734675785.265239 MIRROR_EXTEND to 8 mirrors 000736 1734675785.582363 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d1:0x0]: Invalid argument (22) 000737 1734675785.756987 mapread 0x022ca0 thru 0x027666 (0x49c7 bytes) 000738 1734675785.779098 write 0x01ca08 thru 0x028908 (0xbf01 bytes) 000739 1734675785.814662 mapread 0x00dc8a thru 0x015d30 (0x80a7 bytes) 000741 1734675785.826436 MIRROR_EXTEND to 8 mirrors 000742 1734675786.157074 fallocate 0x01f338 thru 0x01fd1b (0x9e4 bytes) 000744 1734675786.185506 write 0x03e007 thru 0x03ffff (0x1ff9 bytes) 000746 1734675786.189278 trunc from 0x040000 to 0x030374 (0xfc8b bytes) 000747 1734675786.208321 fallocate 0x0136b8 thru 0x01ebd4 (0xb51d bytes) 000748 1734675786.215257 MIRROR_EXTEND to 9 mirrors 000749 1734675786.576807 read 0x00c521 thru 0x018d62 (0xc842 bytes) 000750 1734675786.588467 fallocate 0x004a9e thru 0x0082a3 (0x3806 bytes) 000751 1734675786.617192 fallocate 0x020792 thru 0x029594 (0x8e03 bytes) 000752 1734675786.619902 mapread 0x0075fa thru 0x010907 (0x930e bytes) 000753 1734675786.631817 write 0x03c20f thru 0x03ffff (0x3df1 bytes) 000755 1734675786.634920 fallocate 0x005ddd thru 0x0101e3 (0xa407 bytes) 000756 1734675786.637416 write 0x02aa12 thru 0x031a1c (0x700b bytes) 000757 1734675786.645814 mapwrite 0x01c445 thru 0x01f838 (0x33f4 bytes) 000759 1734675786.673185 read 0x00c155 thru 0x013051 (0x6efd bytes) 000762 1734675786.675631 trunc from 0x040000 to 0x004d86 (0x3b279 bytes) 000763 1734675786.700693 MIRROR_RESYNC 9 mirrors 000766 1734675787.414062 mapread 0x002bfd thru 0x004d86 (0x218a bytes) 000767 1734675787.420442 trunc from 0x004d87 to 0x017614 (0x1288e bytes) 000768 1734675787.449904 fallocate 0x0351a4 thru 0x038f52 (0x3daf bytes) 000769 1734675787.454646 read 0x006d95 thru 0x01397a (0xcbe6 bytes) 000770 1734675787.466030 MIRROR_RESYNC 9 mirrors 000771 1734675788.382097 mapwrite 0x00ccad thru 0x017712 (0xaa66 bytes) 000772 1734675788.494635 punch from 0x010d25 to 0x01809c (0x7378 bytes) 000773 1734675788.504401 MIRROR_EXTEND to 10 mirrors 000774 1734675788.853217 punch from 0x02705a to 0x0353e4 (0xe38b bytes) 000775 1734675788.879807 fallocate 0x006020 thru 0x0109f6 (0xa9d7 bytes) 000776 1734675788.883776 write 0x0385bf thru 0x03ffff (0x7a41 bytes) 000777 1734675788.889009 read 0x01c6ca thru 0x023f5c (0x7893 bytes) 000778 1734675788.895313 fallocate 0x028301 thru 0x0364c1 (0xe1c1 bytes) 000779 1734675788.902487 mapread 0x03ced8 thru 0x03ffff (0x3128 bytes) 000780 1734675788.903174 write 0x01bed5 thru 0x0211f5 (0x5321 bytes) 000782 1734675788.908660 fallocate 0x0177de thru 0x01ccb6 (0x54d9 bytes) 000785 1734675788.913342 fallocate 0x028956 thru 0x032e81 (0xa52c bytes) 000786 1734675788.923599 MIRROR_RESYNC 10 mirrors 000787 1734675790.457314 MIRROR_RESYNC 10 mirrors 000788 1734675790.586830 write 0x02ff77 thru 0x030b3f (0xbc9 bytes) 000789 1734675790.616376 mapwrite 0x02284a thru 0x0258ba (0x3071 bytes) 000790 1734675790.635847 punch from 0x00c9b0 to 0x017b11 (0xb162 bytes) 000791 1734675790.639165 punch from 0x03d81a to 0x03ffff (0x27e6 bytes) 000793 1734675790.642559 trunc from 0x040000 to 0x017930 (0x286cf bytes) 000794 1734675790.660674 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d1:0x0]: Invalid argument (22) 000795 1734675790.784760 punch from 0x01537b to 0x017930 (0x25b6 bytes) 000797 1734675790.789069 trunc from 0x017931 to 0x00bef0 (0xba40 bytes) 000799 1734675790.803381 fallocate 0x02320c thru 0x02d043 (0x9e38 bytes) 000800 1734675790.808248 punch from 0x007ca6 to 0x00deb8 (0x6213 bytes) 000802 1734675790.812001 mapread 0x012d88 thru 0x013a6e (0xce7 bytes) 000803 1734675790.820582 read 0x01c8ef thru 0x024beb (0x82fd bytes) 000804 1734675790.825751 write 0x01a382 thru 0x0269bd (0xc63c bytes) 000805 1734675790.834394 fallocate 0x01856f thru 0x019842 (0x12d4 bytes) 000807 1734675790.839199 mapwrite 0x033b1a thru 0x039f3d (0x6424 bytes) 000808 1734675790.881675 write 0x00eb0d thru 0x01d817 (0xed0b bytes) 000809 1734675790.888776 write 0x031895 thru 0x03f0f9 (0xd865 bytes) 000810 1734675790.889853 trunc from 0x03f0fa to 0x012fbd (0x2c13c bytes) 000811 1734675790.905358 write 0x032cf1 thru 0x0337f9 (0xb09 bytes) 000812 1734675790.906314 trunc from 0x0337fa to 0x02f594 (0x4265 bytes) 000813 1734675790.916560 trunc from 0x02f595 to 0x005806 (0x29d8e bytes) 000815 1734675790.928199 read 0x000c09 thru 0x003828 (0x2c20 bytes) 000816 1734675790.932670 trunc from 0x005807 to 0x0187fb (0x12ff5 bytes) 000817 1734675790.942849 mapwrite 0x027442 thru 0x02f459 (0x8018 bytes) 000818 1734675790.974173 fallocate 0x02e1b1 thru 0x037fee (0x9e3e bytes) 000820 1734675790.979806 mapwrite 0x037ee3 thru 0x03ffff (0x811d bytes) 000821 1734675791.012264 write 0x03cbd3 thru 0x03ffff (0x342d bytes) 000822 1734675791.014347 trunc from 0x040000 to 0x001d46 (0x3e2b9 bytes) 000823 1734675791.029068 mapwrite 0x03d24a thru 0x03ffff (0x2db6 bytes) 000824 1734675791.066908 MIRROR_RESYNC 9 mirrors 000825 1734675791.809050 punch from 0x018a8e to 0x01fbbd (0x7130 bytes) 000826 1734675791.846105 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d1:0x0]: Invalid argument (22) 000827 1734675792.006502 mapwrite 0x03d612 thru 0x03ffff (0x29ee bytes) 000828 1734675792.046241 MIRROR_RESYNC 8 mirrors 000829 1734675792.880751 MIRROR_EXTEND to 9 mirrors 000830 1734675793.293115 punch from 0x02769e to 0x036725 (0xf088 bytes) 000831 1734675793.329159 mapread 0x00e602 thru 0x01a458 (0xbe57 bytes) 000832 1734675793.349903 fallocate 0x02ae5a thru 0x034f39 (0xa0e0 bytes) 000833 1734675793.354031 read 0x00b299 thru 0x017745 (0xc4ad bytes) 000834 1734675793.354545 fallocate 0x012c7c thru 0x020722 (0xdaa7 bytes) 000835 1734675793.363676 MIRROR_EXTEND to 10 mirrors 000837 1734675793.752089 mapread 0x02272d thru 0x02ebb2 (0xc486 bytes) 000838 1734675793.793982 mapwrite 0x008296 thru 0x012c74 (0xa9df bytes) 000839 1734675793.877989 punch from 0x006951 to 0x00f5e4 (0x8c94 bytes) 000841 1734675793.882110 punch from 0x01d2c9 to 0x025e9f (0x8bd7 bytes) 000843 1734675793.897769 trunc from 0x040000 to 0x03ae9a (0x5165 bytes) 000846 1734675793.924438 read 0x03876e thru 0x0388dc (0x16f bytes) 000847 1734675793.931310 trunc from 0x03ae9b to 0x039e9e (0xffc bytes) 000849 1734675793.954325 write 0x005c53 thru 0x00cfaa (0x7358 bytes) 000850 1734675793.965444 trunc from 0x039e9f to 0x026efa (0x12fa4 bytes) 000851 1734675793.984467 punch from 0x00c51d to 0x01667d (0xa161 bytes) 000852 1734675793.997631 write 0x01c74d thru 0x023768 (0x701c bytes) 000853 1734675794.012294 write 0x02b2bb thru 0x037076 (0xbdbc bytes) 000854 1734675794.013459 write 0x015244 thru 0x01663b (0x13f8 bytes) 000855 1734675794.014027 write 0x00debf thru 0x00e155 (0x297 bytes) 000856 1734675794.028486 MIRROR_EXTEND to 11 mirrors 000857 1734675794.568338 fallocate 0x03eb5b thru 0x03ffff (0x14a5 bytes) 000858 1734675794.598695 fallocate 0x02d985 thru 0x0316b3 (0x3d2f bytes) 000859 1734675794.603614 fallocate 0x01c55b thru 0x025c60 (0x9706 bytes) 000860 1734675794.613143 MIRROR_RESYNC 11 mirrors 000861 1734675796.359588 fallocate 0x0387f9 thru 0x03e699 (0x5ea1 bytes) 000863 1734675796.396080 fallocate 0x03af41 thru 0x03b956 (0xa16 bytes) 000864 1734675796.408841 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d1:0x0]: Invalid argument (22) 000867 1734675796.542781 write 0x00f44f thru 0x01cd5b (0xd90d bytes) 000868 1734675796.554320 write 0x00b66f thru 0x016762 (0xb0f4 bytes) 000870 1734675796.562432 mapread 0x02694b thru 0x02f230 (0x88e6 bytes) 000871 1734675796.575140 write 0x00f51e thru 0x0178ab (0x838e bytes) 000872 1734675796.575982 write 0x03fb59 thru 0x03ffff (0x4a7 bytes) 000873 1734675796.583979 mapread 0x00b185 thru 0x0183e5 (0xd261 bytes) 000875 1734675796.585422 read 0x008647 thru 0x00ab40 (0x24fa bytes) 000876 1734675796.586394 punch from 0x02d254 to 0x03653e (0x92eb bytes) 000877 1734675796.591682 trunc from 0x040000 to 0x01b5d3 (0x24a2c bytes) 000878 1734675796.608753 write 0x013543 thru 0x01d99a (0xa458 bytes) 000879 1734675796.609703 fallocate 0x03e424 thru 0x03ffff (0x1bdc bytes) 000880 1734675796.615014 write 0x0361d5 thru 0x03f457 (0x9283 bytes) 000881 1734675796.628337 punch from 0x004598 to 0x005800 (0x1269 bytes) 000882 1734675796.632780 write 0x03f56a thru 0x03ffff (0xa96 bytes) 000883 1734675796.633306 trunc from 0x040000 to 0x023aa1 (0x1c55e bytes) 000884 1734675796.650894 MIRROR_RESYNC 10 mirrors 000885 1734675797.754747 punch from 0x013494 to 0x01c127 (0x8c94 bytes) 000886 1734675797.813837 punch from 0x010e66 to 0x018405 (0x75a0 bytes) 000887 1734675797.829953 read 0x00d8e7 thru 0x01cec9 (0xf5e3 bytes) 000888 1734675797.842408 mapwrite 0x021bca thru 0x0309f7 (0xee2e bytes) 000889 1734675797.892827 mapwrite 0x02a5ec thru 0x02e1be (0x3bd3 bytes) 000890 1734675797.905515 fallocate 0x029467 thru 0x032f95 (0x9b2f bytes) 000891 1734675797.913623 fallocate 0x014955 thru 0x015e31 (0x14dd bytes) 000892 1734675797.918364 mapread 0x02a863 thru 0x02c3e0 (0x1b7e bytes) 000894 1734675797.920347 mapwrite 0x025b5f thru 0x032089 (0xc52b bytes) 000895 1734675797.962838 punch from 0x02cc2a to 0x032089 (0x5460 bytes) 000896 1734675797.968806 mapread 0x00760b thru 0x013c44 (0xc63a bytes) 000898 1734675797.975410 read 0x016532 thru 0x0249a0 (0xe46f bytes) 000899 1734675797.975872 read 0x01e8de thru 0x02364c (0x4d6f bytes) 000900 1734675797.976607 read 0x002851 thru 0x00d37c (0xab2c bytes) 000902 1734675797.986110 write 0x035e98 thru 0x038cae (0x2e17 bytes) 000903 1734675797.987475 trunc from 0x038caf to 0x015d3a (0x22f74 bytes) 000904 1734675798.003854 MIRROR_EXTEND to 11 mirrors 000905 1734675798.325248 fallocate 0x02c7f5 thru 0x02ef74 (0x2780 bytes) 000906 1734675798.364029 write 0x000d6d thru 0x00ccfb (0xbf8f bytes) 000907 1734675798.375230 trunc from 0x02ef75 to 0x036037 (0x70c3 bytes) 000908 1734675798.400545 mapwrite 0x001f99 thru 0x005eb0 (0x3f18 bytes) 000909 1734675798.426601 MIRROR_EXTEND to 12 mirrors 000910 1734675798.753891 punch from 0x00ccf9 to 0x013441 (0x6749 bytes) 000911 1734675798.804261 mapwrite 0x00f48d thru 0x01db80 (0xe6f4 bytes) 000912 1734675798.841240 mapwrite 0x03d416 thru 0x03ffff (0x2bea bytes) 000914 1734675798.889225 write 0x01d139 thru 0x025508 (0x83d0 bytes) 000915 1734675798.894707 read 0x00d5e4 thru 0x015d26 (0x8743 bytes) 000916 1734675798.895387 write 0x01fdd8 thru 0x022089 (0x22b2 bytes) 000917 1734675798.895994 fallocate 0x01767a thru 0x01ce3c (0x57c3 bytes) 000918 1734675798.902641 punch from 0x030ebc to 0x032384 (0x14c9 bytes) 000919 1734675798.906537 mapread 0x006374 thru 0x01388b (0xd518 bytes) 000921 1734675798.914116 punch from 0x022bbe to 0x02364c (0xa8f bytes) 000922 1734675798.925732 MIRROR_RESYNC 12 mirrors 000923 1734675800.178456 punch from 0x02a5c9 to 0x02fdfe (0x5836 bytes) 000926 1734675800.209539 fallocate 0x02675b thru 0x02af7b (0x4821 bytes) 000927 1734675800.214785 punch from 0x01f231 to 0x025c67 (0x6a37 bytes) 000928 1734675800.218425 trunc from 0x040000 to 0x03b8d4 (0x472b bytes) 000929 1734675800.232811 mapread 0x0333cc thru 0x03b8d4 (0x8509 bytes) 000930 1734675800.244317 fallocate 0x01342b thru 0x018f90 (0x5b66 bytes) 000932 1734675800.247705 mapwrite 0x01562b thru 0x01badf (0x64b5 bytes) 000933 1734675800.297488 mapread 0x006fae thru 0x011b7f (0xabd2 bytes) 000936 1734675800.306952 read 0x0393ba thru 0x03b8d4 (0x251b bytes) 000937 1734675800.307692 read 0x011bc6 thru 0x02042a (0xe865 bytes) 000939 1734675800.318345 mapread 0x01bad6 thru 0x025ba6 (0xa0d1 bytes) 000940 1734675800.320886 write 0x024c70 thru 0x032f4d (0xe2de bytes) 000942 1734675800.322340 fallocate 0x03efca thru 0x03ffff (0x1036 bytes) 000943 1734675800.329073 read 0x0312ab thru 0x0381a7 (0x6efd bytes) 000944 1734675800.329478 punch from 0x03af3a to 0x03b8d4 (0x99b bytes) 000945 1734675800.333386 fallocate 0x00db0b thru 0x010b00 (0x2ff6 bytes) 000946 1734675800.337489 read 0x003f3f thru 0x004c5d (0xd1f bytes) 000947 1734675800.337861 trunc from 0x03b8d5 to 0x0233fb (0x184d9 bytes) 000948 1734675800.353783 fallocate 0x01f886 thru 0x021b1d (0x2298 bytes) 000949 1734675800.359040 punch from 0x0223dd to 0x0233fb (0x101f bytes) 000950 1734675800.363279 read 0x0068bc thru 0x00ef1a (0x865f bytes) 000952 1734675800.365785 fallocate 0x000f03 thru 0x005329 (0x4427 bytes) 000953 1734675800.369620 fallocate 0x000cac thru 0x00b8f5 (0xac4a bytes) 000954 1734675800.374301 trunc from 0x0233fc to 0x0297ef (0x63f4 bytes) 000956 1734675800.392263 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d1:0x0]: Invalid argument (22) 000958 1734675800.590766 punch from 0x00b8f8 to 0x012649 (0x6d52 bytes) 000959 1734675800.594902 punch from 0x016240 to 0x01f65e (0x941f bytes) 000960 1734675800.618430 MIRROR_RESYNC 11 mirrors 000961 1734675802.034515 mapwrite 0x03b9cb thru 0x03ffff (0x4635 bytes) 000963 1734675802.113209 punch from 0x015a8c to 0x01bf64 (0x64d9 bytes) 000964 1734675802.119580 mapwrite 0x0100eb thru 0x01c3be (0xc2d4 bytes) 000965 1734675802.155689 trunc from 0x040000 to 0x025e19 (0x1a1e6 bytes) 000966 1734675802.170407 write 0x02d1c8 thru 0x033a89 (0x68c2 bytes) 000967 1734675802.171284 mapread 0x020aa0 thru 0x02aee2 (0xa443 bytes) 000968 1734675802.181326 mapread 0x018f1b thru 0x024c92 (0xbd78 bytes) 000969 1734675802.182388 fallocate 0x00d89f thru 0x01aba1 (0xd303 bytes) 000970 1734675802.186596 fallocate 0x015b5c thru 0x019e8d (0x4332 bytes) 000971 1734675802.190504 trunc from 0x033a8a to 0x02291b (0x1116e bytes) 000972 1734675802.210394 MIRROR_RESYNC 11 mirrors 000973 1734675803.467903 write 0x033df5 thru 0x035471 (0x167d bytes) 000974 1734675803.491167 punch from 0x015b83 to 0x025745 (0xfbc3 bytes) 000975 1734675803.494834 fallocate 0x0198c2 thru 0x02778d (0xdecc bytes) 000976 1734675803.498177 punch from 0x01fe17 to 0x02a5cb (0xa7b5 bytes) 000977 1734675803.504477 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d1:0x0]: Invalid argument (22) 000979 1734675803.653518 read 0x00b94d thru 0x010e5d (0x5511 bytes) 000980 1734675803.659787 trunc from 0x035472 to 0x00c996 (0x28adb bytes) 000981 1734675803.673765 punch from 0x00ba45 to 0x00c996 (0xf52 bytes) 000983 1734675803.676796 trunc from 0x00c997 to 0x01ed0a (0x12374 bytes) 000984 1734675803.685192 read 0x01203a thru 0x01ed0a (0xccd1 bytes) 000985 1734675803.692223 punch from 0x01125c to 0x019716 (0x84bb bytes) 000987 1734675803.698319 mapwrite 0x0380aa thru 0x03b2fb (0x3252 bytes) 000988 1734675803.734839 trunc from 0x03b2fc to 0x0398ef (0x1a0c bytes) 000990 1734675803.746849 mapwrite 0x0197f7 thru 0x026695 (0xce9f bytes) 000991 1734675803.767226 MIRROR_RESYNC 10 mirrors 000993 1734675804.777824 MIRROR_RESYNC 10 mirrors 000994 1734675804.925128 punch from 0x02dfbe to 0x033366 (0x53a9 bytes) 000996 1734675804.950276 mapwrite 0x00f7c5 thru 0x0170dd (0x7919 bytes) 000997 1734675804.978827 mapread 0x026a0b thru 0x027299 (0x88f bytes) 000998 1734675804.986868 mapread 0x0117d1 thru 0x01d103 (0xb933 bytes) 000999 1734675804.992147 write 0x005369 thru 0x00ec94 (0x992c bytes) 001000 1734675805.006093 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5d1:0x0]: Invalid argument (22) All operations completed A-OK! PASS 70a (42s) == sanity-flr test 200a: stress test ===================== 01:23:27 (1734675807) Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre2 Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7720672 .. Extending file size to 4414048 .. done Extending file size to 8050176 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5544832 .. Extending file size to 8464032 .. Extending file size to 8564416 .. Extending file size to 7594400 .. Extending file size to 8891008 .. Extending file size to 8821024 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8928960 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9187040 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9266432 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 338 339 341 342 344 done 192.168.202.103@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg203-client.virtnet /mnt/lustre2 (opts:) 192.168.202.103@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg203-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (67s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 01:24:34 (1734675874) Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre2 Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2462144 .. Extending file size to 6731840 .. done Extending file size to 7297024 .. Extending file size to 8277344 .. Extending file size to 8162400 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9015168 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8640896 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9183712 .. Extending file size to 8874944 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 2514 2516 2517 2518 2519 192.168.202.103@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg203-client.virtnet /mnt/lustre2 (opts:) 192.168.202.103@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg203-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (67s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 01:25:41 (1734675941) Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3016 1284672 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1916 1285772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 22724 3584296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 14756 3592264 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37480 7176560 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3016 1284672 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1916 1285772 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 22724 3584296 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 14756 3592264 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37480 7176560 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0401845 s, 52.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0660746 s, 31.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.103@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg203-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 01:25:46 (1734675946) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.063876 s, 32.8 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 01:25:49 (1734675949) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x618:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x688:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x688:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x689:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x6a6:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 01:25:52 (1734675952) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0436347 s, 48.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x33:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x33:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x34:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 01:25:55 (1734675955) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0509985 s, 41.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x68a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x68b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6a7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x68a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x68b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6a7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6a8:0x0] } PASS 204b (4s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 01:25:59 (1734675959) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0290582 s, 72.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x68c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6a9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x68c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6a9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6aa:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 01:26:24 (1734675984) Waiting for MDT destroys to complete oleg203-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 oleg203-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg203-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0254787 s, 41.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x34:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x36:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x35:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x34:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x36:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x35:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x35:0x0] } PASS 204d (35s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 01:26:59 (1734676019) Waiting for MDT destroys to complete oleg203-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 oleg203-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg203-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6ab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6ac:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0353329 s, 59.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6ab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6ac:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x68d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6ab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6ac:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x68d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (36s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 01:27:35 (1734676055) Waiting for MDT destroys to complete oleg203-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 oleg203-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg203-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg203-server: Pool lustre.test_204f created Adding targets to pool oleg203-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg203-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0317692 s, 66.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x68e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x68e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg203-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg203-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg203-server: Pool lustre.test_204f destroyed PASS 204f (47s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 01:28:22 (1734676102) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 01:28:25 (1734676105) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 01:28:27 (1734676107) oleg203-server: Pool lustre.test_206 created Adding targets to pool oleg203-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg203-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg203-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg203-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg203-server: Pool lustre.test_206 destroyed PASS 206 (18s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 01:28:45 (1734676125) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0316706 s, 33.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0290334 s, 36.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xfeb004ad Mirror 3: 0xfeb004ad PASS 207 (4s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 01:28:49 (1734676129) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 01:28:51 (1734676131) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 01:28:53 (1734676133) Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0439225 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xa20000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x8a0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x8a0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x8a0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xda0000 Read this (4 bytes): BBBB 192.168.202.103@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg203-client.virtnet /mnt/lustre2 (opts:) PASS 209a (18s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 01:29:11 (1734676151) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00802728 s, 1.0 MB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 01:29:14 (1734676154) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0244877 s, 42.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x639 lmm_fid: [0x200000405:0x639:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x639 lmm_fid: [0x200000405:0x639:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x699:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339612729 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-12-20 01:29:15.000000000 -0500 Modify: 2024-12-20 01:29:15.000000000 -0500 Change: 2024-12-20 01:29:16.000000000 -0500 Birth: - PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 01:29:18 (1734676158) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0133421 s, 78.6 MB/s oleg203-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg203-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg203-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 oleg203-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 oleg203-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg203-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg203-server: O/280000401/d23/6b7: File not found by ext2_lookup PASS 210b (12s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 01:29:30 (1734676170) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0455072 s, 900 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0128013 s, 320 kB/s size after second write -rw-r--r-- 1 root root 4096 Dec 20 01:29 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x63e:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Dec 20 01:29 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test complete, duration 926 sec ============ 01:29:33 (1734676173) === sanity-flr: start cleanup 01:29:34 (1734676174) === === sanity-flr: finish cleanup 01:29:37 (1734676177) ===