-----============= acceptance-small: sanity-flr ============----- Thu Dec 5 06:13:48 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 6 201 44c === sanity-flr: start setup 06:13:53 (1733397233) === oleg310-client.virtnet: executing check_config_client /mnt/lustre oleg310-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg310-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b6091800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6091800.idle_timeout=debug disable quota as required oleg310-server: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 06:14:02 (1733397242) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f7566] == sanity-flr test 0a: lfs mirror create with -N option == 06:14:03 (1733397243) PASS 0a (4s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 06:14:07 (1733397247) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 06:14:08 (1733397248) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 06:14:09 (1733397249) PASS 0d (7s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 06:14:16 (1733397256) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 06:14:17 (1733397257) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 06:14:18 (1733397258) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b6091800.stats=clear osc.lustre-OST0001-osc-ffff8800b6091800.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 06:14:20 (1733397260) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 06:14:23 (1733397263) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 06:14:25 (1733397265) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 06:14:26 (1733397266) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 06:14:29 (1733397269) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 06:14:31 (1733397271) PASS 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 06:14:33 (1733397273) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 06:14:35 (1733397275) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 06:14:38 (1733397278) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 06:14:40 (1733397280) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.23817 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0302569 s, 34.7 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 06:14:42 (1733397282) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.0615904 s, 136 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 06:14:44 (1733397284) Stopping /mnt/lustre-ost1 (opts:) on oleg310-server oleg310-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6091800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6091800.ost_server_uuid in DISCONN state after 0 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg310-server: oleg310-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg310-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg310-server oleg310-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6091800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6091800.ost_server_uuid in DISCONN state after 4 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg310-server: oleg310-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg310-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (25s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 06:15:09 (1733397309) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB) copied, 0.121008 s, 60.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg310-server oleg310-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6091800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6091800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg310-server: oleg310-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg310-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (10s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 06:15:19 (1733397319) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg310-server oleg310-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6091800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6091800.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg310-server: oleg310-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg310-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg310-server oleg310-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6091800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6091800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg310-server: oleg310-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg310-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (24s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 06:15:43 (1733397343) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.832185 s, 63.0 MB/s -rw-r--r-- 1 root root 52428800 Dec 5 06:15 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.227s user 0m0.001s sys 0m0.132s Stopping /mnt/lustre-ost1 (opts:) on oleg310-server oleg310-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6091800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6091800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.226s user 0m0.000s sys 0m0.101s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg310-server: oleg310-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg310-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg310-server oleg310-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6091800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6091800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.241s user 0m0.000s sys 0m0.128s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg310-server: oleg310-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg310-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg310-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b6091800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6091800.ost_server_uuid in FULL state after 0 sec PASS 33b (21s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 06:16:05 (1733397365) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 06:16:05 (1733397365) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 06:16:06 (1733397366) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 06:16:07 (1733397367) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 06:16:09 (1733397369) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.155974 s, 80.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 06:16:12 (1733397372) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116629 s, 89.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109052 s, 96.2 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 06:16:14 (1733397374) PASS 36c (2s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 06:16:16 (1733397376) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000605559 s, 1.7 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000576775 s, 1.8 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000632959 s, 1.6 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000618078 s, 1.7 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 06:16:19 (1733397379) create test files with size 15752522 .. 20a9ece1cbdbde6fea32b0a0dec5350d - a13820359b81fbf3cee903cd38917f48 - 4793d18ee25656250575c15d372dd994 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg310-client.virtnet /mnt/lustre (opts:) Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre PASS 37 (6s) == sanity-flr test 38: resync ============================ 06:16:24 (1733397384) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 06:16:27 (1733397387) PASS 39 (1s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 06:16:28 (1733397388) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0222608 s, 47.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0110673 s, 94.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x41:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 06:16:30 (1733397390) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0211678 s, 99.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0896644 s, 46.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0411433 s, 51.0 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 06:16:33 (1733397393) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 06:16:34 (1733397394) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (0s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 06:16:34 (1733397394) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00927573 s, 113 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 06:16:36 (1733397396) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0658588 s, 47.8 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x10:0x0]: Invalid argument (22) PASS 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 06:16:38 (1733397398) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4a:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x48:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x15:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 06:16:40 (1733397400) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0790897 s, 133 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg310-client.virtnet /mnt/lustre (opts:) Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (3s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 06:16:43 (1733397403) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800a5ac0800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0727373 s, 144 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg310-client.virtnet /mnt/lustre (opts:) Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800a1d03800.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 06:16:46 (1733397406) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (8s) == sanity-flr test 46: Verify setstripe --copy option ==== 06:16:54 (1733397414) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (6s) == sanity-flr test 47: Verify mirror obj alloc =========== 06:17:00 (1733397420) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 06:17:01 (1733397421) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0694665 s, 45.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0547082 s, 57.5 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: b5748b0f7138693c7a2cb853a9db7d25 - mirror 1 checksum: 17dc421e609a18c9d6d26810a3dcd372 - mirror 2 checksum: b5748b0f7138693c7a2cb853a9db7d25 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Dec 5 06:17:01 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Dec 5 06:17:01 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: b5748b0f7138693c7a2cb853a9db7d25 - mirror 1 checksum: 17dc421e609a18c9d6d26810a3dcd372 - mirror 2 checksum: 17dc421e609a18c9d6d26810a3dcd372 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Dec 5 06:17:01 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Dec 5 06:17:01 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } PASS 48 (2s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 06:17:03 (1733397423) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00902515 s, 116 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 529408.. 530431: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 06:17:05 (1733397425) Failing mds1 on oleg310-server Stopping /mnt/lustre-mds1 (opts:) on oleg310-server 06:17:12 (1733397432) shut down Failover mds1 to oleg310-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg310-server: oleg310-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-MDT0000 06:17:25 (1733397445) targets are mounted 06:17:25 (1733397445) facet_failover done oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0103674 s, 101 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x56:0x0] } PASS 50A (28s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 06:17:33 (1733397453) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0078299 s, 2.6 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0045676 s, 4.5 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 06:17:35 (1733397455) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00763645 s, 2.7 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00206807 s, 9.9 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 06:17:36 (1733397456) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00181985 s, 9.0 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 06:17:38 (1733397458) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0101668 s, 2.0 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 06:17:39 (1733397459) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00170733 s, 2.4 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00560565 s, 731 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 06:17:41 (1733397461) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (3s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 06:17:44 (1733397464) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 06:18:02 (1733397482) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x47:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 06:18:19 (1733397499) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (16s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 06:18:35 (1733397515) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 06:18:37 (1733397517) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 26046 mirror split pid 26047 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6724: 26047 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6724: 26047 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4855: 26046 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4855: 26047 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 06:19:39 (1733397579) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 7579 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read skipping zero size read fallocating to largest ever: 0x14eeb 000005 1733397579.660946 fallocate 0x006696 thru 0x014eea (0xe855 bytes) truncating to largest ever: 0x2d0e7 000006 1733397579.662390 trunc from 0x014eeb to 0x02d0e6 (0x181fc bytes) 000007 1733397579.665440 fallocate 0x002096 thru 0x00c479 (0xa3e4 bytes) 000010 1733397579.667055 mapwrite 0x032d4b thru 0x03ffff (0xd2b5 bytes) 000012 1733397579.682640 write 0x012fd2 thru 0x018296 (0x52c5 bytes) 000013 1733397579.684792 mapread 0x014ed4 thru 0x02206d (0xd19a bytes) 000014 1733397579.686728 write 0x036861 thru 0x03f1d8 (0x8978 bytes) 000016 1733397579.687775 mapwrite 0x0203c2 thru 0x024407 (0x4046 bytes) truncating to largest ever: 0x37d34 000017 1733397579.692047 trunc from 0x040000 to 0x037d33 (0x82cc bytes) 000019 1733397579.695670 read 0x007988 thru 0x01167b (0x9cf4 bytes) 000020 1733397579.697062 write 0x034926 thru 0x03555b (0xc36 bytes) 000021 1733397579.697459 fallocate 0x00a41c thru 0x012342 (0x7f27 bytes) 000022 1733397579.698542 write 0x035cc6 thru 0x03ffff (0xa33a bytes) 000023 1733397579.699112 write 0x039246 thru 0x03ffff (0x6dba bytes) 000024 1733397579.699627 mapwrite 0x038123 thru 0x03ab25 (0x2a03 bytes) 000025 1733397579.703587 fallocate 0x01e83f thru 0x02dd7e (0xf540 bytes) 000028 1733397579.705098 punch from 0x02709b to 0x02e7c4 (0x772a bytes) 000029 1733397579.706308 punch from 0x00680a to 0x00bafb (0x52f2 bytes) 000032 1733397579.707958 punch from 0x033fe1 to 0x03ffff (0xc01f bytes) 000033 1733397579.709167 mapwrite 0x027fa9 thru 0x030e4c (0x8ea4 bytes) 000035 1733397579.715604 read 0x026b34 thru 0x02d736 (0x6c03 bytes) 000038 1733397579.717075 mapwrite 0x026257 thru 0x02650d (0x2b7 bytes) 000040 1733397579.719963 punch from 0x029d31 to 0x039926 (0xfbf6 bytes) 000041 1733397579.721093 mapread 0x02b249 thru 0x03587b (0xa633 bytes) 000042 1733397579.724592 MIRROR_EXTEND to 2 mirrors 000044 1733397579.814100 punch from 0x01c656 to 0x025fb7 (0x9962 bytes) 000045 1733397579.821428 read 0x01441b thru 0x0229bc (0xe5a2 bytes) 000046 1733397579.823917 mapwrite 0x00ef17 thru 0x01caea (0xdbd4 bytes) 000049 1733397579.834721 write 0x02f0cd thru 0x0367d5 (0x7709 bytes) 000050 1733397579.837784 mapread 0x00cc12 thru 0x00ef57 (0x2346 bytes) 000051 1733397579.839132 MIRROR_EXTEND to 3 mirrors truncating to largest ever: 0x3fc68 000052 1733397579.933000 trunc from 0x040000 to 0x03fc67 (0x398 bytes) 000055 1733397579.944832 punch from 0x036b43 to 0x03fc67 (0x9125 bytes) 000056 1733397579.945969 punch from 0x03738f to 0x03fc67 (0x88d9 bytes) 000057 1733397579.946963 trunc from 0x03fc68 to 0x002b30 (0x3d137 bytes) 000058 1733397579.951142 mapread 0x000add thru 0x002b30 (0x2054 bytes) 000059 1733397579.954611 read 0x000c40 thru 0x002b30 (0x1ef1 bytes) 000060 1733397579.954959 read 0x002848 thru 0x002b30 (0x2e9 bytes) 000061 1733397579.955309 fallocate 0x00b293 thru 0x010415 (0x5183 bytes) 000063 1733397579.956736 trunc from 0x002b31 to 0x002f44 (0x414 bytes) 000064 1733397579.959569 mapwrite 0x02443f thru 0x0294f1 (0x50b3 bytes) 000065 1733397579.968837 mapwrite 0x02f75c thru 0x039e4f (0xa6f4 bytes) fallocating to largest ever: 0x40000 000066 1733397579.978714 fallocate 0x03e6f1 thru 0x03ffff (0x190f bytes) 000067 1733397579.979978 trunc from 0x039e50 to 0x00a43d (0x2fa12 bytes) 000068 1733397579.983399 mapwrite 0x021b2e thru 0x02a604 (0x8ad7 bytes) 000069 1733397580.000930 mapread 0x0221ac thru 0x02a604 (0x8459 bytes) 000072 1733397580.002203 fallocate 0x005c77 thru 0x00c563 (0x68ed bytes) 000073 1733397580.003706 trunc from 0x02a605 to 0x01ce2e (0xd7d6 bytes) 000074 1733397580.006947 trunc from 0x01ce2f to 0x01771c (0x5712 bytes) 000075 1733397580.009859 mapwrite 0x0275d3 thru 0x03374a (0xc178 bytes) 000076 1733397580.029449 trunc from 0x03374b to 0x00cedf (0x2686b bytes) 000077 1733397580.032594 mapwrite 0x037387 thru 0x03ffff (0x8c79 bytes) 000079 1733397580.044204 read 0x002c5f thru 0x00d4e0 (0xa882 bytes) 000080 1733397580.046618 mapread 0x01c5d0 thru 0x0253c1 (0x8df2 bytes) 000081 1733397580.047038 mapwrite 0x00369f thru 0x00547c (0x1dde bytes) 000082 1733397580.050488 mapread 0x02e406 thru 0x0395ba (0xb1b5 bytes) 000083 1733397580.050976 mapread 0x019d2e thru 0x024fd9 (0xb2ac bytes) 000084 1733397580.051466 mapwrite 0x02ffbf thru 0x03d3cb (0xd40d bytes) 000085 1733397580.056292 mapread 0x0096e1 thru 0x00edf3 (0x5713 bytes) 000086 1733397580.056733 fallocate 0x037dd0 thru 0x03ffff (0x8230 bytes) 000088 1733397580.058045 trunc from 0x040000 to 0x00acf3 (0x3530c bytes) 000089 1733397580.061466 trunc from 0x00acf4 to 0x0228ce (0x17bdb bytes) 000090 1733397580.064342 read 0x01d9e4 thru 0x0228ce (0x4eeb bytes) 000092 1733397580.066561 mapwrite 0x02f61a thru 0x038ddb (0x97c2 bytes) 000094 1733397580.075525 mapread 0x0321e9 thru 0x038ddb (0x6bf3 bytes) 000095 1733397580.075915 read 0x004484 thru 0x0098fa (0x5477 bytes) 000097 1733397580.076538 fallocate 0x028758 thru 0x03120c (0x8ab5 bytes) 000102 1733397580.078481 fallocate 0x00c447 thru 0x00cf63 (0xb1d bytes) 000103 1733397580.079528 read 0x035417 thru 0x038ddb (0x39c5 bytes) 000104 1733397580.079747 mapread 0x000ebe thru 0x001c6e (0xdb1 bytes) 000105 1733397580.080087 fallocate 0x011d7f thru 0x013dc5 (0x2047 bytes) 000107 1733397580.081571 mapread 0x036352 thru 0x038ddb (0x2a8a bytes) 000109 1733397580.082164 read 0x023858 thru 0x02b527 (0x7cd0 bytes) 000110 1733397580.082510 fallocate 0x0178e4 thru 0x01ab79 (0x3296 bytes) 000111 1733397580.083612 fallocate 0x0077f9 thru 0x010042 (0x884a bytes) 000112 1733397580.084764 write 0x01323d thru 0x016956 (0x371a bytes) 000113 1733397580.085255 punch from 0x0128b6 to 0x01b94d (0x9098 bytes) 000114 1733397580.087224 fallocate 0x01eb67 thru 0x02a287 (0xb721 bytes) 000115 1733397580.088212 read 0x024f27 thru 0x02e778 (0x9852 bytes) 000116 1733397580.088517 mapwrite 0x0335a1 thru 0x03ffff (0xca5f bytes) 000118 1733397580.098014 trunc from 0x040000 to 0x002511 (0x3daee bytes) 000119 1733397580.100955 fallocate 0x03c1b0 thru 0x03ce67 (0xcb8 bytes) 000120 1733397580.102192 read 0x000b54 thru 0x00147a (0x927 bytes) 000121 1733397580.102653 mapread 0x0008c2 thru 0x002511 (0x1c50 bytes) 000122 1733397580.103007 mapwrite 0x008a0f thru 0x00ec26 (0x6218 bytes) 000124 1733397580.112872 punch from 0x0057e0 to 0x00ec26 (0x9447 bytes) 000126 1733397580.114329 trunc from 0x00ec27 to 0x007807 (0x741f bytes) 000128 1733397580.117089 punch from 0x000ca2 to 0x0046e9 (0x3a48 bytes) 000129 1733397580.118225 write 0x020cb0 thru 0x02219f (0x14f0 bytes) 000131 1733397580.118839 write 0x03b852 thru 0x03ffff (0x47ae bytes) 000132 1733397580.119250 fallocate 0x02ea8d thru 0x039c5f (0xb1d3 bytes) 000133 1733397580.120533 trunc from 0x040000 to 0x03744f (0x8bb0 bytes) 000134 1733397580.123432 punch from 0x01eb48 to 0x02b4de (0xc997 bytes) 000135 1733397580.125738 mapwrite 0x034d58 thru 0x03935e (0x4607 bytes) 000136 1733397580.135621 mapwrite 0x0326b5 thru 0x03438b (0x1cd7 bytes) 000137 1733397580.138552 trunc from 0x03935f to 0x010e68 (0x284f6 bytes) 000139 1733397580.141804 mapread 0x0074b1 thru 0x007ce9 (0x839 bytes) 000140 1733397580.143187 read 0x00e374 thru 0x010e68 (0x2af5 bytes) 000141 1733397580.145124 punch from 0x00825a to 0x010e68 (0x8c0f bytes) 000142 1733397580.146311 punch from 0x00aade to 0x00b875 (0xd98 bytes) 000145 1733397580.147943 mapread 0x008f12 thru 0x00ff0a (0x6ff9 bytes) 000146 1733397580.155141 read 0x006126 thru 0x009b0c (0x39e7 bytes) 000147 1733397580.155539 trunc from 0x010e69 to 0x016309 (0x54a1 bytes) 000148 1733397580.158612 write 0x028a0f thru 0x029d09 (0x12fb bytes) 000149 1733397580.159040 punch from 0x012c2d to 0x0167aa (0x3b7e bytes) 000150 1733397580.160132 fallocate 0x0257ac thru 0x0264fb (0xd50 bytes) 000151 1733397580.162097 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa25:0x0]: Invalid argument (22) 000152 1733397580.195716 fallocate 0x01d43f thru 0x028295 (0xae57 bytes) 000153 1733397580.197071 trunc from 0x029d0a to 0x0383a8 (0xe69f bytes) 000154 1733397580.202043 write 0x0153fb thru 0x0198b6 (0x44bc bytes) 000155 1733397580.203967 mapwrite 0x012ad9 thru 0x0184c1 (0x59e9 bytes) 000156 1733397580.209794 read 0x02b562 thru 0x03227e (0x6d1d bytes) 000157 1733397580.211459 mapread 0x0310f3 thru 0x0383a8 (0x72b6 bytes) 000158 1733397580.213495 mapwrite 0x03d92b thru 0x03ffff (0x26d5 bytes) 000159 1733397580.220790 trunc from 0x040000 to 0x02265a (0x1d9a5 bytes) 000161 1733397580.224140 punch from 0x01150f to 0x01de5f (0xc951 bytes) 000162 1733397580.225522 punch from 0x0015cf to 0x0114e2 (0xff14 bytes) 000163 1733397580.226733 read 0x011ccf thru 0x014f56 (0x3288 bytes) 000165 1733397580.229366 punch from 0x011840 to 0x018717 (0x6ed8 bytes) 000168 1733397580.231033 trunc from 0x02265b to 0x005d45 (0x1c915 bytes) 000169 1733397580.234413 write 0x00cb55 thru 0x01039b (0x3847 bytes) 000170 1733397580.234886 read 0x00a9cd thru 0x01039b (0x59cf bytes) 000171 1733397580.237756 MIRROR_RESYNC 2 mirrors 000173 1733397580.301414 fallocate 0x027361 thru 0x02fe46 (0x8ae6 bytes) 000174 1733397580.307434 write 0x00df31 thru 0x01927a (0xb34a bytes) 000175 1733397580.310913 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa25:0x0]: Invalid argument (22) 000177 1733397580.334620 mapread 0x02ca74 thru 0x02fe46 (0x33d3 bytes) 000181 1733397580.339390 read 0x02938a thru 0x02fe46 (0x6abd bytes) 000182 1733397580.339723 mapwrite 0x017552 thru 0x024694 (0xd143 bytes) 000183 1733397580.348200 mapwrite 0x01870e thru 0x020d41 (0x8634 bytes) 000186 1733397580.352985 punch from 0x023826 to 0x024eca (0x16a5 bytes) 000188 1733397580.354371 read 0x009dc6 thru 0x014976 (0xabb1 bytes) 000189 1733397580.357078 MIRROR_EXTEND to 2 mirrors 000190 1733397580.435179 read 0x012b1c thru 0x01e818 (0xbcfd bytes) 000192 1733397580.437097 read 0x01613c thru 0x01a374 (0x4239 bytes) 000193 1733397580.437251 mapwrite 0x01597f thru 0x0233f1 (0xda73 bytes) 000194 1733397580.473580 fallocate 0x00f5aa thru 0x010039 (0xa90 bytes) 000196 1733397580.475054 fallocate 0x01edd2 thru 0x024727 (0x5956 bytes) 000197 1733397580.476235 punch from 0x02ef05 to 0x02fe46 (0xf42 bytes) 000198 1733397580.477307 trunc from 0x02fe47 to 0x0184f5 (0x17951 bytes) 000201 1733397580.485084 mapread 0x00a578 thru 0x00b529 (0xfb2 bytes) 000202 1733397580.488261 MIRROR_RESYNC 2 mirrors 000203 1733397580.549209 MIRROR_RESYNC 2 mirrors 000204 1733397580.574901 mapread 0x001aee thru 0x00f732 (0xdc45 bytes) 000205 1733397580.579954 fallocate 0x018027 thru 0x023f4d (0xbf27 bytes) 000206 1733397580.586465 punch from 0x0000f4 to 0x009ecc (0x9dd9 bytes) 000207 1733397580.587559 trunc from 0x0184f6 to 0x003af0 (0x14a05 bytes) 000208 1733397580.593957 read 0x002ab0 thru 0x003af0 (0x1041 bytes) 000210 1733397580.595668 mapread 0x001f2d thru 0x003af0 (0x1bc4 bytes) 000211 1733397580.597027 mapread 0x000dd5 thru 0x003af0 (0x2d1c bytes) 000213 1733397580.598491 write 0x0198e3 thru 0x01db10 (0x422e bytes) 000214 1733397580.598951 read 0x0167a1 thru 0x01db10 (0x7370 bytes) 000215 1733397580.601756 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa25:0x0]: Invalid argument (22) 000216 1733397580.624573 read 0x0090ac thru 0x009ea2 (0xdf7 bytes) 000217 1733397580.625920 mapwrite 0x006342 thru 0x012778 (0xc437 bytes) 000219 1733397580.635858 read 0x005296 thru 0x01297d (0xd6e8 bytes) 000220 1733397580.636131 trunc from 0x01db11 to 0x030add (0x12fcd bytes) 000221 1733397580.639464 trunc from 0x030ade to 0x01cabc (0x14021 bytes) 000223 1733397580.642427 read 0x0152e5 thru 0x01cabc (0x77d8 bytes) 000224 1733397580.644290 fallocate 0x033e56 thru 0x03587a (0x1a25 bytes) 000225 1733397580.645555 mapwrite 0x01ef0d thru 0x0211f1 (0x22e5 bytes) 000226 1733397580.654087 trunc from 0x0211f2 to 0x00a168 (0x17089 bytes) 000227 1733397580.657186 write 0x026bcb thru 0x02f4fc (0x8932 bytes) 000228 1733397580.657959 read 0x007a5d thru 0x009832 (0x1dd6 bytes) 000229 1733397580.658189 fallocate 0x03baa0 thru 0x03ffff (0x4560 bytes) 000230 1733397580.659556 fallocate 0x037bae thru 0x03f781 (0x7bd4 bytes) 000231 1733397580.660751 read 0x016792 thru 0x02441c (0xdc8b bytes) 000237 1733397580.664433 write 0x026fc1 thru 0x02da62 (0x6aa2 bytes) 000240 1733397580.665348 fallocate 0x011bb9 thru 0x0162ca (0x4712 bytes) 000242 1733397580.666822 read 0x021bc3 thru 0x03083f (0xec7d bytes) 000244 1733397580.667357 trunc from 0x040000 to 0x01c568 (0x23a97 bytes) 000245 1733397580.671241 MIRROR_EXTEND to 2 mirrors 000246 1733397580.746636 mapwrite 0x029133 thru 0x0316aa (0x8578 bytes) 000247 1733397580.767019 MIRROR_RESYNC 2 mirrors 000248 1733397580.838688 mapwrite 0x012033 thru 0x01891f (0x68ed bytes) 000249 1733397580.864760 punch from 0x0195bf to 0x0252ce (0xbd10 bytes) 000250 1733397580.866087 read 0x019401 thru 0x0255ba (0xc1ba bytes) 000251 1733397580.868934 MIRROR_EXTEND to 3 mirrors 000252 1733397580.951325 fallocate 0x015659 thru 0x020722 (0xb0ca bytes) 000253 1733397580.959442 MIRROR_EXTEND to 4 mirrors 000254 1733397581.047949 mapread 0x022a4b thru 0x02493d (0x1ef3 bytes) 000255 1733397581.051961 fallocate 0x024bdb thru 0x025ffa (0x1420 bytes) 000256 1733397581.059026 mapwrite 0x03f039 thru 0x03ffff (0xfc7 bytes) 000257 1733397581.067950 write 0x01342e thru 0x023076 (0xfc49 bytes) 000262 1733397581.072153 write 0x02006a thru 0x02dd0f (0xdca6 bytes) 000263 1733397581.073884 mapwrite 0x03cd7a thru 0x03f0ab (0x2332 bytes) 000264 1733397581.077977 mapread 0x028c0e thru 0x033668 (0xaa5b bytes) 000265 1733397581.080303 fallocate 0x0074ab thru 0x0166da (0xf230 bytes) 000266 1733397581.081655 mapwrite 0x02958d thru 0x02a843 (0x12b7 bytes) 000267 1733397581.084227 mapwrite 0x00db72 thru 0x0170a9 (0x9538 bytes) 000269 1733397581.088741 fallocate 0x007aa8 thru 0x00804c (0x5a5 bytes) 000271 1733397581.090216 mapread 0x016513 thru 0x01db41 (0x762f bytes) 000273 1733397581.091122 mapread 0x00f026 thru 0x010055 (0x1030 bytes) 000274 1733397581.092743 MIRROR_EXTEND to 5 mirrors 000275 1733397581.177761 trunc from 0x040000 to 0x00a6bd (0x35942 bytes) 000276 1733397581.188908 fallocate 0x03a0b6 thru 0x03ffff (0x5f4a bytes) 000277 1733397581.190165 punch from 0x000977 to 0x00a6bd (0x9d47 bytes) 000278 1733397581.191323 punch from 0x003873 to 0x005c67 (0x23f5 bytes) 000279 1733397581.192441 mapread 0x006f55 thru 0x00a6bd (0x3769 bytes) 000280 1733397581.196028 punch from 0x009673 to 0x00a6bd (0x104b bytes) 000281 1733397581.197028 mapread 0x001cd0 thru 0x00847b (0x67ac bytes) 000282 1733397581.198352 MIRROR_EXTEND to 6 mirrors 000283 1733397581.270325 mapread 0x003650 thru 0x00a6bd (0x706e bytes) 000284 1733397581.276350 mapread 0x0053b1 thru 0x00a6bd (0x530d bytes) 000285 1733397581.279201 punch from 0x003854 to 0x003dbd (0x56a bytes) 000286 1733397581.287063 MIRROR_RESYNC 6 mirrors 000287 1733397581.413552 write 0x032598 thru 0x039495 (0x6efe bytes) 000288 1733397581.419711 trunc from 0x039496 to 0x035b3f (0x3956 bytes) 000289 1733397581.426632 fallocate 0x03d9f9 thru 0x03ffff (0x2607 bytes) 000291 1733397581.428219 fallocate 0x031151 thru 0x03624d (0x50fd bytes) 000292 1733397581.429681 read 0x03196b thru 0x035b3f (0x41d5 bytes) 000293 1733397581.430980 mapwrite 0x01e795 thru 0x023d1c (0x5588 bytes) 000294 1733397581.439173 trunc from 0x035b40 to 0x00e5fa (0x27545 bytes) 000296 1733397581.442931 trunc from 0x00e5fb to 0x011881 (0x3287 bytes) 000297 1733397581.447050 MIRROR_EXTEND to 7 mirrors 000298 1733397581.524590 mapwrite 0x0041c6 thru 0x00e5da (0xa415 bytes) 000300 1733397581.552963 punch from 0x0084e0 to 0x0116a2 (0x91c3 bytes) 000301 1733397581.554241 punch from 0x00db52 to 0x00e489 (0x938 bytes) 000302 1733397581.555406 read 0x0045ff thru 0x010321 (0xbd23 bytes) 000304 1733397581.557606 punch from 0x004aa5 to 0x00b817 (0x6d73 bytes) 000305 1733397581.558777 fallocate 0x0398be thru 0x03ffff (0x6742 bytes) 000306 1733397581.560073 mapread 0x00d8c3 thru 0x017865 (0x9fa3 bytes) 000307 1733397581.562238 read 0x01565a thru 0x01920b (0x3bb2 bytes) 000309 1733397581.562792 fallocate 0x022516 thru 0x022bbd (0x6a8 bytes) 000310 1733397581.564157 write 0x000594 thru 0x004ffc (0x4a69 bytes) 000311 1733397581.567324 punch from 0x009dac to 0x017ff8 (0xe24d bytes) 000312 1733397581.568526 trunc from 0x040000 to 0x026aff (0x19500 bytes) 000313 1733397581.575884 MIRROR_EXTEND to 8 mirrors 000314 1733397581.648238 write 0x03bf1a thru 0x03ffff (0x40e6 bytes) 000315 1733397581.657011 MIRROR_RESYNC 8 mirrors 000316 1733397581.794161 punch from 0x028f49 to 0x036174 (0xd22c bytes) 000317 1733397581.800880 read 0x003dee thru 0x009fcd (0x61e0 bytes) 000319 1733397581.802647 trunc from 0x040000 to 0x025f39 (0x1a0c6 bytes) 000320 1733397581.809978 trunc from 0x025f3a to 0x0263d7 (0x49e bytes) 000321 1733397581.812981 mapread 0x000c20 thru 0x004cdd (0x40be bytes) 000322 1733397581.816911 fallocate 0x01fc3f thru 0x0213ce (0x1790 bytes) 000323 1733397581.819343 MIRROR_RESYNC 8 mirrors 000324 1733397581.970582 mapread 0x01d91e thru 0x024590 (0x6c73 bytes) 000325 1733397581.974342 fallocate 0x007661 thru 0x0110c6 (0x9a66 bytes) 000326 1733397581.982452 read 0x015e85 thru 0x01b289 (0x5405 bytes) 000328 1733397581.984467 write 0x01088a thru 0x01e79b (0xdf12 bytes) 000329 1733397581.987501 fallocate 0x0331df thru 0x03ad7e (0x7ba0 bytes) 000330 1733397581.988877 trunc from 0x03ad7f to 0x03d095 (0x2317 bytes) 000331 1733397581.996039 mapwrite 0x02ad45 thru 0x02d1ac (0x2468 bytes) 000332 1733397582.002879 read 0x02f3ab thru 0x031324 (0x1f7a bytes) 000333 1733397582.003294 fallocate 0x025de4 thru 0x027479 (0x1696 bytes) 000334 1733397582.004625 punch from 0x01b838 to 0x022152 (0x691b bytes) 000335 1733397582.005912 fallocate 0x00996d thru 0x01274b (0x8ddf bytes) 000336 1733397582.007156 read 0x025bcf thru 0x02bd93 (0x61c5 bytes) 000337 1733397582.007495 read 0x01cad3 thru 0x01ea85 (0x1fb3 bytes) 000338 1733397582.009872 punch from 0x024330 to 0x024b14 (0x7e5 bytes) 000339 1733397582.011003 trunc from 0x03d096 to 0x033e2d (0x9268 bytes) 000340 1733397582.014296 trunc from 0x033e2e to 0x02fed2 (0x3f5b bytes) 000341 1733397582.017059 write 0x03157f thru 0x03ec58 (0xd6da bytes) 000342 1733397582.017921 mapwrite 0x026006 thru 0x02ba78 (0x5a73 bytes) 000343 1733397582.022829 mapwrite 0x0023f2 thru 0x004ee5 (0x2af4 bytes) 000344 1733397582.025871 punch from 0x035200 to 0x0386f3 (0x34f4 bytes) 000345 1733397582.027055 mapwrite 0x01afc2 thru 0x01c8d6 (0x1915 bytes) 000347 1733397582.030669 write 0x022f65 thru 0x02f0e5 (0xc181 bytes) 000348 1733397582.031665 punch from 0x0177d8 to 0x01b6b4 (0x3edd bytes) 000350 1733397582.033716 punch from 0x02dcd9 to 0x0378f1 (0x9c19 bytes) 000351 1733397582.037963 write 0x03d2bf thru 0x03dffd (0xd3f bytes) 000352 1733397582.038584 mapread 0x00cf59 thru 0x00cf64 (0xc bytes) 000353 1733397582.039107 fallocate 0x02c686 thru 0x02e526 (0x1ea1 bytes) 000355 1733397582.042136 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa25:0x0]: Invalid argument (22) 000356 1733397582.084583 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa25:0x0]: Invalid argument (22) 000357 1733397582.121847 read 0x00dbc8 thru 0x00dde4 (0x21d bytes) 000359 1733397582.123644 trunc from 0x03ec59 to 0x0397af (0x54a9 bytes) 000360 1733397582.127337 mapwrite 0x034e6c thru 0x0397b9 (0x494e bytes) 000361 1733397582.138413 punch from 0x007a1c to 0x00d0ec (0x56d1 bytes) 000362 1733397582.141129 MIRROR_EXTEND to 7 mirrors 000363 1733397582.246759 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa25:0x0]: Invalid argument (22) 000365 1733397582.311196 write 0x021122 thru 0x0303f2 (0xf2d1 bytes) 000366 1733397582.322001 fallocate 0x01e890 thru 0x024872 (0x5fe3 bytes) 000368 1733397582.323714 punch from 0x00b5a4 to 0x01a32d (0xed8a bytes) 000369 1733397582.325147 read 0x015503 thru 0x0170db (0x1bd9 bytes) 000370 1733397582.326793 fallocate 0x035b88 thru 0x03b77f (0x5bf8 bytes) 000371 1733397582.328273 mapread 0x01e368 thru 0x0299c2 (0xb65b bytes) 000372 1733397582.330888 mapwrite 0x000837 thru 0x0042f8 (0x3ac2 bytes) 000373 1733397582.339202 mapread 0x02af72 thru 0x038ef3 (0xdf82 bytes) 000374 1733397582.341832 fallocate 0x02304b thru 0x02fab9 (0xca6f bytes) 000375 1733397582.343427 fallocate 0x00e782 thru 0x01280e (0x408d bytes) 000376 1733397582.345317 mapwrite 0x039822 thru 0x03ffff (0x67de bytes) 000377 1733397582.362078 read 0x00ae78 thru 0x01125b (0x63e4 bytes) 000378 1733397582.364169 mapread 0x01c11a thru 0x024df7 (0x8cde bytes) 000379 1733397582.364697 trunc from 0x040000 to 0x03d37b (0x2c84 bytes) 000380 1733397582.368358 mapwrite 0x003e92 thru 0x00601e (0x218d bytes) 000381 1733397582.372781 read 0x0253d0 thru 0x025657 (0x288 bytes) 000382 1733397582.373031 mapwrite 0x00d40e thru 0x01a4c0 (0xd0b3 bytes) 000383 1733397582.378246 mapwrite 0x0246ff thru 0x02d084 (0x8986 bytes) 000384 1733397582.383741 write 0x028bec thru 0x02e7b9 (0x5bce bytes) 000385 1733397582.385665 MIRROR_RESYNC 6 mirrors 000386 1733397582.637838 read 0x00f414 thru 0x0194a6 (0xa093 bytes) 000387 1733397582.640002 read 0x02a12a thru 0x033ef4 (0x9dcb bytes) 000388 1733397582.642964 mapwrite 0x03e457 thru 0x03ffff (0x1ba9 bytes) 000390 1733397582.662236 trunc from 0x040000 to 0x031452 (0xebad bytes) 000391 1733397582.667324 trunc from 0x031453 to 0x039699 (0x8247 bytes) 000393 1733397582.672210 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa25:0x0]: Invalid argument (22) 000394 1733397582.711092 fallocate 0x00966a thru 0x015759 (0xc0f0 bytes) 000395 1733397582.712541 fallocate 0x01393d thru 0x01dabf (0xa183 bytes) 000396 1733397582.713764 read 0x0256f0 thru 0x031881 (0xc192 bytes) 000399 1733397582.716783 trunc from 0x03969a to 0x02fa04 (0x9c95 bytes) 000400 1733397582.721516 read 0x02d4dc thru 0x02fa04 (0x2529 bytes) 000401 1733397582.721916 trunc from 0x02fa05 to 0x026e66 (0x8b9e bytes) 000402 1733397582.725277 punch from 0x011404 to 0x021254 (0xfe51 bytes) 000403 1733397582.728046 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa25:0x0]: Invalid argument (22) 000404 1733397582.766455 read 0x01f6be thru 0x026e66 (0x77a9 bytes) 000405 1733397582.768320 mapread 0x00abca thru 0x014725 (0x9b5c bytes) 000406 1733397582.773738 mapread 0x003e15 thru 0x00f9d9 (0xbbc5 bytes) 000407 1733397582.774324 trunc from 0x026e67 to 0x039b85 (0x12d1f bytes) 000408 1733397582.779285 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa25:0x0]: Invalid argument (22) 000409 1733397582.818618 write 0x00821f thru 0x00aea3 (0x2c85 bytes) 000410 1733397582.821985 trunc from 0x039b86 to 0x028ab8 (0x110cd bytes) 000411 1733397582.826444 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa25:0x0]: Invalid argument (22) 000412 1733397582.874352 mapwrite 0x01e310 thru 0x01fe01 (0x1af2 bytes) 000413 1733397582.881761 punch from 0x010ff7 to 0x013899 (0x28a3 bytes) 000414 1733397582.883454 fallocate 0x025c8f thru 0x025ffa (0x36c bytes) 000416 1733397582.885600 trunc from 0x028ab9 to 0x03cea9 (0x143f1 bytes) 000417 1733397582.890633 write 0x02f84a thru 0x033560 (0x3d17 bytes) 000418 1733397582.893390 mapread 0x029664 thru 0x03845f (0xedfc bytes) 000419 1733397582.896119 read 0x015d2a thru 0x017276 (0x154d bytes) 000420 1733397582.898010 mapwrite 0x03043b thru 0x030748 (0x30e bytes) 000421 1733397582.903163 read 0x01442a thru 0x017b58 (0x372f bytes) 000422 1733397582.907436 MIRROR_RESYNC 2 mirrors 000423 1733397583.010430 mapwrite 0x00284c thru 0x0101f0 (0xd9a5 bytes) 000424 1733397583.039946 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa25:0x0]: Invalid argument (22) 000425 1733397583.065524 trunc from 0x03ceaa to 0x0384b2 (0x49f7 bytes) 000426 1733397583.072499 punch from 0x02cba9 to 0x035a0d (0x8e65 bytes) 000427 1733397583.073690 fallocate 0x012dc7 thru 0x01f1c1 (0xc3fb bytes) 000428 1733397583.074752 punch from 0x018ac5 to 0x022c0a (0xa146 bytes) 000430 1733397583.076192 write 0x025fec thru 0x02f097 (0x90ac bytes) 000431 1733397583.078869 trunc from 0x0384b3 to 0x00211a (0x36398 bytes) 000433 1733397583.082994 mapwrite 0x00e78d thru 0x012e34 (0x46a8 bytes) 000436 1733397583.093410 write 0x024f0b thru 0x029536 (0x462c bytes) 000438 1733397583.094401 trunc from 0x029537 to 0x00a19a (0x1f39c bytes) 000439 1733397583.097822 read 0x00a176 thru 0x00a19a (0x25 bytes) 000442 1733397583.098879 read 0x004e3c thru 0x00a19a (0x535f bytes) 000443 1733397583.101024 trunc from 0x00a19b to 0x007e0b (0x238f bytes) 000444 1733397583.104558 read 0x005972 thru 0x007e0b (0x249a bytes) 000447 1733397583.105240 write 0x002d78 thru 0x0060c1 (0x334a bytes) 000450 1733397583.106404 trunc from 0x007e0c to 0x00131a (0x6af1 bytes) 000451 1733397583.109555 fallocate 0x029f98 thru 0x02e887 (0x48f0 bytes) 000452 1733397583.111183 write 0x011b49 thru 0x012264 (0x71c bytes) 000453 1733397583.114155 mapread 0x01f59f thru 0x0295eb (0xa04d bytes) 000455 1733397583.115134 write 0x004348 thru 0x00ca9b (0x8754 bytes) 000456 1733397583.115866 punch from 0x00d742 to 0x00fb64 (0x2423 bytes) 000457 1733397583.117379 fallocate 0x00a989 thru 0x00f28f (0x4907 bytes) 000458 1733397583.118713 punch from 0x00ec64 to 0x014a92 (0x5e2f bytes) 000459 1733397583.122080 read 0x022223 thru 0x02c22c (0xa00a bytes) 000460 1733397583.122488 fallocate 0x00f1de thru 0x012be2 (0x3a05 bytes) 000462 1733397583.124388 write 0x02d737 thru 0x03bf0f (0xe7d9 bytes) 000463 1733397583.125403 fallocate 0x01fbc4 thru 0x02af95 (0xb3d2 bytes) 000464 1733397583.126831 mapwrite 0x014a2d thru 0x0210bd (0xc691 bytes) 000465 1733397583.135778 write 0x026a8b thru 0x02b1e7 (0x475d bytes) 000466 1733397583.136465 write 0x005f86 thru 0x01427d (0xe2f8 bytes) 000467 1733397583.137082 fallocate 0x03a1a0 thru 0x03ffff (0x5e60 bytes) 000468 1733397583.138506 fallocate 0x02b01f thru 0x0373c3 (0xc3a5 bytes) 000469 1733397583.139933 fallocate 0x02aa26 thru 0x0358b4 (0xae8f bytes) 000470 1733397583.141363 trunc from 0x040000 to 0x020374 (0x1fc8b bytes) 000471 1733397583.144998 trunc from 0x020375 to 0x026c9a (0x6926 bytes) 000472 1733397583.148027 mapread 0x01748a thru 0x01c9f6 (0x556d bytes) 000474 1733397583.148900 write 0x03ac56 thru 0x03ffff (0x53aa bytes) 000475 1733397583.149600 write 0x0078df thru 0x015a70 (0xe192 bytes) 000476 1733397583.150402 trunc from 0x040000 to 0x032ed9 (0xd126 bytes) 000477 1733397583.154131 read 0x0237cd thru 0x032ed9 (0xf70d bytes) 000478 1733397583.156564 mapwrite 0x005bcb thru 0x00eb1e (0x8f54 bytes) 000479 1733397583.163548 mapwrite 0x037a1f thru 0x038c19 (0x11fb bytes) 000480 1733397583.174283 fallocate 0x02dee1 thru 0x032430 (0x4550 bytes) 000481 1733397583.175792 write 0x027ffb thru 0x02b665 (0x366b bytes) 000483 1733397583.176648 read 0x03389e thru 0x038c19 (0x537c bytes) 000484 1733397583.176967 punch from 0x009bb1 to 0x018028 (0xe478 bytes) 000486 1733397583.178737 mapwrite 0x02b822 thru 0x039c13 (0xe3f2 bytes) 000487 1733397583.190416 fallocate 0x01e41d thru 0x02a00c (0xbbf0 bytes) 000489 1733397583.191976 punch from 0x02a972 to 0x03280a (0x7e99 bytes) 000491 1733397583.193407 read 0x027cf5 thru 0x02d744 (0x5a50 bytes) 000493 1733397583.195951 mapread 0x030734 thru 0x039c13 (0x94e0 bytes) 000494 1733397583.196568 fallocate 0x005c7f thru 0x015b47 (0xfec9 bytes) 000497 1733397583.198238 punch from 0x02b968 to 0x034ea7 (0x9540 bytes) 000498 1733397583.199446 write 0x02da72 thru 0x030c0d (0x319c bytes) 000500 1733397583.202016 mapwrite 0x036d94 thru 0x03ffff (0x926c bytes) 000501 1733397583.211699 punch from 0x03e2f6 to 0x03ffff (0x1d0a bytes) 000503 1733397583.213001 punch from 0x011c66 to 0x01cb4c (0xaee7 bytes) 000505 1733397583.214655 trunc from 0x040000 to 0x0219e0 (0x1e61f bytes) 000506 1733397583.217752 fallocate 0x03c1fb thru 0x03ffff (0x3e05 bytes) 000507 1733397583.219005 punch from 0x002916 to 0x010776 (0xde61 bytes) 000509 1733397583.220711 read 0x032abb thru 0x037225 (0x476b bytes) 000510 1733397583.223313 write 0x0398d1 thru 0x03ffff (0x672f bytes) 000511 1733397583.223822 read 0x02cce8 thru 0x03816f (0xb488 bytes) 000512 1733397583.224058 mapread 0x007bbf thru 0x012512 (0xa954 bytes) 000513 1733397583.224459 write 0x02d842 thru 0x02e9eb (0x11aa bytes) 000514 1733397583.224792 trunc from 0x040000 to 0x008cfa (0x37305 bytes) 000515 1733397583.228519 mapread 0x000f8b thru 0x006abe (0x5b34 bytes) 000516 1733397583.229027 trunc from 0x008cfb to 0x02382b (0x1ab31 bytes) 000517 1733397583.232102 write 0x039146 thru 0x03a1eb (0x10a6 bytes) 000520 1733397583.233332 punch from 0x01837d to 0x026ea5 (0xeb29 bytes) 000521 1733397583.234775 write 0x018114 thru 0x01a7cb (0x26b8 bytes) 000522 1733397583.237605 read 0x0020ed thru 0x00f4df (0xd3f3 bytes) 000523 1733397583.237988 punch from 0x00187c to 0x00ac42 (0x93c7 bytes) 000524 1733397583.239439 read 0x02fc76 thru 0x03a1eb (0xa576 bytes) 000525 1733397583.239865 fallocate 0x0395a0 thru 0x03c1a7 (0x2c08 bytes) 000526 1733397583.241399 read 0x009ce7 thru 0x014d61 (0xb07b bytes) 000528 1733397583.243844 mapwrite 0x03b14b thru 0x03ffff (0x4eb5 bytes) 000529 1733397583.253243 fallocate 0x016232 thru 0x024d94 (0xeb63 bytes) 000530 1733397583.254883 fallocate 0x00046c thru 0x008fd7 (0x8b6c bytes) 000531 1733397583.256396 fallocate 0x010f9f thru 0x01cfe8 (0xc04a bytes) 000533 1733397583.258123 trunc from 0x040000 to 0x00b481 (0x34b7e bytes) 000534 1733397583.261875 punch from 0x00344e to 0x00b481 (0x8034 bytes) 000537 1733397583.263566 trunc from 0x00b482 to 0x012ced (0x786c bytes) 000538 1733397583.266784 mapread 0x00e38a thru 0x012ced (0x4964 bytes) 000539 1733397583.272196 trunc from 0x012cee to 0x0191e2 (0x64f5 bytes) 000540 1733397583.275580 write 0x038089 thru 0x03a869 (0x27e1 bytes) 000541 1733397583.276188 fallocate 0x039263 thru 0x03fb62 (0x6900 bytes) 000542 1733397583.277538 read 0x01300b thru 0x021485 (0xe47b bytes) 000543 1733397583.280907 write 0x02b84b thru 0x02c543 (0xcf9 bytes) 000544 1733397583.281500 mapwrite 0x038526 thru 0x03fd3c (0x7817 bytes) 000545 1733397583.292059 punch from 0x007f9a to 0x012906 (0xa96d bytes) 000550 1733397583.294661 punch from 0x01be5c to 0x024885 (0x8a2a bytes) 000551 1733397583.296141 mapwrite 0x034915 thru 0x038770 (0x3e5c bytes) 000555 1733397583.302658 mapread 0x0244cc thru 0x030261 (0xbd96 bytes) 000556 1733397583.304443 mapwrite 0x0054e1 thru 0x00d27c (0x7d9c bytes) 000559 1733397583.317359 mapwrite 0x02263e thru 0x02ad53 (0x8716 bytes) 000560 1733397583.325447 read 0x005936 thru 0x014e00 (0xf4cb bytes) 000561 1733397583.327812 mapwrite 0x00f776 thru 0x015b3f (0x63ca bytes) 000562 1733397583.332149 read 0x007d9a thru 0x00c870 (0x4ad7 bytes) 000563 1733397583.332565 mapwrite 0x001f15 thru 0x00cec2 (0xafae bytes) 000566 1733397583.338718 mapwrite 0x0039ed thru 0x012e79 (0xf48d bytes) 000567 1733397583.345157 fallocate 0x003ac7 thru 0x0066aa (0x2be4 bytes) 000569 1733397583.346655 write 0x019987 thru 0x021f38 (0x85b2 bytes) 000571 1733397583.347625 punch from 0x03c797 to 0x03fd3c (0x35a6 bytes) 000572 1733397583.348761 fallocate 0x01a97a thru 0x027aa6 (0xd12d bytes) 000573 1733397583.350000 write 0x002358 thru 0x007614 (0x52bd bytes) 000574 1733397583.350539 write 0x0110c2 thru 0x01adb1 (0x9cf0 bytes) 000575 1733397583.351095 punch from 0x03a195 to 0x03fd3c (0x5ba8 bytes) 000576 1733397583.352309 trunc from 0x03fd3d to 0x03e794 (0x15a8 bytes) 000578 1733397583.355998 write 0x00f7e2 thru 0x019713 (0x9f32 bytes) 000579 1733397583.356582 write 0x008965 thru 0x0109e1 (0x807d bytes) 000580 1733397583.357117 punch from 0x0038f7 to 0x0099a4 (0x60ae bytes) 000581 1733397583.361032 mapread 0x02ad2f thru 0x035e64 (0xb136 bytes) 000582 1733397583.361661 fallocate 0x02b21b thru 0x032c34 (0x7a1a bytes) 000583 1733397583.363019 punch from 0x030c0c to 0x035cf3 (0x50e8 bytes) 000584 1733397583.364246 read 0x000814 thru 0x004728 (0x3f15 bytes) 000585 1733397583.366914 punch from 0x021e78 to 0x02e96a (0xcaf3 bytes) 000589 1733397583.369541 trunc from 0x03e795 to 0x0039a5 (0x3adef bytes) 000590 1733397583.373274 fallocate 0x039997 thru 0x03ffff (0x6669 bytes) 000591 1733397583.374767 mapwrite 0x03e911 thru 0x03ffff (0x16ef bytes) 000592 1733397583.383312 mapread 0x01c164 thru 0x0217d2 (0x566f bytes) 000593 1733397583.389547 punch from 0x026036 to 0x026320 (0x2eb bytes) 000594 1733397583.390693 read 0x006d10 thru 0x0097f7 (0x2ae8 bytes) 000595 1733397583.393648 mapwrite 0x03dd4e thru 0x03ffff (0x22b2 bytes) 000596 1733397583.397737 fallocate 0x016239 thru 0x020487 (0xa24f bytes) 000597 1733397583.399197 trunc from 0x040000 to 0x03eada (0x1525 bytes) 000599 1733397583.402841 trunc from 0x03eadb to 0x01cb55 (0x21f85 bytes) 000600 1733397583.407160 MIRROR_EXTEND to 2 mirrors 000601 1733397583.486874 read 0x00a756 thru 0x0165d5 (0xbe80 bytes) 000603 1733397583.489031 read 0x003954 thru 0x00e9bd (0xb06a bytes) 000604 1733397583.492319 MIRROR_RESYNC 2 mirrors 000605 1733397583.525705 write 0x02261e thru 0x030077 (0xda5a bytes) 000606 1733397583.535317 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa25:0x0]: Invalid argument (22) 000607 1733397583.566384 MIRROR_EXTEND to 2 mirrors 000610 1733397583.657868 read 0x01a730 thru 0x02164d (0x6f1e bytes) 000611 1733397583.659681 fallocate 0x0061bf thru 0x0113e9 (0xb22b bytes) 000612 1733397583.666674 write 0x009ec1 thru 0x0185c5 (0xe705 bytes) 000614 1733397583.670360 trunc from 0x030078 to 0x0282be (0x7db9 bytes) 000615 1733397583.678033 mapwrite 0x0177d1 thru 0x01e87d (0x70ad bytes) 000616 1733397583.685884 punch from 0x0081d8 to 0x016b5c (0xe985 bytes) 000617 1733397583.687325 trunc from 0x0282bf to 0x03138e (0x90d0 bytes) 000618 1733397583.690739 punch from 0x01cdae to 0x0226d2 (0x5925 bytes) 000619 1733397583.692191 write 0x01c414 thru 0x01e26c (0x1e59 bytes) 000620 1733397583.694195 trunc from 0x03138f to 0x038248 (0x6eba bytes) 000621 1733397583.697327 write 0x004d33 thru 0x00f9b9 (0xac87 bytes) 000622 1733397583.699796 read 0x007b44 thru 0x008e66 (0x1323 bytes) 000623 1733397583.700128 mapread 0x002ead thru 0x00c0fe (0x9252 bytes) 000624 1733397583.700527 read 0x03680b thru 0x038248 (0x1a3e bytes) 000625 1733397583.703027 trunc from 0x038249 to 0x01e16b (0x1a0dd bytes) 000627 1733397583.706637 punch from 0x00007c to 0x00f692 (0xf617 bytes) 000628 1733397583.710321 trunc from 0x01e16c to 0x027c0e (0x9aa3 bytes) 000629 1733397583.713905 mapwrite 0x02ab63 thru 0x02e76c (0x3c0a bytes) 000630 1733397583.725901 trunc from 0x02e76d to 0x008dd8 (0x25994 bytes) 000631 1733397583.728803 punch from 0x004599 to 0x008dd8 (0x4840 bytes) 000632 1733397583.729850 write 0x03b0c9 thru 0x03ffff (0x4f37 bytes) 000633 1733397583.730404 write 0x014773 thru 0x01c15e (0x79ec bytes) 000635 1733397583.733004 fallocate 0x022da8 thru 0x0328d9 (0xfb32 bytes) 000636 1733397583.734330 fallocate 0x00d41e thru 0x01a229 (0xce0c bytes) 000637 1733397583.735638 mapread 0x0201ad thru 0x020f59 (0xdad bytes) 000638 1733397583.736991 trunc from 0x040000 to 0x0130cc (0x2cf33 bytes) 000639 1733397583.739872 write 0x000919 thru 0x009170 (0x8858 bytes) 000641 1733397583.742284 trunc from 0x0130cd to 0x01b994 (0x88c8 bytes) 000642 1733397583.744962 mapread 0x0123c2 thru 0x01708b (0x4cca bytes) 000643 1733397583.750335 fallocate 0x00eefd thru 0x01c4e7 (0xd5eb bytes) 000644 1733397583.751413 trunc from 0x01b995 to 0x0199bb (0x1fd9 bytes) 000645 1733397583.754098 trunc from 0x0199bc to 0x02baa8 (0x120ed bytes) 000647 1733397583.757031 read 0x01acc7 thru 0x01fa55 (0x4d8f bytes) 000648 1733397583.759214 mapwrite 0x031485 thru 0x03ed77 (0xd8f3 bytes) 000649 1733397583.769691 mapread 0x02816e thru 0x02c459 (0x42ec bytes) 000650 1733397583.770049 trunc from 0x03ed78 to 0x01fa38 (0x1f33f bytes) 000652 1733397583.773021 write 0x02226c thru 0x02d956 (0xb6eb bytes) 000653 1733397583.773621 punch from 0x01837d to 0x023f58 (0xbbdc bytes) 000654 1733397583.776429 punch from 0x00e790 to 0x01809d (0x990e bytes) 000655 1733397583.778437 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa25:0x0]: Invalid argument (22) 000656 1733397583.800573 read 0x00a162 thru 0x01087d (0x671c bytes) 000658 1733397583.802881 write 0x000ee8 thru 0x00ad07 (0x9e20 bytes) 000659 1733397583.805189 write 0x034d41 thru 0x03d502 (0x87c2 bytes) 000661 1733397583.806277 write 0x02296d thru 0x02b481 (0x8b15 bytes) 000664 1733397583.808856 mapread 0x025dc4 thru 0x029de7 (0x4024 bytes) 000666 1733397583.810528 MIRROR_EXTEND to 2 mirrors 000667 1733397583.905352 trunc from 0x03d503 to 0x028a63 (0x14a9f bytes) 000669 1733397583.916721 mapwrite 0x008349 thru 0x0175ca (0xf282 bytes) 000670 1733397583.928271 write 0x0061cf thru 0x0118db (0xb70d bytes) 000671 1733397583.929110 mapread 0x00b076 thru 0x0123c4 (0x734f bytes) 000672 1733397583.929501 fallocate 0x02e3d9 thru 0x038d1f (0xa947 bytes) 000675 1733397583.931376 fallocate 0x03f154 thru 0x03ffff (0xeac bytes) 000676 1733397583.932701 write 0x0127a0 thru 0x0135ad (0xe0e bytes) 000677 1733397583.933311 mapread 0x00eee0 thru 0x00fc52 (0xd73 bytes) 000680 1733397583.934427 trunc from 0x038d20 to 0x0285e9 (0x10736 bytes) 000681 1733397583.938351 trunc from 0x0285ea to 0x01647f (0x1216a bytes) 000683 1733397583.941469 trunc from 0x016480 to 0x03c0ae (0x25c2f bytes) 000684 1733397583.944374 mapread 0x00f15c thru 0x00f7a0 (0x645 bytes) 000685 1733397583.944779 fallocate 0x00ac2c thru 0x00dd96 (0x316b bytes) 000687 1733397583.946675 write 0x03cd96 thru 0x03ffff (0x326a bytes) 000688 1733397583.949643 MIRROR_EXTEND to 3 mirrors 000689 1733397584.049452 mapread 0x012552 thru 0x0194de (0x6f8d bytes) 000690 1733397584.055657 mapread 0x01c519 thru 0x0290c8 (0xcbb0 bytes) 000691 1733397584.061226 mapread 0x039a9c thru 0x03b72f (0x1c94 bytes) 000692 1733397584.065815 punch from 0x012152 to 0x01c848 (0xa6f7 bytes) 000693 1733397584.074085 fallocate 0x025df4 thru 0x0262c4 (0x4d1 bytes) 000694 1733397584.075436 trunc from 0x040000 to 0x031fa5 (0xe05a bytes) 000696 1733397584.080565 write 0x0265e5 thru 0x03415a (0xdb76 bytes) 000697 1733397584.083450 mapread 0x007413 thru 0x013a6f (0xc65d bytes) 000698 1733397584.087207 trunc from 0x03415b to 0x01cb9d (0x175bd bytes) 000699 1733397584.090429 mapwrite 0x00ae05 thru 0x012b3e (0x7d3a bytes) 000700 1733397584.095342 mapread 0x019e56 thru 0x01cb9d (0x2d48 bytes) 000702 1733397584.097211 fallocate 0x018930 thru 0x02794d (0xf01e bytes) 000703 1733397584.098557 trunc from 0x02794e to 0x019e6e (0xdadf bytes) 000704 1733397584.101813 trunc from 0x019e6f to 0x022194 (0x8326 bytes) 000705 1733397584.104942 read 0x004cc4 thru 0x00f41d (0xa75a bytes) 000706 1733397584.106343 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa25:0x0]: Invalid argument (22) 000707 1733397584.147878 read 0x010fa4 thru 0x018c77 (0x7cd4 bytes) 000709 1733397584.151961 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa25:0x0]: Invalid argument (22) 000712 1733397584.179510 read 0x01d312 thru 0x022194 (0x4e83 bytes) 000713 1733397584.181687 write 0x00421c thru 0x00ccb0 (0x8a95 bytes) 000714 1733397584.184862 mapread 0x007b07 thru 0x014874 (0xcd6e bytes) 000715 1733397584.187127 read 0x01431c thru 0x01f164 (0xae49 bytes) 000716 1733397584.187550 read 0x018f4f thru 0x02170f (0x87c1 bytes) 000718 1733397584.188041 mapread 0x008a6b thru 0x010c94 (0x822a bytes) 000719 1733397584.188437 mapread 0x012635 thru 0x01e8b4 (0xc280 bytes) 000720 1733397584.188948 mapwrite 0x01d144 thru 0x027180 (0xa03d bytes) 000721 1733397584.200592 punch from 0x00be9e to 0x015499 (0x95fc bytes) 000722 1733397584.202417 trunc from 0x027181 to 0x00bb69 (0x1b617 bytes) 000723 1733397584.206338 trunc from 0x00bb6a to 0x0160df (0xa576 bytes) 000724 1733397584.209346 read 0x010051 thru 0x0160df (0x608f bytes) 000725 1733397584.211762 punch from 0x00b591 to 0x0160df (0xab4f bytes) 000726 1733397584.213352 punch from 0x00ae76 to 0x00be2a (0xfb5 bytes) 000727 1733397584.214886 mapwrite 0x02094d thru 0x02b9ac (0xb060 bytes) 000728 1733397584.226428 trunc from 0x02b9ad to 0x01d799 (0xe213 bytes) 000729 1733397584.229460 write 0x011171 thru 0x0112aa (0x13a bytes) 000730 1733397584.230853 punch from 0x00a50d to 0x01244a (0x7f3e bytes) 000732 1733397584.233542 trunc from 0x01d79a to 0x0110f7 (0xc6a2 bytes) 000733 1733397584.236776 mapread 0x00516a thru 0x00d12c (0x7fc3 bytes) 000735 1733397584.241611 mapwrite 0x039c1d thru 0x03daf2 (0x3ed6 bytes) 000736 1733397584.252841 mapread 0x03c3f1 thru 0x03daf2 (0x1702 bytes) 000737 1733397584.253375 trunc from 0x03daf3 to 0x024ade (0x19014 bytes) 000738 1733397584.257086 fallocate 0x027fb9 thru 0x02de37 (0x5e7f bytes) 000739 1733397584.258634 read 0x000bf6 thru 0x00b9fa (0xae05 bytes) 000741 1733397584.259162 trunc from 0x024adf to 0x0243fd (0x6e1 bytes) 000742 1733397584.262230 trunc from 0x0243fe to 0x00a2e4 (0x1a119 bytes) 000743 1733397584.265037 write 0x00974c thru 0x015592 (0xbe47 bytes) 000744 1733397584.265687 trunc from 0x015593 to 0x0177c1 (0x222f bytes) 000747 1733397584.269008 mapread 0x002535 thru 0x00d4be (0xaf8a bytes) 000748 1733397584.269448 mapread 0x003b71 thru 0x0094ec (0x597c bytes) 000749 1733397584.269851 read 0x008be4 thru 0x00d5c9 (0x49e6 bytes) 000750 1733397584.270111 mapwrite 0x0081c4 thru 0x012e0a (0xac47 bytes) 000751 1733397584.275682 read 0x009514 thru 0x009879 (0x366 bytes) 000754 1733397584.276531 write 0x032850 thru 0x0398d7 (0x7088 bytes) 000755 1733397584.277119 mapread 0x002718 thru 0x00c5b0 (0x9e99 bytes) 000756 1733397584.277629 trunc from 0x0398d8 to 0x027735 (0x121a2 bytes) 000757 1733397584.280813 mapread 0x007d79 thru 0x013b74 (0xbdfc bytes) 000758 1733397584.281208 read 0x003782 thru 0x008e5d (0x56dc bytes) 000760 1733397584.282804 MIRROR_EXTEND to 2 mirrors 000761 1733397584.364098 punch from 0x001d09 to 0x0023a8 (0x6a0 bytes) 000762 1733397584.371123 write 0x020a3c thru 0x0274ed (0x6ab2 bytes) 000763 1733397584.374152 fallocate 0x00fe93 thru 0x016569 (0x66d7 bytes) 000766 1733397584.375882 mapread 0x018a84 thru 0x02312b (0xa6a8 bytes) 000768 1733397584.377966 read 0x0200dd thru 0x023660 (0x3584 bytes) 000769 1733397584.378235 trunc from 0x027736 to 0x00bce3 (0x1ba52 bytes) 000770 1733397584.385039 punch from 0x00869c to 0x00bce3 (0x3648 bytes) 000771 1733397584.386172 write 0x0159ad thru 0x01dfd8 (0x862c bytes) 000773 1733397584.387064 punch from 0x00fa4c to 0x01dfd8 (0xe58d bytes) 000774 1733397584.389886 mapread 0x010d67 thru 0x01a6a1 (0x993b bytes) 000775 1733397584.394257 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa25:0x0]: Invalid argument (22) 000776 1733397584.418903 fallocate 0x02d8da thru 0x031459 (0x3b80 bytes) 000777 1733397584.420356 fallocate 0x016064 thru 0x021faa (0xbf47 bytes) 000778 1733397584.421575 write 0x019230 thru 0x028724 (0xf4f5 bytes) 000781 1733397584.425409 mapread 0x00d66a thru 0x010f10 (0x38a7 bytes) 000782 1733397584.427690 mapwrite 0x03c709 thru 0x03ca9c (0x394 bytes) 000783 1733397584.436166 write 0x01399b thru 0x015435 (0x1a9b bytes) 000784 1733397584.436649 trunc from 0x03ca9d to 0x02ecaa (0xddf2 bytes) 000785 1733397584.439496 mapwrite 0x02793c thru 0x037481 (0xfb46 bytes) 000787 1733397584.451402 read 0x01afcb thru 0x027f19 (0xcf4f bytes) 000789 1733397584.451947 mapwrite 0x023020 thru 0x028f75 (0x5f56 bytes) 000790 1733397584.455398 mapread 0x026c67 thru 0x02e869 (0x7c03 bytes) 000791 1733397584.455896 fallocate 0x00e140 thru 0x01b3e8 (0xd2a9 bytes) 000793 1733397584.457316 write 0x00bb03 thru 0x0135df (0x7add bytes) 000794 1733397584.457974 trunc from 0x037482 to 0x0259df (0x11aa2 bytes) 000795 1733397584.460909 mapwrite 0x020f36 thru 0x02d375 (0xc440 bytes) 000796 1733397584.471276 read 0x017fbf thru 0x01fd81 (0x7dc3 bytes) 000798 1733397584.472044 trunc from 0x02d376 to 0x0141be (0x191b7 bytes) 000801 1733397584.476181 mapread 0x00f9b9 thru 0x0141be (0x4806 bytes) 000802 1733397584.476676 punch from 0x01363d to 0x0141be (0xb82 bytes) 000804 1733397584.478328 write 0x014531 thru 0x01b586 (0x7056 bytes) 000805 1733397584.480620 trunc from 0x01b587 to 0x005218 (0x1636e bytes) 000806 1733397584.483738 fallocate 0x00c52e thru 0x0168ed (0xa3c0 bytes) 000807 1733397584.486269 MIRROR_EXTEND to 2 mirrors 000808 1733397584.567290 punch from 0x00395c to 0x004768 (0xe0d bytes) 000809 1733397584.574809 fallocate 0x01a936 thru 0x01d65e (0x2d29 bytes) 000810 1733397584.576124 punch from 0x0149d1 to 0x01d65e (0x8c8e bytes) 000811 1733397584.578210 MIRROR_EXTEND to 3 mirrors 000813 1733397584.658554 read 0x00bcf5 thru 0x0134ba (0x77c6 bytes) 000814 1733397584.663027 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa25:0x0]: Invalid argument (22) 000815 1733397584.701010 mapread 0x003c62 thru 0x009d18 (0x60b7 bytes) 000817 1733397584.705166 write 0x035c1f thru 0x038da1 (0x3183 bytes) 000818 1733397584.709712 MIRROR_RESYNC 2 mirrors 000819 1733397584.795298 MIRROR_RESYNC 2 mirrors 000820 1733397584.828128 MIRROR_RESYNC 2 mirrors 000821 1733397584.857342 mapwrite 0x0208fb thru 0x025749 (0x4e4f bytes) 000822 1733397584.882943 mapread 0x02dd91 thru 0x032386 (0x45f6 bytes) 000824 1733397584.885699 write 0x030a33 thru 0x0320cd (0x169b bytes) 000825 1733397584.886142 punch from 0x029bb9 to 0x0376f6 (0xdb3e bytes) 000827 1733397584.889042 trunc from 0x038da2 to 0x025ae9 (0x132b8 bytes) 000828 1733397584.896023 write 0x028568 thru 0x032e79 (0xa912 bytes) 000829 1733397584.896766 write 0x0145f4 thru 0x018924 (0x4331 bytes) 000831 1733397584.898951 mapwrite 0x004b01 thru 0x00f6bc (0xabbc bytes) 000832 1733397584.906079 mapwrite 0x03e2b4 thru 0x03ffff (0x1d4c bytes) 000833 1733397584.913298 write 0x021309 thru 0x02bdab (0xaaa3 bytes) 000834 1733397584.914792 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa25:0x0]: Invalid argument (22) 000835 1733397584.937617 mapread 0x00b4b2 thru 0x00f0d2 (0x3c21 bytes) 000836 1733397584.941128 fallocate 0x02ccc3 thru 0x037c22 (0xaf60 bytes) 000837 1733397584.942194 read 0x02ce99 thru 0x033a1a (0x6b82 bytes) 000838 1733397584.944444 MIRROR_EXTEND to 2 mirrors 000839 1733397585.042881 mapwrite 0x009cce thru 0x00dd40 (0x4073 bytes) 000842 1733397585.067338 punch from 0x02959c to 0x033669 (0xa0ce bytes) 000843 1733397585.068419 trunc from 0x040000 to 0x028cce (0x17331 bytes) 000844 1733397585.072593 write 0x03e996 thru 0x03ffff (0x166a bytes) 000845 1733397585.073145 mapwrite 0x00f62f thru 0x014aab (0x547d bytes) 000846 1733397585.079910 MIRROR_EXTEND to 3 mirrors 000847 1733397585.169941 punch from 0x01f04c to 0x02af4b (0xbf00 bytes) 000848 1733397585.176834 trunc from 0x040000 to 0x03d668 (0x2997 bytes) 000849 1733397585.184411 MIRROR_EXTEND to 4 mirrors 000850 1733397585.266011 punch from 0x01c043 to 0x0212ef (0x52ad bytes) 000851 1733397585.273273 write 0x033e08 thru 0x035b1b (0x1d14 bytes) 000853 1733397585.276593 fallocate 0x0398db thru 0x03ffff (0x6725 bytes) 000854 1733397585.277769 mapwrite 0x0356cb thru 0x03b0bb (0x59f1 bytes) 000855 1733397585.284312 trunc from 0x040000 to 0x02e5e1 (0x11a1e bytes) 000856 1733397585.291091 write 0x006fe7 thru 0x009a91 (0x2aab bytes) 000857 1733397585.294044 MIRROR_RESYNC 4 mirrors 000859 1733397585.390996 trunc from 0x02e5e2 to 0x0399d6 (0xb3f5 bytes) 000861 1733397585.401665 MIRROR_RESYNC 4 mirrors 000862 1733397585.484399 punch from 0x0219c2 to 0x028a25 (0x7064 bytes) 000863 1733397585.490566 trunc from 0x0399d7 to 0x0089f3 (0x30fe3 bytes) 000864 1733397585.496775 punch from 0x007b8c to 0x0089f3 (0xe68 bytes) 000865 1733397585.497968 read 0x00263c thru 0x0089f3 (0x63b8 bytes) 000866 1733397585.499763 fallocate 0x02e7cd thru 0x032476 (0x3caa bytes) 000867 1733397585.501176 trunc from 0x0089f4 to 0x037038 (0x2e645 bytes) 000868 1733397585.504186 fallocate 0x024825 thru 0x031af4 (0xd2d0 bytes) 000869 1733397585.505610 trunc from 0x037039 to 0x02ac5b (0xc3dd bytes) 000870 1733397585.508748 trunc from 0x02ac5c to 0x00b19a (0x1fac1 bytes) 000871 1733397585.511969 trunc from 0x00b19b to 0x034522 (0x29388 bytes) 000872 1733397585.514808 read 0x015edc thru 0x01b523 (0x5648 bytes) 000873 1733397585.517329 trunc from 0x034523 to 0x016d91 (0x1d791 bytes) 000874 1733397585.520191 mapwrite 0x02e57c thru 0x033cdb (0x5760 bytes) 000875 1733397585.531861 fallocate 0x01fd12 thru 0x02f227 (0xf516 bytes) 000876 1733397585.532984 punch from 0x006712 to 0x0095f5 (0x2ee4 bytes) 000877 1733397585.534224 mapread 0x00770f thru 0x014ebc (0xd7ae bytes) 000878 1733397585.537232 mapwrite 0x00bae4 thru 0x0163d3 (0xa8f0 bytes) 000879 1733397585.541695 write 0x025d98 thru 0x02cbac (0x6e15 bytes) 000880 1733397585.543911 write 0x017d6a thru 0x01ccdd (0x4f74 bytes) 000881 1733397585.546122 mapwrite 0x03f24f thru 0x03ffff (0xdb1 bytes) 000882 1733397585.554734 mapread 0x01a540 thru 0x01f9c8 (0x5489 bytes) 000883 1733397585.557878 trunc from 0x040000 to 0x0128de (0x2d721 bytes) 000884 1733397585.561094 fallocate 0x01cacc thru 0x02b1bd (0xe6f2 bytes) 000885 1733397585.562370 fallocate 0x01973f thru 0x024eea (0xb7ac bytes) 000886 1733397585.563537 punch from 0x00d9c5 to 0x01131e (0x395a bytes) 000887 1733397585.564712 fallocate 0x035916 thru 0x03aa63 (0x514e bytes) 000888 1733397585.566017 fallocate 0x02aef0 thru 0x039e06 (0xef17 bytes) 000889 1733397585.567391 read 0x01f85f thru 0x02342a (0x3bcc bytes) 000893 1733397585.570452 mapread 0x022371 thru 0x02305b (0xceb bytes) 000895 1733397585.570982 punch from 0x024855 to 0x024eea (0x696 bytes) 000896 1733397585.573009 fallocate 0x009f03 thru 0x01537d (0xb47b bytes) 000897 1733397585.574389 read 0x021180 thru 0x024eea (0x3d6b bytes) 000898 1733397585.575787 fallocate 0x0392ea thru 0x03ab50 (0x1867 bytes) 000899 1733397585.577058 write 0x02a35f thru 0x032e51 (0x8af3 bytes) 000900 1733397585.578719 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa25:0x0]: Invalid argument (22) 000904 1733397585.615176 write 0x03212e thru 0x038d0e (0x6be1 bytes) 000907 1733397585.617946 mapwrite 0x0000f4 thru 0x006251 (0x615e bytes) 000908 1733397585.625409 write 0x01e20e thru 0x0243b6 (0x61a9 bytes) 000910 1733397585.627646 punch from 0x005f69 to 0x00aa07 (0x4a9f bytes) 000911 1733397585.628823 punch from 0x02f972 to 0x038d0e (0x939d bytes) 000912 1733397585.629841 fallocate 0x0202d9 thru 0x02a29d (0x9fc5 bytes) 000913 1733397585.631923 MIRROR_EXTEND to 4 mirrors 000914 1733397585.723418 read 0x00b985 thru 0x01398f (0x800b bytes) 000916 1733397585.725379 write 0x02e708 thru 0x032d89 (0x4682 bytes) 000917 1733397585.733187 mapwrite 0x017bd7 thru 0x022653 (0xaa7d bytes) 000918 1733397585.742140 read 0x00d6d9 thru 0x01c13c (0xea64 bytes) 000919 1733397585.744220 trunc from 0x038d0f to 0x0211ac (0x17b62 bytes) 000920 1733397585.751236 fallocate 0x030656 thru 0x03e06e (0xda19 bytes) 000921 1733397585.752561 mapread 0x00caa4 thru 0x015cde (0x923b bytes) 000922 1733397585.753167 punch from 0x0029e2 to 0x0122a9 (0xf8c8 bytes) 000923 1733397585.754769 punch from 0x00adec to 0x016497 (0xb6ac bytes) 000925 1733397585.756211 trunc from 0x0211ad to 0x02c4dd (0xb331 bytes) 000926 1733397585.759433 write 0x020ae6 thru 0x02165a (0xb75 bytes) 000928 1733397585.760180 trunc from 0x02c4de to 0x01fe4c (0xc691 bytes) 000929 1733397585.763085 read 0x007e69 thru 0x0126db (0xa873 bytes) 000930 1733397585.766152 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa25:0x0]: Invalid argument (22) 000931 1733397585.802152 mapwrite 0x00ac81 thru 0x00d054 (0x23d4 bytes) 000932 1733397585.810566 MIRROR_RESYNC 3 mirrors 000934 1733397585.905632 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa25:0x0]: Invalid argument (22) 000935 1733397585.944762 fallocate 0x0131b0 thru 0x022dd7 (0xfc28 bytes) 000936 1733397585.951092 write 0x0118b8 thru 0x01e65e (0xcda7 bytes) 000937 1733397585.954889 MIRROR_RESYNC 2 mirrors 000938 1733397586.038251 mapread 0x01772b thru 0x01b377 (0x3c4d bytes) 000940 1733397586.042561 trunc from 0x022dd8 to 0x03699b (0x13bc4 bytes) 000941 1733397586.050253 trunc from 0x03699c to 0x03fc74 (0x92d9 bytes) 000943 1733397586.054447 fallocate 0x02a52f thru 0x02b685 (0x1157 bytes) 000944 1733397586.055786 read 0x0293c5 thru 0x035b26 (0xc762 bytes) 000945 1733397586.057898 trunc from 0x03fc75 to 0x017758 (0x2851c bytes) 000946 1733397586.060992 fallocate 0x0320b7 thru 0x03f0ab (0xcff5 bytes) 000947 1733397586.062313 trunc from 0x03f0ac to 0x03a24d (0x4e5e bytes) 000948 1733397586.066921 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa25:0x0]: Invalid argument (22) 000949 1733397586.088596 mapread 0x02b6ee thru 0x032a6d (0x7380 bytes) 000951 1733397586.092863 mapwrite 0x02a49e thru 0x02e4b7 (0x401a bytes) 000952 1733397586.097094 mapread 0x02f7ce thru 0x0340a1 (0x48d4 bytes) 000953 1733397586.097527 read 0x0161ee thru 0x01b79e (0x55b1 bytes) 000954 1733397586.099104 fallocate 0x0169d2 thru 0x01c3d4 (0x5a03 bytes) 000955 1733397586.100297 read 0x00c9af thru 0x01c4ed (0xfb3f bytes) 000956 1733397586.102518 mapwrite 0x012b32 thru 0x0198c4 (0x6d93 bytes) 000957 1733397586.106595 read 0x00d7bf thru 0x016913 (0x9155 bytes) 000958 1733397586.106901 fallocate 0x02eb2c thru 0x030ed0 (0x23a5 bytes) 000960 1733397586.108626 read 0x01fbdd thru 0x020b9e (0xfc2 bytes) 000961 1733397586.108846 write 0x01ba7a thru 0x01e490 (0x2a17 bytes) 000962 1733397586.110155 write 0x0151c3 thru 0x01b50f (0x634d bytes) 000963 1733397586.110743 punch from 0x026b91 to 0x02e11b (0x758b bytes) 000964 1733397586.113031 MIRROR_EXTEND to 2 mirrors 000965 1733397586.203996 MIRROR_EXTEND to 3 mirrors 000966 1733397586.297496 mapread 0x02e3d0 thru 0x0339d9 (0x560a bytes) 000967 1733397586.304251 read 0x028cce thru 0x030929 (0x7c5c bytes) 000968 1733397586.306686 mapread 0x002c18 thru 0x01137d (0xe766 bytes) 000969 1733397586.312559 read 0x020037 thru 0x02cfad (0xcf77 bytes) 000970 1733397586.315892 punch from 0x024929 to 0x032314 (0xd9ec bytes) 000971 1733397586.323083 mapwrite 0x022cc0 thru 0x02f68b (0xc9cc bytes) 000972 1733397586.339923 punch from 0x00e3b3 to 0x012776 (0x43c4 bytes) 000973 1733397586.341179 mapwrite 0x018abe thru 0x026c63 (0xe1a6 bytes) 000974 1733397586.354530 write 0x03178e thru 0x032dfb (0x166e bytes) 000975 1733397586.357385 read 0x013a4f thru 0x0166c8 (0x2c7a bytes) 000976 1733397586.359646 mapread 0x032877 thru 0x03a24d (0x79d7 bytes) 000977 1733397586.360081 write 0x00aacc thru 0x00ce34 (0x2369 bytes) 000979 1733397586.360818 punch from 0x015737 to 0x022aaa (0xd374 bytes) 000981 1733397586.362157 trunc from 0x03a24e to 0x02d7f8 (0xca55 bytes) 000982 1733397586.369006 read 0x00ea17 thru 0x011e12 (0x33fc bytes) 000983 1733397586.369310 punch from 0x02c2a1 to 0x02d7f8 (0x1558 bytes) 000984 1733397586.370371 trunc from 0x02d7f9 to 0x01b051 (0x127a7 bytes) 000985 1733397586.373323 mapread 0x004320 thru 0x00b725 (0x7406 bytes) 000986 1733397586.373805 read 0x000c50 thru 0x00c3c9 (0xb77a bytes) 000988 1733397586.374309 mapread 0x003354 thru 0x00695e (0x360b bytes) 000989 1733397586.374718 fallocate 0x018f56 thru 0x02853c (0xf5e7 bytes) 000990 1733397586.375850 read 0x00c2a7 thru 0x00f485 (0x31df bytes) 000992 1733397586.376357 punch from 0x018b0f to 0x01b051 (0x2543 bytes) 000993 1733397586.378370 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa25:0x0]: Invalid argument (22) 000994 1733397586.416466 punch from 0x00e05c to 0x01652b (0x84d0 bytes) 000995 1733397586.417794 fallocate 0x03dffc thru 0x03ffff (0x2004 bytes) 000996 1733397586.419019 trunc from 0x01b052 to 0x033c4d (0x18bfc bytes) 000998 1733397586.423039 write 0x02bc55 thru 0x030c49 (0x4ff5 bytes) 000999 1733397586.426415 MIRROR_EXTEND to 3 mirrors 001000 1733397586.529406 read 0x013c38 thru 0x01ebda (0xafa3 bytes) All operations completed A-OK! PASS 70a (8s) == sanity-flr test 200a: stress test ===================== 06:19:47 (1733397587) Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre2 Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4723904 .. Extending file size to 1980512 .. done Extending file size to 5861472 .. Extending file size to 5890752 .. Extending file size to 6033856 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8500416 .. Extending file size to 7550752 .. Extending file size to 8746400 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8680512 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8949536 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9154944 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8958176 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 8010 8011 8013 8014 8017 192.168.203.110@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg310-client.virtnet /mnt/lustre2 (opts:) 192.168.203.110@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg310-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (63s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 06:20:50 (1733397650) Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre2 Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8395456 .. Extending file size to 4408128 .. Extending file size to 6276864 .. Extending file size to 7823264 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8489312 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7979680 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8893920 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8470560 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9100416 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9200960 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9055808 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 9274880 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9330464 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 11735 11736 11738 11740 11742 192.168.203.110@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg310-client.virtnet /mnt/lustre2 (opts:) 192.168.203.110@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg310-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 06:21:54 (1733397714) Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3952 1283736 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1912 1285776 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 21904 3585116 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 15852 3591168 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37756 7176284 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3952 1283736 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1912 1285776 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 21904 3585116 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 15852 3591168 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37756 7176284 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0317851 s, 66.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0514823 s, 40.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.110@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg310-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 06:21:59 (1733397719) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0234195 s, 89.5 MB/s PASS 202 (1s) == sanity-flr test 203: mirror file preserve mirror ID === 06:22:00 (1733397720) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa62:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x1068:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x1068:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x1069:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x1067:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 06:22:02 (1733397722) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0193085 s, 109 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (1s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 06:22:03 (1733397723) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0212993 s, 98.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x106a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x106b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1068:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x106a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x106b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1068:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1069:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 06:22:05 (1733397725) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0217311 s, 96.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2e:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 06:22:28 (1733397748) Waiting for MDT destroys to complete oleg310-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg310-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg310-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0119746 s, 87.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x30:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x30:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2d:0x0] } PASS 204d (30s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 06:22:58 (1733397778) Waiting for MDT destroys to complete oleg310-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg310-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg310-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x106a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x106b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0197532 s, 106 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x106a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x106b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x106c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x106a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x106b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x106c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (33s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 06:23:31 (1733397811) Waiting for MDT destroys to complete oleg310-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg310-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg310-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool oleg310-server: Pool lustre.test_204f created Adding targets to pool oleg310-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg310-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0207544 s, 101 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x31:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x31:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg310-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg310-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg310-server: Pool lustre.test_204f destroyed PASS 204f (48s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 06:24:19 (1733397859) PASS 205a (1s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 06:24:20 (1733397860) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 06:24:22 (1733397862) oleg310-server: Pool lustre.test_206 created Adding targets to pool oleg310-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg310-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg310-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg310-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg310-server: Pool lustre.test_206 destroyed PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 06:24:35 (1733397875) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0202447 s, 51.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00970066 s, 108 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xbf199ac9 Mirror 3: 0xbf199ac9 PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 06:24:37 (1733397877) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 06:24:38 (1733397878) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 06:24:39 (1733397879) Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.00877553 s, 0.5 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2530000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1d40000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1d40000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1d40000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xd80000 Read this (4 bytes): BBBB 192.168.203.110@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg310-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 06:24:52 (1733397892) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.0026334 s, 3.1 MB/s the very first read read with warm cache PASS 209b (1s) == sanity-flr test 210a: handle broken mirrored lovea ==== 06:24:53 (1733397893) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101334 s, 103 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa7f lmm_fid: [0x200000405:0xa7f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa7f lmm_fid: [0x200000405:0xa7f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1075:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339613823 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-12-05 06:24:54.000000000 -0500 Modify: 2024-12-05 06:24:54.000000000 -0500 Change: 2024-12-05 06:24:54.000000000 -0500 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 06:24:55 (1733397895) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00916164 s, 114 MB/s oleg310-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg310-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg310-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg310-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 oleg310-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg310-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg310-server: O/280000401/d22/1076: File not found by ext2_lookup PASS 210b (13s) == sanity-flr test complete, duration 679 sec ============ 06:25:08 (1733397908) === sanity-flr: start cleanup 06:25:08 (1733397908) === === sanity-flr: finish cleanup 06:25:10 (1733397910) ===