-----============= acceptance-small: sanity-flr ============----- Tue Aug 27 11:42:35 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 6 201 44c === sanity-flr: start setup 11:42:40 (1724773360) === oleg414-client.virtnet: executing check_config_client /mnt/lustre oleg414-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg414-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b620d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b620d000.idle_timeout=debug disable quota as required oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 11:42:49 (1724773369) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f7601] == sanity-flr test 0a: lfs mirror create with -N option == 11:42:49 (1724773369) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 11:42:52 (1724773372) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 11:42:53 (1724773373) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (0s) == sanity-flr test 0d: lfs mirror extend with -N option == 11:42:54 (1724773374) PASS 0d (6s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 11:42:59 (1724773379) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 11:43:00 (1724773380) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 11:43:01 (1724773381) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b620d000.stats=clear osc.lustre-OST0001-osc-ffff8800b620d000.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 11:43:03 (1724773383) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 11:43:05 (1724773385) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 11:43:07 (1724773387) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 11:43:08 (1724773388) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 11:43:10 (1724773390) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 11:43:12 (1724773392) PASS 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 11:43:13 (1724773393) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 11:43:15 (1724773395) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 11:43:18 (1724773398) PASS 7 (1s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 11:43:19 (1724773399) 20+0 records in 20+0 records out 20971520 bytes (21 MB) copied, 0.522594 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0281147 s, 37.3 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 11:43:22 (1724773402) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB) copied, 0.0580171 s, 127 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 11:43:24 (1724773404) Stopping /mnt/lustre-ost1 (opts:) on oleg414-server oleg414-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b620d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b620d000.ost_server_uuid in DISCONN state after 2 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg414-server: oleg414-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg414-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg414-server oleg414-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b620d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b620d000.ost_server_uuid in DISCONN state after 3 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg414-server: oleg414-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg414-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (27s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 11:43:51 (1724773431) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB) copied, 0.125919 s, 58.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg414-server oleg414-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b620d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b620d000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg414-server: oleg414-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg414-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (11s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 11:44:02 (1724773442) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg414-server oleg414-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b620d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b620d000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg414-server: oleg414-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg414-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg414-server oleg414-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b620d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b620d000.ost_server_uuid in DISCONN state after 3 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg414-server: oleg414-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg414-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (23s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 11:44:25 (1724773465) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.808353 s, 64.9 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 11:44 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.231s user 0m0.000s sys 0m0.116s Stopping /mnt/lustre-ost1 (opts:) on oleg414-server oleg414-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b620d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b620d000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.218s user 0m0.000s sys 0m0.109s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg414-server: oleg414-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg414-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg414-server oleg414-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b620d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b620d000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost1)... real 0m0.227s user 0m0.001s sys 0m0.103s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg414-server: oleg414-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg414-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg414-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b620d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b620d000.ost_server_uuid in FULL state after 0 sec PASS 33b (22s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 11:44:47 (1724773487) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 11:44:48 (1724773488) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 11:44:49 (1724773489) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 11:44:50 (1724773490) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 11:44:52 (1724773492) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.177125 s, 71.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 11:44:55 (1724773495) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0105334 s, 99.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106863 s, 98.1 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 11:44:57 (1724773497) PASS 36c (2s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 11:44:59 (1724773499) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000591296 s, 1.7 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00052872 s, 1.9 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000916265 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000545071 s, 1.9 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 11:45:02 (1724773502) create test files with size 15746038 .. 291348d87987cc0b1ca991c449da2b14 - a8b29e068c806f17c70d17174285ab05 - 7008897bab261f089dd225a7f427f182 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg414-client.virtnet /mnt/lustre (opts:) Starting client: oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre PASS 37 (5s) == sanity-flr test 38: resync ============================ 11:45:07 (1724773507) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 11:45:10 (1724773510) PASS 39 (1s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 11:45:11 (1724773511) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0223887 s, 46.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000401:0x3a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104135 s, 101 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000401:0x3c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x2c0000401:0x3f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 11:45:13 (1724773513) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0250814 s, 83.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0991897 s, 42.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0464594 s, 45.1 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 11:45:16 (1724773516) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 11:45:17 (1724773517) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 11:45:18 (1724773518) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.011003 s, 95.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 11:45:20 (1724773520) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0747538 s, 42.1 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x12:0x0]: Invalid argument (22) PASS 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 11:45:22 (1724773522) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000401:0x47:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x2c0000401:0x4a:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x17:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (1s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 11:45:24 (1724773524) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0733011 s, 143 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg414-client.virtnet /mnt/lustre (opts:) Starting client: oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (2s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 11:45:26 (1724773526) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff88012a605000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0840013 s, 125 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg414-client.virtnet /mnt/lustre (opts:) Starting client: oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800b60c8000.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 11:45:30 (1724773530) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (8s) == sanity-flr test 46: Verify setstripe --copy option ==== 11:45:38 (1724773538) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (7s) == sanity-flr test 47: Verify mirror obj alloc =========== 11:45:45 (1724773545) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 11:45:46 (1724773546) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0778689 s, 40.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0505818 s, 62.2 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 118ccce5e1d24256d95c0bef0ddc8b60 - mirror 1 checksum: 5d2368ef947609e0d55a9c582ace144f - mirror 2 checksum: 118ccce5e1d24256d95c0bef0ddc8b60 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Aug 27 11:45:47 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Aug 27 11:45:47 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000401:0x53:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 118ccce5e1d24256d95c0bef0ddc8b60 - mirror 1 checksum: 5d2368ef947609e0d55a9c582ace144f - mirror 2 checksum: 5d2368ef947609e0d55a9c582ace144f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Aug 27 11:45:47 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Aug 27 11:45:47 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000401:0x53:0x0] } PASS 48 (3s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 11:45:49 (1724773549) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109211 s, 96.0 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 11:45:51 (1724773551) Failing mds1 on oleg414-server Stopping /mnt/lustre-mds1 (opts:) on oleg414-server 11:45:58 (1724773558) shut down Failover mds1 to oleg414-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 11:46:11 (1724773571) targets are mounted 11:46:11 (1724773571) facet_failover done oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x11:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00998723 s, 105 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x11 lmm_fid: [0x200000405:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000401:0x55:0x0] } PASS 50A (27s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 11:46:18 (1724773578) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0114165 s, 1.8 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00510598 s, 4.0 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 11:46:21 (1724773581) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0119183 s, 1.7 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00442617 s, 4.6 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 11:46:23 (1724773583) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00233833 s, 7.0 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 11:46:25 (1724773585) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00900664 s, 2.3 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 11:46:27 (1724773587) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00355067 s, 1.2 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0101764 s, 403 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 11:46:29 (1724773589) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) PASS 60b (4s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 11:46:33 (1724773593) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 11:46:51 (1724773611) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x49:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 11:47:09 (1724773629) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 11:47:26 (1724773646) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (1s) == sanity-flr test 70: mirror create and split race ====== 11:47:27 (1724773647) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 25431 mirror split pid 25433 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7464: 25431 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4819: 25433 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 11:48:29 (1724773709) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 3709 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported 000002 1724773709.975981 write 0x033e60 thru 0x03681d (0x29be bytes) 000003 1724773709.978156 mapread 0x003e6a thru 0x012d21 (0xeeb8 bytes) 000005 1724773709.984853 punch from 0x01f9c1 to 0x02bdda (0xc41a bytes) 000006 1724773709.986160 read 0x01fb40 thru 0x025ae1 (0x5fa2 bytes) 000008 1724773709.988447 read 0x00b6be thru 0x019cf9 (0xe63c bytes) 000009 1724773709.992351 MIRROR_EXTEND to 2 mirrors fallocating to largest ever: 0x5ca1 000012 1724773710.101760 fallocate 0x004292 thru 0x005ca0 (0x1a0f bytes) truncating to largest ever: 0x348a9 000013 1724773710.109314 trunc from 0x03681e to 0x0348a8 (0x1f75 bytes) 000014 1724773710.115279 write 0x013ffa thru 0x020c40 (0xcc47 bytes) 000016 1724773710.118596 read 0x01e4e0 thru 0x02cd1e (0xe83f bytes) 000017 1724773710.121325 mapwrite 0x018f31 thru 0x01ac75 (0x1d45 bytes) 000019 1724773710.130528 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000020 1724773710.161614 mapwrite 0x01a689 thru 0x02043a (0x5db2 bytes) 000021 1724773710.171888 read 0x024164 thru 0x02a2c2 (0x615f bytes) 000022 1724773710.175365 fallocate 0x02ddf8 thru 0x03065f (0x2868 bytes) 000023 1724773710.177267 mapwrite 0x01493b thru 0x015c4e (0x1314 bytes) 000024 1724773710.183338 trunc from 0x0348a9 to 0x02b38d (0x951b bytes) 000025 1724773710.189130 punch from 0x013510 to 0x016401 (0x2ef2 bytes) 000026 1724773710.190981 read 0x00e271 thru 0x00f096 (0xe26 bytes) 000027 1724773710.191372 write 0x011686 thru 0x01a5e3 (0x8f5e bytes) 000028 1724773710.192229 mapread 0x00cef1 thru 0x01a5ce (0xd6de bytes) 000029 1724773710.195399 mapwrite 0x029e8d thru 0x034660 (0xa7d4 bytes) 000030 1724773710.213000 MIRROR_EXTEND to 2 mirrors 000032 1724773710.309887 mapwrite 0x0073f3 thru 0x00b671 (0x427f bytes) 000033 1724773710.339388 write 0x00b7e0 thru 0x019a79 (0xe29a bytes) 000034 1724773710.342781 read 0x00e7a7 thru 0x015cdb (0x7535 bytes) 000035 1724773710.343164 mapread 0x01ec46 thru 0x024fd1 (0x638c bytes) 000037 1724773710.345884 read 0x015053 thru 0x0249e6 (0xf994 bytes) 000038 1724773710.348210 punch from 0x024ee3 to 0x02c822 (0x7940 bytes) fallocating to largest ever: 0x40000 000040 1724773710.349962 fallocate 0x0362b2 thru 0x03ffff (0x9d4e bytes) 000041 1724773710.351378 write 0x02fbaf thru 0x0384e0 (0x8932 bytes) 000042 1724773710.353202 mapwrite 0x013807 thru 0x01d820 (0xa01a bytes) 000043 1724773710.358981 fallocate 0x01593c thru 0x01cfe6 (0x76ab bytes) 000044 1724773710.360456 fallocate 0x022417 thru 0x0282b3 (0x5e9d bytes) 000045 1724773710.361980 mapwrite 0x007e67 thru 0x00e46b (0x6605 bytes) 000046 1724773710.367203 write 0x009ef6 thru 0x00d2b5 (0x33c0 bytes) 000048 1724773710.368017 mapwrite 0x030159 thru 0x03e661 (0xe509 bytes) 000049 1724773710.376415 write 0x039725 thru 0x03ffff (0x68db bytes) 000052 1724773710.377679 write 0x03e834 thru 0x03ffff (0x17cc bytes) 000053 1724773710.378246 mapread 0x00ad0d thru 0x011be6 (0x6eda bytes) 000054 1724773710.378663 trunc from 0x040000 to 0x00234c (0x3dcb3 bytes) 000056 1724773710.388746 punch from 0x001039 to 0x00234c (0x1314 bytes) 000057 1724773710.390443 write 0x02ee67 thru 0x030513 (0x16ad bytes) 000059 1724773710.391407 read 0x0132c0 thru 0x01aa37 (0x7778 bytes) 000061 1724773710.394637 trunc from 0x030514 to 0x02aeff (0x5614 bytes) 000062 1724773710.399007 fallocate 0x00ed71 thru 0x01e1db (0xf46b bytes) 000063 1724773710.400807 read 0x01ab3f thru 0x020444 (0x5906 bytes) 000064 1724773710.401237 write 0x01beac thru 0x023db4 (0x7f09 bytes) 000065 1724773710.402071 trunc from 0x02af00 to 0x03990b (0xea0c bytes) 000066 1724773710.406514 fallocate 0x009e93 thru 0x00af39 (0x10a7 bytes) 000068 1724773710.408514 write 0x002d34 thru 0x00c56e (0x983b bytes) 000070 1724773710.409600 mapwrite 0x01c6ec thru 0x01e5f8 (0x1f0d bytes) 000071 1724773710.415964 write 0x03cbe0 thru 0x03ffff (0x3420 bytes) 000073 1724773710.417020 mapwrite 0x012e4c thru 0x0169cc (0x3b81 bytes) 000074 1724773710.423260 fallocate 0x022a1e thru 0x02ebed (0xc1d0 bytes) 000075 1724773710.425409 punch from 0x03758f to 0x03d371 (0x5de3 bytes) 000076 1724773710.428410 MIRROR_EXTEND to 3 mirrors 000077 1724773710.547254 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000079 1724773710.605462 punch from 0x011fa6 to 0x01e639 (0xc694 bytes) 000080 1724773710.614790 fallocate 0x03cde3 thru 0x03ffff (0x321d bytes) 000081 1724773710.617657 MIRROR_EXTEND to 3 mirrors 000082 1724773710.749132 punch from 0x01b5cd to 0x01f056 (0x3a8a bytes) 000083 1724773710.760032 trunc from 0x040000 to 0x034e8f (0xb170 bytes) 000084 1724773710.773634 read 0x01932f thru 0x024219 (0xaeeb bytes) 000085 1724773710.776811 fallocate 0x03522e thru 0x03ffff (0xadd2 bytes) 000086 1724773710.779099 write 0x0318d4 thru 0x03e416 (0xcb43 bytes) 000087 1724773710.781353 mapwrite 0x00f751 thru 0x01ef11 (0xf7c1 bytes) 000088 1724773710.794020 mapwrite 0x02a738 thru 0x02ea67 (0x4330 bytes) 000089 1724773710.803048 punch from 0x013a4f to 0x01f073 (0xb625 bytes) 000090 1724773710.804744 fallocate 0x01d743 thru 0x0223f0 (0x4cae bytes) 000091 1724773710.806200 read 0x00e1e4 thru 0x00e633 (0x450 bytes) 000092 1724773710.806530 mapwrite 0x0156a5 thru 0x0157bf (0x11b bytes) 000093 1724773710.812531 trunc from 0x03e417 to 0x02a0b3 (0x14363 bytes) 000094 1724773710.817226 write 0x03ad99 thru 0x03ffff (0x5267 bytes) 000095 1724773710.817976 mapread 0x032ed9 thru 0x03c19c (0x92c4 bytes) 000096 1724773710.829111 mapwrite 0x01439b thru 0x022cf6 (0xe95c bytes) 000097 1724773710.837347 trunc from 0x040000 to 0x03d8f2 (0x270d bytes) 000099 1724773710.843195 write 0x02c918 thru 0x031a75 (0x515e bytes) 000100 1724773710.847478 mapread 0x009933 thru 0x015c78 (0xc346 bytes) 000102 1724773710.848494 read 0x019dc0 thru 0x020544 (0x6785 bytes) 000103 1724773710.848881 mapwrite 0x02fa2f thru 0x0391f1 (0x97c3 bytes) 000104 1724773710.857912 MIRROR_RESYNC 3 mirrors 000106 1724773710.990513 trunc from 0x03d8f3 to 0x021305 (0x1c5ed bytes) 000108 1724773711.003683 trunc from 0x021306 to 0x02a764 (0x945f bytes) 000110 1724773711.009030 MIRROR_RESYNC 3 mirrors 000112 1724773711.158174 punch from 0x004589 to 0x00ee59 (0xa8d1 bytes) 000113 1724773711.166069 trunc from 0x02a765 to 0x00469f (0x260c5 bytes) 000114 1724773711.170105 punch from 0x001216 to 0x00469f (0x348a bytes) 000115 1724773711.171420 mapwrite 0x000f43 thru 0x00ea0d (0xdacb bytes) 000117 1724773711.188319 read 0x002d8a thru 0x00a0a8 (0x731f bytes) 000118 1724773711.190015 MIRROR_RESYNC 3 mirrors 000119 1724773711.286655 punch from 0x00af40 to 0x00e091 (0x3152 bytes) 000120 1724773711.297432 mapwrite 0x033685 thru 0x03c629 (0x8fa5 bytes) 000121 1724773711.322240 MIRROR_EXTEND to 4 mirrors 000124 1724773711.424391 read 0x01205a thru 0x0218d4 (0xf87b bytes) 000126 1724773711.426998 mapwrite 0x00409f thru 0x006fa1 (0x2f03 bytes) 000127 1724773711.453865 fallocate 0x0136bb thru 0x013e07 (0x74d bytes) 000128 1724773711.455463 fallocate 0x030ac7 thru 0x031113 (0x64d bytes) 000129 1724773711.456921 mapread 0x0120b6 thru 0x01758d (0x54d8 bytes) 000130 1724773711.459730 mapwrite 0x03249e thru 0x03ff8e (0xdaf1 bytes) 000131 1724773711.478328 trunc from 0x03ff8f to 0x01f458 (0x20b36 bytes) 000132 1724773711.482952 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000133 1724773711.523219 punch from 0x00963f to 0x010766 (0x7128 bytes) 000134 1724773711.524794 trunc from 0x01f459 to 0x02a386 (0xaf2e bytes) 000135 1724773711.529506 trunc from 0x02a387 to 0x01ef07 (0xb47f bytes) 000136 1724773711.532500 write 0x0155b8 thru 0x0183f7 (0x2e40 bytes) 000137 1724773711.535974 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000138 1724773711.576896 mapread 0x002dd3 thru 0x01283c (0xfa6a bytes) 000139 1724773711.582421 write 0x00f108 thru 0x01e30c (0xf205 bytes) 000140 1724773711.585414 read 0x002c28 thru 0x012a94 (0xfe6d bytes) 000141 1724773711.585707 mapwrite 0x02b9e8 thru 0x03b376 (0xf98f bytes) 000142 1724773711.600424 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000143 1724773711.625755 trunc from 0x03b377 to 0x0197f7 (0x21b7f bytes) 000144 1724773711.631982 read 0x00ba7d thru 0x0197f7 (0xdd7b bytes) 000145 1724773711.634171 read 0x011a4b thru 0x015a1d (0x3fd3 bytes) 000146 1724773711.634467 fallocate 0x039cca thru 0x03ffff (0x6336 bytes) 000147 1724773711.635812 mapread 0x003581 thru 0x010d5a (0xd7da bytes) 000148 1724773711.637780 mapread 0x002842 thru 0x008957 (0x6116 bytes) 000150 1724773711.638320 mapread 0x00b985 thru 0x0175e3 (0xbc5f bytes) 000151 1724773711.638657 read 0x006142 thru 0x00fd62 (0x9c21 bytes) 000153 1724773711.640357 MIRROR_EXTEND to 2 mirrors 000154 1724773711.726584 trunc from 0x0197f8 to 0x03cc69 (0x23472 bytes) 000155 1724773711.740270 mapwrite 0x03f6e4 thru 0x03f8df (0x1fc bytes) 000156 1724773711.749925 write 0x0146bd thru 0x01de27 (0x976b bytes) 000157 1724773711.752620 mapread 0x02b858 thru 0x032630 (0x6dd9 bytes) 000158 1724773711.754744 read 0x000d9c thru 0x00e30d (0xd572 bytes) 000159 1724773711.758253 MIRROR_RESYNC 2 mirrors 000160 1724773711.861465 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000161 1724773711.887504 mapread 0x01466b thru 0x01f036 (0xa9cc bytes) 000163 1724773711.894014 punch from 0x02dee0 to 0x034c9e (0x6dbf bytes) 000164 1724773711.898146 fallocate 0x037a81 thru 0x03ffff (0x857f bytes) 000165 1724773711.899732 punch from 0x03af0f to 0x03f8df (0x49d1 bytes) 000167 1724773711.901519 punch from 0x0037d4 to 0x00c541 (0x8d6e bytes) 000169 1724773711.903774 trunc from 0x03f8e0 to 0x03473c (0xb1a3 bytes) 000170 1724773711.914902 trunc from 0x03473d to 0x034ef1 (0x7b5 bytes) 000171 1724773711.919036 mapwrite 0x01b5f0 thru 0x0264f5 (0xaf06 bytes) 000172 1724773711.931164 mapread 0x002ba0 thru 0x0122fc (0xf75d bytes) 000173 1724773711.937124 write 0x033cf1 thru 0x0360a0 (0x23b0 bytes) 000176 1724773711.940134 punch from 0x0312a9 to 0x0360a0 (0x4df8 bytes) 000177 1724773711.943647 write 0x0356f1 thru 0x03ffff (0xa90f bytes) 000178 1724773711.946568 mapread 0x029dd1 thru 0x02cac3 (0x2cf3 bytes) 000180 1724773711.947291 fallocate 0x018756 thru 0x02046b (0x7d16 bytes) 000181 1724773711.948853 mapwrite 0x002819 thru 0x0028ba (0xa2 bytes) 000184 1724773711.955352 MIRROR_EXTEND to 2 mirrors 000185 1724773712.062410 mapread 0x0024c8 thru 0x007ce3 (0x581c bytes) 000187 1724773712.066373 punch from 0x03059c to 0x03ffff (0xfa64 bytes) 000188 1724773712.074444 punch from 0x00f186 to 0x01aec6 (0xbd41 bytes) 000190 1724773712.076508 mapwrite 0x02621a thru 0x02943c (0x3223 bytes) 000191 1724773712.083899 mapwrite 0x0297aa thru 0x0396d3 (0xff2a bytes) 000192 1724773712.094909 trunc from 0x040000 to 0x02923c (0x16dc3 bytes) 000193 1724773712.101322 punch from 0x0086d6 to 0x01166d (0x8f98 bytes) 000196 1724773712.103933 read 0x0153a6 thru 0x01ec0d (0x9868 bytes) 000197 1724773712.108477 MIRROR_RESYNC 2 mirrors 000198 1724773712.200589 mapread 0x019ef1 thru 0x01bc80 (0x1d90 bytes) 000200 1724773712.206102 fallocate 0x02bab9 thru 0x0311d3 (0x571b bytes) 000201 1724773712.215049 mapread 0x01da44 thru 0x026ee6 (0x94a3 bytes) 000203 1724773712.219750 read 0x023248 thru 0x02923c (0x5ff5 bytes) 000204 1724773712.220199 write 0x037d3a thru 0x03cce6 (0x4fad bytes) 000205 1724773712.220833 write 0x028fd1 thru 0x0367e8 (0xd818 bytes) 000206 1724773712.223129 trunc from 0x03cce7 to 0x03cf01 (0x21b bytes) 000207 1724773712.231669 trunc from 0x03cf02 to 0x03fe34 (0x2f33 bytes) 000208 1724773712.235052 trunc from 0x03fe35 to 0x033d88 (0xc0ac bytes) 000209 1724773712.238250 trunc from 0x033d89 to 0x02a103 (0x9c85 bytes) 000210 1724773712.241667 mapread 0x0158f6 thru 0x02426e (0xe979 bytes) 000211 1724773712.244090 mapwrite 0x036f21 thru 0x03ffff (0x90df bytes) 000212 1724773712.257585 punch from 0x000554 to 0x004d6f (0x481c bytes) 000214 1724773712.259741 trunc from 0x040000 to 0x03ba8b (0x4574 bytes) 000215 1724773712.263747 mapwrite 0x00c703 thru 0x01463b (0x7f39 bytes) 000216 1724773712.273116 fallocate 0x00bdad thru 0x01ae11 (0xf065 bytes) 000219 1724773712.275733 write 0x00d70f thru 0x01650d (0x8dff bytes) 000220 1724773712.278343 MIRROR_EXTEND to 3 mirrors 000222 1724773712.397282 mapread 0x01f870 thru 0x02d7a6 (0xdf37 bytes) 000223 1724773712.410177 punch from 0x01686f to 0x018706 (0x1e98 bytes) 000224 1724773712.422481 MIRROR_RESYNC 3 mirrors 000225 1724773712.587743 fallocate 0x02c5ed thru 0x02f27b (0x2c8f bytes) 000226 1724773712.597531 trunc from 0x03ba8c to 0x026924 (0x15167 bytes) 000227 1724773712.607780 write 0x010952 thru 0x018421 (0x7ad0 bytes) 000228 1724773712.612245 MIRROR_RESYNC 3 mirrors 000229 1724773712.745305 mapwrite 0x031b52 thru 0x035ae3 (0x3f92 bytes) 000230 1724773712.772230 mapwrite 0x00d93d thru 0x011846 (0x3f0a bytes) 000231 1724773712.781344 trunc from 0x035ae4 to 0x00af14 (0x2abcf bytes) 000232 1724773712.788325 read 0x000245 thru 0x005ff8 (0x5db4 bytes) 000234 1724773712.792255 trunc from 0x00af15 to 0x03767e (0x2c76a bytes) 000235 1724773712.798630 read 0x02dd5a thru 0x036adc (0x8d83 bytes) 000236 1724773712.802469 read 0x03640a thru 0x03767e (0x1275 bytes) 000237 1724773712.802884 fallocate 0x020325 thru 0x02cc1c (0xc8f8 bytes) 000238 1724773712.805127 mapread 0x018288 thru 0x019da7 (0x1b20 bytes) 000239 1724773712.805656 write 0x009a66 thru 0x00e64b (0x4be6 bytes) 000240 1724773712.806525 write 0x02e6cd thru 0x02fd33 (0x1667 bytes) 000241 1724773712.807008 trunc from 0x03767f to 0x01d972 (0x19d0c bytes) 000243 1724773712.814079 punch from 0x00f6b8 to 0x01c75e (0xd0a7 bytes) 000244 1724773712.816119 punch from 0x00fc1e to 0x01ccd9 (0xd0bc bytes) 000245 1724773712.817517 mapwrite 0x00ab0c thru 0x012609 (0x7afe bytes) 000246 1724773712.829712 mapwrite 0x02b756 thru 0x0319ea (0x6295 bytes) 000247 1724773712.848275 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000248 1724773712.899619 trunc from 0x0319eb to 0x038f3d (0x7553 bytes) 000250 1724773712.906620 mapwrite 0x031394 thru 0x037c11 (0x687e bytes) 000251 1724773712.920358 trunc from 0x038f3e to 0x0347a1 (0x479c bytes) 000252 1724773712.927079 mapread 0x02dfcb thru 0x0347a1 (0x67d7 bytes) 000253 1724773712.927716 write 0x017ef9 thru 0x020389 (0x8491 bytes) 000254 1724773712.932427 fallocate 0x01b1a9 thru 0x01be36 (0xc8e bytes) 000255 1724773712.934142 trunc from 0x0347a2 to 0x018b2f (0x1bc72 bytes) 000256 1724773712.940141 mapread 0x014ccc thru 0x018b2f (0x3e64 bytes) 000257 1724773712.940769 trunc from 0x018b30 to 0x010c74 (0x7ebb bytes) 000259 1724773712.945673 punch from 0x006b7b to 0x00c15a (0x55e0 bytes) 000260 1724773712.947665 mapread 0x005321 thru 0x00a0be (0x4d9e bytes) 000261 1724773712.953131 MIRROR_EXTEND to 3 mirrors 000262 1724773713.080670 fallocate 0x030acc thru 0x031777 (0xcac bytes) 000265 1724773713.091729 trunc from 0x010c75 to 0x0016cb (0xf5a9 bytes) 000266 1724773713.098602 mapwrite 0x0387a5 thru 0x03ffff (0x785b bytes) 000267 1724773713.113922 trunc from 0x040000 to 0x0297a6 (0x16859 bytes) 000268 1724773713.119478 write 0x016029 thru 0x0173bd (0x1395 bytes) 000269 1724773713.128445 MIRROR_RESYNC 3 mirrors 000270 1724773713.316278 trunc from 0x0297a7 to 0x028111 (0x1695 bytes) 000271 1724773713.328570 write 0x02c633 thru 0x0373c3 (0xad91 bytes) 000273 1724773713.329683 mapwrite 0x018229 thru 0x026cac (0xea84 bytes) 000274 1724773713.347920 fallocate 0x0265fc thru 0x030a77 (0xa47c bytes) 000275 1724773713.350181 fallocate 0x02b313 thru 0x036486 (0xb174 bytes) 000277 1724773713.352700 read 0x00be56 thru 0x0196f0 (0xd89b bytes) 000279 1724773713.357084 trunc from 0x0373c4 to 0x016f22 (0x204a1 bytes) 000280 1724773713.363981 mapwrite 0x034677 thru 0x0378bf (0x3249 bytes) 000281 1724773713.378839 MIRROR_EXTEND to 4 mirrors 000282 1724773713.503610 write 0x02d135 thru 0x03a2a0 (0xd16c bytes) 000283 1724773713.517694 mapwrite 0x02e239 thru 0x030ceb (0x2ab3 bytes) 000287 1724773713.528025 MIRROR_EXTEND to 5 mirrors 000289 1724773713.661253 mapwrite 0x03ffa1 thru 0x03ffff (0x5f bytes) 000290 1724773713.689383 mapread 0x00609a thru 0x006f9b (0xf02 bytes) 000291 1724773713.692792 MIRROR_EXTEND to 6 mirrors 000292 1724773713.836955 fallocate 0x033012 thru 0x033e2b (0xe1a bytes) 000293 1724773713.851321 read 0x02be54 thru 0x02c7f4 (0x9a1 bytes) 000294 1724773713.853506 punch from 0x01775d to 0x01d9b0 (0x6254 bytes) 000295 1724773713.855324 fallocate 0x013fe1 thru 0x0230b6 (0xf0d6 bytes) 000296 1724773713.857132 mapwrite 0x0227ba thru 0x030afc (0xe343 bytes) 000297 1724773713.878391 read 0x02fc28 thru 0x031554 (0x192d bytes) 000298 1724773713.878866 mapread 0x01cee7 thru 0x026819 (0x9933 bytes) 000299 1724773713.882593 mapread 0x000b87 thru 0x00f2c2 (0xe73c bytes) 000300 1724773713.883182 write 0x0045ed thru 0x013ac2 (0xf4d6 bytes) 000301 1724773713.886314 mapread 0x009673 thru 0x014be3 (0xb571 bytes) 000302 1724773713.886885 punch from 0x03c16a to 0x03cdb0 (0xc47 bytes) 000303 1724773713.888870 punch from 0x01a1e9 to 0x027211 (0xd029 bytes) 000304 1724773713.891233 trunc from 0x040000 to 0x03b63c (0x49c3 bytes) 000305 1724773713.905878 mapread 0x000089 thru 0x00f533 (0xf4ab bytes) 000306 1724773713.906626 fallocate 0x00ca64 thru 0x015cbb (0x9258 bytes) 000307 1724773713.909514 read 0x00b611 thru 0x01891a (0xd30a bytes) 000308 1724773713.909965 read 0x000510 thru 0x00b89e (0xb38f bytes) 000309 1724773713.910300 punch from 0x0368a2 to 0x03b63c (0x4d9b bytes) 000310 1724773713.912375 trunc from 0x03b63d to 0x00ca34 (0x2ec08 bytes) 000311 1724773713.917685 write 0x005546 thru 0x00a49e (0x4f59 bytes) 000313 1724773713.918870 mapwrite 0x021622 thru 0x027b4d (0x652c bytes) 000314 1724773713.938963 mapread 0x01cbb3 thru 0x02216b (0x55b9 bytes) 000315 1724773713.939382 punch from 0x0073e4 to 0x009566 (0x2183 bytes) 000316 1724773713.941512 fallocate 0x02a94b thru 0x03a141 (0xf7f7 bytes) 000319 1724773713.945972 MIRROR_EXTEND to 7 mirrors 000320 1724773714.109094 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000321 1724773714.190570 MIRROR_RESYNC 6 mirrors 000322 1724773714.471641 punch from 0x018799 to 0x0222ef (0x9b57 bytes) 000323 1724773714.481367 mapwrite 0x032754 thru 0x03a8a0 (0x814d bytes) 000324 1724773714.499204 mapread 0x0093dc thru 0x01640e (0xd033 bytes) 000325 1724773714.504271 mapread 0x02998f thru 0x02b45d (0x1acf bytes) 000326 1724773714.507149 mapwrite 0x02dbd5 thru 0x0334a0 (0x58cc bytes) 000327 1724773714.514617 fallocate 0x03a632 thru 0x03f427 (0x4df6 bytes) 000328 1724773714.516328 trunc from 0x03a8a1 to 0x017ff1 (0x228af bytes) 000329 1724773714.521537 mapwrite 0x01b4bd thru 0x02a033 (0xeb77 bytes) 000330 1724773714.540479 trunc from 0x02a034 to 0x01a4ab (0xfb88 bytes) 000331 1724773714.545402 write 0x02b5c7 thru 0x033fef (0x8a29 bytes) 000333 1724773714.546630 fallocate 0x0208d7 thru 0x02b507 (0xac31 bytes) 000334 1724773714.548849 fallocate 0x00bb21 thru 0x011f8b (0x646b bytes) 000335 1724773714.550663 mapread 0x0203e9 thru 0x026aa0 (0x66b8 bytes) 000336 1724773714.555752 MIRROR_RESYNC 6 mirrors 000337 1724773714.862639 fallocate 0x005eda thru 0x010ecd (0xaff4 bytes) 000338 1724773714.872959 MIRROR_EXTEND to 7 mirrors 000339 1724773714.989432 punch from 0x00aae2 to 0x015275 (0xa794 bytes) 000340 1724773714.997933 fallocate 0x008890 thru 0x01016f (0x78e0 bytes) 000341 1724773715.000506 MIRROR_EXTEND to 8 mirrors 000343 1724773715.109689 punch from 0x01ec81 to 0x0205ab (0x192b bytes) 000344 1724773715.120881 fallocate 0x02d91b thru 0x034240 (0x6926 bytes) 000345 1724773715.122680 mapread 0x02b971 thru 0x033fef (0x867f bytes) 000346 1724773715.129217 write 0x019753 thru 0x01aac4 (0x1372 bytes) 000347 1724773715.133278 fallocate 0x01b68a thru 0x0283a4 (0xcd1b bytes) 000349 1724773715.135372 mapread 0x0182f3 thru 0x01a793 (0x24a1 bytes) 000350 1724773715.135887 mapread 0x010872 thru 0x011f86 (0x1715 bytes) 000352 1724773715.139367 punch from 0x0069a0 to 0x0159ad (0xf00e bytes) 000354 1724773715.141803 trunc from 0x033ff0 to 0x01c56d (0x17a82 bytes) 000355 1724773715.147295 mapread 0x00da00 thru 0x01c369 (0xe96a bytes) 000356 1724773715.152933 write 0x038789 thru 0x03f59f (0x6e17 bytes) 000357 1724773715.153809 mapwrite 0x026fff thru 0x02d1b6 (0x61b8 bytes) 000360 1724773715.164616 mapwrite 0x01aa57 thru 0x028da2 (0xe34c bytes) 000361 1724773715.173719 fallocate 0x02a37f thru 0x02d988 (0x360a bytes) 000362 1724773715.175192 trunc from 0x03f5a0 to 0x01fa19 (0x1fb86 bytes) 000363 1724773715.179142 punch from 0x0177cf to 0x01fa19 (0x824b bytes) 000364 1724773715.180792 write 0x02a7aa thru 0x02b063 (0x8ba bytes) 000365 1724773715.181232 write 0x00a67b thru 0x0165f1 (0xbf77 bytes) 000366 1724773715.181952 punch from 0x02904d to 0x02b063 (0x2017 bytes) 000367 1724773715.185701 mapread 0x02727c thru 0x02b063 (0x3de8 bytes) 000369 1724773715.188810 fallocate 0x033ef8 thru 0x0368ca (0x29d3 bytes) 000370 1724773715.190442 read 0x022309 thru 0x02b063 (0x8d5b bytes) 000371 1724773715.190856 mapwrite 0x03e8e9 thru 0x03fd9f (0x14b7 bytes) 000372 1724773715.201633 mapread 0x013f25 thru 0x021bf8 (0xdcd4 bytes) 000373 1724773715.205392 trunc from 0x03fda0 to 0x021b02 (0x1e29d bytes) 000374 1724773715.208895 read 0x001994 thru 0x009be8 (0x8255 bytes) 000375 1724773715.212439 MIRROR_EXTEND to 9 mirrors 000376 1724773715.328331 read 0x013db3 thru 0x01a3c2 (0x6610 bytes) 000377 1724773715.332298 read 0x01a702 thru 0x021b02 (0x7401 bytes) 000378 1724773715.336898 mapwrite 0x03cb2f thru 0x03ffff (0x34d1 bytes) 000379 1724773715.363207 mapwrite 0x022404 thru 0x02caf0 (0xa6ed bytes) 000381 1724773715.372019 write 0x0095ee thru 0x0176ec (0xe0ff bytes) 000382 1724773715.379425 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000383 1724773715.434217 mapwrite 0x011f6c thru 0x0196ba (0x774f bytes) 000385 1724773715.447550 mapread 0x02c2c4 thru 0x02d5af (0x12ec bytes) 000386 1724773715.450606 mapread 0x03295d thru 0x0335fb (0xc9f bytes) 000387 1724773715.451104 mapread 0x01b987 thru 0x028d4d (0xd3c7 bytes) 000388 1724773715.457038 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000389 1724773715.503847 mapread 0x03a2ab thru 0x03ffff (0x5d55 bytes) 000391 1724773715.508714 read 0x00f27f thru 0x01a891 (0xb613 bytes) 000392 1724773715.513607 MIRROR_RESYNC 7 mirrors 000393 1724773715.995663 write 0x01499b thru 0x021cd9 (0xd33f bytes) 000394 1724773716.007620 fallocate 0x006187 thru 0x00c951 (0x67cb bytes) 000395 1724773716.009413 write 0x00c6df thru 0x0103ac (0x3cce bytes) 000396 1724773716.012465 read 0x01d232 thru 0x026bfd (0x99cc bytes) 000397 1724773716.015004 read 0x031d09 thru 0x031e0c (0x104 bytes) 000398 1724773716.018163 trunc from 0x040000 to 0x00750f (0x38af0 bytes) 000399 1724773716.028751 trunc from 0x007510 to 0x00254e (0x4fc1 bytes) 000400 1724773716.032854 write 0x02f17c thru 0x03b5cf (0xc454 bytes) 000402 1724773716.034130 mapwrite 0x00c1af thru 0x013618 (0x746a bytes) 000403 1724773716.056148 trunc from 0x03b5d0 to 0x013d95 (0x2783a bytes) 000404 1724773716.060723 mapwrite 0x0251f1 thru 0x031d92 (0xcba2 bytes) 000405 1724773716.094072 mapread 0x00d528 thru 0x012af7 (0x55d0 bytes) 000406 1724773716.094594 mapwrite 0x0246e1 thru 0x026efd (0x281d bytes) 000407 1724773716.100726 mapwrite 0x00f40d thru 0x01409f (0x4c93 bytes) 000408 1724773716.108818 fallocate 0x00d3c7 thru 0x00f36d (0x1fa7 bytes) 000409 1724773716.110626 punch from 0x022bdc to 0x029339 (0x675e bytes) 000410 1724773716.112394 fallocate 0x028475 thru 0x02cc05 (0x4791 bytes) 000411 1724773716.113954 punch from 0x02e325 to 0x031d92 (0x3a6e bytes) 000414 1724773716.117531 MIRROR_EXTEND to 8 mirrors 000416 1724773716.257824 trunc from 0x031d93 to 0x02aa96 (0x72fc bytes) 000417 1724773716.272149 read 0x0075e1 thru 0x010775 (0x9195 bytes) 000418 1724773716.276156 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000419 1724773716.326362 write 0x01e2bf thru 0x01e8c9 (0x60b bytes) 000420 1724773716.329560 punch from 0x01aff2 to 0x029245 (0xe254 bytes) 000421 1724773716.333170 mapread 0x01bcb5 thru 0x0204fd (0x4849 bytes) 000422 1724773716.339037 write 0x02da15 thru 0x03a9d9 (0xcfc5 bytes) 000423 1724773716.340187 mapwrite 0x0269f8 thru 0x029da6 (0x33af bytes) 000425 1724773716.350580 mapwrite 0x00e000 thru 0x013217 (0x5218 bytes) 000426 1724773716.360091 trunc from 0x03a9da to 0x036baf (0x3e2a bytes) 000427 1724773716.367101 fallocate 0x00b5ea thru 0x01b0be (0xfad5 bytes) 000428 1724773716.369241 trunc from 0x036bb0 to 0x038aae (0x1eff bytes) 000429 1724773716.375124 mapwrite 0x03c713 thru 0x03f23e (0x2b2c bytes) 000430 1724773716.390927 mapwrite 0x00e36b thru 0x01b430 (0xd0c6 bytes) 000431 1724773716.401000 trunc from 0x03f23f to 0x020cad (0x1e591 bytes) 000432 1724773716.406628 punch from 0x01f233 to 0x020cad (0x1a7b bytes) 000433 1724773716.408808 trunc from 0x020cae to 0x03d911 (0x1cc64 bytes) 000435 1724773716.414286 read 0x00ab4c thru 0x013f32 (0x93e7 bytes) 000436 1724773716.414748 mapread 0x0147f7 thru 0x01814c (0x3956 bytes) 000437 1724773716.415361 punch from 0x030b24 to 0x037e3d (0x731a bytes) 000438 1724773716.417271 write 0x008d98 thru 0x00e7ad (0x5a16 bytes) 000439 1724773716.417959 mapread 0x01d972 thru 0x02539a (0x7a29 bytes) 000440 1724773716.422968 MIRROR_RESYNC 7 mirrors 000441 1724773716.742816 write 0x000a66 thru 0x00178f (0xd2a bytes) 000443 1724773716.758723 write 0x031a1b thru 0x033f92 (0x2578 bytes) 000444 1724773716.766144 trunc from 0x03d912 to 0x035359 (0x85b8 bytes) 000446 1724773716.788685 MIRROR_EXTEND to 8 mirrors 000447 1724773716.948763 mapwrite 0x004c3c thru 0x00648c (0x1851 bytes) 000449 1724773716.985364 mapread 0x02344d thru 0x02dd55 (0xa909 bytes) 000450 1724773716.991423 trunc from 0x03535a to 0x0015b7 (0x33da2 bytes) 000451 1724773717.009775 mapread 0x00135e thru 0x0015b7 (0x25a bytes) 000452 1724773717.010487 mapwrite 0x0354a4 thru 0x039c54 (0x47b1 bytes) 000453 1724773717.028301 trunc from 0x039c55 to 0x01585a (0x243fa bytes) 000454 1724773717.032909 write 0x02f843 thru 0x03ede9 (0xf5a7 bytes) 000457 1724773717.034635 fallocate 0x02f303 thru 0x035fae (0x6cac bytes) 000458 1724773717.037018 write 0x00b0d0 thru 0x0126b3 (0x75e4 bytes) 000459 1724773717.041289 read 0x013004 thru 0x01ce0b (0x9e08 bytes) 000461 1724773717.044691 read 0x0040a5 thru 0x00bc99 (0x7bf5 bytes) 000463 1724773717.047043 MIRROR_EXTEND to 9 mirrors 000464 1724773717.189322 trunc from 0x03edea to 0x02d438 (0x119b1 bytes) 000465 1724773717.209088 fallocate 0x02c5f7 thru 0x02da26 (0x1430 bytes) 000466 1724773717.210936 read 0x008977 thru 0x011610 (0x8c9a bytes) 000467 1724773717.214121 fallocate 0x0047b0 thru 0x00f2e9 (0xab3a bytes) 000468 1724773717.215853 trunc from 0x02d439 to 0x0299e5 (0x3a53 bytes) 000469 1724773717.221338 read 0x02654a thru 0x0299e5 (0x349c bytes) 000470 1724773717.225846 fallocate 0x03b13d thru 0x03ffff (0x4ec3 bytes) 000471 1724773717.228338 trunc from 0x040000 to 0x02a92e (0x156d1 bytes) 000472 1724773717.234137 mapwrite 0x033366 thru 0x0347c4 (0x145f bytes) 000473 1724773717.250122 write 0x00fb7b thru 0x0132c5 (0x374b bytes) 000474 1724773717.251463 mapread 0x0346ba thru 0x0347c4 (0x10b bytes) 000475 1724773717.252030 mapread 0x012025 thru 0x020c42 (0xec1e bytes) 000477 1724773717.252883 fallocate 0x016765 thru 0x0171cf (0xa6b bytes) 000478 1724773717.254916 trunc from 0x0347c5 to 0x027e21 (0xc9a3 bytes) 000479 1724773717.260307 read 0x0165f2 thru 0x016944 (0x353 bytes) 000482 1724773717.261349 trunc from 0x027e22 to 0x034452 (0xc631 bytes) 000484 1724773717.267799 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000485 1724773717.337364 read 0x0180ca thru 0x01c465 (0x439c bytes) 000486 1724773717.340679 trunc from 0x034453 to 0x034edd (0xa8b bytes) 000487 1724773717.349607 write 0x01d00d thru 0x0268c5 (0x98b9 bytes) 000488 1724773717.353416 mapread 0x029f01 thru 0x034edd (0xafdd bytes) 000489 1724773717.359406 write 0x012c5b thru 0x013a3e (0xde4 bytes) 000491 1724773717.362819 punch from 0x01cedc to 0x028880 (0xb9a5 bytes) 000492 1724773717.367051 trunc from 0x034ede to 0x0116d2 (0x2380b bytes) 000493 1724773717.371478 mapread 0x00031d thru 0x00aa5d (0xa741 bytes) 000494 1724773717.377716 fallocate 0x00592f thru 0x011863 (0xbf35 bytes) 000495 1724773717.379249 trunc from 0x0116d3 to 0x023f99 (0x128c7 bytes) 000498 1724773717.384181 mapwrite 0x0166b8 thru 0x019a6f (0x33b8 bytes) 000499 1724773717.394684 MIRROR_EXTEND to 9 mirrors 000500 1724773717.531440 mapwrite 0x01f47a thru 0x020347 (0xece bytes) 000501 1724773717.574702 read 0x016d5a thru 0x018d0b (0x1fb2 bytes) 000503 1724773717.577038 read 0x01e106 thru 0x023f99 (0x5e94 bytes) 000504 1724773717.577406 mapwrite 0x0128b9 thru 0x0173f7 (0x4b3f bytes) 000505 1724773717.585788 mapwrite 0x01f8b3 thru 0x02e42b (0xeb79 bytes) 000506 1724773717.607885 mapread 0x002ca5 thru 0x010a56 (0xddb2 bytes) 000507 1724773717.608651 fallocate 0x00ec2d thru 0x01910e (0xa4e2 bytes) 000508 1724773717.610336 read 0x010769 thru 0x01ca1f (0xc2b7 bytes) 000509 1724773717.610703 read 0x019c4c thru 0x025f3b (0xc2f0 bytes) 000510 1724773717.610987 read 0x00f4d0 thru 0x01226a (0x2d9b bytes) 000511 1724773717.611294 punch from 0x013d80 to 0x01bfa6 (0x8227 bytes) 000512 1724773717.612920 trunc from 0x02e42c to 0x00cf84 (0x214a7 bytes) 000514 1724773717.616895 read 0x003028 thru 0x00a1e1 (0x71ba bytes) 000516 1724773717.617620 punch from 0x001527 to 0x00191d (0x3f7 bytes) 000517 1724773717.619626 punch from 0x0020e6 to 0x00cf84 (0xae9f bytes) 000518 1724773717.621621 trunc from 0x00cf85 to 0x033e82 (0x26efe bytes) 000519 1724773717.627609 fallocate 0x00ae65 thru 0x00e4ee (0x368a bytes) 000520 1724773717.629538 mapwrite 0x0187de thru 0x020350 (0x7b73 bytes) 000522 1724773717.641593 mapread 0x020e8f thru 0x02afb7 (0xa129 bytes) 000524 1724773717.642484 punch from 0x0119be to 0x01bd08 (0xa34b bytes) 000527 1724773717.645315 write 0x00b9b7 thru 0x01500e (0x9658 bytes) 000528 1724773717.648074 trunc from 0x033e83 to 0x03be03 (0x7f81 bytes) 000529 1724773717.653498 trunc from 0x03be04 to 0x018d59 (0x230aa bytes) 000530 1724773717.660878 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000532 1724773717.722632 mapread 0x005fde thru 0x014621 (0xe644 bytes) 000533 1724773717.731301 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000534 1724773717.783095 fallocate 0x00f969 thru 0x01c224 (0xc8bc bytes) 000536 1724773717.785375 read 0x002fd5 thru 0x010b39 (0xdb65 bytes) 000537 1724773717.788419 punch from 0x009f0f to 0x00f3e7 (0x54d9 bytes) 000538 1724773717.790227 fallocate 0x005b54 thru 0x00c6c6 (0x6b73 bytes) 000539 1724773717.792064 mapread 0x0137bd thru 0x018d59 (0x559d bytes) 000540 1724773717.798658 write 0x01dbe9 thru 0x01ffe9 (0x2401 bytes) 000542 1724773717.801283 read 0x005b29 thru 0x00a843 (0x4d1b bytes) 000543 1724773717.804690 read 0x019a50 thru 0x01f963 (0x5f14 bytes) 000546 1724773717.805974 trunc from 0x01ffea to 0x00f243 (0x10da6 bytes) 000547 1724773717.812281 trunc from 0x00f244 to 0x01acb8 (0xba75 bytes) 000548 1724773717.816928 trunc from 0x01acb9 to 0x035460 (0x1a7a8 bytes) 000551 1724773717.823336 mapread 0x015039 thru 0x0165c0 (0x1588 bytes) 000552 1724773717.827526 write 0x02ac46 thru 0x03602c (0xb3e7 bytes) 000553 1724773717.828439 mapwrite 0x03e29e thru 0x03ffff (0x1d62 bytes) 000555 1724773717.844154 mapwrite 0x019013 thru 0x020e91 (0x7e7f bytes) 000556 1724773717.852667 MIRROR_EXTEND to 8 mirrors 000557 1724773717.991433 fallocate 0x03affe thru 0x03ffff (0x5002 bytes) 000558 1724773718.002733 trunc from 0x040000 to 0x01454f (0x2bab0 bytes) 000559 1724773718.014887 MIRROR_RESYNC 8 mirrors 000560 1724773718.293138 fallocate 0x01970e thru 0x01f781 (0x6074 bytes) 000561 1724773718.303638 trunc from 0x014550 to 0x02f623 (0x1b0d4 bytes) 000563 1724773718.309466 write 0x0106f8 thru 0x01b327 (0xac30 bytes) 000564 1724773718.312841 fallocate 0x03721e thru 0x03ffff (0x8de2 bytes) 000565 1724773718.314525 mapread 0x015e6c thru 0x0251c3 (0xf358 bytes) 000567 1724773718.321317 read 0x025739 thru 0x03550e (0xfdd6 bytes) 000568 1724773718.325802 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000569 1724773718.377050 write 0x01a530 thru 0x023e1e (0x98ef bytes) 000570 1724773718.382239 trunc from 0x040000 to 0x008dc5 (0x3723a bytes) 000572 1724773718.387053 punch from 0x006819 to 0x008dc5 (0x25ad bytes) 000573 1724773718.388555 punch from 0x0074c4 to 0x008dc5 (0x1902 bytes) 000574 1724773718.390084 fallocate 0x0201c5 thru 0x02adcc (0xac08 bytes) 000575 1724773718.391878 punch from 0x0119dc to 0x02191d (0xff42 bytes) 000576 1724773718.393426 punch from 0x028616 to 0x02adcc (0x27b7 bytes) 000577 1724773718.394912 fallocate 0x02b11f thru 0x03a179 (0xf05b bytes) 000579 1724773718.396876 trunc from 0x03a17a to 0x02be04 (0xe375 bytes) 000580 1724773718.401423 punch from 0x01179f to 0x018f02 (0x7764 bytes) 000581 1724773718.403179 mapread 0x02a7dc thru 0x02be04 (0x1629 bytes) 000582 1724773718.406164 mapwrite 0x028f6b thru 0x02a39a (0x1430 bytes) 000583 1724773718.411652 mapread 0x019b44 thru 0x019ec1 (0x37e bytes) 000585 1724773718.415038 fallocate 0x031180 thru 0x038a9d (0x791e bytes) 000586 1724773718.416588 write 0x01579c thru 0x019175 (0x39da bytes) 000587 1724773718.419549 MIRROR_RESYNC 7 mirrors 000589 1724773718.897253 punch from 0x007cb2 to 0x00fca8 (0x7ff7 bytes) 000590 1724773718.908085 fallocate 0x01a859 thru 0x029617 (0xedbf bytes) 000591 1724773718.911623 punch from 0x02d47d to 0x035444 (0x7fc8 bytes) 000592 1724773718.913491 punch from 0x01c770 to 0x029b99 (0xd42a bytes) 000593 1724773718.915292 read 0x030dd5 thru 0x038a9d (0x7cc9 bytes) 000594 1724773718.919335 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000595 1724773718.973841 MIRROR_RESYNC 6 mirrors 000596 1724773719.292358 write 0x00fda3 thru 0x014796 (0x49f4 bytes) 000597 1724773719.302676 trunc from 0x038a9e to 0x0070fa (0x319a3 bytes) 000598 1724773719.308260 write 0x038a94 thru 0x03c134 (0x36a1 bytes) 000599 1724773719.309002 punch from 0x01c441 to 0x028cac (0xc86c bytes) 000600 1724773719.310674 punch from 0x007a66 to 0x016e5d (0xf3f8 bytes) 000601 1724773719.312161 read 0x00ba4b thru 0x01367d (0x7c33 bytes) 000603 1724773719.314535 fallocate 0x01d87f thru 0x023e65 (0x65e7 bytes) 000605 1724773719.316451 mapread 0x01f62b thru 0x02cf94 (0xd96a bytes) 000606 1724773719.321414 trunc from 0x03c135 to 0x025112 (0x17022 bytes) 000607 1724773719.325862 fallocate 0x016c53 thru 0x01d4d8 (0x6886 bytes) 000608 1724773719.327502 mapread 0x01064d thru 0x01ea9e (0xe452 bytes) 000609 1724773719.331350 trunc from 0x025113 to 0x031180 (0xc06e bytes) 000610 1724773719.335044 mapwrite 0x000651 thru 0x00f0f8 (0xeaa8 bytes) 000611 1724773719.344229 mapread 0x00af63 thru 0x00e6d5 (0x3773 bytes) 000612 1724773719.344768 read 0x004e06 thru 0x011af7 (0xccf2 bytes) 000613 1724773719.346588 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000615 1724773719.392503 mapwrite 0x0240a2 thru 0x031d00 (0xdc5f bytes) 000617 1724773719.411464 punch from 0x01ffd4 to 0x0201e6 (0x213 bytes) 000618 1724773719.413071 punch from 0x004c0f to 0x00c39d (0x778f bytes) 000619 1724773719.414511 fallocate 0x005359 thru 0x00c5e9 (0x7291 bytes) 000621 1724773719.416513 mapread 0x00fe70 thru 0x0156c0 (0x5851 bytes) 000623 1724773719.419723 read 0x00c47b thru 0x0158b1 (0x9437 bytes) 000624 1724773719.421475 mapread 0x00227d thru 0x009c0e (0x7992 bytes) 000625 1724773719.425486 MIRROR_RESYNC 5 mirrors 000627 1724773719.573819 mapread 0x00db61 thru 0x01683e (0x8cde bytes) 000628 1724773719.578660 mapread 0x009533 thru 0x01731d (0xddeb bytes) 000629 1724773719.580704 fallocate 0x01e6da thru 0x02e4fc (0xfe23 bytes) 000630 1724773719.589155 write 0x03e649 thru 0x03fcf5 (0x16ad bytes) 000631 1724773719.589762 mapwrite 0x016dff thru 0x020245 (0x9447 bytes) 000632 1724773719.601090 fallocate 0x024e60 thru 0x02dc0b (0x8dac bytes) 000633 1724773719.603310 mapwrite 0x02a1cd thru 0x035e7f (0xbcb3 bytes) 000635 1724773719.613180 read 0x02ca9a thru 0x033de6 (0x734d bytes) 000637 1724773719.613823 mapread 0x02e03f thru 0x03bc9d (0xdc5f bytes) 000639 1724773719.614547 punch from 0x038165 to 0x03fcf5 (0x7b91 bytes) 000643 1724773719.616935 mapwrite 0x005409 thru 0x0112e1 (0xbed9 bytes) 000645 1724773719.625596 trunc from 0x03fcf6 to 0x00ebcf (0x31126 bytes) 000646 1724773719.634776 mapread 0x006670 thru 0x00e9a1 (0x8332 bytes) 000647 1724773719.635309 write 0x033489 thru 0x034110 (0xc88 bytes) 000648 1724773719.637585 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000649 1724773719.690892 mapwrite 0x00ae8e thru 0x00d348 (0x24bb bytes) 000650 1724773719.702447 fallocate 0x027888 thru 0x028ee9 (0x1662 bytes) 000651 1724773719.704318 mapread 0x019fb7 thru 0x0214bf (0x7509 bytes) 000652 1724773719.709106 trunc from 0x034111 to 0x01ab7e (0x19592 bytes) 000653 1724773719.715138 MIRROR_RESYNC 4 mirrors 000654 1724773719.821369 punch from 0x0108e9 to 0x0143a1 (0x3ab9 bytes) 000655 1724773719.828858 write 0x001b28 thru 0x00bde8 (0xa2c1 bytes) 000656 1724773719.831306 mapwrite 0x013d34 thru 0x02392a (0xfbf7 bytes) 000657 1724773719.851806 read 0x00a490 thru 0x00f094 (0x4c05 bytes) 000658 1724773719.852255 fallocate 0x01f4ee thru 0x022cde (0x37f1 bytes) 000659 1724773719.853905 fallocate 0x03f670 thru 0x03ffff (0x990 bytes) 000660 1724773719.855579 punch from 0x0249c7 to 0x02a85c (0x5e96 bytes) 000661 1724773719.858075 MIRROR_RESYNC 4 mirrors 000662 1724773720.005401 mapread 0x018bb7 thru 0x025530 (0xc97a bytes) 000663 1724773720.012329 fallocate 0x01750e thru 0x0262b4 (0xeda7 bytes) 000664 1724773720.022148 fallocate 0x01ef79 thru 0x027e9d (0x8f25 bytes) 000665 1724773720.026823 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000667 1724773720.071910 write 0x032cf0 thru 0x03872d (0x5a3e bytes) 000668 1724773720.075574 punch from 0x0313d5 to 0x03c253 (0xae7f bytes) 000669 1724773720.080295 fallocate 0x00cc41 thru 0x00e4c5 (0x1885 bytes) 000670 1724773720.082006 read 0x025982 thru 0x02e8c4 (0x8f43 bytes) 000672 1724773720.084534 mapread 0x03182c thru 0x0380cd (0x68a2 bytes) 000673 1724773720.087102 mapwrite 0x037942 thru 0x039da0 (0x245f bytes) 000674 1724773720.092236 mapwrite 0x01ac62 thru 0x02733a (0xc6d9 bytes) 000675 1724773720.103027 fallocate 0x03bf15 thru 0x03ffff (0x40eb bytes) 000676 1724773720.106071 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000677 1724773720.168347 write 0x0266cf thru 0x02c68d (0x5fbf bytes) 000678 1724773720.173309 punch from 0x02b13f to 0x02d120 (0x1fe2 bytes) 000679 1724773720.178840 write 0x00a469 thru 0x019992 (0xf52a bytes) 000681 1724773720.185651 mapwrite 0x034c30 thru 0x038fe2 (0x43b3 bytes) 000682 1724773720.197544 MIRROR_RESYNC 2 mirrors 000683 1724773720.312751 mapread 0x007a48 thru 0x00bb53 (0x410c bytes) 000684 1724773720.317777 trunc from 0x040000 to 0x017933 (0x286cc bytes) 000685 1724773720.334798 MIRROR_EXTEND to 3 mirrors 000686 1724773720.430982 MIRROR_RESYNC 3 mirrors 000688 1724773720.512550 mapread 0x0062ce thru 0x01478a (0xe4bd bytes) 000689 1724773720.519973 write 0x017edb thru 0x024e0f (0xcf35 bytes) 000690 1724773720.531734 mapwrite 0x02c5b5 thru 0x03b282 (0xecce bytes) 000691 1724773720.553631 mapread 0x006cbe thru 0x0106bb (0x99fe bytes) 000693 1724773720.562680 mapwrite 0x02eea2 thru 0x039893 (0xa9f2 bytes) 000694 1724773720.572132 read 0x036b09 thru 0x03b282 (0x477a bytes) 000695 1724773720.574350 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000697 1724773720.637746 MIRROR_RESYNC 2 mirrors 000698 1724773720.753295 punch from 0x022bef to 0x031ef1 (0xf303 bytes) 000699 1724773720.762882 write 0x006eb4 thru 0x00c9f6 (0x5b43 bytes) 000700 1724773720.766107 read 0x004fe6 thru 0x00fd10 (0xad2b bytes) 000701 1724773720.768318 mapwrite 0x025298 thru 0x03168e (0xc3f7 bytes) 000702 1724773720.781275 write 0x00532f thru 0x00d4b2 (0x8184 bytes) 000703 1724773720.783861 MIRROR_RESYNC 2 mirrors 000704 1724773720.881172 punch from 0x01c1ca to 0x0255e2 (0x9419 bytes) 000705 1724773720.889196 trunc from 0x03b283 to 0x02da8f (0xd7f3 bytes) 000706 1724773720.898238 mapread 0x01446f thru 0x02353f (0xf0d1 bytes) 000707 1724773720.904917 trunc from 0x02da90 to 0x014539 (0x19556 bytes) 000709 1724773720.910309 write 0x01728c thru 0x01b34b (0x40c0 bytes) 000710 1724773720.911564 write 0x021038 thru 0x026f9c (0x5f65 bytes) 000711 1724773720.912368 trunc from 0x026f9d to 0x00cd6f (0x1a22d bytes) 000712 1724773720.918417 punch from 0x00807a to 0x00bdf8 (0x3d7f bytes) 000713 1724773720.921631 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000714 1724773720.954799 MIRROR_EXTEND to 2 mirrors 000715 1724773721.076135 MIRROR_RESYNC 2 mirrors 000716 1724773721.116195 mapread 0x0000e0 thru 0x00cd6f (0xcc90 bytes) 000718 1724773721.125901 MIRROR_EXTEND to 3 mirrors 000719 1724773721.282406 mapwrite 0x01f471 thru 0x01f492 (0x22 bytes) 000721 1724773721.306417 read 0x006d4c thru 0x008204 (0x14b9 bytes) 000722 1724773721.309326 mapwrite 0x005827 thru 0x00eacf (0x92a9 bytes) 000723 1724773721.322520 punch from 0x00fdc9 to 0x01412e (0x4366 bytes) 000725 1724773721.325541 read 0x01ba89 thru 0x01f492 (0x3a0a bytes) 000726 1724773721.329509 write 0x01b74a thru 0x0254b8 (0x9d6f bytes) 000727 1724773721.330373 fallocate 0x01f988 thru 0x02eb72 (0xf1eb bytes) 000728 1724773721.332577 punch from 0x009f97 to 0x00f8e2 (0x594c bytes) 000729 1724773721.334859 mapread 0x00e8eb thru 0x0153f3 (0x6b09 bytes) 000730 1724773721.338616 mapread 0x0232ab thru 0x0254b8 (0x220e bytes) 000731 1724773721.339218 write 0x025f59 thru 0x02df77 (0x801f bytes) 000732 1724773721.340157 fallocate 0x03b0ce thru 0x03ffff (0x4f32 bytes) 000733 1724773721.343058 read 0x03f3fb thru 0x03ffff (0xc05 bytes) 000734 1724773721.347325 write 0x008175 thru 0x00c411 (0x429d bytes) 000735 1724773721.348082 read 0x003edc thru 0x004001 (0x126 bytes) 000736 1724773721.348470 read 0x015f1e thru 0x01c3a1 (0x6484 bytes) 000739 1724773721.349531 fallocate 0x02f404 thru 0x036895 (0x7492 bytes) 000740 1724773721.352913 fallocate 0x003391 thru 0x012e9e (0xfb0e bytes) 000741 1724773721.354955 mapwrite 0x024001 thru 0x02a169 (0x6169 bytes) 000742 1724773721.362100 trunc from 0x040000 to 0x02aecd (0x15132 bytes) 000743 1724773721.369822 mapwrite 0x017317 thru 0x01a11e (0x2e08 bytes) 000744 1724773721.375462 read 0x007600 thru 0x017432 (0xfe33 bytes) 000745 1724773721.375983 mapwrite 0x02117c thru 0x02892f (0x77b4 bytes) 000747 1724773721.382441 mapread 0x0221d3 thru 0x0282f3 (0x6121 bytes) 000750 1724773721.383668 fallocate 0x01e100 thru 0x022cd0 (0x4bd1 bytes) 000751 1724773721.385626 fallocate 0x038966 thru 0x03958a (0xc25 bytes) 000752 1724773721.387879 punch from 0x01b00c to 0x02097c (0x5971 bytes) 000753 1724773721.389834 punch from 0x028da4 to 0x02a25e (0x14bb bytes) 000754 1724773721.391834 read 0x01cd53 thru 0x02acc7 (0xdf75 bytes) 000758 1724773721.396603 mapwrite 0x019384 thru 0x0215cb (0x8248 bytes) 000761 1724773721.406923 write 0x006dea thru 0x012685 (0xb89c bytes) 000762 1724773721.407875 fallocate 0x017e90 thru 0x01e322 (0x6493 bytes) 000764 1724773721.410777 mapread 0x02c222 thru 0x0384bc (0xc29b bytes) 000765 1724773721.413520 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000766 1724773721.494779 write 0x00d68d thru 0x014f89 (0x78fd bytes) 000767 1724773721.500161 punch from 0x02e577 to 0x037035 (0x8abf bytes) 000768 1724773721.502164 fallocate 0x00ee90 thru 0x010b7e (0x1cef bytes) 000770 1724773721.504143 mapwrite 0x003868 thru 0x00f4f6 (0xbc8f bytes) 000773 1724773721.519632 mapwrite 0x023ee6 thru 0x02abb2 (0x6ccd bytes) 000774 1724773721.532036 fallocate 0x036f90 thru 0x03c80a (0x587b bytes) 000775 1724773721.534094 write 0x03539d thru 0x03a540 (0x51a4 bytes) 000776 1724773721.537919 punch from 0x006853 to 0x009268 (0x2a16 bytes) 000778 1724773721.540848 write 0x01310e thru 0x019996 (0x6889 bytes) 000779 1724773721.543991 trunc from 0x03c80b to 0x02df20 (0xe8ea bytes) 000780 1724773721.550338 read 0x00babf thru 0x0158e1 (0x9e23 bytes) 000781 1724773721.550813 fallocate 0x038127 thru 0x03ffff (0x7ed9 bytes) 000782 1724773721.552912 punch from 0x03b5f3 to 0x03ffff (0x4a0d bytes) 000783 1724773721.554844 trunc from 0x040000 to 0x03ee81 (0x117e bytes) 000784 1724773721.560197 mapread 0x005c45 thru 0x015afe (0xfeba bytes) 000787 1724773721.563610 write 0x034bde thru 0x035f17 (0x133a bytes) 000788 1724773721.567520 punch from 0x00f73d to 0x018719 (0x8fdd bytes) 000789 1724773721.572206 read 0x02e7ba thru 0x03d959 (0xf1a0 bytes) 000790 1724773721.575016 write 0x021ff7 thru 0x02be37 (0x9e41 bytes) 000791 1724773721.575716 write 0x008a0c thru 0x010daa (0x839f bytes) 000792 1724773721.576431 fallocate 0x0227a6 thru 0x025cb6 (0x3511 bytes) 000793 1724773721.578577 read 0x03abf1 thru 0x03ee81 (0x4291 bytes) 000794 1724773721.579008 punch from 0x00ab00 to 0x0179d9 (0xceda bytes) 000795 1724773721.585371 write 0x019b09 thru 0x02681a (0xcd12 bytes) 000796 1724773721.586318 mapread 0x009ce8 thru 0x00c288 (0x25a1 bytes) 000798 1724773721.589725 mapwrite 0x03cd07 thru 0x03ffff (0x32f9 bytes) 000799 1724773721.602398 trunc from 0x040000 to 0x01b37b (0x24c84 bytes) 000800 1724773721.608791 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000801 1724773721.646886 read 0x013556 thru 0x01b37b (0x7e26 bytes) 000802 1724773721.652905 MIRROR_EXTEND to 2 mirrors 000803 1724773721.781250 punch from 0x003f64 to 0x008349 (0x43e6 bytes) 000804 1724773721.796710 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000805 1724773721.833214 mapread 0x00744b thru 0x01136e (0x9f24 bytes) 000806 1724773721.840776 trunc from 0x01b37c to 0x02a031 (0xecb6 bytes) 000807 1724773721.852285 punch from 0x0071f3 to 0x007268 (0x76 bytes) 000808 1724773721.855412 MIRROR_EXTEND to 2 mirrors 000809 1724773721.993403 write 0x026ecb thru 0x03384e (0xc984 bytes) 000812 1724773722.009917 MIRROR_RESYNC 2 mirrors 000813 1724773722.190142 mapread 0x017ba8 thru 0x01d4f6 (0x594f bytes) 000814 1724773722.196653 mapread 0x00cba2 thru 0x01674f (0x9bae bytes) 000815 1724773722.201480 MIRROR_EXTEND to 3 mirrors 000816 1724773722.359983 MIRROR_EXTEND to 4 mirrors 000817 1724773722.519178 punch from 0x02265d to 0x025fef (0x3993 bytes) 000819 1724773722.531400 mapwrite 0x019d98 thru 0x026fe4 (0xd24d bytes) 000820 1724773722.549815 trunc from 0x03384f to 0x02ee2c (0x4a22 bytes) 000821 1724773722.558516 write 0x008a8f thru 0x00e4a2 (0x5a14 bytes) 000822 1724773722.561495 mapread 0x007a04 thru 0x012c7e (0xb27b bytes) 000823 1724773722.563935 write 0x030412 thru 0x03a694 (0xa283 bytes) 000825 1724773722.565269 mapwrite 0x00f0fa thru 0x017ae1 (0x89e8 bytes) 000826 1724773722.573450 read 0x013f9c thru 0x0193c7 (0x542c bytes) 000827 1724773722.573863 mapwrite 0x0278ee thru 0x034aba (0xd1cd bytes) 000828 1724773722.585527 read 0x021e10 thru 0x023f69 (0x215a bytes) 000829 1724773722.585989 punch from 0x013447 to 0x01b95c (0x8516 bytes) 000830 1724773722.589149 mapwrite 0x003dc5 thru 0x011189 (0xd3c5 bytes) 000831 1724773722.597417 trunc from 0x03a695 to 0x00d537 (0x2d15d bytes) 000832 1724773722.601857 read 0x00b480 thru 0x00d537 (0x20b8 bytes) 000833 1724773722.602234 punch from 0x00bd12 to 0x00d537 (0x1826 bytes) 000835 1724773722.604193 write 0x004040 thru 0x014029 (0xffea bytes) 000836 1724773722.605290 read 0x010ebf thru 0x0137d2 (0x2914 bytes) 000837 1724773722.605653 punch from 0x00dc6c to 0x014029 (0x63be bytes) 000839 1724773722.610619 mapread 0x0130b5 thru 0x014029 (0xf75 bytes) 000840 1724773722.614002 mapread 0x013df0 thru 0x014029 (0x23a bytes) 000841 1724773722.614573 write 0x022a50 thru 0x02a81e (0x7dcf bytes) 000843 1724773722.615569 mapwrite 0x019479 thru 0x02660f (0xd197 bytes) 000848 1724773722.628300 write 0x0302de thru 0x030ae6 (0x809 bytes) 000849 1724773722.628929 punch from 0x00874a to 0x00c93b (0x41f2 bytes) 000851 1724773722.630891 fallocate 0x01a55a thru 0x025d84 (0xb82b bytes) 000852 1724773722.632608 mapwrite 0x00d01c thru 0x017957 (0xa93c bytes) 000853 1724773722.638893 punch from 0x002f8a to 0x00d4b9 (0xa530 bytes) 000854 1724773722.640434 mapwrite 0x011f52 thru 0x01be99 (0x9f48 bytes) 000855 1724773722.648261 trunc from 0x030ae7 to 0x001e35 (0x2ecb1 bytes) 000856 1724773722.653903 write 0x03e2d5 thru 0x03ffff (0x1d2b bytes) 000857 1724773722.654825 punch from 0x007609 to 0x007e3e (0x836 bytes) 000859 1724773722.656832 trunc from 0x040000 to 0x01e73e (0x218c1 bytes) 000860 1724773722.663195 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000861 1724773722.718214 punch from 0x00e5c7 to 0x014cc3 (0x66fd bytes) 000862 1724773722.719699 mapread 0x003e53 thru 0x005932 (0x1ae0 bytes) 000863 1724773722.725423 write 0x03cfac thru 0x03ffff (0x3054 bytes) 000865 1724773722.728253 write 0x015a9a thru 0x01bdd1 (0x6338 bytes) 000866 1724773722.731891 write 0x01b6b0 thru 0x027e6c (0xc7bd bytes) 000867 1724773722.734386 punch from 0x01dbe8 to 0x02ba25 (0xde3e bytes) 000868 1724773722.739519 mapread 0x00d1e5 thru 0x014c38 (0x7a54 bytes) 000869 1724773722.739971 mapread 0x02deb3 thru 0x030573 (0x26c1 bytes) 000870 1724773722.743566 mapwrite 0x0218e2 thru 0x02bb0a (0xa229 bytes) 000871 1724773722.757340 MIRROR_RESYNC 3 mirrors 000873 1724773722.930169 punch from 0x01fcc7 to 0x02998e (0x9cc8 bytes) 000875 1724773722.941774 punch from 0x02ddc6 to 0x0302c0 (0x24fb bytes) 000876 1724773722.943500 mapread 0x0248d6 thru 0x02af3a (0x6665 bytes) 000877 1724773722.950336 mapread 0x023d6d thru 0x027274 (0x3508 bytes) 000878 1724773722.950915 write 0x00c941 thru 0x00ef7a (0x263a bytes) 000880 1724773722.956422 mapwrite 0x0185cd thru 0x01b4e2 (0x2f16 bytes) 000881 1724773722.967035 MIRROR_RESYNC 3 mirrors 000882 1724773723.159745 MIRROR_RESYNC 3 mirrors 000883 1724773723.213039 mapwrite 0x028f19 thru 0x02f8c3 (0x69ab bytes) 000884 1724773723.257407 punch from 0x02998a to 0x034323 (0xa99a bytes) 000885 1724773723.259636 fallocate 0x0214cf thru 0x02b47b (0x9fad bytes) 000886 1724773723.261585 mapread 0x00793d thru 0x010fbd (0x9681 bytes) 000887 1724773723.268585 punch from 0x025db6 to 0x02c15b (0x63a6 bytes) 000889 1724773723.271085 mapwrite 0x0089c2 thru 0x0145a8 (0xbbe7 bytes) 000890 1724773723.279123 mapwrite 0x00f01c thru 0x0125d6 (0x35bb bytes) 000892 1724773723.286191 mapread 0x0146b8 thru 0x024591 (0xfeda bytes) 000893 1724773723.290930 write 0x019b50 thru 0x01e861 (0x4d12 bytes) 000895 1724773723.293761 punch from 0x007200 to 0x0101ab (0x8fac bytes) 000896 1724773723.297148 read 0x03658b thru 0x03ffff (0x9a75 bytes) 000897 1724773723.301052 mapread 0x0336bf thru 0x0381c2 (0x4b04 bytes) 000898 1724773723.304310 fallocate 0x01baaa thru 0x026b30 (0xb087 bytes) 000899 1724773723.308061 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000900 1724773723.370635 read 0x023375 thru 0x030b48 (0xd7d4 bytes) 000901 1724773723.373573 write 0x002814 thru 0x009d8f (0x757c bytes) 000902 1724773723.378370 fallocate 0x03fc2f thru 0x03ffff (0x3d1 bytes) 000903 1724773723.379894 fallocate 0x00e15f thru 0x00fe61 (0x1d03 bytes) 000904 1724773723.381718 read 0x022b60 thru 0x02bbda (0x907b bytes) 000906 1724773723.386263 read 0x02caf8 thru 0x02d4bc (0x9c5 bytes) 000907 1724773723.387928 MIRROR_RESYNC 2 mirrors 000908 1724773723.500359 read 0x0356b0 thru 0x03b332 (0x5c83 bytes) 000909 1724773723.503028 read 0x0091a8 thru 0x00b889 (0x26e2 bytes) 000910 1724773723.506561 punch from 0x018259 to 0x024c13 (0xc9bb bytes) 000913 1724773723.518970 write 0x034420 thru 0x03ffff (0xbbe0 bytes) 000914 1724773723.522754 mapread 0x001f04 thru 0x0105b7 (0xe6b4 bytes) 000917 1724773723.545259 punch from 0x01fcea to 0x02cc25 (0xcf3c bytes) 000918 1724773723.547825 mapread 0x036479 thru 0x038a54 (0x25dc bytes) 000919 1724773723.548424 punch from 0x00ecfc to 0x017168 (0x846d bytes) 000920 1724773723.550573 mapread 0x038eb1 thru 0x03ffff (0x714f bytes) 000922 1724773723.551509 write 0x03f7be thru 0x03ffff (0x842 bytes) 000923 1724773723.551915 mapread 0x02c765 thru 0x02f131 (0x29cd bytes) 000924 1724773723.559460 trunc from 0x040000 to 0x00c1f9 (0x33e06 bytes) 000925 1724773723.576169 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000926 1724773723.608148 punch from 0x0082f8 to 0x00c1f9 (0x3f02 bytes) 000929 1724773723.610445 mapread 0x008aeb thru 0x00c1f9 (0x370f bytes) 000930 1724773723.615711 punch from 0x007b20 to 0x00c1f9 (0x46da bytes) 000931 1724773723.617282 punch from 0x001894 to 0x009ccf (0x843c bytes) 000933 1724773723.619588 read 0x00b04e thru 0x00c1f9 (0x11ac bytes) 000935 1724773723.622198 mapread 0x0059ba thru 0x009d14 (0x435b bytes) 000936 1724773723.624974 fallocate 0x009576 thru 0x014a33 (0xb4be bytes) 000937 1724773723.626635 read 0x000a9c thru 0x00c1f9 (0xb75e bytes) 000938 1724773723.628832 mapread 0x004e4c thru 0x009347 (0x44fc bytes) 000939 1724773723.629336 punch from 0x0057e4 to 0x00c1f9 (0x6a16 bytes) 000943 1724773723.632410 read 0x00044e thru 0x00554f (0x5102 bytes) 000945 1724773723.635397 read 0x0093e7 thru 0x00c1f9 (0x2e13 bytes) 000946 1724773723.635783 write 0x039e4d thru 0x03ffff (0x61b3 bytes) 000947 1724773723.638441 write 0x00e7c3 thru 0x0144c0 (0x5cfe bytes) 000948 1724773723.641388 fallocate 0x00fd4c thru 0x012130 (0x23e5 bytes) 000949 1724773723.643008 punch from 0x0363ca to 0x03ffff (0x9c36 bytes) 000950 1724773723.648351 fallocate 0x020da1 thru 0x02c733 (0xb993 bytes) 000951 1724773723.650698 trunc from 0x040000 to 0x03b042 (0x4fbd bytes) 000953 1724773723.656319 fallocate 0x037c43 thru 0x03f411 (0x77cf bytes) 000954 1724773723.658392 punch from 0x0137d2 to 0x02078f (0xcfbe bytes) 000955 1724773723.660179 trunc from 0x03f412 to 0x0078b5 (0x37b5c bytes) 000956 1724773723.664325 mapread 0x0030be thru 0x0078b5 (0x47f8 bytes) 000957 1724773723.664896 punch from 0x004cba to 0x0078b5 (0x2bfc bytes) 000958 1724773723.666962 fallocate 0x00f58b thru 0x01cb92 (0xd608 bytes) 000959 1724773723.669699 punch from 0x00ddb1 to 0x017d88 (0x9fd8 bytes) 000960 1724773723.671255 trunc from 0x01cb93 to 0x024db9 (0x8227 bytes) 000961 1724773723.678484 write 0x01641d thru 0x0246df (0xe2c3 bytes) 000963 1724773723.682381 mapwrite 0x0304d3 thru 0x039eef (0x9a1d bytes) 000964 1724773723.712170 MIRROR_EXTEND to 2 mirrors 000965 1724773723.851525 mapread 0x0208f9 thru 0x0297e3 (0x8eeb bytes) 000967 1724773723.861507 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) 000969 1724773723.906116 punch from 0x038d3d to 0x039eef (0x11b3 bytes) 000971 1724773723.911836 fallocate 0x00f67e thru 0x015aa3 (0x6426 bytes) 000972 1724773723.913966 fallocate 0x0070d9 thru 0x010b71 (0x9a99 bytes) 000973 1724773723.916343 read 0x018611 thru 0x01ba0b (0x33fb bytes) 000974 1724773723.918036 trunc from 0x039ef0 to 0x02d49a (0xca55 bytes) 000975 1724773723.930090 mapwrite 0x02474e thru 0x0259a7 (0x125a bytes) 000976 1724773723.940628 read 0x02a681 thru 0x02d49a (0x2e1a bytes) 000977 1724773723.944154 fallocate 0x01634a thru 0x01f74f (0x9406 bytes) 000978 1724773723.946076 write 0x018602 thru 0x025423 (0xce22 bytes) 000979 1724773723.946984 mapread 0x007e20 thru 0x00e7d6 (0x69b7 bytes) 000980 1724773723.947815 write 0x036cbf thru 0x03ee38 (0x817a bytes) 000981 1724773723.948578 trunc from 0x03ee39 to 0x014e4e (0x29fea bytes) 000982 1724773723.954146 punch from 0x008806 to 0x014e21 (0xc61c bytes) 000984 1724773723.956371 punch from 0x002fce to 0x00b0a5 (0x80d8 bytes) 000985 1724773723.958837 mapwrite 0x00006b thru 0x00708a (0x7020 bytes) 000986 1724773723.975819 mapwrite 0x02575d thru 0x03311f (0xd9c3 bytes) 000988 1724773724.003217 MIRROR_EXTEND to 2 mirrors 000989 1724773724.138964 fallocate 0x029d58 thru 0x02a6b1 (0x95a bytes) 000990 1724773724.147442 read 0x02262e thru 0x0241e7 (0x1bba bytes) 000991 1724773724.150061 MIRROR_EXTEND to 3 mirrors 000992 1724773724.272384 mapwrite 0x0243ef thru 0x02e898 (0xa4aa bytes) 000993 1724773724.316062 write 0x0044e5 thru 0x013a86 (0xf5a2 bytes) 000994 1724773724.324826 mapread 0x015bc0 thru 0x023e65 (0xe2a6 bytes) 000995 1724773724.332568 MIRROR_EXTEND to 4 mirrors 000996 1724773724.461074 fallocate 0x00179b thru 0x0018b5 (0x11b bytes) 000999 1724773724.472614 fallocate 0x036886 thru 0x0375bd (0xd38 bytes) 001000 1724773724.476581 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) All operations completed A-OK! PASS 70a (16s) == sanity-flr test 200a: stress test ===================== 11:48:45 (1724773725) Starting client: oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre2 Starting client: oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3164704 .. Extending file size to 2454240 .. failed Extending file size to 8061984 .. Extending file size to 4183104 .. Extending file size to 4419328 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8143616 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8960384 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8254272 .. Extending file size to 8834976 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8987264 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9079296 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9180128 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9120608 .. done Waiting 2837 2838 2839 2840 2843 192.168.204.114@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg414-client.virtnet /mnt/lustre2 (opts:) 192.168.204.114@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg414-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (65s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 11:49:50 (1724773790) Starting client: oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre2 Starting client: oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3465888 .. Extending file size to 5561568 .. Extending file size to 3953536 .. Extending file size to 7702624 .. Extending file size to 5936480 .. Extending file size to 7517248 .. done Extending file size to 8873344 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7717216 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7812800 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8692160 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8751168 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8782784 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9206496 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9082816 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9199872 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 5857 5858 5860 5862 5864 192.168.204.114@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg414-client.virtnet /mnt/lustre2 (opts:) 192.168.204.114@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg414-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 11:50:54 (1724773854) Starting client: oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2748 1284940 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2668 1285020 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 15856 3591164 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 21712 3585308 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37568 7176472 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2748 1284940 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2668 1285020 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 15856 3591164 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 21712 3585308 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37568 7176472 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0365475 s, 57.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0431919 s, 48.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.114@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg414-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 11:50:58 (1724773858) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0249549 s, 84.0 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 11:51:00 (1724773860) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9f:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x2c0000401:0xcb:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x2c0000401:0xcb:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x2c0000401:0xcc:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000401:0xca:0x0] } PASS 203 (1s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 11:51:01 (1724773861) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0272743 s, 76.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9ee:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x2c0000400:0x9ef:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x2c0000400:0x9f0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 11:51:03 (1724773863) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0256667 s, 81.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x2c0000401:0xcd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x2c0000401:0xce:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000401:0xcb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x2c0000401:0xcd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x2c0000401:0xce:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000401:0xcb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000401:0xcc:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 11:51:05 (1724773865) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0270754 s, 77.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x2c0000401:0xcf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000401:0xcd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x2c0000401:0xcf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000401:0xcd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000401:0xce:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 11:51:28 (1724773888) Waiting for MDT destroys to complete oleg414-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg414-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg414-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0138226 s, 75.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9ef:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x2c0000400:0x9f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x2c0000400:0x9f1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9ef:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x2c0000400:0x9f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x2c0000400:0x9f1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9f0:0x0] } PASS 204d (33s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 11:52:01 (1724773921) Waiting for MDT destroys to complete oleg414-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg414-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg414-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x2c0000401:0xd0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x2c0000401:0xd1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0207307 s, 101 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x2c0000401:0xd0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x2c0000401:0xd1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000401:0xcf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x2c0000401:0xd0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x2c0000401:0xd1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000401:0xcf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (36s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 11:52:37 (1724773957) Waiting for MDT destroys to complete oleg414-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 oleg414-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg414-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool oleg414-server: Pool lustre.test_204f created Adding targets to pool oleg414-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg414-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0192207 s, 109 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x2c0000401:0xd2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000401:0xd0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x2c0000401:0xd2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000401:0xd0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg414-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg414-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg414-server: Pool lustre.test_204f destroyed PASS 204f (46s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 11:53:23 (1724774003) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 11:53:25 (1724774005) PASS 205b (1s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 11:53:26 (1724774006) oleg414-server: Pool lustre.test_206 created Adding targets to pool oleg414-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg414-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg414-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg414-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg414-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (15s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 11:53:42 (1724774022) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.018099 s, 57.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00813138 s, 129 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xfc1c93a Mirror 3: 0xfc1c93a PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 11:53:43 (1724774023) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 11:53:44 (1724774024) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 11:53:45 (1724774025) Starting client: oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.00688467 s, 0.6 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x6a0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1800000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1800000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1800000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x17e0000 Read this (4 bytes): BBBB 192.168.204.114@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg414-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 11:53:58 (1724774038) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00245134 s, 3.3 MB/s the very first read read with warm cache PASS 209b (1s) == sanity-flr test 210a: handle broken mirrored lovea ==== 11:53:59 (1724774039) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00945375 s, 111 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc0 lmm_fid: [0x200000405:0xc0:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc0 lmm_fid: [0x200000405:0xc0:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000401:0xda:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611328 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-08-27 11:54:00.000000000 -0400 Modify: 2024-08-27 11:54:00.000000000 -0400 Change: 2024-08-27 11:54:00.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 11:54:01 (1724774041) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00845962 s, 124 MB/s oleg414-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg414-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg414-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg414-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 oleg414-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg414-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg414-server: O/2c0000401/d29/dd: File not found by ext2_lookup PASS 210b (11s) == sanity-flr test complete, duration 697 sec ============ 11:54:12 (1724774052) === sanity-flr: start cleanup 11:54:13 (1724774053) === === sanity-flr: finish cleanup 11:54:14 (1724774054) ===