-----============= acceptance-small: sanity-flr ============----- Thu Mar 13 16:31:28 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg117-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 16:31:42 (1741897902) === oleg117-client.virtnet: executing check_config_client /mnt/lustre oleg117-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg117-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b5879800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b5879800.idle_timeout=debug disable quota as required oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 16:32:03 (1741897923) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7662] == sanity-flr test 0a: lfs mirror create with -N option == 16:32:04 (1741897924) PASS 0a (10s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 16:32:14 (1741897934) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 16:32:16 (1741897936) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 16:32:19 (1741897939) PASS 0d (13s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 16:32:33 (1741897953) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 16:32:35 (1741897955) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 16:32:37 (1741897957) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b5879800.stats=clear osc.lustre-OST0001-osc-ffff8800b5879800.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 16:32:40 (1741897960) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 16:32:44 (1741897964) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 16:32:49 (1741897969) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 16:32:50 (1741897970) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (5s) == sanity-flr test 2: create components from existing files ========================================================== 16:32:55 (1741897975) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 16:32:59 (1741897979) PASS 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 16:33:03 (1741897983) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td lfs extend: cannot get the last init comp error: lfs extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (5s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 16:33:08 (1741897988) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 16:33:13 (1741897993) PASS 7 (4s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 16:33:17 (1741897997) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.74307 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0823258 s, 12.7 MB/s PASS 21 (6s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 16:33:23 (1741898003) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0203068 s, 51.6 MB/s PASS 22 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 16:33:27 (1741898007) Stopping /mnt/lustre-ost1 (opts:) on oleg117-server oleg117-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5879800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5879800.ost_server_uuid in DISCONN state after 0 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg117-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg117-server oleg117-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5879800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5879800.ost_server_uuid in DISCONN state after 2 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg117-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (38s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 16:34:05 (1741898045) 11+0 records in 11+0 records out 11534336 bytes (12 MB) copied, 0.282035 s, 40.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg117-server oleg117-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5879800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5879800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg117-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (18s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 16:34:23 (1741898063) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg117-server oleg117-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5879800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5879800.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg117-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg117-server oleg117-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5879800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5879800.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg117-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (38s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 16:35:02 (1741898102) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 1.76269 s, 29.7 MB/s -rw-r--r-- 1 root root 52428800 Mar 13 16:35 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.521s user 0m0.000s sys 0m0.175s Stopping /mnt/lustre-ost1 (opts:) on oleg117-server oleg117-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5879800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5879800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.694s user 0m0.000s sys 0m0.194s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg117-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg117-server oleg117-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5879800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5879800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.615s user 0m0.000s sys 0m0.171s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg117-server: oleg117-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg117-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg117-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b5879800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5879800.ost_server_uuid in FULL state after 0 sec PASS 33b (41s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 16:35:43 (1741898143) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 16:35:45 (1741898145) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 16:35:46 (1741898146) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 16:35:48 (1741898148) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 16:35:52 (1741898152) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.253237 s, 49.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (6s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 16:35:57 (1741898157) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0203162 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0224695 s, 46.7 MB/s PASS 36b (4s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 16:36:01 (1741898161) PASS 36c (5s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 16:36:06 (1741898166) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00192599 s, 532 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00138875 s, 737 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00120116 s, 853 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00171594 s, 597 kB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 16:36:10 (1741898170) create test files with size 15755353 .. 42c5824b841757d83d8d419106804e8a - 3b46d529efaa62e14f046cc36e5f3087 - d7cab99f2068feb3228da08bc2139998 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg117-client.virtnet /mnt/lustre (opts:) Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre PASS 37 (10s) == sanity-flr test 38: resync ============================ 16:36:20 (1741898180) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (5s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 16:36:25 (1741898185) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 16:36:28 (1741898188) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0450216 s, 23.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x37:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x38:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0205256 s, 51.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3d:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 16:36:33 (1741898193) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0509144 s, 41.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.16393 s, 25.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.131052 s, 16.0 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (6s) == sanity-flr test 42: lfs mirror verify ================= 16:36:38 (1741898198) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 16:36:40 (1741898200) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 16:36:41 (1741898201) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.030659 s, 34.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 16:36:45 (1741898205) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.16106 s, 19.5 MB/s PASS 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 16:36:49 (1741898209) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x42:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x47:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x47:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 16:36:53 (1741898213) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.127035 s, 82.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg117-client.virtnet /mnt/lustre (opts:) Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (15s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 16:37:08 (1741898228) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800aa76f000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.203937 s, 51.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg117-client.virtnet /mnt/lustre (opts:) Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800aa769000.xattr_cache=1 PASS 44e (8s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 16:37:16 (1741898236) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (15s) == sanity-flr test 46: Verify setstripe --copy option ==== 16:37:32 (1741898252) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (15s) == sanity-flr test 47: Verify mirror obj alloc =========== 16:37:47 (1741898267) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 16:37:49 (1741898269) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.131677 s, 23.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0865587 s, 36.3 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: e2f1dd58ba3ab742ddc32d388486306c - mirror 1 checksum: 452d656ecb1099c2e8d4a84a969a7712 - mirror 2 checksum: e2f1dd58ba3ab742ddc32d388486306c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Mar 13 16:37:51 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4c:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Mar 13 16:37:51 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: e2f1dd58ba3ab742ddc32d388486306c - mirror 1 checksum: 452d656ecb1099c2e8d4a84a969a7712 - mirror 2 checksum: 452d656ecb1099c2e8d4a84a969a7712 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Mar 13 16:37:51 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4c:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Mar 13 16:37:51 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } PASS 48 (4s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 16:37:53 (1741898273) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0251197 s, 41.7 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 529408.. 530431: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (3s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 16:37:56 (1741898276) Failing mds1 on oleg117-server Stopping /mnt/lustre-mds1 (opts:) on oleg117-server 16:38:04 (1741898284) shut down facet: mds1 facet_host: oleg117-server facet_failover_host: oleg117-server Failover mds1 to oleg117-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 16:38:18 (1741898298) targets are mounted 16:38:18 (1741898298) facet_failover done oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0145955 s, 71.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } PASS 50A (31s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 16:38:27 (1741898307) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0200815 s, 1.0 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0165284 s, 1.2 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 16:38:31 (1741898311) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0180823 s, 1.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00583355 s, 3.5 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 16:38:34 (1741898314) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00324184 s, 5.1 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 16:38:36 (1741898316) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0202839 s, 1.0 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 16:38:39 (1741898319) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00228966 s, 1.8 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00891998 s, 459 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 16:38:41 (1741898321) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) PASS 60b (8s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 16:38:49 (1741898329) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (21s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 16:39:10 (1741898350) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 16:39:28 (1741898368) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 16:39:47 (1741898387) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 16:39:51 (1741898391) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 26467 mirror split pid 26469 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4866: 26467 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4866: 26469 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 16:40:54 (1741898454) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 8454 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0xe35e 000001 1741898455.056981 trunc from 00000000 to 0x00e35d (0xe35e bytes) 000002 1741898455.065684 punch from 0x00763f to 0x009e32 (0x27f4 bytes) 000003 1741898455.069284 mapwrite 0x0302dd thru 0x03f8e3 (0xf607 bytes) 000004 1741898455.133249 mapread 0x03211e thru 0x03f8e3 (0xd7c6 bytes) 000005 1741898455.137707 MIRROR_EXTEND to 2 mirrors fallocating to largest ever: 0x2ec84 000006 1741898455.429291 fallocate 0x021392 thru 0x02ec83 (0xd8f2 bytes) 000010 1741898455.452116 fallocate 0x0240cd thru 0x02df6b (0x9e9f bytes) 000011 1741898455.456162 mapwrite 0x002b80 thru 0x0074a8 (0x4929 bytes) 000013 1741898455.487846 punch from 0x02c1e8 to 0x02cf86 (0xd9f bytes) fallocating to largest ever: 0x40000 000015 1741898455.494032 fallocate 0x03e71e thru 0x03ffff (0x18e2 bytes) 000016 1741898455.502795 read 0x010804 thru 0x01535b (0x4b58 bytes) 000018 1741898455.511331 mapread 0x037e81 thru 0x0382c1 (0x441 bytes) 000019 1741898455.521716 fallocate 0x029b5c thru 0x039157 (0xf5fc bytes) 000020 1741898455.526560 mapread 0x03cc6b thru 0x03ffff (0x3395 bytes) 000021 1741898455.527550 mapread 0x015b80 thru 0x01bfe7 (0x6468 bytes) 000022 1741898455.532895 mapread 0x021d11 thru 0x02d1c6 (0xb4b6 bytes) 000025 1741898455.543299 read 0x02c722 thru 0x0363e3 (0x9cc2 bytes) 000027 1741898455.544297 read 0x00ab68 thru 0x00c31d (0x17b6 bytes) 000028 1741898455.545410 read 0x02fd26 thru 0x031bd2 (0x1ead bytes) 000029 1741898455.546310 mapwrite 0x020b56 thru 0x027812 (0x6cbd bytes) 000030 1741898455.565469 read 0x03a080 thru 0x03b62e (0x15af bytes) 000031 1741898455.566012 fallocate 0x03fd27 thru 0x03ffff (0x2d9 bytes) 000032 1741898455.568424 mapwrite 0x022c43 thru 0x02cd6b (0xa129 bytes) 000033 1741898455.586353 read 0x006cc5 thru 0x009828 (0x2b64 bytes) 000034 1741898455.586880 mapwrite 0x0013a8 thru 0x00ecd4 (0xd92d bytes) 000035 1741898455.605231 mapwrite 0x027f35 thru 0x02c5f9 (0x46c5 bytes) 000036 1741898455.619575 trunc from 0x040000 to 0x034514 (0xbaeb bytes) 000037 1741898455.648804 mapread 0x01b47f thru 0x02849e (0xd020 bytes) 000039 1741898455.657303 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000040 1741898455.773230 fallocate 0x02117d thru 0x02d523 (0xc3a7 bytes) 000041 1741898455.778900 MIRROR_EXTEND to 2 mirrors 000043 1741898456.035638 fallocate 0x02c44e thru 0x033731 (0x72e4 bytes) 000044 1741898456.061691 fallocate 0x028844 thru 0x02e70f (0x5ecc bytes) 000045 1741898456.068833 write 0x03bcf0 thru 0x03df3b (0x224c bytes) 000046 1741898456.075435 punch from 0x003b4f to 0x006948 (0x2dfa bytes) 000047 1741898456.079047 mapread 0x012b02 thru 0x01f814 (0xcd13 bytes) 000048 1741898456.099561 write 0x00703b thru 0x013bbe (0xcb84 bytes) 000049 1741898456.110180 trunc from 0x03df3c to 0x0180d0 (0x25e6b bytes) 000050 1741898456.139199 mapwrite 0x029008 thru 0x02e56f (0x5568 bytes) 000052 1741898456.172878 fallocate 0x03a042 thru 0x03de44 (0x3e03 bytes) 000053 1741898456.176209 read 0x001b8e thru 0x00cdaa (0xb21d bytes) 000055 1741898456.177638 read 0x01b88d thru 0x01bf84 (0x6f8 bytes) 000056 1741898456.185689 read 0x009e44 thru 0x00ff77 (0x6134 bytes) 000057 1741898456.186182 fallocate 0x031c6e thru 0x0347c4 (0x2b57 bytes) 000058 1741898456.189154 mapread 0x010edd thru 0x01e289 (0xd3ad bytes) 000060 1741898456.190325 mapwrite 0x020e48 thru 0x02b26f (0xa428 bytes) 000062 1741898456.204431 trunc from 0x03de45 to 0x02eb23 (0xf321 bytes) 000063 1741898456.210659 fallocate 0x01acb4 thru 0x029a6c (0xedb9 bytes) 000064 1741898456.213869 write 0x01e6bf thru 0x0213fc (0x2d3e bytes) 000065 1741898456.215153 fallocate 0x00d477 thru 0x01aa28 (0xd5b2 bytes) 000066 1741898456.218825 read 0x011160 thru 0x012249 (0x10ea bytes) 000067 1741898456.221384 MIRROR_EXTEND to 3 mirrors 000068 1741898456.488390 trunc from 0x02eb24 to 0x01f6a8 (0xf47b bytes) 000070 1741898456.529892 fallocate 0x013287 thru 0x0177f9 (0x4573 bytes) 000071 1741898456.532839 write 0x006a58 thru 0x00c2df (0x5888 bytes) 000072 1741898456.537270 mapwrite 0x003cd2 thru 0x00bcc4 (0x7ff3 bytes) 000075 1741898456.553843 trunc from 0x01f6a9 to 0x01948f (0x6219 bytes) 000076 1741898456.567129 fallocate 0x003bab thru 0x00dc9b (0xa0f1 bytes) 000078 1741898456.570411 write 0x0390b2 thru 0x03cffb (0x3f4a bytes) 000079 1741898456.571696 read 0x01286d thru 0x01a72b (0x7ebf bytes) 000080 1741898456.577172 punch from 0x034584 to 0x03579b (0x1218 bytes) 000081 1741898456.580308 punch from 0x027d60 to 0x031235 (0x94d6 bytes) 000082 1741898456.585790 write 0x03e753 thru 0x03ffff (0x18ad bytes) 000083 1741898456.586639 punch from 0x002175 to 0x009257 (0x70e3 bytes) 000084 1741898456.593164 MIRROR_RESYNC 3 mirrors 000085 1741898457.050001 write 0x02cb14 thru 0x03937b (0xc868 bytes) 000086 1741898457.073145 read 0x00b53f thru 0x015237 (0x9cf9 bytes) 000087 1741898457.077432 punch from 0x035e88 to 0x03b67c (0x57f5 bytes) 000088 1741898457.086922 trunc from 0x040000 to 0x014fdf (0x2b020 bytes) 000089 1741898457.109546 punch from 0x001d10 to 0x00a75e (0x8a4f bytes) 000090 1741898457.113405 trunc from 0x014fe0 to 0x02f4b9 (0x1a4da bytes) 000091 1741898457.126126 punch from 0x009e72 to 0x016181 (0xc310 bytes) 000092 1741898457.128768 punch from 0x007e17 to 0x011b96 (0x9d80 bytes) 000093 1741898457.134919 mapread 0x01b09c thru 0x01f483 (0x43e8 bytes) 000094 1741898457.143257 write 0x0088e9 thru 0x010cb5 (0x83cd bytes) 000095 1741898457.152539 fallocate 0x02c0ba thru 0x033e16 (0x7d5d bytes) 000096 1741898457.159800 trunc from 0x033e17 to 0x023e35 (0xffe1 bytes) 000097 1741898457.172735 read 0x010ace thru 0x01996e (0x8ea1 bytes) 000098 1741898457.180927 write 0x033639 thru 0x03d1ec (0x9bb4 bytes) 000099 1741898457.182989 mapread 0x030a5a thru 0x03d1ec (0xc793 bytes) 000100 1741898457.194224 mapwrite 0x027b01 thru 0x02dbf2 (0x60f2 bytes) 000101 1741898457.227201 punch from 0x006d54 to 0x012dd1 (0xc07e bytes) 000102 1741898457.230741 fallocate 0x01ee67 thru 0x02d301 (0xe49b bytes) 000104 1741898457.241402 MIRROR_EXTEND to 4 mirrors 000105 1741898457.476231 fallocate 0x029578 thru 0x038e3b (0xf8c4 bytes) 000106 1741898457.495170 trunc from 0x03d1ed to 0x01d8a4 (0x1f948 bytes) 000107 1741898457.518019 fallocate 0x033567 thru 0x03546d (0x1f07 bytes) 000108 1741898457.523685 MIRROR_EXTEND to 5 mirrors 000109 1741898457.743049 fallocate 0x02c1c3 thru 0x02ce9c (0xcda bytes) 000110 1741898457.758713 trunc from 0x02ce9d to 0x01c2c6 (0x10bd6 bytes) 000111 1741898457.773013 write 0x00fcd2 thru 0x0191dc (0x950b bytes) 000113 1741898457.777210 write 0x00f5a0 thru 0x01af6b (0xb9cc bytes) 000115 1741898457.783063 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000116 1741898457.899198 write 0x026414 thru 0x034d03 (0xe8f0 bytes) 000118 1741898457.903711 punch from 0x034b7d to 0x034d03 (0x187 bytes) 000119 1741898457.912840 punch from 0x017b2a to 0x01dfdc (0x64b3 bytes) 000120 1741898457.918629 mapread 0x012b46 thru 0x016b03 (0x3fbe bytes) 000121 1741898457.939705 mapwrite 0x0048c7 thru 0x007dcd (0x3507 bytes) 000122 1741898457.954266 fallocate 0x02111c thru 0x02cd6f (0xbc54 bytes) 000123 1741898457.959305 trunc from 0x034d04 to 0x00831f (0x2c9e4 bytes) 000124 1741898457.980080 read 0x000b3f thru 0x0048a0 (0x3d62 bytes) 000126 1741898457.989208 punch from 0x00524e to 0x00831f (0x30d2 bytes) 000127 1741898457.992898 mapread 0x00133a thru 0x00831f (0x6fe6 bytes) 000130 1741898458.005312 write 0x0200f4 thru 0x023ff3 (0x3f00 bytes) 000131 1741898458.007799 trunc from 0x023ff4 to 0x002900 (0x216f3 bytes) 000133 1741898458.018790 trunc from 0x002901 to 0x031e69 (0x2f569 bytes) 000134 1741898458.033494 write 0x02a2bb thru 0x038b4d (0xe893 bytes) 000135 1741898458.041328 trunc from 0x038b4e to 0x0263f6 (0x12757 bytes) 000136 1741898458.051333 mapwrite 0x02d5e1 thru 0x0381d3 (0xabf3 bytes) 000137 1741898458.075650 read 0x01dd2e thru 0x023b93 (0x5e66 bytes) 000139 1741898458.082692 mapread 0x02e32c thru 0x03235b (0x4030 bytes) 000140 1741898458.083800 write 0x03f103 thru 0x03fc18 (0xb16 bytes) 000141 1741898458.090230 mapwrite 0x01a308 thru 0x02295c (0x8655 bytes) 000142 1741898458.099363 write 0x02b666 thru 0x0347f4 (0x918f bytes) 000143 1741898458.101533 trunc from 0x03fc19 to 0x00152f (0x3e6e9 bytes) 000144 1741898458.110304 mapwrite 0x030c63 thru 0x0332ef (0x268d bytes) 000145 1741898458.130926 punch from 0x02faa3 to 0x0332ef (0x384d bytes) 000146 1741898458.133336 mapwrite 0x008524 thru 0x00f1f7 (0x6cd4 bytes) 000147 1741898458.157731 mapread 0x02e1e5 thru 0x0332ef (0x510b bytes) 000149 1741898458.173484 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000151 1741898458.278207 mapwrite 0x033865 thru 0x03d34e (0x9aea bytes) 000152 1741898458.329825 read 0x0018f6 thru 0x00753d (0x5c48 bytes) 000154 1741898458.339421 MIRROR_RESYNC 3 mirrors 000155 1741898458.742855 MIRROR_EXTEND to 4 mirrors 000156 1741898459.019723 fallocate 0x009f87 thru 0x00dfca (0x4044 bytes) 000157 1741898459.038856 read 0x0062b6 thru 0x015226 (0xef71 bytes) 000158 1741898459.045213 read 0x015bbd thru 0x01b18f (0x55d3 bytes) 000160 1741898459.056783 read 0x01fe72 thru 0x029347 (0x94d6 bytes) 000161 1741898459.057241 mapread 0x022b4c thru 0x02a9b2 (0x7e67 bytes) 000162 1741898459.062134 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000163 1741898459.174038 mapwrite 0x02670d thru 0x029bc9 (0x34bd bytes) 000164 1741898459.196496 read 0x03af35 thru 0x03d34e (0x241a bytes) 000165 1741898459.200121 mapread 0x03abe9 thru 0x03d34e (0x2766 bytes) 000167 1741898459.201475 mapread 0x034946 thru 0x03d34e (0x8a09 bytes) 000168 1741898459.207335 mapread 0x03b468 thru 0x03d34e (0x1ee7 bytes) 000169 1741898459.213747 MIRROR_EXTEND to 4 mirrors 000170 1741898459.490429 trunc from 0x03d34f to 0x02d711 (0xfc3d bytes) 000171 1741898459.542805 fallocate 0x019fc7 thru 0x01bcba (0x1cf4 bytes) 000172 1741898459.548245 fallocate 0x0002cc thru 0x001db8 (0x1aed bytes) 000173 1741898459.553264 punch from 0x02c797 to 0x02d711 (0xf7b bytes) 000174 1741898459.555926 read 0x00db88 thru 0x013a39 (0x5eb2 bytes) 000175 1741898459.559093 mapread 0x029c6b thru 0x02d711 (0x3aa7 bytes) 000176 1741898459.572507 MIRROR_EXTEND to 5 mirrors 000177 1741898459.824791 write 0x016a35 thru 0x0232a0 (0xc86c bytes) 000178 1741898459.851829 MIRROR_EXTEND to 6 mirrors 000179 1741898460.163272 MIRROR_EXTEND to 7 mirrors 000180 1741898460.439258 fallocate 0x0010db thru 0x0030ae (0x1fd4 bytes) 000182 1741898460.463247 MIRROR_RESYNC 7 mirrors 000183 1741898461.276381 MIRROR_EXTEND to 8 mirrors 000184 1741898461.521422 fallocate 0x02fe53 thru 0x03b5aa (0xb758 bytes) 000185 1741898461.540215 write 0x02c845 thru 0x035ff6 (0x97b2 bytes) 000186 1741898461.550246 mapread 0x00b4b3 thru 0x0159e4 (0xa532 bytes) 000188 1741898461.560309 mapwrite 0x0370ba thru 0x03ffff (0x8f46 bytes) 000190 1741898461.594619 write 0x01ec46 thru 0x0270ff (0x84ba bytes) 000191 1741898461.602225 write 0x0039df thru 0x011ba0 (0xe1c2 bytes) 000192 1741898461.610248 read 0x02ec01 thru 0x03a95e (0xbd5e bytes) 000193 1741898461.611235 mapwrite 0x006168 thru 0x00b84f (0x56e8 bytes) 000194 1741898461.625884 MIRROR_RESYNC 8 mirrors 000196 1741898462.283292 write 0x00e3ec thru 0x01e3aa (0xffbf bytes) 000197 1741898462.310529 mapread 0x011b0c thru 0x0214cd (0xf9c2 bytes) 000198 1741898462.316719 write 0x01a196 thru 0x020333 (0x619e bytes) 000199 1741898462.317858 mapwrite 0x00336c thru 0x011b6c (0xe801 bytes) 000200 1741898462.343625 mapwrite 0x02775d thru 0x034a4e (0xd2f2 bytes) 000201 1741898462.374092 MIRROR_EXTEND to 9 mirrors 000202 1741898462.561351 trunc from 0x040000 to 0x014ee4 (0x2b11b bytes) 000203 1741898462.590547 punch from 0x0124e7 to 0x014ee4 (0x29fe bytes) 000204 1741898462.593537 punch from 0x010892 to 0x014ee4 (0x4653 bytes) 000205 1741898462.596199 fallocate 0x02bc43 thru 0x033390 (0x774e bytes) 000207 1741898462.599725 write 0x01f36c thru 0x026dd6 (0x7a6b bytes) 000208 1741898462.606852 trunc from 0x033391 to 0x03886b (0x54db bytes) 000209 1741898462.613750 trunc from 0x03886c to 0x00a06d (0x2e7fe bytes) 000210 1741898462.622738 write 0x002fba thru 0x0031f8 (0x23f bytes) 000211 1741898462.631118 mapwrite 0x01fd7a thru 0x02714b (0x73d2 bytes) 000212 1741898462.657794 trunc from 0x02714c to 0x0013cc (0x25d7f bytes) 000214 1741898462.669168 mapread 0x000345 thru 0x0013cc (0x1088 bytes) 000215 1741898462.675626 mapwrite 0x037b8f thru 0x03b7e8 (0x3c5a bytes) 000216 1741898462.703409 mapwrite 0x00b573 thru 0x016006 (0xaa94 bytes) 000217 1741898462.722205 punch from 0x00c28c to 0x0165c6 (0xa33b bytes) 000218 1741898462.724542 mapread 0x036ffc thru 0x03b7e8 (0x47ed bytes) 000219 1741898462.729935 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000220 1741898462.827659 mapread 0x037a01 thru 0x03b7e8 (0x3de8 bytes) 000221 1741898462.841798 mapwrite 0x022e76 thru 0x026746 (0x38d1 bytes) 000222 1741898462.860321 mapread 0x01055a thru 0x015c01 (0x56a8 bytes) 000223 1741898462.869312 mapwrite 0x031fcd thru 0x035d15 (0x3d49 bytes) 000226 1741898462.886040 write 0x009e2e thru 0x00fe09 (0x5fdc bytes) 000227 1741898462.893569 read 0x0085d7 thru 0x0106a8 (0x80d2 bytes) 000229 1741898462.894289 read 0x038a0e thru 0x03b7e8 (0x2ddb bytes) 000230 1741898462.897567 read 0x02276b thru 0x024a86 (0x231c bytes) 000231 1741898462.898071 fallocate 0x033e06 thru 0x03ffff (0xc1fa bytes) 000232 1741898462.907361 punch from 0x028751 to 0x02fe8c (0x773c bytes) 000233 1741898462.911350 trunc from 0x040000 to 0x02c298 (0x13d67 bytes) 000234 1741898462.923738 punch from 0x0190ee to 0x0203b8 (0x72cb bytes) 000236 1741898462.931540 punch from 0x0075eb to 0x010ba7 (0x95bd bytes) 000237 1741898462.940983 write 0x0199a4 thru 0x019fce (0x62b bytes) 000238 1741898462.947975 fallocate 0x032cc3 thru 0x03b2bb (0x85f9 bytes) 000239 1741898462.950325 mapwrite 0x026050 thru 0x02cdb6 (0x6d67 bytes) 000240 1741898462.959182 write 0x0183ac thru 0x022499 (0xa0ee bytes) 000241 1741898462.960298 punch from 0x035ee2 to 0x03b2bb (0x53da bytes) 000242 1741898462.962685 mapwrite 0x00d805 thru 0x011304 (0x3b00 bytes) 000244 1741898462.974052 mapwrite 0x005c91 thru 0x008231 (0x25a1 bytes) 000247 1741898462.983036 read 0x01b220 thru 0x0221ae (0x6f8f bytes) 000249 1741898462.984410 punch from 0x0192d2 to 0x025692 (0xc3c1 bytes) 000250 1741898462.987592 mapread 0x025cab thru 0x02e4c2 (0x8818 bytes) 000251 1741898462.998319 read 0x01701a thru 0x01ad5e (0x3d45 bytes) 000252 1741898463.000393 mapread 0x01839e thru 0x01ca74 (0x46d7 bytes) 000253 1741898463.001290 trunc from 0x03b2bc to 0x035962 (0x5959 bytes) 000254 1741898463.014775 read 0x028def thru 0x0342ce (0xb4e0 bytes) 000255 1741898463.015331 fallocate 0x034537 thru 0x03ffff (0xbac9 bytes) 000256 1741898463.017730 read 0x01267f thru 0x018141 (0x5ac3 bytes) 000257 1741898463.018315 fallocate 0x025818 thru 0x028af0 (0x32d9 bytes) 000258 1741898463.022416 MIRROR_EXTEND to 9 mirrors 000259 1741898463.299336 fallocate 0x0185d1 thru 0x02809c (0xfacc bytes) 000260 1741898463.316388 fallocate 0x00394d thru 0x011a05 (0xe0b9 bytes) 000261 1741898463.324531 MIRROR_EXTEND to 10 mirrors 000262 1741898463.629423 MIRROR_EXTEND to 11 mirrors 000263 1741898463.901045 trunc from 0x040000 to 0x01a76e (0x25891 bytes) 000264 1741898463.936370 trunc from 0x01a76f to 0x03236f (0x17c01 bytes) 000265 1741898463.947111 mapwrite 0x02d5c8 thru 0x034245 (0x6c7e bytes) 000266 1741898463.992389 write 0x031c7a thru 0x03e232 (0xc5b9 bytes) 000267 1741898463.994499 trunc from 0x03e233 to 0x038fcb (0x5267 bytes) 000268 1741898464.002724 mapwrite 0x007962 thru 0x00d387 (0x5a26 bytes) 000269 1741898464.021548 mapread 0x006f08 thru 0x00b3c7 (0x44c0 bytes) 000270 1741898464.022475 read 0x022f00 thru 0x02768f (0x4790 bytes) 000271 1741898464.030524 MIRROR_EXTEND to 12 mirrors 000272 1741898464.379683 mapwrite 0x00c0eb thru 0x00c47c (0x392 bytes) 000274 1741898464.438876 read 0x00c1ae thru 0x018662 (0xc4b5 bytes) 000275 1741898464.444757 fallocate 0x00bcac thru 0x00fc93 (0x3fe8 bytes) 000276 1741898464.449485 trunc from 0x038fcc to 0x0321c4 (0x6e07 bytes) 000278 1741898464.480501 fallocate 0x036262 thru 0x037c18 (0x19b7 bytes) 000279 1741898464.484249 mapread 0x01d62a thru 0x0253f5 (0x7dcc bytes) 000280 1741898464.495748 write 0x03719c thru 0x03ffff (0x8e64 bytes) 000281 1741898464.498093 fallocate 0x03941f thru 0x03ffff (0x6be1 bytes) 000282 1741898464.506325 mapread 0x0083bd thru 0x00c5d2 (0x4216 bytes) 000283 1741898464.515709 read 0x02e0dd thru 0x03b08c (0xcfb0 bytes) 000284 1741898464.524016 trunc from 0x040000 to 0x037246 (0x8db9 bytes) 000286 1741898464.539783 fallocate 0x030afd thru 0x037157 (0x665b bytes) 000289 1741898464.546767 mapread 0x006349 thru 0x007fee (0x1ca6 bytes) 000291 1741898464.551962 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000292 1741898464.694176 fallocate 0x02d603 thru 0x037365 (0x9d63 bytes) 000293 1741898464.697956 punch from 0x0146c4 to 0x01fe60 (0xb79d bytes) 000294 1741898464.701117 trunc from 0x037366 to 0x02721a (0x1014b bytes) 000295 1741898464.717345 punch from 0x018666 to 0x02721a (0xebb5 bytes) 000296 1741898464.719710 mapwrite 0x020954 thru 0x027215 (0x68c2 bytes) 000297 1741898464.755376 MIRROR_EXTEND to 12 mirrors 000298 1741898465.081168 write 0x00b2f5 thru 0x019b62 (0xe86e bytes) 000299 1741898465.115607 fallocate 0x03be0e thru 0x03c5d0 (0x7c3 bytes) 000300 1741898465.119229 mapwrite 0x03710b thru 0x03ffff (0x8ef5 bytes) 000301 1741898465.189065 mapread 0x008d33 thru 0x00932b (0x5f9 bytes) 000302 1741898465.200791 write 0x01ad8c thru 0x0269ba (0xbc2f bytes) 000304 1741898465.217382 MIRROR_RESYNC 12 mirrors 000305 1741898466.642607 mapread 0x01f279 thru 0x022b08 (0x3890 bytes) 000306 1741898466.666065 fallocate 0x014057 thru 0x0235f9 (0xf5a3 bytes) 000307 1741898466.711799 MIRROR_RESYNC 12 mirrors 000310 1741898468.569404 trunc from 0x040000 to 0x0268ef (0x19710 bytes) 000312 1741898468.604325 punch from 0x00b637 to 0x010c21 (0x55eb bytes) 000313 1741898468.613265 punch from 0x01a5dd to 0x01b6f7 (0x111b bytes) 000314 1741898468.620404 mapread 0x01797f thru 0x017b26 (0x1a8 bytes) 000315 1741898468.635353 MIRROR_EXTEND to 13 mirrors 000316 1741898469.041420 read 0x0195c5 thru 0x01d754 (0x4190 bytes) 000317 1741898469.051948 fallocate 0x020512 thru 0x02f887 (0xf376 bytes) 000318 1741898469.086298 MIRROR_EXTEND to 14 mirrors 000319 1741898469.434506 MIRROR_EXTEND to 15 mirrors 000320 1741898469.735899 trunc from 0x0268f0 to 0x016231 (0x106be bytes) 000321 1741898469.773624 MIRROR_RESYNC 15 mirrors 000322 1741898471.613134 mapread 0x00fcfe thru 0x016231 (0x6534 bytes) 000323 1741898471.624197 trunc from 0x016232 to 0x02b458 (0x15227 bytes) 000324 1741898471.654791 mapread 0x005ae3 thru 0x00ae41 (0x535f bytes) 000326 1741898471.661353 mapwrite 0x007047 thru 0x0132bb (0xc275 bytes) 000327 1741898471.684626 mapread 0x028b09 thru 0x02b458 (0x2950 bytes) 000328 1741898471.692119 MIRROR_SPLIT mirror 1 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000331 1741898471.828282 mapread 0x016135 thru 0x022128 (0xbff4 bytes) 000332 1741898471.860424 trunc from 0x02b459 to 0x00124f (0x2a209 bytes) 000333 1741898471.884168 MIRROR_SPLIT mirror 12 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000334 1741898472.031872 MIRROR_EXTEND to 14 mirrors 000335 1741898472.263406 fallocate 0x01c8ba thru 0x026b08 (0xa24f bytes) 000336 1741898472.284161 write 0x003772 thru 0x013537 (0xfdc6 bytes) 000337 1741898472.288845 read 0x00a25c thru 0x013537 (0x92dc bytes) 000338 1741898472.289352 fallocate 0x02dcb5 thru 0x03b390 (0xd6dc bytes) 000339 1741898472.293560 mapread 0x012b5b thru 0x013537 (0x9dd bytes) 000340 1741898472.294823 mapread 0x00ea7c thru 0x013537 (0x4abc bytes) 000342 1741898472.295915 write 0x00c78a thru 0x00df2c (0x17a3 bytes) 000343 1741898472.296514 trunc from 0x013538 to 0x02c5c9 (0x19092 bytes) 000344 1741898472.312180 fallocate 0x002c20 thru 0x00d6f4 (0xaad5 bytes) 000345 1741898472.314985 mapwrite 0x022de6 thru 0x023f05 (0x1120 bytes) 000346 1741898472.340629 write 0x00fcc1 thru 0x01ddb3 (0xe0f3 bytes) 000347 1741898472.344347 trunc from 0x02c5ca to 0x016714 (0x15eb5 bytes) 000349 1741898472.356584 punch from 0x004b1d to 0x006ced (0x21d1 bytes) 000350 1741898472.359876 write 0x0392d9 thru 0x039b2d (0x855 bytes) 000351 1741898472.360997 read 0x01689b thru 0x021b12 (0xb278 bytes) 000352 1741898472.372720 mapwrite 0x011972 thru 0x01473f (0x2dce bytes) 000354 1741898472.386353 read 0x035185 thru 0x039b2d (0x49a9 bytes) 000355 1741898472.387890 mapread 0x006ced thru 0x00b85f (0x4b73 bytes) 000356 1741898472.389070 trunc from 0x039b2e to 0x01623a (0x238f3 bytes) 000357 1741898472.404775 MIRROR_SPLIT mirror 4 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000358 1741898472.549464 write 0x034eac thru 0x03ffff (0xb154 bytes) 000359 1741898472.556531 MIRROR_RESYNC 13 mirrors 000360 1741898474.078148 mapwrite 0x028fe1 thru 0x038f19 (0xff39 bytes) 000361 1741898474.181001 write 0x0351cf thru 0x03ffff (0xae31 bytes) 000362 1741898474.184396 mapread 0x02bfb3 thru 0x02e7fa (0x2848 bytes) 000365 1741898474.188686 mapread 0x01519f thru 0x0157a4 (0x606 bytes) 000366 1741898474.196778 read 0x019a5d thru 0x0217fc (0x7da0 bytes) 000367 1741898474.205078 read 0x00cddf thru 0x00d741 (0x963 bytes) 000368 1741898474.219172 MIRROR_EXTEND to 14 mirrors 000369 1741898474.557105 mapwrite 0x00640b thru 0x0106c7 (0xa2bd bytes) 000370 1741898474.625409 mapread 0x02a638 thru 0x0302d4 (0x5c9d bytes) 000371 1741898474.632969 write 0x01a28a thru 0x01df33 (0x3caa bytes) 000373 1741898474.647691 trunc from 0x040000 to 0x029417 (0x16be8 bytes) 000374 1741898474.670853 read 0x020033 thru 0x029417 (0x93e5 bytes) 000375 1741898474.677244 mapread 0x012637 thru 0x01b499 (0x8e63 bytes) 000376 1741898474.680870 read 0x016758 thru 0x01a4cd (0x3d76 bytes) 000377 1741898474.681425 mapwrite 0x014342 thru 0x01505c (0xd1b bytes) 000378 1741898474.688138 mapread 0x004c42 thru 0x0111cb (0xc58a bytes) 000379 1741898474.689251 punch from 0x01f559 to 0x029417 (0x9ebf bytes) 000380 1741898474.693515 mapread 0x004f6b thru 0x007312 (0x23a8 bytes) 000381 1741898474.696169 MIRROR_SPLIT mirror 10 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000382 1741898474.806200 MIRROR_EXTEND to 14 mirrors 000383 1741898475.078343 fallocate 0x025c92 thru 0x02c143 (0x64b2 bytes) 000384 1741898475.112109 MIRROR_EXTEND to 15 mirrors 000385 1741898475.480562 MIRROR_RESYNC 15 mirrors 000386 1741898477.310152 mapread 0x01a28a thru 0x0242b7 (0xa02e bytes) 000387 1741898477.328948 write 0x02b729 thru 0x0330a0 (0x7978 bytes) 000390 1741898477.350729 mapwrite 0x037677 thru 0x03d4c6 (0x5e50 bytes) 000391 1741898477.395494 read 0x004929 thru 0x005637 (0xd0f bytes) 000393 1741898477.400424 mapwrite 0x033bee thru 0x037a89 (0x3e9c bytes) 000394 1741898477.415357 punch from 0x00c8d8 to 0x00e570 (0x1c99 bytes) 000395 1741898477.418326 fallocate 0x01e6f5 thru 0x022991 (0x429d bytes) 000396 1741898477.422321 mapwrite 0x0278d2 thru 0x0361c9 (0xe8f8 bytes) 000397 1741898477.437775 fallocate 0x01daae thru 0x0250d4 (0x7627 bytes) 000399 1741898477.440992 mapwrite 0x01e4d8 thru 0x026afb (0x8624 bytes) 000401 1741898477.459019 read 0x01b215 thru 0x02599c (0xa788 bytes) 000402 1741898477.459422 mapwrite 0x00967e thru 0x0153ec (0xbd6f bytes) 000403 1741898477.479240 mapwrite 0x0102e0 thru 0x019c55 (0x9976 bytes) 000404 1741898477.492880 mapwrite 0x0314cd thru 0x03af11 (0x9a45 bytes) 000405 1741898477.507734 MIRROR_SPLIT mirror 5 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000406 1741898477.605810 trunc from 0x03d4c7 to 0x021955 (0x1bb71 bytes) 000407 1741898477.623803 write 0x02189b thru 0x02e9de (0xd144 bytes) 000408 1741898477.632721 MIRROR_RESYNC 14 mirrors 000410 1741898478.778593 punch from 0x0143f5 to 0x019d91 (0x599d bytes) 000411 1741898478.805122 punch from 0x01d92f to 0x01f53c (0x1c0e bytes) 000412 1741898478.811742 fallocate 0x011418 thru 0x017d82 (0x696b bytes) 000416 1741898478.819404 punch from 0x00d3de to 0x00d6a8 (0x2cb bytes) 000417 1741898478.822296 punch from 0x00332f to 0x007512 (0x41e4 bytes) 000419 1741898478.825057 mapread 0x0188db thru 0x01c6dc (0x3e02 bytes) 000420 1741898478.841505 write 0x01c9a1 thru 0x02a545 (0xdba5 bytes) 000421 1741898478.854116 read 0x01e72e thru 0x01ebc8 (0x49b bytes) 000422 1741898478.860393 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000423 1741898479.034416 write 0x004b51 thru 0x011c43 (0xd0f3 bytes) 000424 1741898479.045304 fallocate 0x03da04 thru 0x03ffff (0x25fc bytes) 000428 1741898479.053667 MIRROR_RESYNC 13 mirrors 000429 1741898480.095048 fallocate 0x03b5e7 thru 0x03ffff (0x4a19 bytes) 000430 1741898480.122099 MIRROR_SPLIT mirror 8 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000431 1741898480.276531 write 0x01f3b1 thru 0x020cf2 (0x1942 bytes) 000432 1741898480.283502 MIRROR_EXTEND to 13 mirrors 000433 1741898480.607311 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000434 1741898480.795559 write 0x015f89 thru 0x01f292 (0x930a bytes) 000435 1741898480.826759 mapwrite 0x0279fe thru 0x031243 (0x9846 bytes) 000436 1741898480.878338 trunc from 0x031244 to 0x02cd9d (0x44a6 bytes) 000440 1741898480.897336 trunc from 0x02cd9e to 0x02e88f (0x1af2 bytes) 000441 1741898480.907246 MIRROR_RESYNC 12 mirrors 000443 1741898481.810654 fallocate 0x0344d7 thru 0x03d089 (0x8bb3 bytes) 000444 1741898481.829768 write 0x01e0c2 thru 0x02183e (0x377d bytes) 000445 1741898481.836567 punch from 0x027361 to 0x02b1fb (0x3e9b bytes) 000446 1741898481.842736 write 0x035c28 thru 0x03ffff (0xa3d8 bytes) 000448 1741898481.844835 mapread 0x03cde7 thru 0x03ffff (0x3219 bytes) 000449 1741898481.848890 MIRROR_SPLIT mirror 2 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000450 1741898481.955469 MIRROR_EXTEND to 12 mirrors 000451 1741898482.278193 read 0x039018 thru 0x03ffff (0x6fe8 bytes) 000452 1741898482.291064 read 0x0144c1 thru 0x019965 (0x54a5 bytes) 000453 1741898482.313175 mapwrite 0x026f39 thru 0x02b37b (0x4443 bytes) 000455 1741898482.385000 trunc from 0x040000 to 0x036142 (0x9ebd bytes) 000457 1741898482.395193 read 0x029f76 thru 0x02e311 (0x439c bytes) 000458 1741898482.402612 read 0x032a49 thru 0x032e25 (0x3dd bytes) 000459 1741898482.403276 fallocate 0x026e9c thru 0x0281d2 (0x1337 bytes) 000460 1741898482.406763 read 0x01e69f thru 0x02274c (0x40ae bytes) 000461 1741898482.407166 trunc from 0x036143 to 0x01e8d0 (0x17872 bytes) 000462 1741898482.419595 trunc from 0x01e8d1 to 0x0278ad (0x8fdd bytes) 000463 1741898482.435767 read 0x01c883 thru 0x01e3f7 (0x1b75 bytes) 000465 1741898482.436983 mapread 0x024219 thru 0x0278ad (0x3695 bytes) 000467 1741898482.445407 read 0x01f90f thru 0x027166 (0x7858 bytes) 000468 1741898482.446313 fallocate 0x01f137 thru 0x02b94c (0xc816 bytes) 000470 1741898482.457196 MIRROR_RESYNC 12 mirrors 000472 1741898483.494910 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000473 1741898483.776812 write 0x016e57 thru 0x0209e5 (0x9b8f bytes) 000475 1741898483.813633 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000476 1741898484.024514 mapread 0x009c0f thru 0x014755 (0xab47 bytes) 000477 1741898484.043924 MIRROR_EXTEND to 11 mirrors 000478 1741898484.416172 write 0x02af62 thru 0x02cd96 (0x1e35 bytes) 000479 1741898484.439084 fallocate 0x027c0a thru 0x02d4b3 (0x58aa bytes) 000481 1741898484.442857 mapread 0x00432b thru 0x00d86a (0x9540 bytes) 000482 1741898484.459225 fallocate 0x00427a thru 0x006a1c (0x27a3 bytes) 000483 1741898484.465127 MIRROR_EXTEND to 12 mirrors 000484 1741898484.701864 read 0x02c3d8 thru 0x02d4b3 (0x10dc bytes) 000485 1741898484.709029 read 0x00e016 thru 0x01a5d0 (0xc5bb bytes) 000486 1741898484.724511 read 0x00cd75 thru 0x0104b1 (0x373d bytes) 000487 1741898484.735221 trunc from 0x02d4b4 to 0x02136e (0xc145 bytes) 000488 1741898484.772631 mapread 0x000655 thru 0x00a46f (0x9e1b bytes) 000489 1741898484.781752 mapread 0x02022f thru 0x02136e (0x1140 bytes) 000490 1741898484.782761 read 0x01ad56 thru 0x01edf8 (0x40a3 bytes) 000491 1741898484.783266 mapread 0x00f05c thru 0x013b66 (0x4b0b bytes) 000493 1741898484.785703 read 0x01578b thru 0x02136e (0xbbe4 bytes) 000494 1741898484.786270 mapread 0x001da3 thru 0x003013 (0x1271 bytes) 000496 1741898484.788772 read 0x0009f2 thru 0x00b692 (0xaca1 bytes) 000497 1741898484.790262 punch from 0x003573 to 0x006672 (0x3100 bytes) 000498 1741898484.794755 mapwrite 0x0175c6 thru 0x018838 (0x1273 bytes) 000499 1741898484.813244 mapread 0x00647d thru 0x0160ee (0xfc72 bytes) 000500 1741898484.820512 write 0x009a3d thru 0x017157 (0xd71b bytes) 000503 1741898484.824322 trunc from 0x02136f to 0x01cfb8 (0x43b6 bytes) 000504 1741898484.835486 fallocate 0x010679 thru 0x01b587 (0xaf0f bytes) 000505 1741898484.838145 punch from 0x0142d0 to 0x01cfb8 (0x8ce9 bytes) 000506 1741898484.855261 read 0x011d0a thru 0x018701 (0x69f8 bytes) 000507 1741898484.864387 mapread 0x011fba thru 0x01cfb8 (0xafff bytes) 000508 1741898484.867661 fallocate 0x008966 thru 0x00f471 (0x6b0c bytes) 000509 1741898484.873333 trunc from 0x01cfb9 to 0x03e669 (0x216b1 bytes) 000512 1741898484.896711 write 0x0232d9 thru 0x027779 (0x44a1 bytes) 000513 1741898484.909027 write 0x005379 thru 0x00ed70 (0x99f8 bytes) 000514 1741898484.910368 mapread 0x01a2c9 thru 0x01b7ae (0x14e6 bytes) 000518 1741898484.913206 mapwrite 0x027823 thru 0x02cd1d (0x54fb bytes) 000519 1741898484.931787 punch from 0x022838 to 0x02dace (0xb297 bytes) 000520 1741898484.939980 fallocate 0x004132 thru 0x00966c (0x553b bytes) 000521 1741898484.945428 punch from 0x03477d to 0x03e669 (0x9eed bytes) 000522 1741898484.948324 trunc from 0x03e66a to 0x029845 (0x14e24 bytes) 000523 1741898484.957322 write 0x039892 thru 0x03ffff (0x676e bytes) 000524 1741898484.958479 write 0x03ae41 thru 0x03ffff (0x51bf bytes) 000525 1741898484.959293 fallocate 0x02d009 thru 0x0395be (0xc5b6 bytes) 000526 1741898484.965414 mapwrite 0x00d7a7 thru 0x013ea9 (0x6703 bytes) 000527 1741898484.983783 mapread 0x01317f thru 0x018922 (0x57a4 bytes) 000529 1741898484.986299 punch from 0x01f154 to 0x025529 (0x63d6 bytes) 000532 1741898484.995331 write 0x031e92 thru 0x038bc1 (0x6d30 bytes) 000533 1741898485.003071 punch from 0x032a3e to 0x03dab8 (0xb07b bytes) 000534 1741898485.023961 MIRROR_RESYNC 12 mirrors 000535 1741898486.755808 read 0x026cb5 thru 0x02c507 (0x5853 bytes) 000536 1741898486.765363 write 0x01e25f thru 0x02c69e (0xe440 bytes) 000539 1741898486.809550 write 0x038de2 thru 0x03ffff (0x721e bytes) 000540 1741898486.819036 read 0x0109f8 thru 0x013c78 (0x3281 bytes) 000541 1741898486.828080 trunc from 0x040000 to 0x01d7a7 (0x22858 bytes) 000542 1741898486.842851 punch from 0x001d68 to 0x00ba23 (0x9cbc bytes) 000543 1741898486.846333 mapread 0x0078b0 thru 0x015ff1 (0xe742 bytes) 000544 1741898486.858250 MIRROR_SPLIT mirror 4 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000545 1741898486.992872 mapread 0x013789 thru 0x01d7a7 (0xa01f bytes) 000546 1741898487.018220 mapwrite 0x03c653 thru 0x03f04a (0x29f8 bytes) 000547 1741898487.039492 mapread 0x00469a thru 0x00c232 (0x7b99 bytes) 000548 1741898487.045121 fallocate 0x039733 thru 0x03b792 (0x2060 bytes) 000549 1741898487.054395 MIRROR_EXTEND to 12 mirrors 000550 1741898487.408595 fallocate 0x03f374 thru 0x03ffff (0xc8c bytes) 000551 1741898487.453742 mapwrite 0x0254aa thru 0x026ff2 (0x1b49 bytes) 000552 1741898487.475920 punch from 0x03ba6b to 0x03d9d5 (0x1f6b bytes) 000553 1741898487.479906 mapread 0x028801 thru 0x035fd6 (0xd7d6 bytes) 000555 1741898487.491282 fallocate 0x02ce07 thru 0x03360e (0x6808 bytes) 000556 1741898487.498775 MIRROR_SPLIT mirror 8 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000557 1741898487.605575 write 0x02882e thru 0x0292ab (0xa7e bytes) 000559 1741898487.615243 read 0x01420f thru 0x01569e (0x1490 bytes) 000560 1741898487.617768 write 0x03a7f8 thru 0x03ffff (0x5808 bytes) 000562 1741898487.629003 MIRROR_EXTEND to 12 mirrors 000563 1741898487.977977 mapread 0x02be06 thru 0x03a818 (0xea13 bytes) 000564 1741898488.008176 write 0x036f3e thru 0x03e370 (0x7433 bytes) 000565 1741898488.047514 trunc from 0x040000 to 0x016ab5 (0x2954a bytes) 000566 1741898488.070894 mapwrite 0x014c84 thru 0x020242 (0xb5bf bytes) 000567 1741898488.096137 fallocate 0x02159b thru 0x0268b3 (0x5319 bytes) 000568 1741898488.099750 fallocate 0x01a671 thru 0x01abff (0x58f bytes) 000570 1741898488.104991 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000571 1741898488.202169 mapwrite 0x00d018 thru 0x01add4 (0xddbd bytes) 000572 1741898488.219239 read 0x008cd4 thru 0x017cb7 (0xefe4 bytes) 000573 1741898488.220957 read 0x007efc thru 0x00d965 (0x5a6a bytes) 000574 1741898488.224639 mapread 0x001cf9 thru 0x011196 (0xf49e bytes) 000577 1741898488.226354 read 0x00cff9 thru 0x01888d (0xb895 bytes) 000579 1741898488.227464 mapwrite 0x03a991 thru 0x03e55f (0x3bcf bytes) 000580 1741898488.247939 write 0x03a35e thru 0x03ffff (0x5ca2 bytes) 000582 1741898488.251885 MIRROR_RESYNC 11 mirrors 000583 1741898489.916171 mapwrite 0x020ac3 thru 0x028c32 (0x8170 bytes) 000585 1741898490.006529 read 0x0369d8 thru 0x03daab (0x70d4 bytes) 000588 1741898490.014401 punch from 0x019981 to 0x021573 (0x7bf3 bytes) 000590 1741898490.022598 mapwrite 0x03ca02 thru 0x03ffff (0x35fe bytes) 000592 1741898490.043364 MIRROR_EXTEND to 12 mirrors 000593 1741898490.445028 fallocate 0x00599b thru 0x005ec7 (0x52d bytes) 000594 1741898490.465393 mapwrite 0x01fd48 thru 0x02aecb (0xb184 bytes) 000596 1741898490.498557 punch from 0x010bb3 to 0x01fcf5 (0xf143 bytes) 000597 1741898490.501861 fallocate 0x010e8b thru 0x01c3a1 (0xb517 bytes) 000598 1741898490.505007 trunc from 0x040000 to 0x0008f8 (0x3f707 bytes) 000601 1741898490.544895 trunc from 0x0008f9 to 0x0261ea (0x258f2 bytes) 000603 1741898490.557632 fallocate 0x0018d4 thru 0x00c10f (0xa83c bytes) 000604 1741898490.563856 mapwrite 0x01891b thru 0x026c5d (0xe343 bytes) 000605 1741898490.606549 trunc from 0x026c5e to 0x03ddf0 (0x17193 bytes) 000606 1741898490.619350 trunc from 0x03ddf1 to 0x030252 (0xdb9e bytes) 000607 1741898490.628664 mapwrite 0x0156c6 thru 0x017d6e (0x26a9 bytes) 000608 1741898490.644515 mapwrite 0x037276 thru 0x03a59f (0x332a bytes) 000609 1741898490.673699 write 0x00a6e0 thru 0x00f5b0 (0x4ed1 bytes) 000611 1741898490.679307 mapread 0x01cbbd thru 0x029178 (0xc5bc bytes) 000612 1741898490.687850 write 0x02e595 thru 0x03a2dd (0xbd49 bytes) 000615 1741898490.694812 MIRROR_SPLIT mirror 5 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000616 1741898490.853998 mapread 0x0294d8 thru 0x030962 (0x748b bytes) 000618 1741898490.877327 mapread 0x024339 thru 0x0266be (0x2386 bytes) 000619 1741898490.884536 fallocate 0x0388a4 thru 0x03ffff (0x775c bytes) 000620 1741898490.891831 write 0x02dc47 thru 0x02e1ab (0x565 bytes) 000622 1741898490.899582 write 0x014a51 thru 0x014ff0 (0x5a0 bytes) 000623 1741898490.906762 mapwrite 0x015edc thru 0x01ddf9 (0x7f1e bytes) 000624 1741898490.937152 mapread 0x00121c thru 0x0104d4 (0xf2b9 bytes) 000625 1741898490.953246 read 0x024919 thru 0x0284a8 (0x3b90 bytes) 000626 1741898490.960624 MIRROR_EXTEND to 12 mirrors 000627 1741898491.371019 MIRROR_RESYNC 12 mirrors 000630 1741898492.758417 trunc from 0x03a5a0 to 0x003499 (0x37106 bytes) 000631 1741898492.799682 MIRROR_EXTEND to 13 mirrors 000634 1741898493.122870 fallocate 0x033d73 thru 0x03947d (0x570b bytes) 000635 1741898493.153324 mapread 0x02ad91 thru 0x02eeec (0x415c bytes) 000637 1741898493.174199 trunc from 0x03947e to 0x0393cf (0xae bytes) 000639 1741898493.195603 MIRROR_RESYNC 13 mirrors 000641 1741898494.407054 trunc from 0x0393d0 to 0x0115e6 (0x27de9 bytes) 000642 1741898494.441313 write 0x01254f thru 0x021760 (0xf212 bytes) 000643 1741898494.443165 mapread 0x01fcba thru 0x021760 (0x1aa7 bytes) 000644 1741898494.444418 read 0x013ad9 thru 0x01b83e (0x7d66 bytes) 000645 1741898494.445246 fallocate 0x00e4b8 thru 0x012e21 (0x496a bytes) 000646 1741898494.452635 write 0x00df38 thru 0x0182de (0xa3a7 bytes) 000647 1741898494.457203 mapread 0x01e426 thru 0x01f747 (0x1322 bytes) 000648 1741898494.457950 mapread 0x0194f1 thru 0x021760 (0x8270 bytes) 000649 1741898494.458657 write 0x000231 thru 0x00c540 (0xc310 bytes) 000650 1741898494.468693 punch from 0x00a665 to 0x01a602 (0xff9e bytes) 000652 1741898494.494902 MIRROR_EXTEND to 14 mirrors 000653 1741898494.854877 MIRROR_SPLIT mirror 10 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000655 1741898495.115610 punch from 0x004c3a to 0x00f428 (0xa7ef bytes) 000656 1741898495.152724 trunc from 0x021761 to 0x00ff7f (0x117e1 bytes) 000657 1741898495.185318 trunc from 0x00ff80 to 0x025256 (0x152d7 bytes) 000658 1741898495.205124 write 0x0134b3 thru 0x01db88 (0xa6d6 bytes) 000659 1741898495.214232 fallocate 0x0132b1 thru 0x02308b (0xfddb bytes) 000660 1741898495.217939 read 0x009814 thru 0x00d128 (0x3915 bytes) 000661 1741898495.222313 mapwrite 0x03dbb0 thru 0x03ffff (0x2450 bytes) 000662 1741898495.266331 fallocate 0x02820d thru 0x036da3 (0xeb97 bytes) 000664 1741898495.279852 write 0x027a76 thru 0x02f7e0 (0x7d6b bytes) 000666 1741898495.289298 read 0x0298dd thru 0x031d44 (0x8468 bytes) 000668 1741898495.298474 write 0x03047c thru 0x03b90c (0xb491 bytes) 000670 1741898495.303655 MIRROR_SPLIT mirror 2 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000671 1741898495.469057 fallocate 0x025c26 thru 0x034c4b (0xf026 bytes) 000672 1741898495.472579 write 0x02e702 thru 0x03c75e (0xe05d bytes) 000673 1741898495.492300 MIRROR_RESYNC 12 mirrors 000675 1741898496.861732 write 0x00c41b thru 0x0103e3 (0x3fc9 bytes) 000676 1741898496.884490 mapread 0x022b55 thru 0x028c1c (0x60c8 bytes) 000677 1741898496.893153 MIRROR_RESYNC 12 mirrors 000678 1741898498.322724 write 0x008e30 thru 0x018814 (0xf9e5 bytes) 000679 1741898498.366848 mapread 0x0061f8 thru 0x011bb9 (0xb9c2 bytes) 000680 1741898498.378242 mapwrite 0x028323 thru 0x02c2a9 (0x3f87 bytes) 000681 1741898498.407808 mapread 0x013d2d thru 0x023d1f (0xfff3 bytes) 000682 1741898498.413259 trunc from 0x040000 to 0x03b89d (0x4762 bytes) 000683 1741898498.428274 fallocate 0x03d14e thru 0x03e39e (0x1251 bytes) 000684 1741898498.431044 fallocate 0x0345a0 thru 0x03b45a (0x6ebb bytes) 000686 1741898498.434109 mapwrite 0x0146ac thru 0x018a93 (0x43e8 bytes) 000687 1741898498.445702 mapwrite 0x009c0d thru 0x00c3f2 (0x27e6 bytes) 000688 1741898498.457491 punch from 0x0265b4 to 0x02f683 (0x90d0 bytes) 000689 1741898498.464283 write 0x004675 thru 0x005647 (0xfd3 bytes) 000690 1741898498.465129 write 0x014233 thru 0x01dcb2 (0x9a80 bytes) 000691 1741898498.466842 trunc from 0x03e39f to 0x030f24 (0xd47a bytes) 000692 1741898498.476379 mapread 0x0283b8 thru 0x030c73 (0x88bc bytes) 000693 1741898498.482112 read 0x02527f thru 0x0253a8 (0x12a bytes) 000694 1741898498.482859 write 0x00be08 thru 0x011905 (0x5afe bytes) 000695 1741898498.483875 mapwrite 0x029ec1 thru 0x035ece (0xc00e bytes) 000696 1741898498.518900 read 0x0102b4 thru 0x0181fe (0x7f4b bytes) 000697 1741898498.522024 trunc from 0x035ecf to 0x036d86 (0xeb8 bytes) 000699 1741898498.534323 read 0x0046d8 thru 0x01111d (0xca46 bytes) 000700 1741898498.534883 mapwrite 0x004835 thru 0x007919 (0x30e5 bytes) 000701 1741898498.552992 write 0x031eeb thru 0x03fb33 (0xdc49 bytes) 000702 1741898498.556465 fallocate 0x0094c3 thru 0x011183 (0x7cc1 bytes) 000703 1741898498.561202 punch from 0x0266df to 0x026bf7 (0x519 bytes) 000704 1741898498.567386 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000705 1741898498.677625 fallocate 0x00f846 thru 0x01f4e2 (0xfc9d bytes) 000707 1741898498.680680 write 0x027118 thru 0x02889d (0x1786 bytes) 000708 1741898498.688151 read 0x017201 thru 0x0251aa (0xdfaa bytes) 000709 1741898498.693223 trunc from 0x03fb34 to 0x028f66 (0x16bcd bytes) 000710 1741898498.699489 fallocate 0x00026d thru 0x00e958 (0xe6ec bytes) 000711 1741898498.704035 MIRROR_RESYNC 11 mirrors 000714 1741898499.605834 write 0x00a19b thru 0x00db2e (0x3994 bytes) 000716 1741898499.628196 read 0x003ae9 thru 0x00f6e5 (0xbbfd bytes) 000718 1741898499.637750 punch from 0x007b55 to 0x00af46 (0x33f2 bytes) 000719 1741898499.646861 trunc from 0x028f67 to 0x0114e7 (0x17a7f bytes) 000720 1741898499.659916 mapwrite 0x03f228 thru 0x03f92e (0x707 bytes) 000723 1741898499.704220 MIRROR_RESYNC 11 mirrors 000724 1741898500.771612 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000725 1741898501.003854 write 0x03706e thru 0x03c75c (0x56ef bytes) 000727 1741898501.045901 write 0x000806 thru 0x00153a (0xd35 bytes) 000728 1741898501.054870 read 0x03dec9 thru 0x03f92e (0x1a66 bytes) 000729 1741898501.062050 punch from 0x004f43 to 0x00b9c6 (0x6a84 bytes) 000732 1741898501.068113 MIRROR_RESYNC 10 mirrors 000733 1741898502.248887 mapread 0x021e35 thru 0x02b609 (0x97d5 bytes) 000734 1741898502.269285 mapread 0x02f4ef thru 0x0302dd (0xdef bytes) 000735 1741898502.269698 mapread 0x024936 thru 0x028b33 (0x41fe bytes) 000736 1741898502.270684 punch from 0x03d53f to 0x03f92e (0x23f0 bytes) 000737 1741898502.290359 punch from 0x00f4bf to 0x00fcfb (0x83d bytes) 000738 1741898502.295902 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000739 1741898502.445416 MIRROR_EXTEND to 10 mirrors 000742 1741898502.759339 mapwrite 0x02e531 thru 0x031899 (0x3369 bytes) 000743 1741898502.813914 trunc from 0x03f92f to 0x0295c1 (0x1636d bytes) 000745 1741898502.838855 MIRROR_EXTEND to 11 mirrors 000746 1741898503.104479 mapwrite 0x027923 thru 0x029505 (0x1be3 bytes) 000747 1741898503.177128 punch from 0x029232 to 0x0295c1 (0x390 bytes) 000748 1741898503.180757 punch from 0x001f9e to 0x00f73b (0xd79e bytes) 000749 1741898503.184404 write 0x0234d7 thru 0x02af98 (0x7ac2 bytes) 000750 1741898503.190662 fallocate 0x02b815 thru 0x0300e1 (0x48cd bytes) 000751 1741898503.195283 read 0x029081 thru 0x0300e1 (0x7061 bytes) 000752 1741898503.201814 mapwrite 0x008db3 thru 0x00aedd (0x212b bytes) 000753 1741898503.215090 mapread 0x003fcf thru 0x004a94 (0xac6 bytes) 000754 1741898503.221226 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000755 1741898503.355964 write 0x00608b thru 0x014d48 (0xecbe bytes) 000756 1741898503.364303 trunc from 0x0300e2 to 0x03ffd4 (0xfef3 bytes) 000757 1741898503.382883 write 0x03f11d thru 0x03ffff (0xee3 bytes) 000758 1741898503.390682 fallocate 0x0127a9 thru 0x01d28b (0xaae3 bytes) 000759 1741898503.394816 punch from 0x00fa2f to 0x010b1e (0x10f0 bytes) 000760 1741898503.403070 mapwrite 0x00a437 thru 0x01401a (0x9be4 bytes) 000763 1741898503.429169 write 0x03b7cd thru 0x03c402 (0xc36 bytes) 000764 1741898503.430602 mapread 0x02a71e thru 0x03a40f (0xfcf2 bytes) 000766 1741898503.447002 mapwrite 0x006b45 thru 0x01074e (0x9c0a bytes) 000767 1741898503.464434 read 0x03874e thru 0x03ceba (0x476d bytes) 000768 1741898503.465962 punch from 0x03b56d to 0x03ffff (0x4a93 bytes) 000770 1741898503.474879 punch from 0x035c09 to 0x03ffff (0xa3f7 bytes) 000772 1741898503.482496 fallocate 0x0044ff thru 0x011c14 (0xd716 bytes) 000774 1741898503.488771 read 0x00829e thru 0x015562 (0xd2c5 bytes) 000775 1741898503.489498 mapread 0x03fde1 thru 0x03ffff (0x21f bytes) 000776 1741898503.500643 fallocate 0x03a595 thru 0x03ffff (0x5a6b bytes) 000779 1741898503.508413 fallocate 0x03d1a1 thru 0x03ffff (0x2e5f bytes) 000781 1741898503.516271 mapwrite 0x01a5bd thru 0x02a04e (0xfa92 bytes) 000785 1741898503.540160 fallocate 0x0003e5 thru 0x00b85b (0xb477 bytes) 000786 1741898503.550024 MIRROR_RESYNC 10 mirrors 000787 1741898504.359307 trunc from 0x040000 to 0x01027c (0x2fd83 bytes) 000788 1741898504.392677 read 0x007e1e thru 0x01027c (0x845f bytes) 000789 1741898504.398999 read 0x0017dc thru 0x00c34d (0xab72 bytes) 000790 1741898504.406101 trunc from 0x01027d to 0x02a3b2 (0x1a136 bytes) 000791 1741898504.422482 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000792 1741898504.562714 mapread 0x0227f0 thru 0x027fdb (0x57ec bytes) 000793 1741898504.574403 mapwrite 0x0039d3 thru 0x013997 (0xffc5 bytes) 000794 1741898504.623186 MIRROR_RESYNC 9 mirrors 000795 1741898505.450831 punch from 0x025a43 to 0x02a3b2 (0x4970 bytes) 000796 1741898505.475491 read 0x002235 thru 0x007d42 (0x5b0e bytes) 000798 1741898505.485407 trunc from 0x02a3b3 to 0x00ece5 (0x1b6cd bytes) 000799 1741898505.503463 punch from 0x00ea0d to 0x00ece5 (0x2d9 bytes) 000800 1741898505.506581 mapread 0x006764 thru 0x00ece5 (0x8582 bytes) 000801 1741898505.515594 read 0x00919a thru 0x00ece5 (0x5b4c bytes) 000802 1741898505.516239 fallocate 0x022e9b thru 0x0281b4 (0x531a bytes) 000803 1741898505.520803 read 0x00d01b thru 0x00ece5 (0x1ccb bytes) 000804 1741898505.521229 read 0x00c011 thru 0x00ece5 (0x2cd5 bytes) 000805 1741898505.521694 trunc from 0x00ece6 to 0x00aa33 (0x42b2 bytes) 000808 1741898505.540700 write 0x00fc13 thru 0x01448e (0x487c bytes) 000809 1741898505.546214 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000810 1741898505.693394 write 0x03f6c4 thru 0x03fc34 (0x571 bytes) 000811 1741898505.697079 write 0x01aa51 thru 0x025e20 (0xb3d0 bytes) 000812 1741898505.703099 mapread 0x00a489 thru 0x0190b4 (0xec2c bytes) 000813 1741898505.714641 mapwrite 0x00165e thru 0x00498e (0x3331 bytes) 000814 1741898505.735050 read 0x0176dd thru 0x024d41 (0xd665 bytes) 000815 1741898505.735803 read 0x0386e7 thru 0x03fc34 (0x754e bytes) 000816 1741898505.746591 write 0x01b5f0 thru 0x023b08 (0x8519 bytes) 000817 1741898505.747866 punch from 0x01e3b2 to 0x029dbe (0xba0d bytes) 000819 1741898505.763780 MIRROR_EXTEND to 9 mirrors 000820 1741898506.097198 MIRROR_RESYNC 9 mirrors 000821 1741898507.163752 fallocate 0x004e3a thru 0x011499 (0xc660 bytes) 000822 1741898507.192312 mapwrite 0x0009cd thru 0x00ddae (0xd3e2 bytes) 000824 1741898507.229646 punch from 0x004efc to 0x013601 (0xe706 bytes) 000825 1741898507.233990 write 0x00b4d6 thru 0x00b7cc (0x2f7 bytes) 000826 1741898507.239689 punch from 0x00caad to 0x018410 (0xb964 bytes) 000828 1741898507.248823 mapread 0x0138a8 thru 0x01d3ed (0x9b46 bytes) 000829 1741898507.259772 read 0x026ae3 thru 0x026ba8 (0xc6 bytes) 000830 1741898507.263174 punch from 0x008edc to 0x0161c4 (0xd2e9 bytes) 000831 1741898507.274286 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000832 1741898507.451094 write 0x019a09 thru 0x01ffe6 (0x65de bytes) 000833 1741898507.469110 punch from 0x03c9d1 to 0x03fc34 (0x3264 bytes) 000837 1741898507.476964 punch from 0x020989 to 0x024435 (0x3aad bytes) 000839 1741898507.485722 fallocate 0x02a36f thru 0x0382aa (0xdf3c bytes) 000840 1741898507.492974 read 0x03016d thru 0x035c29 (0x5abd bytes) 000842 1741898507.505190 fallocate 0x027c30 thru 0x02b952 (0x3d23 bytes) 000843 1741898507.509626 read 0x035229 thru 0x0363f6 (0x11ce bytes) 000844 1741898507.521777 punch from 0x018496 to 0x01969e (0x1209 bytes) 000846 1741898507.542684 fallocate 0x015884 thru 0x02144b (0xbbc8 bytes) 000847 1741898507.547322 mapwrite 0x02ba76 thru 0x03a536 (0xeac1 bytes) 000848 1741898507.573621 MIRROR_EXTEND to 9 mirrors 000849 1741898508.051359 mapread 0x035c7f thru 0x03d04a (0x73cc bytes) 000851 1741898508.094567 mapwrite 0x02febf thru 0x03fe18 (0xff5a bytes) 000852 1741898508.212646 MIRROR_RESYNC 9 mirrors 000854 1741898509.402220 mapwrite 0x01c25f thru 0x02aa5f (0xe801 bytes) 000855 1741898509.465653 fallocate 0x01a460 thru 0x01b28e (0xe2f bytes) 000856 1741898509.468683 mapwrite 0x028a16 thru 0x03557a (0xcb65 bytes) 000857 1741898509.497420 punch from 0x03254a to 0x0394e8 (0x6f9f bytes) 000858 1741898509.501904 mapwrite 0x002d42 thru 0x006cea (0x3fa9 bytes) 000860 1741898509.517539 punch from 0x013417 to 0x01a4d8 (0x70c2 bytes) 000861 1741898509.521314 write 0x00bae0 thru 0x00eccc (0x31ed bytes) 000862 1741898509.525004 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000864 1741898509.677536 MIRROR_RESYNC 8 mirrors 000865 1741898510.966269 trunc from 0x03fe19 to 0x02273d (0x1d6db bytes) 000866 1741898511.005421 punch from 0x020f4a to 0x02273d (0x17f4 bytes) 000867 1741898511.014708 mapread 0x014fba thru 0x0225dd (0xd624 bytes) 000868 1741898511.034658 read 0x018bef thru 0x02273d (0x9b4f bytes) 000870 1741898511.035344 fallocate 0x00353c thru 0x013475 (0xff3a bytes) 000871 1741898511.038335 mapwrite 0x035580 thru 0x03ffff (0xaa80 bytes) 000874 1741898511.074348 read 0x034f50 thru 0x0378df (0x2990 bytes) 000875 1741898511.077953 mapwrite 0x024bf8 thru 0x02d271 (0x867a bytes) 000876 1741898511.101093 mapread 0x032b7a thru 0x03a3ea (0x7871 bytes) 000877 1741898511.103495 fallocate 0x01b41a thru 0x024c6c (0x9853 bytes) 000878 1741898511.108835 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000879 1741898511.291945 fallocate 0x002938 thru 0x00c25f (0x9928 bytes) 000880 1741898511.305120 mapwrite 0x019afe thru 0x029625 (0xfb28 bytes) 000881 1741898511.355710 MIRROR_EXTEND to 8 mirrors 000882 1741898511.819115 trunc from 0x040000 to 0x03b6ab (0x4954 bytes) 000883 1741898511.877867 trunc from 0x03b6ac to 0x0125bc (0x290ef bytes) 000884 1741898511.904259 read 0x00169e thru 0x003609 (0x1f6c bytes) 000886 1741898511.914425 write 0x016cef thru 0x022e9f (0xc1b1 bytes) 000887 1741898511.916259 trunc from 0x022ea0 to 0x037f13 (0x15074 bytes) 000888 1741898511.934081 fallocate 0x00eacd thru 0x010eb9 (0x23ed bytes) 000889 1741898511.939415 punch from 0x01e6bb to 0x0212a3 (0x2be9 bytes) 000890 1741898511.954997 write 0x018939 thru 0x021210 (0x88d8 bytes) 000891 1741898511.968525 mapwrite 0x02ca67 thru 0x030c70 (0x420a bytes) 000892 1741898511.990704 read 0x003948 thru 0x00bee1 (0x859a bytes) 000893 1741898511.998607 fallocate 0x008b91 thru 0x00ec67 (0x60d7 bytes) 000894 1741898512.001402 read 0x015a33 thru 0x01936d (0x393b bytes) 000895 1741898512.001859 trunc from 0x037f14 to 0x03aeaa (0x2f97 bytes) 000896 1741898512.011582 write 0x0185c4 thru 0x01871d (0x15a bytes) 000897 1741898512.014839 MIRROR_RESYNC 8 mirrors 000898 1741898513.671250 punch from 0x01c622 to 0x029b06 (0xd4e5 bytes) 000899 1741898513.726854 read 0x003aaa thru 0x00d1c0 (0x9717 bytes) 000900 1741898513.739501 punch from 0x022339 to 0x02929e (0x6f66 bytes) 000901 1741898513.744226 fallocate 0x023529 thru 0x02eba0 (0xb678 bytes) 000902 1741898513.758293 trunc from 0x03aeab to 0x00805b (0x32e4f bytes) 000903 1741898513.773724 fallocate 0x0016e4 thru 0x009688 (0x7fa5 bytes) 000904 1741898513.783905 fallocate 0x01bf1f thru 0x01d31d (0x13ff bytes) 000905 1741898513.787855 write 0x018c06 thru 0x01ffa4 (0x739f bytes) 000906 1741898513.790483 punch from 0x005daa to 0x012049 (0xc2a0 bytes) 000907 1741898513.795824 trunc from 0x01ffa5 to 0x01a6a4 (0x5900 bytes) 000908 1741898513.817869 mapwrite 0x00526a thru 0x00b8c7 (0x665e bytes) 000910 1741898513.856919 read 0x00b435 thru 0x0171de (0xbdaa bytes) 000911 1741898513.869190 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x45:0x0]: Invalid argument (22) 000912 1741898514.049382 mapread 0x002505 thru 0x011dee (0xf8ea bytes) 000914 1741898514.069896 mapwrite 0x024f42 thru 0x02d659 (0x8718 bytes) 000915 1741898514.124230 punch from 0x019ab2 to 0x028409 (0xe958 bytes) 000917 1741898514.130496 mapwrite 0x00e2fb thru 0x00e5e4 (0x2ea bytes) 000918 1741898514.144789 punch from 0x0156a1 to 0x021b57 (0xc4b7 bytes) 000919 1741898514.149262 mapread 0x0259ef thru 0x02c903 (0x6f15 bytes) 000920 1741898514.157971 write 0x03c18b thru 0x03ffff (0x3e75 bytes) 000921 1741898514.161361 write 0x02c162 thru 0x03459d (0x843c bytes) 000922 1741898514.171906 MIRROR_RESYNC 7 mirrors 000923 1741898515.379283 fallocate 0x02c01f thru 0x0326b8 (0x669a bytes) 000924 1741898515.410278 write 0x01752d thru 0x01fabd (0x8591 bytes) 000925 1741898515.417396 write 0x020b20 thru 0x029b51 (0x9032 bytes) 000926 1741898515.429500 read 0x0023a6 thru 0x00cf6f (0xabca bytes) 000927 1741898515.440716 MIRROR_EXTEND to 8 mirrors 000928 1741898515.878229 mapwrite 0x017ae9 thru 0x0182d4 (0x7ec bytes) 000931 1741898515.968247 write 0x01b496 thru 0x01c1db (0xd46 bytes) 000932 1741898515.974536 punch from 0x019e21 to 0x02527d (0xb45d bytes) 000933 1741898515.990528 trunc from 0x040000 to 0x01115a (0x2eea5 bytes) 000935 1741898516.017169 write 0x00cab6 thru 0x01bf03 (0xf44e bytes) 000936 1741898516.030100 MIRROR_EXTEND to 9 mirrors 000939 1741898516.400562 mapwrite 0x015912 thru 0x024bba (0xf2a9 bytes) 000940 1741898516.490899 trunc from 0x024bbb to 0x01900e (0xbbac bytes) 000944 1741898516.520121 mapwrite 0x021bca thru 0x02f4c8 (0xd8ff bytes) 000945 1741898516.580742 write 0x00f2bb thru 0x01a102 (0xae48 bytes) 000946 1741898516.594268 write 0x03eb06 thru 0x03ffff (0x14fa bytes) 000947 1741898516.595212 trunc from 0x040000 to 0x03aa75 (0x558a bytes) 000948 1741898516.611825 mapread 0x0267e1 thru 0x02f801 (0x9021 bytes) 000949 1741898516.613613 fallocate 0x03f449 thru 0x03ffff (0xbb7 bytes) 000952 1741898516.621083 write 0x00ad07 thru 0x018724 (0xda1e bytes) 000955 1741898516.623066 trunc from 0x03aa76 to 0x01ed5a (0x1bd1b bytes) 000956 1741898516.644854 write 0x01aeae thru 0x0290c7 (0xe21a bytes) 000957 1741898516.646634 punch from 0x00731e to 0x00e353 (0x7036 bytes) 000958 1741898516.671511 trunc from 0x0290c8 to 0x021f43 (0x7184 bytes) 000960 1741898516.690377 trunc from 0x021f44 to 0x0293db (0x7498 bytes) 000962 1741898516.712365 MIRROR_EXTEND to 10 mirrors 000963 1741898517.091198 trunc from 0x0293dc to 0x032b2e (0x9753 bytes) 000964 1741898517.135778 mapread 0x014c92 thru 0x01cbbc (0x7f2b bytes) 000965 1741898517.153522 fallocate 0x02c7d3 thru 0x03b886 (0xf0b4 bytes) 000967 1741898517.171499 MIRROR_RESYNC 10 mirrors 000968 1741898518.428407 write 0x02f06e thru 0x03441d (0x53b0 bytes) 000969 1741898518.474494 mapread 0x0320ea thru 0x03441d (0x2334 bytes) 000970 1741898518.476779 trunc from 0x03441e to 0x016b27 (0x1d8f6 bytes) 000971 1741898518.501821 read 0x0028e2 thru 0x0057e1 (0x2f00 bytes) 000972 1741898518.509589 punch from 0x001871 to 0x00589e (0x402e bytes) 000975 1741898518.522618 MIRROR_EXTEND to 11 mirrors 000978 1741898519.009996 write 0x02ca05 thru 0x02f211 (0x280d bytes) 000979 1741898519.052685 write 0x036f1a thru 0x03ffff (0x90e6 bytes) 000980 1741898519.054268 fallocate 0x0065b7 thru 0x00d2bf (0x6d09 bytes) 000981 1741898519.060734 mapread 0x00246a thru 0x0100a9 (0xdc40 bytes) 000982 1741898519.091371 trunc from 0x040000 to 0x004cc9 (0x3b336 bytes) 000983 1741898519.136939 trunc from 0x004cca to 0x0046bc (0x60d bytes) 000984 1741898519.157267 mapread 0x0021d4 thru 0x0046bc (0x24e9 bytes) 000985 1741898519.163424 mapread 0x000899 thru 0x002e54 (0x25bc bytes) 000987 1741898519.175872 fallocate 0x006152 thru 0x015ab0 (0xf95f bytes) 000988 1741898519.190274 punch from 0x004c7e to 0x00e14b (0x94ce bytes) 000990 1741898519.198433 write 0x03c257 thru 0x03ffff (0x3da9 bytes) 000991 1741898519.204093 mapread 0x009c8a thru 0x010fce (0x7345 bytes) 000992 1741898519.220501 punch from 0x011f5e to 0x01aa39 (0x8adc bytes) 000993 1741898519.231757 punch from 0x03abed to 0x03ffff (0x5413 bytes) 000994 1741898519.249981 write 0x02492d thru 0x02afe2 (0x66b6 bytes) 000995 1741898519.265095 trunc from 0x040000 to 0x03e8de (0x1721 bytes) 000997 1741898519.278552 write 0x02869c thru 0x03834c (0xfcb1 bytes) 000998 1741898519.282418 fallocate 0x03f482 thru 0x03ffff (0xb7e bytes) 000999 1741898519.285022 write 0x02c78b thru 0x02f1ab (0x2a21 bytes) All operations completed A-OK! PASS 70a (67s) == sanity-flr test 71: check mirror extend parent fid ==== 16:42:01 (1741898521) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x200000405:0x41e:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (5s) == sanity-flr test 200a: stress test ===================== 16:42:06 (1741898526) Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre2 Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4022688 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5292800 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 5804832 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7573664 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 5836480 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8076608 .. done Extending file size to 7192736 .. Extending file size to 8177152 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8204448 .. failed Extending file size to 8293408 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8919136 .. Extending file size to 8612160 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8649472 .. Extending file size to 9061920 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8945472 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 31296 31297 31299 31300 31301 192.168.201.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg117-client.virtnet /mnt/lustre2 (opts:) 192.168.201.117@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg117-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (66s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 16:43:12 (1741898592) Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre2 Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 1059072 .. Extending file size to 7463552 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7180320 .. Extending file size to 7689856 .. Extending file size to 8138016 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8315008 .. Extending file size to 8261184 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8802272 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8764832 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9191872 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9252896 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9366016 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 2401 2402 2404 2405 2407 192.168.201.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg117-client.virtnet /mnt/lustre2 (opts:) 192.168.201.117@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg117-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 16:44:16 (1741898656) Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3016 1284672 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2272 1285416 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 23080 3583940 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 14912 3592108 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37992 7176048 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3016 1284672 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2272 1285416 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 23080 3583940 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 14912 3592108 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37992 7176048 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0470056 s, 44.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0746768 s, 28.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg117-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 16:44:23 (1741898663) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0537763 s, 39.0 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 16:44:25 (1741898665) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x42a:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x516:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x516:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x517:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x533:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 16:44:28 (1741898668) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0360479 s, 58.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x61:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x60:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x61:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 16:44:31 (1741898671) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.038119 s, 55.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x518:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x519:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x534:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x518:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x519:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x534:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x535:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 16:44:34 (1741898674) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0388149 s, 54.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x536:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x536:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x537:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 16:44:58 (1741898698) Waiting for MDT destroys to complete oleg117-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg117-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg117-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0375349 s, 27.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x62:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x63:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x62:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x62:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x63:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x62:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x63:0x0] } PASS 204d (33s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 16:45:31 (1741898731) Waiting for MDT destroys to complete oleg117-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 oleg117-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg117-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x538:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x539:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0302966 s, 69.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x538:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x539:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x538:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x539:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (34s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 16:46:05 (1741898765) Waiting for MDT destroys to complete oleg117-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg117-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg117-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg117-server: Pool lustre.test_204f created Adding targets to pool oleg117-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg117-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0398172 s, 52.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x64:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x64:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x64:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x64:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg117-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg117-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg117-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (50s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 16:46:55 (1741898815) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 16:46:57 (1741898817) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 16:47:00 (1741898820) oleg117-server: Pool lustre.test_206 created Adding targets to pool oleg117-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg117-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg117-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg117-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg117-server: Pool lustre.test_206 destroyed PASS 206 (14s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 16:47:14 (1741898834) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0269028 s, 39.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0168439 s, 62.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xe6904a0 Mirror 3: 0xe6904a0 PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 16:47:17 (1741898837) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 16:47:18 (1741898838) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 16:47:19 (1741898839) Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0163427 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1220000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2040000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2040000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2040000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x21b0000 Read this (4 bytes): BBBB 192.168.201.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg117-client.virtnet /mnt/lustre2 (opts:) PASS 209a (17s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 16:47:36 (1741898856) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00481509 s, 1.7 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 16:47:38 (1741898858) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0146558 s, 71.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x449 lmm_fid: [0x200000405:0x449:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x449 lmm_fid: [0x200000405:0x449:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x544:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339612233 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-03-13 16:47:39.000000000 -0400 Modify: 2025-03-13 16:47:39.000000000 -0400 Change: 2025-03-13 16:47:39.000000000 -0400 Birth: - PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 16:47:41 (1741898861) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0183086 s, 57.3 MB/s oleg117-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg117-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg117-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg117-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 oleg117-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg117-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg117-server: O/280000401/d5/545: File not found by ext2_lookup PASS 210b (10s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 16:47:51 (1741898871) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.047976 s, 854 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00924729 s, 443 kB/s size after second write -rw-r--r-- 1 root root 4096 Mar 13 16:47 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x44e:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Mar 13 16:47 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 984 sec ============ 16:47:53 (1741898873) === sanity-flr: start cleanup 16:47:54 (1741898874) === === sanity-flr: finish cleanup 16:47:58 (1741898878) ===