-----============= acceptance-small: sanity-flr ============----- Tue Nov 5 05:16:55 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 6 201 44c === sanity-flr: start setup 05:17:03 (1730801823) === oleg245-client.virtnet: executing check_config_client /mnt/lustre oleg245-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg245-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b58cd800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b58cd800.idle_timeout=debug disable quota as required oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 05:17:20 (1730801840) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f7613] == sanity-flr test 0a: lfs mirror create with -N option == 05:17:21 (1730801841) PASS 0a (8s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 05:17:29 (1730801849) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 05:17:30 (1730801850) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 05:17:32 (1730801852) PASS 0d (11s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 05:17:43 (1730801863) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 05:17:45 (1730801865) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 05:17:46 (1730801866) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b58cd800.stats=clear osc.lustre-OST0001-osc-ffff8800b58cd800.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 05:17:50 (1730801870) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (5s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 05:17:55 (1730801875) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 05:17:59 (1730801879) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 05:18:00 (1730801880) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 05:18:04 (1730801884) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 05:18:08 (1730801888) PASS 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 05:18:11 (1730801891) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 05:18:14 (1730801894) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 05:18:18 (1730801898) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 05:18:21 (1730801901) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB) copied, 0.308652 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0545525 s, 19.2 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 05:18:24 (1730801904) 14+0 records in 14+0 records out 14680064 bytes (15 MB) copied, 0.150849 s, 97.3 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 05:18:27 (1730801907) Stopping /mnt/lustre-ost1 (opts:) on oleg245-server oleg245-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b58cd800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b58cd800.ost_server_uuid in DISCONN state after 0 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg245-server: oleg245-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg245-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg245-server oleg245-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b58cd800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b58cd800.ost_server_uuid in DISCONN state after 3 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg245-server: oleg245-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg245-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (31s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 05:18:58 (1730801938) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.162901 s, 51.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg245-server oleg245-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b58cd800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b58cd800.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg245-server: oleg245-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg245-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (16s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 05:19:15 (1730801955) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg245-server oleg245-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b58cd800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b58cd800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg245-server: oleg245-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg245-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg245-server oleg245-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b58cd800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b58cd800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg245-server: oleg245-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg245-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (27s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 05:19:42 (1730801982) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 1.08198 s, 48.5 MB/s -rw-r--r-- 1 root root 52428800 Nov 5 05:19 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.487s user 0m0.000s sys 0m0.123s Stopping /mnt/lustre-ost1 (opts:) on oleg245-server oleg245-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b58cd800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b58cd800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... real 0m0.358s user 0m0.000s sys 0m0.133s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg245-server: oleg245-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg245-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg245-server oleg245-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b58cd800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b58cd800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.369s user 0m0.000s sys 0m0.120s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg245-server: oleg245-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg245-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg245-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b58cd800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b58cd800.ost_server_uuid in FULL state after 0 sec PASS 33b (32s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 05:20:14 (1730802014) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 05:20:16 (1730802016) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 05:20:17 (1730802017) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 05:20:18 (1730802018) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 05:20:21 (1730802021) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.199857 s, 63.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 05:20:25 (1730802025) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0103334 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0115208 s, 91.0 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 05:20:27 (1730802027) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 05:20:30 (1730802030) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000754775 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00071999 s, 1.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000655342 s, 1.6 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.0009904 s, 1.0 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 05:20:33 (1730802033) create test files with size 15736767 .. 92badd6fb79181925cedf7ec857b549f - 8e7a11e1965ebee34aa45b430d608fee - e1478746b2e21f23ca9e5058b5a1bbfd - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg245-client.virtnet /mnt/lustre (opts:) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre PASS 37 (8s) == sanity-flr test 38: resync ============================ 05:20:41 (1730802041) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (5s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 05:20:46 (1730802046) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 05:20:48 (1730802048) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0328139 s, 32.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x37:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x38:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0130993 s, 80.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3d:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 05:20:51 (1730802051) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.04142 s, 50.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.134557 s, 31.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0823853 s, 25.5 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (5s) == sanity-flr test 42: lfs mirror verify ================= 05:20:56 (1730802056) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 05:20:57 (1730802057) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 05:20:58 (1730802058) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0262498 s, 39.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 05:21:02 (1730802062) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.146183 s, 21.5 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x12:0x0]: Invalid argument (22) PASS 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 05:21:06 (1730802066) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x48:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x44:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x17:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 05:21:09 (1730802069) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.10595 s, 99.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg245-client.virtnet /mnt/lustre (opts:) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (4s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 05:21:13 (1730802073) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800a1d40800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.121912 s, 86.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg245-client.virtnet /mnt/lustre (opts:) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff88013730f800.xattr_cache=1 PASS 44e (6s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 05:21:19 (1730802079) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (12s) == sanity-flr test 46: Verify setstripe --copy option ==== 05:21:31 (1730802091) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (12s) == sanity-flr test 47: Verify mirror obj alloc =========== 05:21:43 (1730802103) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 05:21:45 (1730802105) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.137956 s, 22.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0944981 s, 33.3 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 6349b9ae2694a0975987e9bbb8b706ed - mirror 1 checksum: abfaef102ee0ae2b490e8e9c53c287e8 - mirror 2 checksum: 6349b9ae2694a0975987e9bbb8b706ed - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Nov 5 05:21:47 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Nov 5 05:21:47 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 6349b9ae2694a0975987e9bbb8b706ed - mirror 1 checksum: abfaef102ee0ae2b490e8e9c53c287e8 - mirror 2 checksum: abfaef102ee0ae2b490e8e9c53c287e8 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Nov 5 05:21:47 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Nov 5 05:21:47 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } PASS 48 (5s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 05:21:50 (1730802110) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.013568 s, 77.3 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (3s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 05:21:53 (1730802113) Failing mds1 on oleg245-server Stopping /mnt/lustre-mds1 (opts:) on oleg245-server 05:22:01 (1730802121) shut down Failover mds1 to oleg245-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 05:22:16 (1730802136) targets are mounted 05:22:16 (1730802136) facet_failover done oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0141447 s, 74.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } PASS 50A (31s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 05:22:24 (1730802144) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0179739 s, 1.1 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.01197 s, 1.7 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 05:22:28 (1730802148) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0334171 s, 613 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0170932 s, 1.2 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 05:22:32 (1730802152) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00534278 s, 3.1 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 05:22:35 (1730802155) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0223355 s, 917 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 05:22:38 (1730802158) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00805046 s, 509 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.013132 s, 312 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 05:22:40 (1730802160) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (8s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 05:22:48 (1730802168) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 05:23:08 (1730802188) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 05:23:26 (1730802206) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 05:23:44 (1730802224) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 05:23:47 (1730802227) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 26171 mirror split pid 26172 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7503: 26171 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4851: 26172 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 05:24:49 (1730802289) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 2290 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read 000002 1730802290.396301 write 0x01f5f9 thru 0x02ab34 (0xb53c bytes) truncating to largest ever: 0x5f76 000004 1730802290.399517 trunc from 0x02ab35 to 0x005f75 (0x24bbf bytes) 000005 1730802290.411196 punch from 0x004bc8 to 0x005f75 (0x13ae bytes) 000006 1730802290.416136 mapwrite 0x023c9d thru 0x02f64b (0xb9af bytes) 000007 1730802290.465714 write 0x03d02b thru 0x03ffff (0x2fd5 bytes) fallocating to largest ever: 0x3fee4 000009 1730802290.466772 fallocate 0x03e010 thru 0x03fee3 (0x1ed4 bytes) 000010 1730802290.469350 fallocate 0x022545 thru 0x02b088 (0x8b44 bytes) 000011 1730802290.472121 mapwrite 0x01bf99 thru 0x02870a (0xc772 bytes) 000012 1730802290.501993 write 0x029de6 thru 0x02d106 (0x3321 bytes) 000013 1730802290.502752 mapwrite 0x002481 thru 0x0101a6 (0xdd26 bytes) 000014 1730802290.536176 MIRROR_EXTEND to 2 mirrors 000016 1730802290.831032 punch from 0x032d0b to 0x03ce2b (0xa121 bytes) 000017 1730802290.851901 trunc from 0x040000 to 0x016b0d (0x294f2 bytes) 000018 1730802290.884527 mapwrite 0x02ab2f thru 0x0316af (0x6b81 bytes) 000019 1730802290.934548 read 0x0169f0 thru 0x01b72c (0x4d3d bytes) 000020 1730802290.939701 trunc from 0x0316b0 to 0x023106 (0xe5a9 bytes) 000021 1730802290.949327 fallocate 0x0180ca thru 0x01df39 (0x5e70 bytes) 000022 1730802290.953363 read 0x00e901 thru 0x0127ca (0x3eca bytes) 000023 1730802290.962166 read 0x003473 thru 0x007ea0 (0x4a2e bytes) 000024 1730802290.962536 trunc from 0x023107 to 0x0003b3 (0x22d53 bytes) 000025 1730802290.976242 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x52d:0x0]: Invalid argument (22) 000026 1730802291.049223 mapread 0x000269 thru 0x0003b3 (0x14b bytes) fallocating to largest ever: 0x40000 000027 1730802291.054997 fallocate 0x03f0d3 thru 0x03ffff (0xf2d bytes) 000028 1730802291.059300 read 0x000280 thru 0x0003b3 (0x134 bytes) 000029 1730802291.059779 mapwrite 0x03f9b4 thru 0x03ffff (0x64c bytes) 000030 1730802291.102358 fallocate 0x002885 thru 0x009815 (0x6f91 bytes) 000031 1730802291.108881 punch from 0x004778 to 0x007f34 (0x37bd bytes) 000032 1730802291.115207 write 0x031e7c thru 0x03d8c6 (0xba4b bytes) 000033 1730802291.121401 read 0x0117f6 thru 0x01e9d1 (0xd1dc bytes) 000035 1730802291.129260 read 0x0167d9 thru 0x023dd2 (0xd5fa bytes) 000037 1730802291.129682 read 0x02daef thru 0x03ab36 (0xd048 bytes) 000040 1730802291.130568 punch from 0x01939a to 0x02828a (0xeef1 bytes) 000041 1730802291.133384 read 0x00adac thru 0x01416a (0x93bf bytes) 000042 1730802291.133824 trunc from 0x040000 to 0x02f22a (0x10dd5 bytes) 000045 1730802291.143369 mapwrite 0x0027d9 thru 0x004d6d (0x2595 bytes) 000048 1730802291.154142 punch from 0x003403 to 0x01085d (0xd45b bytes) 000049 1730802291.159716 mapread 0x017e81 thru 0x01d677 (0x57f7 bytes) 000051 1730802291.172260 punch from 0x005126 to 0x0063ff (0x12da bytes) 000052 1730802291.175854 read 0x00e09a thru 0x00e965 (0x8cc bytes) 000053 1730802291.176363 trunc from 0x02f22b to 0x00e80f (0x20a1b bytes) 000054 1730802291.187581 mapread 0x003f87 thru 0x00bc4a (0x7cc4 bytes) 000055 1730802291.191644 fallocate 0x017005 thru 0x019658 (0x2654 bytes) 000056 1730802291.195766 mapread 0x00ed98 thru 0x019658 (0xa8c1 bytes) 000058 1730802291.202148 mapread 0x00601c thru 0x015158 (0xf13d bytes) 000059 1730802291.203319 write 0x038e7f thru 0x03ffff (0x7181 bytes) 000060 1730802291.204195 fallocate 0x01e327 thru 0x021e28 (0x3b02 bytes) 000061 1730802291.206507 fallocate 0x03e04d thru 0x03e138 (0xec bytes) 000062 1730802291.208509 mapread 0x0282c4 thru 0x028cba (0x9f7 bytes) 000063 1730802291.213500 read 0x01d5cb thru 0x023f06 (0x693c bytes) 000064 1730802291.213987 mapread 0x007737 thru 0x00be5e (0x4728 bytes) 000065 1730802291.214675 fallocate 0x02af5f thru 0x02c8a4 (0x1946 bytes) 000066 1730802291.218629 mapwrite 0x00bfc4 thru 0x01185b (0x5898 bytes) 000068 1730802291.236870 punch from 0x017c52 to 0x0186b8 (0xa67 bytes) 000069 1730802291.242038 write 0x0109d5 thru 0x017aa0 (0x70cc bytes) 000071 1730802291.249168 write 0x01c333 thru 0x027362 (0xb030 bytes) 000072 1730802291.251185 trunc from 0x040000 to 0x00f95d (0x306a2 bytes) 000074 1730802291.265300 mapread 0x00b23d thru 0x00f95d (0x4721 bytes) 000076 1730802291.267285 write 0x007e8b thru 0x00a11c (0x2292 bytes) 000077 1730802291.268560 write 0x033f83 thru 0x0364ed (0x256b bytes) 000078 1730802291.273453 MIRROR_EXTEND to 2 mirrors 000079 1730802291.595228 write 0x038753 thru 0x03af0b (0x27b9 bytes) 000080 1730802291.622452 read 0x00c191 thru 0x011452 (0x52c2 bytes) 000081 1730802291.630393 mapread 0x01702c thru 0x018fdf (0x1fb4 bytes) 000083 1730802291.639466 fallocate 0x029fac thru 0x02a1d8 (0x22d bytes) 000084 1730802291.642268 mapread 0x03772c thru 0x03af0b (0x37e0 bytes) 000085 1730802291.649333 read 0x007cae thru 0x015331 (0xd684 bytes) 000087 1730802291.657433 mapread 0x026b67 thru 0x02c2ec (0x5786 bytes) 000088 1730802291.658410 read 0x00960d thru 0x00c7b5 (0x31a9 bytes) 000091 1730802291.659183 mapread 0x00129b thru 0x00efaa (0xdd10 bytes) 000092 1730802291.663309 MIRROR_RESYNC 2 mirrors 000093 1730802292.045384 trunc from 0x03af0c to 0x019540 (0x219cb bytes) 000094 1730802292.109587 trunc from 0x019541 to 0x01ea75 (0x5535 bytes) 000095 1730802292.122448 mapwrite 0x00772a thru 0x010887 (0x915e bytes) 000096 1730802292.152321 fallocate 0x0317cf thru 0x034cde (0x3510 bytes) 000097 1730802292.156048 mapread 0x0026b3 thru 0x011f73 (0xf8c1 bytes) 000098 1730802292.156658 mapread 0x02e947 thru 0x034cde (0x6398 bytes) 000099 1730802292.161281 fallocate 0x012e74 thru 0x0169dc (0x3b69 bytes) 000100 1730802292.163598 mapread 0x01f0e9 thru 0x02e03b (0xef53 bytes) 000101 1730802292.172471 write 0x01dbc2 thru 0x029572 (0xb9b1 bytes) 000102 1730802292.174496 read 0x032ca9 thru 0x034cde (0x2036 bytes) 000103 1730802292.174874 punch from 0x03250d to 0x0344e1 (0x1fd5 bytes) 000104 1730802292.180660 write 0x0268bb thru 0x02cf6a (0x66b0 bytes) 000105 1730802292.181799 punch from 0x01d219 to 0x01ee28 (0x1c10 bytes) 000107 1730802292.193831 write 0x0113aa thru 0x01410c (0x2d63 bytes) 000108 1730802292.200108 trunc from 0x034cdf to 0x02506f (0xfc6f bytes) 000109 1730802292.211864 MIRROR_EXTEND to 3 mirrors 000110 1730802292.571710 read 0x02363a thru 0x02506f (0x1a36 bytes) 000111 1730802292.585009 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x52d:0x0]: Invalid argument (22) 000112 1730802292.754567 write 0x011822 thru 0x013836 (0x2015 bytes) 000113 1730802292.783590 trunc from 0x025070 to 0x03bb4b (0x16adc bytes) 000114 1730802292.794888 mapwrite 0x01282d thru 0x014e5b (0x262f bytes) 000115 1730802292.813058 write 0x0312a6 thru 0x03bed3 (0xac2e bytes) 000116 1730802292.820204 trunc from 0x03bed4 to 0x037df4 (0x40df bytes) 000118 1730802292.831329 write 0x024c1f thru 0x029ece (0x52b0 bytes) 000119 1730802292.837074 fallocate 0x03ee5b thru 0x03ffff (0x11a5 bytes) 000120 1730802292.842893 punch from 0x00a6af to 0x01709e (0xc9f0 bytes) 000121 1730802292.846263 mapwrite 0x00f318 thru 0x016acb (0x77b4 bytes) 000122 1730802292.873881 trunc from 0x040000 to 0x0088ca (0x37735 bytes) 000123 1730802292.887984 write 0x00fdf9 thru 0x01f399 (0xf5a1 bytes) 000124 1730802292.891433 mapwrite 0x00ef11 thru 0x010207 (0x12f7 bytes) 000125 1730802292.916439 trunc from 0x01f39a to 0x03c4b8 (0x1d11f bytes) 000126 1730802292.925359 fallocate 0x028ebf thru 0x037071 (0xe1b3 bytes) 000127 1730802292.928230 write 0x022bfd thru 0x02a957 (0x7d5b bytes) 000129 1730802292.933315 fallocate 0x016778 thru 0x0186c7 (0x1f50 bytes) 000130 1730802292.936821 read 0x02364a thru 0x02a396 (0x6d4d bytes) 000131 1730802292.937276 write 0x034761 thru 0x03c21c (0x7abc bytes) 000132 1730802292.946466 mapwrite 0x00ccf7 thru 0x013a19 (0x6d23 bytes) 000133 1730802292.971768 read 0x027f37 thru 0x02e5ab (0x6675 bytes) 000134 1730802292.982919 write 0x035465 thru 0x03555c (0xf8 bytes) 000135 1730802292.983543 trunc from 0x03c4b9 to 0x03c4ea (0x32 bytes) 000136 1730802292.996991 punch from 0x009752 to 0x012d74 (0x9623 bytes) 000138 1730802293.012263 read 0x01cd74 thru 0x01d5cc (0x859 bytes) 000140 1730802293.013812 trunc from 0x03c4eb to 0x0233a1 (0x19149 bytes) 000141 1730802293.029272 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x52d:0x0]: Invalid argument (22) 000142 1730802293.123062 punch from 0x00838c to 0x01315f (0xadd4 bytes) 000143 1730802293.125595 mapread 0x010781 thru 0x01df5d (0xd7dd bytes) 000144 1730802293.146797 mapread 0x012486 thru 0x019f9b (0x7b16 bytes) 000145 1730802293.148693 mapread 0x015a0f thru 0x019566 (0x3b58 bytes) 000146 1730802293.149428 fallocate 0x0024b4 thru 0x011973 (0xf4c0 bytes) 000147 1730802293.156128 fallocate 0x01dfe3 thru 0x02aa59 (0xca77 bytes) 000148 1730802293.161182 fallocate 0x02b264 thru 0x0349a5 (0x9742 bytes) 000150 1730802293.166300 fallocate 0x023199 thru 0x032769 (0xf5d1 bytes) 000151 1730802293.169440 fallocate 0x030f9e thru 0x038cf2 (0x7d55 bytes) 000154 1730802293.177377 MIRROR_EXTEND to 2 mirrors 000155 1730802293.616758 mapread 0x013b29 thru 0x01bbd4 (0x80ac bytes) 000156 1730802293.643163 read 0x026efc thru 0x02ca72 (0x5b77 bytes) 000158 1730802293.678478 punch from 0x028c94 to 0x0349a5 (0xbd12 bytes) 000160 1730802293.713650 read 0x01b7f7 thru 0x0221bf (0x69c9 bytes) 000161 1730802293.729133 trunc from 0x0349a6 to 0x00d78e (0x27217 bytes) 000162 1730802293.750097 write 0x034e97 thru 0x03ffff (0xb169 bytes) 000163 1730802293.751636 read 0x03614a thru 0x03ffff (0x9eb6 bytes) 000164 1730802293.752505 punch from 0x038d47 to 0x03ffff (0x72b9 bytes) 000165 1730802293.761026 write 0x002afd thru 0x003847 (0xd4b bytes) 000166 1730802293.761482 mapread 0x0201b1 thru 0x02b873 (0xb6c3 bytes) 000167 1730802293.772197 trunc from 0x040000 to 0x004c50 (0x3b3af bytes) 000168 1730802293.787376 punch from 0x003d60 to 0x004c50 (0xef1 bytes) 000169 1730802293.799641 read 0x001e57 thru 0x0047e8 (0x2992 bytes) 000170 1730802293.806582 mapread 0x002f86 thru 0x004c50 (0x1ccb bytes) 000171 1730802293.807942 fallocate 0x012daf thru 0x01cd3d (0x9f8f bytes) 000172 1730802293.814272 fallocate 0x01084c thru 0x019f85 (0x973a bytes) 000173 1730802293.819581 fallocate 0x000d5b thru 0x006154 (0x53fa bytes) 000175 1730802293.825602 punch from 0x015ee8 to 0x01cd3d (0x6e56 bytes) 000176 1730802293.837253 MIRROR_RESYNC 2 mirrors 000178 1730802294.187422 read 0x017fbd thru 0x01cd3d (0x4d81 bytes) 000180 1730802294.194580 read 0x008555 thru 0x00f11d (0x6bc9 bytes) 000181 1730802294.202091 write 0x025b74 thru 0x02c79e (0x6c2b bytes) 000182 1730802294.233133 write 0x01dada thru 0x02a97d (0xcea4 bytes) 000183 1730802294.242181 read 0x02b7b2 thru 0x02c79e (0xfed bytes) 000185 1730802294.242935 punch from 0x01bd50 to 0x027fec (0xc29d bytes) 000186 1730802294.261539 punch from 0x01fb23 to 0x020ab9 (0xf97 bytes) 000188 1730802294.266154 read 0x0136eb thru 0x016b3d (0x3453 bytes) 000189 1730802294.273861 read 0x02754c thru 0x02c79e (0x5253 bytes) 000191 1730802294.275327 read 0x029cb2 thru 0x02c79e (0x2aed bytes) 000192 1730802294.275748 fallocate 0x0085b7 thru 0x00b7ea (0x3234 bytes) 000193 1730802294.280041 punch from 0x02c09a to 0x02c79e (0x705 bytes) 000196 1730802294.284332 mapwrite 0x0226b4 thru 0x0236e7 (0x1034 bytes) 000197 1730802294.298940 mapread 0x001046 thru 0x004992 (0x394d bytes) 000198 1730802294.299722 write 0x03d882 thru 0x03ec57 (0x13d6 bytes) 000199 1730802294.300999 trunc from 0x03ec58 to 0x004481 (0x3a7d6 bytes) 000200 1730802294.313314 trunc from 0x004482 to 0x01d308 (0x18e87 bytes) 000201 1730802294.324815 punch from 0x00f1d1 to 0x00f557 (0x387 bytes) 000202 1730802294.329208 read 0x013779 thru 0x01d308 (0x9b90 bytes) 000203 1730802294.334794 read 0x0090bb thru 0x013497 (0xa3dd bytes) 000204 1730802294.335264 trunc from 0x01d309 to 0x0090ed (0x1421b bytes) 000205 1730802294.344940 read 0x006568 thru 0x0090ed (0x2b86 bytes) 000207 1730802294.345851 mapwrite 0x036129 thru 0x03f5a4 (0x947c bytes) 000208 1730802294.381146 mapwrite 0x009076 thru 0x00dd9b (0x4d26 bytes) 000210 1730802294.394794 trunc from 0x03f5a5 to 0x011516 (0x2e08e bytes) 000212 1730802294.410773 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x52d:0x0]: Invalid argument (22) 000215 1730802294.493557 write 0x031aed thru 0x03f08f (0xd5a3 bytes) 000216 1730802294.499033 mapwrite 0x014076 thru 0x01a593 (0x651e bytes) 000217 1730802294.523091 read 0x003d2f thru 0x00a9fb (0x6ccd bytes) 000221 1730802294.530449 fallocate 0x0263bf thru 0x0272f9 (0xf3b bytes) 000222 1730802294.534679 mapread 0x004c7f thru 0x0104e3 (0xb865 bytes) 000223 1730802294.539729 fallocate 0x038a3e thru 0x03a851 (0x1e14 bytes) 000226 1730802294.543412 read 0x0327f8 thru 0x034a61 (0x226a bytes) 000227 1730802294.543826 mapwrite 0x02ddb8 thru 0x031124 (0x336d bytes) 000228 1730802294.562315 read 0x01babf thru 0x02a05b (0xe59d bytes) 000229 1730802294.562922 trunc from 0x03f090 to 0x02da6e (0x11621 bytes) 000231 1730802294.578092 mapread 0x0131bb thru 0x017ab8 (0x48fe bytes) 000232 1730802294.579452 mapwrite 0x004ac5 thru 0x00f342 (0xa87e bytes) 000234 1730802294.605994 write 0x014496 thru 0x01b68d (0x71f8 bytes) 000235 1730802294.611904 mapwrite 0x013045 thru 0x01b6f2 (0x86ae bytes) 000236 1730802294.642166 MIRROR_EXTEND to 2 mirrors 000238 1730802294.904716 mapwrite 0x014657 thru 0x014a01 (0x3ab bytes) 000239 1730802294.971882 write 0x023879 thru 0x02fec0 (0xc648 bytes) 000243 1730802294.986716 mapwrite 0x03b8a0 thru 0x03ffff (0x4760 bytes) 000244 1730802295.047521 mapwrite 0x001946 thru 0x00244b (0xb06 bytes) 000245 1730802295.068707 trunc from 0x040000 to 0x007624 (0x389db bytes) 000246 1730802295.080372 fallocate 0x039f7d thru 0x03ffff (0x6083 bytes) 000247 1730802295.085977 mapread 0x01ef84 thru 0x022f23 (0x3fa0 bytes) 000248 1730802295.093749 trunc from 0x040000 to 0x0222c0 (0x1dd3f bytes) 000249 1730802295.107668 fallocate 0x01be61 thru 0x027522 (0xb6c2 bytes) 000250 1730802295.111617 read 0x0233ee thru 0x027522 (0x4135 bytes) 000251 1730802295.119929 punch from 0x0041b3 to 0x00a17f (0x5fcd bytes) 000252 1730802295.122722 read 0x018bd2 thru 0x01cadc (0x3f0b bytes) 000253 1730802295.129079 read 0x0032dc thru 0x0101f2 (0xcf17 bytes) 000254 1730802295.129418 trunc from 0x027523 to 0x02b744 (0x4222 bytes) 000255 1730802295.141268 mapread 0x00f2e6 thru 0x01886b (0x9586 bytes) 000257 1730802295.143856 punch from 0x012e5a to 0x01ea52 (0xbbf9 bytes) 000258 1730802295.148505 read 0x0013a7 thru 0x0015e5 (0x23f bytes) 000259 1730802295.149544 fallocate 0x008499 thru 0x011e6e (0x99d6 bytes) 000261 1730802295.154104 fallocate 0x02e9f5 thru 0x03371d (0x4d29 bytes) 000263 1730802295.159231 punch from 0x005ec3 to 0x008331 (0x246f bytes) 000264 1730802295.162392 write 0x02f6b6 thru 0x03103d (0x1988 bytes) 000266 1730802295.163666 read 0x017d1c thru 0x02796d (0xfc52 bytes) 000268 1730802295.179059 MIRROR_RESYNC 2 mirrors 000270 1730802295.489103 write 0x03b60d thru 0x03ffff (0x49f3 bytes) 000271 1730802295.516329 fallocate 0x022663 thru 0x027a43 (0x53e1 bytes) 000273 1730802295.522458 read 0x0085e0 thru 0x00a408 (0x1e29 bytes) 000274 1730802295.534897 write 0x01ec42 thru 0x02d373 (0xe732 bytes) 000275 1730802295.546638 write 0x008c68 thru 0x00aed7 (0x2270 bytes) 000276 1730802295.550630 fallocate 0x02f978 thru 0x038855 (0x8ede bytes) 000277 1730802295.562562 MIRROR_EXTEND to 3 mirrors 000278 1730802295.936534 mapwrite 0x027b3f thru 0x02b890 (0x3d52 bytes) 000279 1730802296.017655 punch from 0x00b3a5 to 0x00e3c0 (0x301c bytes) 000280 1730802296.029862 MIRROR_EXTEND to 4 mirrors 000282 1730802296.465277 mapwrite 0x000f89 thru 0x00fd35 (0xedad bytes) 000283 1730802296.563517 mapwrite 0x01d3df thru 0x0258df (0x8501 bytes) 000286 1730802296.584805 mapread 0x01a784 thru 0x02513b (0xa9b8 bytes) 000287 1730802296.586011 mapwrite 0x032f2e thru 0x03a620 (0x76f3 bytes) 000289 1730802296.607470 trunc from 0x040000 to 0x00d7b6 (0x32849 bytes) 000291 1730802296.621080 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x52d:0x0]: Invalid argument (22) 000293 1730802296.767060 write 0x018a5d thru 0x01e11d (0x56c1 bytes) 000294 1730802296.777997 read 0x00c030 thru 0x0157a2 (0x9773 bytes) 000297 1730802296.786675 trunc from 0x01e11e to 0x001f58 (0x1c1c5 bytes) 000298 1730802296.800041 mapwrite 0x0360cf thru 0x03ffff (0x9f31 bytes) 000299 1730802296.842843 MIRROR_EXTEND to 4 mirrors 000300 1730802297.140136 write 0x02fe72 thru 0x03fcca (0xfe59 bytes) 000302 1730802297.172245 mapread 0x02e255 thru 0x0358ac (0x7658 bytes) 000304 1730802297.181804 mapread 0x03eed6 thru 0x03f1d6 (0x301 bytes) 000306 1730802297.184713 mapwrite 0x01186e thru 0x01581d (0x3fb0 bytes) 000307 1730802297.212262 mapwrite 0x014c0b thru 0x01d3a7 (0x879d bytes) 000308 1730802297.225805 read 0x007af3 thru 0x016785 (0xec93 bytes) 000309 1730802297.230047 punch from 0x00258e to 0x009c4f (0x76c2 bytes) 000310 1730802297.232858 fallocate 0x00a52e thru 0x014001 (0x9ad4 bytes) 000311 1730802297.235948 punch from 0x013ec0 to 0x0220f1 (0xe232 bytes) 000312 1730802297.239476 fallocate 0x01e831 thru 0x021121 (0x28f1 bytes) 000314 1730802297.242305 read 0x00948d thru 0x00a765 (0x12d9 bytes) 000315 1730802297.247270 fallocate 0x015b4d thru 0x02053a (0xa9ee bytes) 000317 1730802297.250873 mapread 0x028777 thru 0x0353da (0xcc64 bytes) 000318 1730802297.251656 mapwrite 0x02abe5 thru 0x02b427 (0x843 bytes) 000319 1730802297.264768 write 0x01970c thru 0x028c0c (0xf501 bytes) 000320 1730802297.266257 fallocate 0x00148b thru 0x0049e2 (0x3558 bytes) 000321 1730802297.274640 MIRROR_EXTEND to 5 mirrors 000323 1730802297.545630 fallocate 0x0126b1 thru 0x013552 (0xea2 bytes) 000324 1730802297.572509 write 0x0377ea thru 0x03ffff (0x8816 bytes) 000325 1730802297.578695 mapread 0x012d80 thru 0x017b4b (0x4dcc bytes) 000326 1730802297.591450 trunc from 0x040000 to 0x03455e (0xbaa1 bytes) 000328 1730802297.624763 MIRROR_EXTEND to 6 mirrors 000329 1730802297.902675 write 0x02ce89 thru 0x036639 (0x97b1 bytes) 000330 1730802297.931470 mapread 0x021e5f thru 0x0282f2 (0x6494 bytes) 000331 1730802297.948506 fallocate 0x03038e thru 0x03d979 (0xd5ec bytes) 000332 1730802297.955016 mapread 0x01964e thru 0x0265bd (0xcf70 bytes) 000333 1730802297.963882 fallocate 0x021fde thru 0x02944b (0x746e bytes) 000334 1730802297.966937 punch from 0x031174 to 0x03d979 (0xc806 bytes) 000335 1730802297.973496 fallocate 0x02ccd2 thru 0x03858d (0xb8bc bytes) 000336 1730802297.975775 trunc from 0x03d97a to 0x02500d (0x1896c bytes) 000337 1730802297.993206 trunc from 0x02500e to 0x0083f2 (0x1cc1b bytes) 000339 1730802298.003339 mapread 0x0063ba thru 0x0083f2 (0x2039 bytes) 000340 1730802298.011774 write 0x0127c1 thru 0x01a8e5 (0x8125 bytes) 000341 1730802298.012674 read 0x008745 thru 0x00a892 (0x214e bytes) 000343 1730802298.018880 punch from 0x0036f8 to 0x00cef8 (0x9801 bytes) 000344 1730802298.029800 MIRROR_EXTEND to 7 mirrors 000346 1730802298.310829 mapread 0x000a7b thru 0x007a62 (0x6fe8 bytes) 000347 1730802298.328993 trunc from 0x01a8e6 to 0x01cf79 (0x2694 bytes) 000348 1730802298.363445 MIRROR_RESYNC 7 mirrors 000349 1730802299.346498 fallocate 0x009fc4 thru 0x00ed07 (0x4d44 bytes) 000350 1730802299.372738 MIRROR_RESYNC 7 mirrors 000351 1730802300.192102 trunc from 0x01cf7a to 0x0103c0 (0xcbb9 bytes) 000352 1730802300.220460 mapwrite 0x019439 thru 0x0204ed (0x70b5 bytes) 000353 1730802300.256535 read 0x014dea thru 0x017f94 (0x31ab bytes) 000354 1730802300.257033 mapwrite 0x034079 thru 0x03f755 (0xb6dd bytes) 000355 1730802300.293228 write 0x035165 thru 0x03e18f (0x902b bytes) 000356 1730802300.294281 fallocate 0x03386f thru 0x03aafa (0x728c bytes) 000358 1730802300.297074 trunc from 0x03f756 to 0x02c712 (0x13043 bytes) 000359 1730802300.305519 punch from 0x009642 to 0x00d243 (0x3c02 bytes) 000360 1730802300.309994 write 0x034a0e thru 0x03ffff (0xb5f2 bytes) 000361 1730802300.311602 punch from 0x017a54 to 0x023392 (0xb93f bytes) 000362 1730802300.320996 MIRROR_EXTEND to 8 mirrors 000364 1730802300.689224 mapwrite 0x03bdaf thru 0x03ffff (0x4251 bytes) 000365 1730802300.765763 fallocate 0x0058c0 thru 0x013181 (0xd8c2 bytes) 000366 1730802300.770613 trunc from 0x040000 to 0x00c957 (0x336a8 bytes) 000368 1730802300.792200 fallocate 0x02b3b6 thru 0x03584c (0xa497 bytes) 000369 1730802300.797928 write 0x02a906 thru 0x038400 (0xdafb bytes) 000370 1730802300.803144 MIRROR_RESYNC 8 mirrors 000372 1730802302.003478 fallocate 0x03e848 thru 0x03ffff (0x17b8 bytes) 000373 1730802302.032657 trunc from 0x038401 to 0x015659 (0x22da7 bytes) 000374 1730802302.046582 read 0x00b53d thru 0x015659 (0xa11d bytes) 000375 1730802302.052523 punch from 0x008ab0 to 0x0152f6 (0xc847 bytes) 000376 1730802302.058394 mapwrite 0x02f784 thru 0x036fc8 (0x7845 bytes) 000378 1730802302.117279 write 0x032d58 thru 0x03c3f9 (0x96a2 bytes) 000379 1730802302.119409 mapread 0x0095fb thru 0x013e00 (0xa806 bytes) 000381 1730802302.143685 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x52d:0x0]: Invalid argument (22) 000382 1730802302.296908 read 0x009227 thru 0x00c9e4 (0x37be bytes) 000383 1730802302.301276 punch from 0x000887 to 0x0016f2 (0xe6c bytes) 000384 1730802302.306278 mapwrite 0x012a47 thru 0x02235a (0xf914 bytes) 000386 1730802302.353239 write 0x03a267 thru 0x03ffff (0x5d99 bytes) 000387 1730802302.361460 read 0x02eda0 thru 0x03527b (0x64dc bytes) 000388 1730802302.368283 read 0x029cb5 thru 0x039093 (0xf3df bytes) 000389 1730802302.368773 read 0x03e018 thru 0x03ffff (0x1fe8 bytes) 000390 1730802302.369144 fallocate 0x037d3d thru 0x03b0dd (0x33a1 bytes) 000391 1730802302.374910 read 0x032100 thru 0x03ffff (0xdf00 bytes) 000394 1730802302.375930 trunc from 0x040000 to 0x001dc8 (0x3e237 bytes) 000395 1730802302.387197 write 0x03f88a thru 0x03ffff (0x776 bytes) 000396 1730802302.388768 punch from 0x023d74 to 0x025e73 (0x2100 bytes) 000398 1730802302.393213 mapwrite 0x015bdf thru 0x01dfd4 (0x83f6 bytes) 000399 1730802302.414178 mapwrite 0x0183a7 thru 0x022b00 (0xa75a bytes) 000401 1730802302.432307 mapwrite 0x027018 thru 0x032c35 (0xbc1e bytes) 000402 1730802302.456718 MIRROR_RESYNC 7 mirrors 000403 1730802303.355195 mapread 0x007786 thru 0x009893 (0x210e bytes) 000405 1730802303.370813 read 0x035385 thru 0x035b47 (0x7c3 bytes) 000406 1730802303.375797 fallocate 0x009944 thru 0x011628 (0x7ce5 bytes) 000407 1730802303.402724 trunc from 0x040000 to 0x03dc12 (0x23ed bytes) 000408 1730802303.424594 write 0x037361 thru 0x0381fe (0xe9e bytes) 000409 1730802303.434255 mapread 0x00b270 thru 0x00ba82 (0x813 bytes) 000410 1730802303.443777 read 0x01411d thru 0x018382 (0x4266 bytes) 000411 1730802303.451616 read 0x0300e1 thru 0x03dc12 (0xdb32 bytes) 000412 1730802303.452118 read 0x023a75 thru 0x029b75 (0x6101 bytes) 000414 1730802303.452792 fallocate 0x034de6 thru 0x0381a9 (0x33c4 bytes) 000415 1730802303.456547 write 0x01f731 thru 0x026f8d (0x785d bytes) 000417 1730802303.460293 MIRROR_RESYNC 7 mirrors 000418 1730802304.473227 write 0x00bdd4 thru 0x019889 (0xdab6 bytes) 000419 1730802304.517834 read 0x01fe2f thru 0x0265ed (0x67bf bytes) 000421 1730802304.528583 read 0x0169e4 thru 0x026369 (0xf986 bytes) 000422 1730802304.536976 mapwrite 0x003f22 thru 0x012b54 (0xec33 bytes) 000424 1730802304.553157 mapread 0x0169fb thru 0x01bceb (0x52f1 bytes) 000426 1730802304.554160 mapwrite 0x027d0d thru 0x02b1c5 (0x34b9 bytes) 000427 1730802304.567702 mapread 0x0378cf thru 0x03dc12 (0x6344 bytes) 000428 1730802304.568935 mapwrite 0x0208fa thru 0x0235ef (0x2cf6 bytes) 000429 1730802304.589341 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x52d:0x0]: Invalid argument (22) 000430 1730802304.767117 fallocate 0x013dd0 thru 0x01ef49 (0xb17a bytes) 000431 1730802304.771806 mapread 0x01865a thru 0x01f9e4 (0x738b bytes) 000432 1730802304.788337 mapread 0x02023b thru 0x0282e2 (0x80a8 bytes) 000433 1730802304.798687 MIRROR_RESYNC 6 mirrors 000434 1730802305.657848 punch from 0x0175ca to 0x021f6e (0xa9a5 bytes) 000435 1730802305.697324 fallocate 0x010421 thru 0x0155ce (0x51ae bytes) 000436 1730802305.701192 read 0x007e06 thru 0x01458a (0xc785 bytes) 000438 1730802305.708247 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x52d:0x0]: Invalid argument (22) 000439 1730802305.870827 read 0x027465 thru 0x0347a4 (0xd340 bytes) 000440 1730802305.876499 mapwrite 0x02b601 thru 0x035700 (0xa100 bytes) 000441 1730802305.908042 mapwrite 0x000b67 thru 0x006266 (0x5700 bytes) 000442 1730802305.930890 mapread 0x013e7e thru 0x01ee19 (0xaf9c bytes) 000443 1730802305.939519 mapwrite 0x00f19a thru 0x01a459 (0xb2c0 bytes) 000444 1730802305.958554 write 0x015f24 thru 0x023390 (0xd46d bytes) 000445 1730802305.964262 read 0x029566 thru 0x02c842 (0x32dd bytes) 000446 1730802305.965908 trunc from 0x03dc13 to 0x002132 (0x3bae0 bytes) 000447 1730802305.987534 punch from 0x0012a4 to 0x002132 (0xe8f bytes) 000450 1730802305.997544 trunc from 0x002133 to 0x014f3c (0x12e0a bytes) 000452 1730802306.021062 read 0x00092b thru 0x005c9f (0x5375 bytes) 000453 1730802306.029725 trunc from 0x014f3d to 0x019ff6 (0x50ba bytes) 000454 1730802306.043946 punch from 0x008b4d to 0x00af81 (0x2435 bytes) 000455 1730802306.048640 punch from 0x010522 to 0x01246d (0x1f4c bytes) 000457 1730802306.053092 punch from 0x005a22 to 0x00ee9d (0x947c bytes) 000458 1730802306.058850 fallocate 0x0365b4 thru 0x03ffff (0x9a4c bytes) 000460 1730802306.067845 read 0x0192df thru 0x01e299 (0x4fbb bytes) 000461 1730802306.078455 fallocate 0x022c9b thru 0x02a261 (0x75c7 bytes) 000462 1730802306.086031 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x52d:0x0]: Invalid argument (22) 000463 1730802306.202481 fallocate 0x028200 thru 0x028254 (0x55 bytes) 000464 1730802306.205400 fallocate 0x03b5e4 thru 0x03ffff (0x4a1c bytes) 000465 1730802306.212059 read 0x00bdec thru 0x01b1d9 (0xf3ee bytes) 000466 1730802306.220283 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x52d:0x0]: Invalid argument (22) 000468 1730802306.351307 MIRROR_EXTEND to 4 mirrors 000469 1730802306.625581 write 0x0124e4 thru 0x01ee78 (0xc995 bytes) 000470 1730802306.648594 trunc from 0x040000 to 0x03e1a7 (0x1e58 bytes) 000471 1730802306.673462 mapwrite 0x016674 thru 0x018198 (0x1b25 bytes) 000472 1730802306.687302 trunc from 0x03e1a8 to 0x01f22c (0x1ef7b bytes) 000473 1730802306.700914 punch from 0x01814a to 0x01acea (0x2ba1 bytes) 000474 1730802306.705497 trunc from 0x01f22d to 0x02b030 (0xbe04 bytes) 000475 1730802306.720506 trunc from 0x02b031 to 0x039d82 (0xed52 bytes) 000476 1730802306.737510 trunc from 0x039d83 to 0x036987 (0x33fb bytes) 000477 1730802306.750493 fallocate 0x02a0ab thru 0x03859d (0xe4f3 bytes) 000478 1730802306.754118 fallocate 0x00fe6d thru 0x013132 (0x32c6 bytes) 000479 1730802306.756745 fallocate 0x011dd1 thru 0x01256f (0x79f bytes) 000480 1730802306.761631 trunc from 0x03859e to 0x01e2eb (0x1a2b2 bytes) 000481 1730802306.777691 trunc from 0x01e2ec to 0x009f4a (0x143a1 bytes) 000482 1730802306.786926 punch from 0x0040bf to 0x009f4a (0x5e8c bytes) 000483 1730802306.789600 read 0x008010 thru 0x009f4a (0x1f3b bytes) 000484 1730802306.793649 trunc from 0x009f4b to 0x002c68 (0x72e2 bytes) 000485 1730802306.809323 write 0x010384 thru 0x01927a (0x8ef7 bytes) 000486 1730802306.810623 mapwrite 0x02c046 thru 0x036dd1 (0xad8c bytes) 000488 1730802306.858760 mapwrite 0x02e0b3 thru 0x0381e8 (0xa136 bytes) 000489 1730802306.894646 trunc from 0x0381e9 to 0x01e59b (0x19c4d bytes) 000490 1730802306.908823 write 0x03b6ec thru 0x03ffff (0x4914 bytes) 000492 1730802306.910352 mapread 0x00e5f4 thru 0x0171de (0x8beb bytes) 000493 1730802306.917283 punch from 0x013c8f to 0x01e4d7 (0xa849 bytes) 000494 1730802306.922431 punch from 0x014f0b to 0x01d177 (0x826d bytes) 000495 1730802306.930428 mapread 0x00ffd0 thru 0x01bfed (0xc01e bytes) 000496 1730802306.944625 mapread 0x00ed39 thru 0x01332c (0x45f4 bytes) 000497 1730802306.945255 read 0x038d8a thru 0x03ffff (0x7276 bytes) 000498 1730802306.956437 MIRROR_EXTEND to 5 mirrors 000499 1730802307.281627 mapread 0x02a858 thru 0x0341ad (0x9956 bytes) 000500 1730802307.303210 trunc from 0x040000 to 0x021425 (0x1ebda bytes) 000501 1730802307.339359 fallocate 0x00325f thru 0x006d00 (0x3aa2 bytes) 000502 1730802307.342161 punch from 0x013908 to 0x01416e (0x867 bytes) 000503 1730802307.346720 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x52d:0x0]: Invalid argument (22) 000504 1730802307.452391 trunc from 0x021426 to 0x01ea76 (0x29af bytes) 000505 1730802307.464668 punch from 0x01aedd to 0x01ea76 (0x3b9a bytes) 000507 1730802307.469784 mapwrite 0x038768 thru 0x03c454 (0x3ced bytes) 000508 1730802307.514061 MIRROR_EXTEND to 5 mirrors 000510 1730802307.808890 punch from 0x00f04a to 0x0132f9 (0x42b0 bytes) 000511 1730802307.830117 mapwrite 0x010907 thru 0x01565a (0x4d54 bytes) 000512 1730802307.859466 mapread 0x030ada thru 0x03af24 (0xa44b bytes) 000513 1730802307.872533 write 0x0345f0 thru 0x03d9a3 (0x93b4 bytes) 000514 1730802307.876094 mapwrite 0x03d840 thru 0x03fc06 (0x23c7 bytes) 000516 1730802307.911520 fallocate 0x031822 thru 0x03abe8 (0x93c7 bytes) 000517 1730802307.914019 write 0x032600 thru 0x03f1eb (0xcbec bytes) 000519 1730802307.915221 mapread 0x00d289 thru 0x01b527 (0xe29f bytes) 000521 1730802307.919508 mapread 0x01fdd3 thru 0x028fc7 (0x91f5 bytes) 000522 1730802307.923783 mapread 0x010543 thru 0x019042 (0x8b00 bytes) 000523 1730802307.924452 mapread 0x00db83 thru 0x018e40 (0xb2be bytes) 000527 1730802307.925725 write 0x021b3e thru 0x02753e (0x5a01 bytes) 000528 1730802307.926565 mapwrite 0x0221b6 thru 0x02ab1e (0x8969 bytes) 000529 1730802307.950203 write 0x00f393 thru 0x0151ac (0x5e1a bytes) 000530 1730802307.951997 mapread 0x00cbda thru 0x01096f (0x3d96 bytes) 000531 1730802307.956903 MIRROR_EXTEND to 6 mirrors 000532 1730802308.252632 mapwrite 0x018981 thru 0x01aa8e (0x210e bytes) 000533 1730802308.319711 mapread 0x00d791 thru 0x01af6e (0xd7de bytes) 000534 1730802308.323946 mapread 0x0030b3 thru 0x012960 (0xf8ae bytes) 000535 1730802308.332172 punch from 0x017198 to 0x025fc4 (0xee2d bytes) 000536 1730802308.338684 MIRROR_RESYNC 6 mirrors 000537 1730802309.038901 mapread 0x01f378 thru 0x02ac91 (0xb91a bytes) 000538 1730802309.059758 punch from 0x032b09 to 0x03bf5e (0x9456 bytes) 000539 1730802309.082943 punch from 0x009f87 to 0x00e973 (0x49ed bytes) 000540 1730802309.090730 MIRROR_RESYNC 6 mirrors 000541 1730802309.953545 mapwrite 0x00c8d9 thru 0x0119d2 (0x50fa bytes) 000542 1730802310.013393 trunc from 0x03fc07 to 0x01c0ad (0x23b59 bytes) 000543 1730802310.024017 fallocate 0x02ed9c thru 0x03da60 (0xecc5 bytes) 000544 1730802310.026437 read 0x013956 thru 0x01ae10 (0x74bb bytes) 000546 1730802310.030398 read 0x009777 thru 0x0183cd (0xec57 bytes) 000550 1730802310.031820 punch from 0x0057e3 to 0x01189f (0xc0bd bytes) 000551 1730802310.038235 fallocate 0x02664e thru 0x02f0cc (0x8a7f bytes) 000552 1730802310.045124 mapread 0x018cc6 thru 0x01c0ad (0x33e8 bytes) 000553 1730802310.053125 MIRROR_EXTEND to 7 mirrors 000554 1730802310.310301 read 0x016ea1 thru 0x01c0ad (0x520d bytes) 000555 1730802310.325387 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x52d:0x0]: Invalid argument (22) 000556 1730802310.472475 trunc from 0x01c0ae to 0x02b948 (0xf89b bytes) 000557 1730802310.521277 read 0x02480a thru 0x02b948 (0x713f bytes) 000560 1730802310.531847 punch from 0x00092a to 0x004e20 (0x44f7 bytes) 000561 1730802310.538075 mapwrite 0x02dfe5 thru 0x0302c3 (0x22df bytes) 000562 1730802310.571354 fallocate 0x018d91 thru 0x021ae8 (0x8d58 bytes) 000563 1730802310.576030 read 0x02294b thru 0x02cd78 (0xa42e bytes) 000566 1730802310.590077 MIRROR_EXTEND to 7 mirrors 000567 1730802310.986906 mapread 0x010183 thru 0x018cc2 (0x8b40 bytes) 000568 1730802311.006684 punch from 0x00ed89 to 0x010fa4 (0x221c bytes) 000569 1730802311.026433 punch from 0x014dff to 0x020d05 (0xbf07 bytes) 000570 1730802311.028916 mapread 0x012706 thru 0x013568 (0xe63 bytes) 000571 1730802311.034930 write 0x00cea6 thru 0x00d271 (0x3cc bytes) 000573 1730802311.042305 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x52d:0x0]: Invalid argument (22) 000574 1730802311.188451 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x52d:0x0]: Invalid argument (22) 000578 1730802311.307065 mapwrite 0x03a793 thru 0x03f7e6 (0x5054 bytes) 000579 1730802311.358719 mapread 0x02ed2f thru 0x0364c4 (0x7796 bytes) 000581 1730802311.370137 MIRROR_EXTEND to 6 mirrors 000582 1730802311.778524 write 0x01cfe0 thru 0x02823f (0xb260 bytes) 000583 1730802311.808465 trunc from 0x03f7e7 to 0x00d1f2 (0x325f4 bytes) 000584 1730802311.831947 trunc from 0x00d1f3 to 0x009177 (0x407b bytes) 000585 1730802311.844895 read 0x006d6c thru 0x009177 (0x240c bytes) 000586 1730802311.848792 fallocate 0x039b7b thru 0x03ffff (0x6485 bytes) 000587 1730802311.852010 mapwrite 0x029915 thru 0x02bcaf (0x239b bytes) 000588 1730802311.877070 punch from 0x00a40b to 0x00f22e (0x4e24 bytes) 000589 1730802311.880110 punch from 0x019b05 to 0x01b6fb (0x1bf7 bytes) 000590 1730802311.883875 fallocate 0x02c0cd thru 0x03b0dc (0xf010 bytes) 000591 1730802311.889106 mapwrite 0x0378f1 thru 0x03ffff (0x870f bytes) 000593 1730802311.927616 write 0x028dfd thru 0x02dfc4 (0x51c8 bytes) 000594 1730802311.938970 MIRROR_RESYNC 6 mirrors 000595 1730802312.676663 trunc from 0x040000 to 0x006f07 (0x390f8 bytes) 000597 1730802312.712679 read 0x002f86 thru 0x006f07 (0x3f82 bytes) 000598 1730802312.718754 mapread 0x00259e thru 0x006f07 (0x496a bytes) 000599 1730802312.719588 read 0x005159 thru 0x006f07 (0x1daf bytes) 000600 1730802312.720011 mapwrite 0x01b445 thru 0x01e308 (0x2ec4 bytes) 000601 1730802312.750843 trunc from 0x01e309 to 0x03eaec (0x207e4 bytes) 000602 1730802312.761377 fallocate 0x004f6f thru 0x013000 (0xe092 bytes) 000603 1730802312.768423 punch from 0x03b920 to 0x03eaec (0x31cd bytes) 000604 1730802312.774844 fallocate 0x008111 thru 0x0168fa (0xe7ea bytes) 000605 1730802312.777458 trunc from 0x03eaed to 0x01db31 (0x20fbb bytes) 000606 1730802312.792010 mapread 0x00cd2a thru 0x018cae (0xbf85 bytes) 000607 1730802312.793236 read 0x00a8c9 thru 0x012619 (0x7d51 bytes) 000608 1730802312.794451 punch from 0x017a94 to 0x0192f7 (0x1864 bytes) 000609 1730802312.799413 punch from 0x019c66 to 0x01db31 (0x3ecc bytes) 000610 1730802312.806293 mapwrite 0x016394 thru 0x023de4 (0xda51 bytes) 000611 1730802312.853529 fallocate 0x015172 thru 0x020e08 (0xbc97 bytes) 000612 1730802312.856699 write 0x0122b7 thru 0x01427a (0x1fc4 bytes) 000613 1730802312.857702 read 0x00c2f3 thru 0x00e67f (0x238d bytes) 000615 1730802312.858527 mapwrite 0x033f08 thru 0x0358da (0x19d3 bytes) 000617 1730802312.889575 MIRROR_EXTEND to 7 mirrors 000618 1730802313.253184 MIRROR_RESYNC 7 mirrors 000620 1730802314.063026 fallocate 0x012a02 thru 0x01c03c (0x963b bytes) 000621 1730802314.086481 MIRROR_RESYNC 7 mirrors 000623 1730802315.037095 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x52d:0x0]: Invalid argument (22) 000624 1730802315.247537 mapread 0x008c91 thru 0x014b19 (0xbe89 bytes) 000625 1730802315.269657 write 0x03dafe thru 0x03ffff (0x2502 bytes) 000626 1730802315.315108 mapwrite 0x00a710 thru 0x014f6a (0xa85b bytes) 000627 1730802315.351524 fallocate 0x02b646 thru 0x034aa4 (0x945f bytes) 000628 1730802315.359077 MIRROR_EXTEND to 7 mirrors 000629 1730802315.780557 read 0x002934 thru 0x002e59 (0x526 bytes) 000630 1730802315.789883 trunc from 0x040000 to 0x00065b (0x3f9a4 bytes) 000631 1730802315.856118 read 0x00058a thru 0x00065b (0xd2 bytes) 000632 1730802315.860270 write 0x020b35 thru 0x027a29 (0x6ef5 bytes) 000633 1730802315.861574 write 0x0162cf thru 0x01f783 (0x94b5 bytes) 000634 1730802315.868850 mapwrite 0x035187 thru 0x03b9f1 (0x686b bytes) 000635 1730802315.912944 mapwrite 0x003e0c thru 0x00511a (0x130f bytes) 000637 1730802315.930101 punch from 0x01138c to 0x01f231 (0xdea6 bytes) 000638 1730802315.938490 MIRROR_RESYNC 7 mirrors 000639 1730802317.111188 MIRROR_RESYNC 7 mirrors 000640 1730802317.290688 mapwrite 0x03c48b thru 0x03ffff (0x3b75 bytes) 000641 1730802317.373443 mapwrite 0x01e13f thru 0x02b92c (0xd7ee bytes) 000643 1730802317.421217 mapwrite 0x01632c thru 0x01dc37 (0x790c bytes) 000644 1730802317.459523 read 0x03af51 thru 0x03dd3d (0x2ded bytes) 000646 1730802317.461880 fallocate 0x018470 thru 0x027cb7 (0xf848 bytes) 000647 1730802317.469869 fallocate 0x0004cf thru 0x00d921 (0xd453 bytes) 000648 1730802317.474072 mapread 0x02fe64 thru 0x03cad9 (0xcc76 bytes) 000649 1730802317.482923 trunc from 0x040000 to 0x03ddf7 (0x2208 bytes) 000650 1730802317.512845 mapread 0x00d3a2 thru 0x01c902 (0xf561 bytes) 000652 1730802317.528193 write 0x0233cd thru 0x026ac9 (0x36fd bytes) 000656 1730802317.530259 write 0x0220cb thru 0x02d85d (0xb793 bytes) 000658 1730802317.532516 trunc from 0x03ddf8 to 0x022239 (0x1bbbe bytes) 000659 1730802317.552818 fallocate 0x008507 thru 0x01445a (0xbf54 bytes) 000661 1730802317.558193 read 0x01b87e thru 0x01cd8c (0x150f bytes) 000662 1730802317.558603 mapread 0x013704 thru 0x01a15c (0x6a59 bytes) 000664 1730802317.559484 punch from 0x00a3ed to 0x014531 (0xa145 bytes) 000665 1730802317.567500 fallocate 0x006a25 thru 0x00ddcc (0x73a8 bytes) 000669 1730802317.573179 punch from 0x00729b to 0x008b50 (0x18b6 bytes) 000671 1730802317.578928 fallocate 0x0226b7 thru 0x02ff82 (0xd8cc bytes) 000673 1730802317.595286 MIRROR_RESYNC 7 mirrors 000674 1730802318.689767 mapread 0x02737b thru 0x02e2eb (0x6f71 bytes) 000675 1730802318.699672 fallocate 0x026149 thru 0x02bbc4 (0x5a7c bytes) 000676 1730802318.736777 punch from 0x0010ed to 0x008462 (0x7376 bytes) 000677 1730802318.742409 mapwrite 0x01b80a thru 0x01babf (0x2b6 bytes) 000678 1730802318.761579 trunc from 0x02ff83 to 0x011d2a (0x1e258 bytes) 000679 1730802318.791570 MIRROR_RESYNC 7 mirrors 000680 1730802319.619123 read 0x00eaad thru 0x011d2a (0x327e bytes) 000682 1730802319.625896 punch from 0x009e85 to 0x011d2a (0x7ea6 bytes) 000683 1730802319.652811 trunc from 0x011d2b to 0x031734 (0x1fa0a bytes) 000685 1730802319.678606 fallocate 0x00dd19 thru 0x01a91c (0xcc04 bytes) 000686 1730802319.687271 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x52d:0x0]: Invalid argument (22) 000689 1730802319.819595 fallocate 0x01a3bb thru 0x027937 (0xd57d bytes) 000690 1730802319.827415 trunc from 0x031735 to 0x01ee36 (0x128fe bytes) 000692 1730802319.859204 MIRROR_RESYNC 6 mirrors 000693 1730802320.569584 trunc from 0x01ee37 to 0x032638 (0x13802 bytes) 000694 1730802320.598479 fallocate 0x011bfa thru 0x0217f8 (0xfbff bytes) 000695 1730802320.604457 MIRROR_RESYNC 6 mirrors 000696 1730802321.502455 read 0x01429c thru 0x0166aa (0x240f bytes) 000697 1730802321.513880 write 0x024095 thru 0x025e39 (0x1da5 bytes) 000698 1730802321.560345 write 0x039abe thru 0x03d6a5 (0x3be8 bytes) 000699 1730802321.561576 mapread 0x032b87 thru 0x03d6a5 (0xab1f bytes) 000701 1730802321.576260 mapread 0x01e0ce thru 0x01f971 (0x18a4 bytes) 000702 1730802321.589842 mapread 0x039732 thru 0x03d6a5 (0x3f74 bytes) 000703 1730802321.591362 read 0x01be8d thru 0x020c9a (0x4e0e bytes) 000704 1730802321.602602 MIRROR_EXTEND to 7 mirrors 000705 1730802322.101253 MIRROR_RESYNC 7 mirrors 000707 1730802323.256407 MIRROR_RESYNC 7 mirrors 000708 1730802323.426831 mapread 0x026ec4 thru 0x0340d1 (0xd20e bytes) 000709 1730802323.463413 punch from 0x026d27 to 0x03324a (0xc524 bytes) 000710 1730802323.505179 MIRROR_EXTEND to 8 mirrors 000711 1730802323.995223 punch from 0x03c70d to 0x03d6a5 (0xf99 bytes) 000712 1730802324.036429 read 0x018d00 thru 0x01a2f1 (0x15f2 bytes) 000713 1730802324.049236 write 0x01f3a1 thru 0x02d676 (0xe2d6 bytes) 000714 1730802324.065929 fallocate 0x027aff thru 0x02d7ce (0x5cd0 bytes) 000715 1730802324.069414 punch from 0x012a1e to 0x01f208 (0xc7eb bytes) 000716 1730802324.082303 MIRROR_RESYNC 8 mirrors 000717 1730802325.461782 read 0x03239a thru 0x036cdd (0x4944 bytes) 000718 1730802325.467312 MIRROR_RESYNC 8 mirrors 000721 1730802325.591795 read 0x0341b3 thru 0x03a047 (0x5e95 bytes) 000723 1730802325.601325 MIRROR_RESYNC 8 mirrors 000726 1730802325.759245 trunc from 0x03d6a6 to 0x008fbe (0x346e7 bytes) 000727 1730802325.802251 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x52d:0x0]: Invalid argument (22) 000728 1730802325.915158 fallocate 0x00f5dd thru 0x01c66f (0xd093 bytes) 000729 1730802325.917885 mapwrite 0x01bda0 thru 0x01e2eb (0x254c bytes) 000730 1730802325.954775 fallocate 0x00b8d9 thru 0x017151 (0xb879 bytes) 000731 1730802325.958070 trunc from 0x01e2ec to 0x008691 (0x15c5a bytes) 000732 1730802325.974843 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x52d:0x0]: Invalid argument (22) 000733 1730802326.085198 punch from 0x000c73 to 0x008691 (0x7a1f bytes) 000734 1730802326.088798 fallocate 0x036ad6 thru 0x03c642 (0x5b6d bytes) 000735 1730802326.092199 write 0x018b30 thru 0x027a1f (0xeef0 bytes) 000736 1730802326.099909 write 0x0343e0 thru 0x03ffff (0xbc20 bytes) 000737 1730802326.107650 punch from 0x031863 to 0x037133 (0x58d1 bytes) 000738 1730802326.121479 write 0x02d83a thru 0x036bf5 (0x93bc bytes) 000739 1730802326.125263 mapwrite 0x00091e thru 0x000f3a (0x61d bytes) 000741 1730802326.144591 mapread 0x02789e thru 0x02c433 (0x4b96 bytes) 000744 1730802326.150140 fallocate 0x03bcb8 thru 0x03ffff (0x4348 bytes) 000745 1730802326.153178 fallocate 0x03661c thru 0x03ffff (0x99e4 bytes) 000746 1730802326.156058 read 0x02b9f1 thru 0x034c3f (0x924f bytes) 000747 1730802326.156774 trunc from 0x040000 to 0x02ef33 (0x110cc bytes) 000750 1730802326.168738 punch from 0x0140fb to 0x015cdd (0x1be3 bytes) 000751 1730802326.171559 write 0x00b5c1 thru 0x00ea07 (0x3447 bytes) 000752 1730802326.172243 write 0x02cc54 thru 0x030c49 (0x3ff6 bytes) 000753 1730802326.172713 trunc from 0x030c4a to 0x0216c4 (0xf585 bytes) 000754 1730802326.185320 mapwrite 0x0170e1 thru 0x01a654 (0x3574 bytes) 000755 1730802326.211382 mapread 0x01e893 thru 0x0216c4 (0x2e32 bytes) 000757 1730802326.214116 mapwrite 0x007689 thru 0x011cd2 (0xa64a bytes) 000758 1730802326.234875 mapwrite 0x004b7d thru 0x00e660 (0x9ae4 bytes) 000760 1730802326.248633 fallocate 0x028f1e thru 0x036192 (0xd275 bytes) 000762 1730802326.252876 mapread 0x0034f3 thru 0x00552d (0x203b bytes) 000763 1730802326.253487 punch from 0x01fbce to 0x0216c4 (0x1af7 bytes) 000764 1730802326.263306 MIRROR_EXTEND to 7 mirrors 000765 1730802326.582288 trunc from 0x0216c5 to 0x01f134 (0x2590 bytes) 000766 1730802326.643750 read 0x01e324 thru 0x01f0da (0xdb7 bytes) 000767 1730802326.647400 trunc from 0x01f135 to 0x01b924 (0x3810 bytes) 000768 1730802326.662138 write 0x02cb1b thru 0x0344b4 (0x799a bytes) 000769 1730802326.664376 trunc from 0x0344b5 to 0x029001 (0xb4b3 bytes) 000770 1730802326.681459 write 0x007f22 thru 0x00e33e (0x641d bytes) 000771 1730802326.688799 mapwrite 0x03b51a thru 0x03b741 (0x228 bytes) 000772 1730802326.720786 trunc from 0x03b742 to 0x014980 (0x26dc1 bytes) 000773 1730802326.729860 write 0x0089e4 thru 0x0155f8 (0xcc15 bytes) 000774 1730802326.731148 mapwrite 0x01bbfe thru 0x0295f4 (0xd9f7 bytes) 000775 1730802326.767059 mapwrite 0x039da5 thru 0x03ca1a (0x2c76 bytes) 000776 1730802326.803588 mapwrite 0x02ef0e thru 0x037547 (0x863a bytes) 000779 1730802326.825832 fallocate 0x0183db thru 0x02116c (0x8d92 bytes) 000780 1730802326.833922 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x52d:0x0]: Invalid argument (22) 000781 1730802326.971680 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x52d:0x0]: Invalid argument (22) 000783 1730802327.147327 punch from 0x015c38 to 0x017ddc (0x21a5 bytes) 000784 1730802327.154029 write 0x00c7bf thru 0x010f94 (0x47d6 bytes) 000785 1730802327.168430 fallocate 0x03ce13 thru 0x03ffff (0x31ed bytes) 000786 1730802327.174665 punch from 0x01ef59 to 0x0250b4 (0x615c bytes) 000787 1730802327.179117 mapread 0x025c74 thru 0x02faed (0x9e7a bytes) 000791 1730802327.195045 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x52d:0x0]: Invalid argument (22) 000792 1730802327.325423 fallocate 0x036a40 thru 0x03ffff (0x95c0 bytes) 000794 1730802327.331752 read 0x007477 thru 0x0131a7 (0xbd31 bytes) 000795 1730802327.339021 trunc from 0x040000 to 0x0142ef (0x2bd10 bytes) 000796 1730802327.355146 mapread 0x0027d9 thru 0x00dea6 (0xb6ce bytes) 000797 1730802327.366935 trunc from 0x0142f0 to 0x0015ee (0x12d01 bytes) 000798 1730802327.376861 trunc from 0x0015ef to 0x01af72 (0x19984 bytes) 000799 1730802327.392788 mapread 0x0035cc thru 0x00cf9d (0x99d2 bytes) 000800 1730802327.405613 fallocate 0x0152f9 thru 0x0198c4 (0x45cc bytes) 000801 1730802327.418107 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x52d:0x0]: Invalid argument (22) 000802 1730802327.588431 punch from 0x00042f to 0x00044b (0x1d bytes) 000803 1730802327.590938 mapwrite 0x004219 thru 0x00a807 (0x65ef bytes) 000804 1730802327.623794 fallocate 0x00a759 thru 0x00c7a8 (0x2050 bytes) 000806 1730802327.628083 mapread 0x002e16 thru 0x007555 (0x4740 bytes) 000807 1730802327.629115 read 0x018d26 thru 0x01af72 (0x224d bytes) 000808 1730802327.632674 mapwrite 0x0009c0 thru 0x00a2c9 (0x990a bytes) 000809 1730802327.655285 punch from 0x0120f3 to 0x01596f (0x387d bytes) 000810 1730802327.661105 read 0x00794f thru 0x00c820 (0x4ed2 bytes) 000811 1730802327.661523 punch from 0x00fe07 to 0x017f5f (0x8159 bytes) 000812 1730802327.667850 write 0x035856 thru 0x03feb8 (0xa663 bytes) 000816 1730802327.677693 MIRROR_RESYNC 3 mirrors 000817 1730802328.110437 fallocate 0x03116a thru 0x03e93a (0xd7d1 bytes) 000818 1730802328.134167 read 0x02f508 thru 0x03ef0a (0xfa03 bytes) 000819 1730802328.139638 mapread 0x031af8 thru 0x03fbab (0xe0b4 bytes) 000820 1730802328.145761 read 0x01be9d thru 0x028d45 (0xcea9 bytes) 000821 1730802328.159621 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x52d:0x0]: Invalid argument (22) 000822 1730802328.300253 trunc from 0x03feb9 to 0x01c9ea (0x234ce bytes) 000823 1730802328.340988 fallocate 0x012944 thru 0x014872 (0x1f2f bytes) 000824 1730802328.345316 read 0x00581e thru 0x00f477 (0x9c5a bytes) 000825 1730802328.351921 read 0x00450f thru 0x00fcf3 (0xb7e5 bytes) 000826 1730802328.359842 punch from 0x00f5a5 to 0x0162b4 (0x6d10 bytes) 000827 1730802328.364997 write 0x037fed thru 0x0380df (0xf3 bytes) 000828 1730802328.365791 punch from 0x0285d9 to 0x0304fe (0x7f26 bytes) 000829 1730802328.371327 fallocate 0x011d2a thru 0x012a88 (0xd5f bytes) 000831 1730802328.375829 trunc from 0x0380e0 to 0x01df6c (0x1a173 bytes) 000832 1730802328.407250 MIRROR_EXTEND to 3 mirrors 000836 1730802328.858389 write 0x012dc4 thru 0x01c781 (0x99be bytes) 000838 1730802328.898860 read 0x0094a9 thru 0x0110e9 (0x7c41 bytes) 000840 1730802328.903253 trunc from 0x01df6d to 0x01b859 (0x2713 bytes) 000841 1730802328.937278 read 0x01719b thru 0x01b859 (0x46bf bytes) 000842 1730802328.937711 read 0x016675 thru 0x01b859 (0x51e5 bytes) 000843 1730802328.938078 write 0x00f5ea thru 0x0175b6 (0x7fcd bytes) 000844 1730802328.938851 trunc from 0x01b85a to 0x01b078 (0x7e1 bytes) 000845 1730802328.964199 mapwrite 0x029f5b thru 0x032126 (0x81cc bytes) 000846 1730802329.013351 mapread 0x006342 thru 0x008330 (0x1fef bytes) 000849 1730802329.015028 read 0x02ae9f thru 0x02ffe5 (0x5147 bytes) 000850 1730802329.015443 mapread 0x01a14b thru 0x01f5cf (0x5485 bytes) 000851 1730802329.016044 punch from 0x0012a3 to 0x00e39b (0xd0f9 bytes) 000852 1730802329.021926 write 0x026266 thru 0x032506 (0xc2a1 bytes) 000853 1730802329.022911 fallocate 0x003ab4 thru 0x00ac41 (0x718e bytes) 000854 1730802329.029984 write 0x00fbdc thru 0x01059f (0x9c4 bytes) 000855 1730802329.030719 write 0x02e6b5 thru 0x0341e0 (0x5b2c bytes) 000857 1730802329.032412 trunc from 0x0341e1 to 0x03907f (0x4e9f bytes) 000860 1730802329.048305 read 0x02e6cd thru 0x02ed3b (0x66f bytes) 000861 1730802329.048740 mapwrite 0x03c3d4 thru 0x03e926 (0x2553 bytes) 000862 1730802329.117411 trunc from 0x03e927 to 0x01ae25 (0x23b01 bytes) 000863 1730802329.141618 punch from 0x018406 to 0x01ae25 (0x2a20 bytes) 000864 1730802329.153217 mapread 0x00f997 thru 0x015c5e (0x62c8 bytes) 000865 1730802329.158940 MIRROR_RESYNC 3 mirrors 000866 1730802329.626817 punch from 0x00d30f to 0x0197ca (0xc4bc bytes) 000867 1730802329.660464 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x52d:0x0]: Invalid argument (22) 000868 1730802329.797920 trunc from 0x01ae26 to 0x03f42b (0x24606 bytes) 000869 1730802329.829142 mapwrite 0x007caf thru 0x00836f (0x6c1 bytes) 000870 1730802329.851237 MIRROR_EXTEND to 3 mirrors 000871 1730802330.209164 mapwrite 0x027f04 thru 0x02bd76 (0x3e73 bytes) 000872 1730802330.307003 write 0x03fc15 thru 0x03ffff (0x3eb bytes) 000873 1730802330.318738 punch from 0x00271b to 0x008e5f (0x6745 bytes) 000875 1730802330.326993 write 0x00fadb thru 0x01ba81 (0xbfa7 bytes) 000876 1730802330.337786 write 0x012de1 thru 0x01a411 (0x7631 bytes) 000877 1730802330.338597 punch from 0x000d09 to 0x00c408 (0xb700 bytes) 000879 1730802330.342335 mapwrite 0x000d96 thru 0x00f6f0 (0xe95b bytes) 000880 1730802330.365641 MIRROR_RESYNC 3 mirrors 000881 1730802330.721587 mapwrite 0x0188d2 thru 0x01b896 (0x2fc5 bytes) 000882 1730802330.795210 punch from 0x023ef6 to 0x02894c (0x4a57 bytes) 000883 1730802330.804476 MIRROR_RESYNC 3 mirrors 000884 1730802331.217850 trunc from 0x040000 to 0x03f7a1 (0x85e bytes) 000885 1730802331.261804 trunc from 0x03f7a2 to 0x03ec07 (0xb9a bytes) 000887 1730802331.288259 fallocate 0x038cb8 thru 0x03d027 (0x4370 bytes) 000888 1730802331.292317 fallocate 0x00d460 thru 0x01a1d9 (0xcd7a bytes) 000889 1730802331.294982 trunc from 0x03ec08 to 0x007ea2 (0x36d65 bytes) 000890 1730802331.309567 read 0x007445 thru 0x007ea2 (0xa5e bytes) 000891 1730802331.315209 punch from 0x002b11 to 0x007ea2 (0x5392 bytes) 000892 1730802331.322642 trunc from 0x007ea3 to 0x007e75 (0x2d bytes) 000893 1730802331.339282 trunc from 0x007e76 to 0x016142 (0xe2cd bytes) 000895 1730802331.356725 read 0x009ec3 thru 0x014256 (0xa394 bytes) 000896 1730802331.363162 read 0x014a4d thru 0x016142 (0x16f6 bytes) 000897 1730802331.365108 read 0x00e2b0 thru 0x00f33f (0x1090 bytes) 000898 1730802331.365536 read 0x008b5a thru 0x009937 (0xdde bytes) 000899 1730802331.365909 mapwrite 0x00e0ba thru 0x011a17 (0x395e bytes) 000900 1730802331.380048 trunc from 0x016143 to 0x003853 (0x128ef bytes) 000901 1730802331.395021 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x52d:0x0]: Invalid argument (22) 000902 1730802331.532447 punch from 0x001a7c to 0x003853 (0x1dd8 bytes) 000903 1730802331.537083 punch from 0x003421 to 0x003853 (0x433 bytes) 000904 1730802331.540622 write 0x00345b thru 0x00e57f (0xb125 bytes) 000906 1730802331.550159 MIRROR_EXTEND to 3 mirrors 000907 1730802331.860427 write 0x004b58 thru 0x0141c7 (0xf670 bytes) 000908 1730802331.895770 mapread 0x000c90 thru 0x009b49 (0x8eba bytes) 000909 1730802331.908302 punch from 0x00911d to 0x00d605 (0x44e9 bytes) 000910 1730802331.920365 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x52d:0x0]: Invalid argument (22) 000911 1730802332.072679 MIRROR_EXTEND to 3 mirrors 000912 1730802332.429538 trunc from 0x0141c8 to 0x03ee24 (0x2ac5d bytes) 000913 1730802332.461246 fallocate 0x001297 thru 0x00f46e (0xe1d8 bytes) 000916 1730802332.466852 mapwrite 0x013f83 thru 0x01d22a (0x92a8 bytes) 000917 1730802332.493138 write 0x022abf thru 0x024899 (0x1ddb bytes) 000918 1730802332.501523 MIRROR_EXTEND to 4 mirrors 000919 1730802332.796514 mapwrite 0x03167a thru 0x03a0f2 (0x8a79 bytes) 000920 1730802332.880730 mapwrite 0x002f72 thru 0x00cf5b (0x9fea bytes) 000922 1730802332.909516 read 0x00b3a3 thru 0x00bf6e (0xbcc bytes) 000923 1730802332.912384 mapwrite 0x0366db thru 0x03ffff (0x9925 bytes) 000924 1730802332.966067 trunc from 0x040000 to 0x03b994 (0x466b bytes) 000925 1730802332.977594 mapwrite 0x03ef32 thru 0x03ffff (0x10ce bytes) 000926 1730802333.011996 MIRROR_RESYNC 4 mirrors 000927 1730802333.654202 mapread 0x03ea50 thru 0x03ffff (0x15b0 bytes) 000928 1730802333.664210 fallocate 0x03e53c thru 0x03ffff (0x1ac4 bytes) 000930 1730802333.700388 trunc from 0x040000 to 0x013dce (0x2c231 bytes) 000931 1730802333.721302 mapwrite 0x00a67c thru 0x011ed4 (0x7859 bytes) 000933 1730802333.753475 trunc from 0x013dcf to 0x020f03 (0xd135 bytes) 000935 1730802333.767471 punch from 0x00c097 to 0x00c167 (0xd1 bytes) 000937 1730802333.771854 mapread 0x0055cf thru 0x00eb5d (0x958f bytes) 000938 1730802333.778018 write 0x017e15 thru 0x01d297 (0x5483 bytes) 000939 1730802333.791097 MIRROR_EXTEND to 5 mirrors 000940 1730802334.121855 punch from 0x0012c8 to 0x006edd (0x5c16 bytes) 000941 1730802334.162893 trunc from 0x020f04 to 0x004da7 (0x1c15c bytes) 000942 1730802334.201423 read 0x002e8f thru 0x004da7 (0x1f19 bytes) 000943 1730802334.206135 trunc from 0x004da8 to 0x004d8b (0x1c bytes) 000945 1730802334.216926 punch from 0x000b6e to 0x004d8b (0x421e bytes) 000946 1730802334.219792 mapwrite 0x006e23 thru 0x00e9b8 (0x7b96 bytes) 000947 1730802334.251688 mapread 0x004408 thru 0x00e9b8 (0xa5b1 bytes) 000948 1730802334.252673 mapread 0x00a14e thru 0x00e9b8 (0x486b bytes) 000950 1730802334.254305 mapwrite 0x035550 thru 0x03c6c0 (0x7171 bytes) 000951 1730802334.293453 fallocate 0x01d3ef thru 0x024ce1 (0x78f3 bytes) 000953 1730802334.296333 punch from 0x0247cf to 0x0338a5 (0xf0d7 bytes) 000955 1730802334.303625 trunc from 0x03c6c1 to 0x00aae8 (0x31bd8 bytes) 000956 1730802334.318527 MIRROR_EXTEND to 6 mirrors 000957 1730802334.621286 mapread 0x00a69e thru 0x00aae8 (0x44b bytes) 000959 1730802334.644114 MIRROR_EXTEND to 7 mirrors 000962 1730802334.951680 mapwrite 0x029e65 thru 0x035e0e (0xbfaa bytes) 000963 1730802335.028824 mapread 0x004304 thru 0x0058b7 (0x15b4 bytes) 000964 1730802335.036794 read 0x0173a3 thru 0x022d8b (0xb9e9 bytes) 000965 1730802335.043369 trunc from 0x035e0f to 0x0229de (0x13430 bytes) 000967 1730802335.067696 fallocate 0x01d4e4 thru 0x02425f (0x6d7c bytes) 000970 1730802335.074720 mapread 0x002d82 thru 0x0108c1 (0xdb40 bytes) 000971 1730802335.076463 punch from 0x00aff4 to 0x012b58 (0x7b65 bytes) 000972 1730802335.088867 MIRROR_RESYNC 7 mirrors 000973 1730802335.790761 read 0x00b041 thru 0x00b6d1 (0x691 bytes) 000975 1730802335.795561 fallocate 0x028669 thru 0x035657 (0xcfef bytes) 000976 1730802335.823147 trunc from 0x0229df to 0x0067ac (0x1c232 bytes) 000977 1730802335.848556 trunc from 0x0067ad to 0x0313c8 (0x2ac1c bytes) 000978 1730802335.862722 mapread 0x02d2a7 thru 0x0313c8 (0x4122 bytes) 000980 1730802335.886899 trunc from 0x0313c9 to 0x00dc45 (0x23783 bytes) 000981 1730802335.897782 write 0x0342d4 thru 0x03ffff (0xbd2c bytes) 000982 1730802335.899273 mapread 0x021d32 thru 0x028b0b (0x6dda bytes) 000983 1730802335.906416 trunc from 0x040000 to 0x02bb0d (0x144f2 bytes) 000984 1730802335.921220 punch from 0x01a7ed to 0x0212d1 (0x6ae5 bytes) 000985 1730802335.929534 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x52d:0x0]: Invalid argument (22) 000987 1730802336.056762 write 0x032cb6 thru 0x03da6b (0xadb6 bytes) 000988 1730802336.062826 read 0x00c265 thru 0x00c9e6 (0x782 bytes) 000989 1730802336.069323 mapread 0x00d686 thru 0x016e07 (0x9782 bytes) 000990 1730802336.083836 read 0x02ce24 thru 0x034d99 (0x7f76 bytes) 000991 1730802336.093233 trunc from 0x03da6c to 0x02f73b (0xe330 bytes) 000992 1730802336.107230 fallocate 0x0205c5 thru 0x02decf (0xd90b bytes) 000993 1730802336.110745 mapread 0x01549b thru 0x023bf3 (0xe759 bytes) 000994 1730802336.114999 MIRROR_RESYNC 6 mirrors 000995 1730802336.756133 MIRROR_EXTEND to 7 mirrors 000996 1730802337.080214 write 0x038fbf thru 0x03ffff (0x7041 bytes) 000998 1730802337.099097 mapread 0x011162 thru 0x0156eb (0x458a bytes) 000999 1730802337.107072 read 0x03a510 thru 0x03ffff (0x5af0 bytes) 001000 1730802337.107502 write 0x0250e5 thru 0x027de6 (0x2d02 bytes) All operations completed A-OK! PASS 70a (49s) == sanity-flr test 200a: stress test ===================== 05:25:39 (1730802339) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre2 Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4452576 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4944416 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6433760 .. Extending file size to 8429888 .. Extending file size to 7694112 .. done Extending file size to 8520192 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8550656 .. Extending file size to 7742240 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 7903648 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9007744 .. failed Extending file size to 8921408 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9187488 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9051040 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 32125 32126 32128 32129 32131 done 192.168.202.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg245-client.virtnet /mnt/lustre2 (opts:) 192.168.202.145@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg245-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (66s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 05:26:45 (1730802405) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre2 Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8362976 .. Extending file size to 4566528 .. Extending file size to 8999360 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8844448 .. Extending file size to 9029632 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 2114 2115 2116 2119 2121 done 192.168.202.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg245-client.virtnet /mnt/lustre2 (opts:) 192.168.202.145@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg245-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (65s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 05:27:50 (1730802470) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2400 1285288 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2540 1285148 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 14576 3592444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 22400 3584620 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 36976 7177064 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2400 1285288 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2540 1285148 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 14576 3592444 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 22400 3584620 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 36976 7177064 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0413879 s, 50.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0572946 s, 36.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg245-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 05:27:55 (1730802475) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0316608 s, 66.2 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 05:27:57 (1730802477) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x61:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8b:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 05:27:59 (1730802479) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0259768 s, 80.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x57c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x57e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x57f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 05:28:01 (1730802481) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0314273 s, 66.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x57d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x57e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x580:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x57d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x57e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x580:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x581:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 05:28:03 (1730802483) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0294687 s, 71.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 05:28:26 (1730802506) Waiting for MDT destroys to complete oleg245-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg245-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg245-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0206906 s, 50.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x57f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x583:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x582:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x57f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x583:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x582:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x580:0x0] } PASS 204d (35s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 05:29:01 (1730802541) Waiting for MDT destroys to complete oleg245-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg245-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg245-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0254745 s, 82.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (34s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 05:29:35 (1730802575) Waiting for MDT destroys to complete oleg245-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 oleg245-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg245-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool oleg245-server: Pool lustre.test_204f created Adding targets to pool oleg245-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg245-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0286082 s, 73.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg245-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg245-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg245-server: Pool lustre.test_204f destroyed PASS 204f (43s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 05:30:18 (1730802618) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 05:30:20 (1730802620) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 05:30:22 (1730802622) oleg245-server: Pool lustre.test_206 created Adding targets to pool oleg245-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg245-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg245-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg245-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg245-server: Pool lustre.test_206 destroyed PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 05:30:35 (1730802635) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0221419 s, 47.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0120671 s, 86.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x2d3eb16a Mirror 3: 0x2d3eb16a PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 05:30:37 (1730802637) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 05:30:38 (1730802638) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 05:30:39 (1730802639) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0137645 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x15d0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1660000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1660000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1660000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2560000 Read this (4 bytes): BBBB 192.168.202.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg245-client.virtnet /mnt/lustre2 (opts:) PASS 209a (14s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 05:30:53 (1730802653) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00372113 s, 2.2 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 05:30:55 (1730802655) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122 s, 85.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x80 lmm_fid: [0x200000405:0x80:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x80 lmm_fid: [0x200000405:0x80:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x99:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611264 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-11-05 05:30:55.000000000 -0500 Modify: 2024-11-05 05:30:55.000000000 -0500 Change: 2024-11-05 05:30:56.000000000 -0500 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 05:30:57 (1730802657) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0103673 s, 101 MB/s oleg245-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg245-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg245-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg245-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 oleg245-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg245-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg245-server: O/280000401/d26/ba: File not found by ext2_lookup PASS 210b (10s) == sanity-flr test complete, duration 851 sec ============ 05:31:07 (1730802667) === sanity-flr: start cleanup 05:31:08 (1730802668) === === sanity-flr: finish cleanup 05:31:10 (1730802670) ===