-----============= acceptance-small: sanity-flr ============----- Thu Apr 17 05:31:24 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg124-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 05:31:31 (1744882291) === oleg124-client.virtnet: executing check_config_client /mnt/lustre oleg124-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg124-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b6382000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6382000.idle_timeout=debug disable quota as required oleg124-server: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 05:31:43 (1744882303) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7644] == sanity-flr test 0a: lfs mirror create with -N option == 05:31:43 (1744882303) PASS 0a (5s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 05:31:48 (1744882308) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 05:31:49 (1744882309) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (0s) == sanity-flr test 0d: lfs mirror extend with -N option == 05:31:49 (1744882309) PASS 0d (4s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 05:31:53 (1744882313) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 05:31:54 (1744882314) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 05:31:55 (1744882315) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b6382000.stats=clear osc.lustre-OST0001-osc-ffff8800b6382000.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 05:31:57 (1744882317) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 05:31:59 (1744882319) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 05:32:01 (1744882321) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (0s) == sanity-flr test 1: create components with setstripe options ========================================================== 05:32:01 (1744882321) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x12:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range cannot extend layout with embed source fid, re-try old way lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x12:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 05:32:04 (1744882324) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 05:32:05 (1744882325) PASS 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 05:32:07 (1744882327) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory error: lfs extend: layout is NULL PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 05:32:09 (1744882329) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (1s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 05:32:11 (1744882331) PASS 7 (2s) == sanity-flr test 8: extend mirror trying to use different OSTs ========================================================== 05:32:13 (1744882333) /mnt/lustre/f8.sanity-flr-1 - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x28:0x0] } - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x18:0x0] } PASS 8 (1s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 05:32:14 (1744882334) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB) copied, 0.131686 s, 55.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0192302 s, 54.5 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 05:32:16 (1744882336) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB) copied, 0.0529667 s, 119 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 05:32:18 (1744882338) Stopping /mnt/lustre-ost1 (opts:) on oleg124-server oleg124-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6382000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6382000.ost_server_uuid in DISCONN state after 1 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg124-server: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg124-server: oleg124-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg124-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg124-server oleg124-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6382000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6382000.ost_server_uuid in DISCONN state after 4 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg124-server: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg124-server: oleg124-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg124-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (26s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 05:32:44 (1744882364) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.040956 s, 51.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg124-server oleg124-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6382000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6382000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg124-server: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg124-server: oleg124-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg124-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (9s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 05:32:53 (1744882373) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg124-server oleg124-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6382000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6382000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg124-server: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg124-server: oleg124-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg124-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg124-server oleg124-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6382000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6382000.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg124-server: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg124-server: oleg124-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg124-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (20s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 05:33:13 (1744882393) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.839046 s, 62.5 MB/s -rw-r--r-- 1 root root 52428800 Apr 17 05:33 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.208s user 0m0.000s sys 0m0.120s Stopping /mnt/lustre-ost1 (opts:) on oleg124-server oleg124-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6382000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6382000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.210s user 0m0.001s sys 0m0.106s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg124-server: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg124-server: oleg124-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg124-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg124-server oleg124-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6382000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6382000.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost1)... real 0m0.242s user 0m0.001s sys 0m0.116s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg124-server: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg124-server: oleg124-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg124-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg124-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b6382000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6382000.ost_server_uuid in FULL state after 0 sec PASS 33b (21s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 05:33:34 (1744882414) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 05:33:35 (1744882415) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 05:33:36 (1744882416) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (0s) == sanity-flr test 35: allow to write to mirrored files == 05:33:36 (1744882416) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 05:33:38 (1744882418) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.129115 s, 97.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 05:33:41 (1744882421) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00970172 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00740161 s, 142 MB/s PASS 36b (1s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 05:33:42 (1744882422) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 05:33:45 (1744882425) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000689859 s, 1.5 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000520974 s, 2.0 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000733979 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000616717 s, 1.7 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 05:33:47 (1744882427) create test files with size 15734467 .. d6817e9c62af3bcb94770f989ae57503 - c5db86b5f48ab46650a9e00671d6a52c - 524d943190ebcfb6b5f59baa09ea7809 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg124-client.virtnet /mnt/lustre (opts:) Starting client: oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre PASS 37 (4s) == sanity-flr test 38: resync ============================ 05:33:51 (1744882431) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 05:33:54 (1744882434) PASS 39 (1s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 05:33:55 (1744882435) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0177749 s, 59.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x44:0x0 - lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x45:0x0 - lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x33:0x0 - lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20007 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00865005 s, 121 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x34:0x0 - lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x35:0x0 - lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x46:0x0 - lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 0x20007 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 05:33:57 (1744882437) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0240166 s, 87.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0842733 s, 49.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0369056 s, 56.8 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) == sanity-flr test 42: lfs mirror verify ================= 05:33:59 (1744882439) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 05:34:00 (1744882440) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 05:34:01 (1744882441) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0121332 s, 86.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (1s) == sanity-flr test 44a: lfs mirror split check =========== 05:34:02 (1744882442) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0877812 s, 35.8 MB/s PASS 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 05:34:05 (1744882445) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x40:0x0] } lcme_id: 0x20001 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } split mirror_id 1 id 0x10001 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 0x20001 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } split mirror_id 2 id 0x20001 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (1s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 05:34:07 (1744882447) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0765779 s, 137 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg124-client.virtnet /mnt/lustre (opts:) Starting client: oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (22s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 05:34:29 (1744882469) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800aaa6e000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0760062 s, 138 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg124-client.virtnet /mnt/lustre (opts:) Starting client: oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800aa963800.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 05:34:33 (1744882473) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (7s) == sanity-flr test 46: Verify setstripe --copy option ==== 05:34:40 (1744882480) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (6s) == sanity-flr test 47: Verify mirror obj alloc =========== 05:34:46 (1744882486) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 05:34:47 (1744882487) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0778502 s, 40.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0463712 s, 67.8 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 9c6e302c10e9802a66c2eff1de9e041e - mirror 1 checksum: a769976f3c0691dcef482caddf13a712 - mirror 2 checksum: 9c6e302c10e9802a66c2eff1de9e041e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Apr 17 05:34:47 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5b:0x0] } - lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Apr 17 05:34:47 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5c:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 9c6e302c10e9802a66c2eff1de9e041e - mirror 1 checksum: a769976f3c0691dcef482caddf13a712 - mirror 2 checksum: a769976f3c0691dcef482caddf13a712 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Apr 17 05:34:47 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5b:0x0] } - lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Apr 17 05:34:47 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5c:0x0] } PASS 48 (2s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 05:34:49 (1744882489) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0119325 s, 87.9 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 05:34:51 (1744882491) Failing mds1 on oleg124-server Stopping /mnt/lustre-mds1 (opts:) on oleg124-server 05:34:58 (1744882498) shut down facet: mds1 facet_host: oleg124-server facet_failover_host: oleg124-server Failover mds1 to oleg124-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg124-server: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-MDT0000 05:35:11 (1744882511) targets are mounted 05:35:11 (1744882511) facet_failover done oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00825889 s, 127 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 0x20001 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5e:0x0] } PASS 50A (27s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 05:35:18 (1744882518) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00706112 s, 2.9 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00424349 s, 4.8 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 05:35:20 (1744882520) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00970512 s, 2.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00272198 s, 7.5 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 05:35:22 (1744882522) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00240635 s, 6.8 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 05:35:24 (1744882524) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0101332 s, 2.0 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 05:35:26 (1744882526) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00171032 s, 2.4 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0063217 s, 648 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 05:35:28 (1744882528) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) PASS 60b (4s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 05:35:32 (1744882532) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 05:35:50 (1744882550) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (16s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 05:36:06 (1744882566) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 05:36:23 (1744882583) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 05:36:25 (1744882585) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 26942 mirror split pid 26943 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6744: 26943 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6744: 26943 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 26942 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 26943 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (61s) == sanity-flr test 70a: flr mode fsx test ================ 05:37:26 (1744882646) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 2647 skipping zero length punch hole skipping zero length punch hole 000003 1744882647.478934 mapwrite 0x039bf5 thru 0x03ffff (0x640b bytes) zero_range to largest ever: 0x3b287 000004 1744882647.495590 trunc from 0x034db9 to 0x03b286 (0x64ce bytes) 000006 1744882647.498931 trunc from 0x040000 to 0x0084c8 (0x37b37 bytes) 000007 1744882647.503608 mapread 0x000f39 thru 0x001f83 (0x104b bytes) 000008 1744882647.507571 mapread 0x0030c3 thru 0x0084c8 (0x5406 bytes) 000009 1744882647.509586 MIRROR_EXTEND to 2 mirrors 000010 1744882647.589773 mapread 0x00663e thru 0x0084c8 (0x1e8b bytes) 000011 1744882647.592233 punch from 0x0026ca to 0x0084c8 (0x5dff bytes) 000012 1744882647.600477 write 0x00c00e thru 0x00c7e9 (0x7dc bytes) 000013 1744882647.602113 mapread 0x0011a9 thru 0x007800 (0x6658 bytes) 000015 1744882647.605101 trunc from 0x00c7ea to 0x02fe44 (0x2365b bytes) 000016 1744882647.613620 mapwrite 0x03d789 thru 0x03ffff (0x2877 bytes) 000017 1744882647.624403 write 0x022159 thru 0x03163b (0xf4e3 bytes) 000018 1744882647.628472 write 0x028b25 thru 0x02c09e (0x357a bytes) 000019 1744882647.629162 mapread 0x03bc7a thru 0x03ffff (0x4386 bytes) zero_range to largest ever: 0x40000 000020 1744882647.629948 trunc from 0x033713 to 0x03ffff (0xc8ed bytes) 000021 1744882647.632050 read 0x02dd20 thru 0x034f6f (0x7250 bytes) 000024 1744882647.634648 write 0x01fb30 thru 0x0213a6 (0x1877 bytes) 000025 1744882647.635469 read 0x031edd thru 0x037e1f (0x5f43 bytes) 000026 1744882647.638622 fallocate 0x015cb3 thru 0x02299d (0xcceb bytes) 000027 1744882647.640487 mapread 0x03b103 thru 0x03ffff (0x4efd bytes) 000028 1744882647.641216 trunc from 0x040000 to 0x01e11c (0x21ee3 bytes) 000029 1744882647.647947 mapread 0x000132 thru 0x00d321 (0xd1f0 bytes) 000030 1744882647.648801 trunc from 0x01e11d to 0x0027b8 (0x1b964 bytes) 000031 1744882647.653805 trunc from 0x0027b9 to 0x0081e6 (0x5a2e bytes) 000032 1744882647.658063 fallocate 0x01b9f6 thru 0x01c26a (0x875 bytes) 000033 1744882647.660139 mapwrite 0x0058a6 thru 0x00fcd8 (0xa433 bytes) 000035 1744882647.679745 mapwrite 0x0243d6 thru 0x033671 (0xf29c bytes) 000036 1744882647.709518 fallocate 0x03ff98 thru 0x03ffff (0x68 bytes) 000038 1744882647.711478 punch from 0x01fdc5 to 0x02a1cb (0xa407 bytes) 000039 1744882647.712954 trunc from 0x01948c to 0x01af78 (0x1aed bytes) 000040 1744882647.714333 read 0x0093b1 thru 0x014230 (0xae80 bytes) 000041 1744882647.718434 punch from 0x006861 to 0x007d81 (0x1521 bytes) 000043 1744882647.720231 fallocate 0x036f09 thru 0x03c4cf (0x55c7 bytes) 000044 1744882647.721650 read 0x01f7a6 thru 0x020034 (0x88f bytes) 000045 1744882647.722060 write 0x01dce2 thru 0x01e9d5 (0xcf4 bytes) 000046 1744882647.722571 write 0x0128db thru 0x01dd2a (0xb450 bytes) 000047 1744882647.723320 mapwrite 0x03c436 thru 0x03ffff (0x3bca bytes) 000048 1744882647.735463 trunc from 0x040000 to 0x03873e (0x78c1 bytes) 000050 1744882647.739735 mapread 0x00cde9 thru 0x01ad82 (0xdf9a bytes) 000051 1744882647.740347 trunc from 0x007290 to 0x00a98d (0x36fe bytes) 000052 1744882647.742053 trunc from 0x033f59 to 0x03873e (0x47e6 bytes) 000053 1744882647.743581 trunc from 0x01fa44 to 0x025906 (0x5ec3 bytes) 000054 1744882647.745060 trunc from 0x03873f to 0x014422 (0x2431c bytes) 000055 1744882647.749188 punch from 0x00909e to 0x00a39a (0x12fd bytes) 000056 1744882647.750637 mapread 0x01145d thru 0x014422 (0x2fc6 bytes) 000057 1744882647.751106 punch from 0x001f30 to 0x00c4e8 (0xa5b9 bytes) 000058 1744882647.752687 fallocate 0x001445 thru 0x00d656 (0xc212 bytes) 000059 1744882647.757273 MIRROR_EXTEND to 3 mirrors 000060 1744882647.863262 trunc from 0x0037a5 to 0x007279 (0x3ad5 bytes) 000061 1744882647.872958 MIRROR_RESYNC 3 mirrors 000062 1744882648.014037 mapread 0x010dc7 thru 0x011a21 (0xc5b bytes) 000063 1744882648.018751 MIRROR_RESYNC 3 mirrors 000065 1744882648.059909 write 0x0397e5 thru 0x03a5df (0xdfb bytes) 000066 1744882648.067250 trunc from 0x03a5e0 to 0x00f7a1 (0x2ae3e bytes) 000067 1744882648.075785 read 0x004a60 thru 0x005ceb (0x128c bytes) 000068 1744882648.077673 trunc from 0x0091c9 to 0x00f7a1 (0x65d9 bytes) 000069 1744882648.079299 fallocate 0x037ce8 thru 0x03ec05 (0x6f1e bytes) 000070 1744882648.081151 punch from 0x009aaf to 0x00f7a1 (0x5cf3 bytes) 000071 1744882648.082589 mapwrite 0x00f8ad thru 0x01783a (0x7f8e bytes) 000073 1744882648.097463 trunc from 0x01783b to 0x00174f (0x160eb bytes) 000074 1744882648.101045 trunc from 0x001750 to 0x015f1a (0x147cb bytes) 000075 1744882648.104350 trunc from 0x015f1b to 0x035df5 (0x1fedb bytes) 000076 1744882648.107981 read 0x003d8d thru 0x0070bf (0x3333 bytes) 000077 1744882648.111024 fallocate 0x0103c6 thru 0x010edf (0xb1a bytes) 000078 1744882648.112432 read 0x00a1a1 thru 0x012e11 (0x8c71 bytes) 000079 1744882648.112723 fallocate 0x004520 thru 0x00a92c (0x640d bytes) 000080 1744882648.114013 write 0x03da42 thru 0x03ffff (0x25be bytes) 000081 1744882648.114647 mapread 0x01893b thru 0x020a79 (0x813f bytes) 000083 1744882648.115535 mapread 0x00e55a thru 0x00e698 (0x13f bytes) 000084 1744882648.116013 punch from 0x013348 to 0x018ab9 (0x5772 bytes) 000085 1744882648.117677 read 0x004076 thru 0x00dade (0x9a69 bytes) 000087 1744882648.118317 write 0x00d039 thru 0x01c61a (0xf5e2 bytes) 000088 1744882648.119068 fallocate 0x00b1e9 thru 0x00e07e (0x2e96 bytes) 000089 1744882648.120619 fallocate 0x0304ee thru 0x03d011 (0xcb24 bytes) 000090 1744882648.122152 mapread 0x006b96 thru 0x00f036 (0x84a1 bytes) 000091 1744882648.122591 trunc from 0x040000 to 0x01d7fd (0x22802 bytes) 000092 1744882648.126400 mapread 0x004e5e thru 0x005650 (0x7f3 bytes) 000093 1744882648.126891 trunc from 0x01d7fe to 0x0042e9 (0x19514 bytes) 000094 1744882648.130911 punch from 0x000b1d to 0x0042e9 (0x37cd bytes) 000095 1744882648.132427 punch from 0x00189f to 0x002edf (0x1641 bytes) 000097 1744882648.134261 mapwrite 0x00882e thru 0x00a9ea (0x21bd bytes) 000098 1744882648.145169 write 0x0351de thru 0x03ab44 (0x5967 bytes) 000099 1744882648.146058 fallocate 0x0366d4 thru 0x03ffff (0x992c bytes) 000100 1744882648.147435 trunc from 0x02c767 to 0x038e5c (0xc6f6 bytes) 000101 1744882648.150364 read 0x0385cc thru 0x03ab44 (0x2579 bytes) 000102 1744882648.153136 read 0x01593a thru 0x01f44d (0x9b14 bytes) 000103 1744882648.153441 write 0x03b2eb thru 0x03ffff (0x4d15 bytes) 000104 1744882648.153972 mapwrite 0x03b6a3 thru 0x03e57b (0x2ed9 bytes) 000105 1744882648.158165 punch from 0x00dc67 to 0x01a7a6 (0xcb40 bytes) 000107 1744882648.159711 trunc from 0x022cdf to 0x026d65 (0x4087 bytes) 000108 1744882648.161139 mapread 0x0180b0 thru 0x01cd45 (0x4c96 bytes) 000109 1744882648.163744 mapwrite 0x00947a thru 0x01460d (0xb194 bytes) 000110 1744882648.168995 read 0x00bee8 thru 0x011c80 (0x5d99 bytes) 000111 1744882648.169306 punch from 0x01ad32 to 0x01d639 (0x2908 bytes) 000112 1744882648.170552 trunc from 0x040000 to 0x02a4dd (0x15b22 bytes) 000114 1744882648.174517 trunc from 0x02a4de to 0x00f3f5 (0x1b0e8 bytes) 000115 1744882648.178793 trunc from 0x00f3f6 to 0x002434 (0xcfc1 bytes) 000116 1744882648.182366 mapwrite 0x03c951 thru 0x03ffff (0x36af bytes) 000117 1744882648.193722 trunc from 0x0328ad to 0x036c25 (0x4379 bytes) 000118 1744882648.195358 trunc from 0x040000 to 0x0061cb (0x39e34 bytes) 000120 1744882648.201766 fallocate 0x00aba3 thru 0x00c08e (0x14ec bytes) 000122 1744882648.205173 MIRROR_EXTEND to 4 mirrors 000123 1744882648.308711 trunc from 0x00c08f to 0x011627 (0x5599 bytes) 000124 1744882648.321347 punch from 0x003fe1 to 0x011627 (0xd647 bytes) 000125 1744882648.322782 write 0x000672 thru 0x00a5eb (0x9f7a bytes) 000126 1744882648.325682 mapread 0x00097c thru 0x00bd8a (0xb40f bytes) 000127 1744882648.327202 punch from 0x00e7e5 to 0x011627 (0x2e43 bytes) 000128 1744882648.328574 mapwrite 0x00f545 thru 0x019b40 (0xa5fc bytes) 000129 1744882648.345772 mapread 0x01841e thru 0x019b40 (0x1723 bytes) 000130 1744882648.346449 mapread 0x0156f7 thru 0x019b40 (0x444a bytes) 000131 1744882648.350529 read 0x00dcc1 thru 0x0140b8 (0x63f8 bytes) 000132 1744882648.350938 mapread 0x014142 thru 0x019b40 (0x59ff bytes) 000133 1744882648.351395 trunc from 0x019b41 to 0x020751 (0x6c11 bytes) 000134 1744882648.355730 mapwrite 0x00476e thru 0x013a91 (0xf324 bytes) 000136 1744882648.366926 mapwrite 0x02d59b thru 0x031867 (0x42cd bytes) 000137 1744882648.383357 mapwrite 0x027c0f thru 0x028c45 (0x1037 bytes) 000138 1744882648.387298 write 0x035c25 thru 0x03ffff (0xa3db bytes) 000139 1744882648.388475 mapread 0x003ef9 thru 0x00e4fb (0xa603 bytes) 000140 1744882648.389177 write 0x030650 thru 0x03ce20 (0xc7d1 bytes) 000141 1744882648.390264 trunc from 0x005f8c to 0x00614b (0x1c0 bytes) 000142 1744882648.392482 mapwrite 0x0230f0 thru 0x0304c4 (0xd3d5 bytes) 000144 1744882648.402552 mapread 0x0321ff thru 0x03f2e0 (0xd0e2 bytes) 000145 1744882648.403109 trunc from 0x040000 to 0x013ec8 (0x2c137 bytes) 000146 1744882648.407692 fallocate 0x03f7ef thru 0x03ffff (0x811 bytes) 000147 1744882648.410912 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xa46:0x0]: Invalid argument (22) 000148 1744882648.465575 trunc from 0x00263d to 0x006ed4 (0x4898 bytes) 000149 1744882648.467642 read 0x00e556 thru 0x010f3d (0x29e8 bytes) 000150 1744882648.469955 fallocate 0x0076ee thru 0x01149a (0x9dad bytes) 000151 1744882648.471739 read 0x007b67 thru 0x009a5d (0x1ef7 bytes) 000152 1744882648.474843 fallocate 0x01c10f thru 0x0213b8 (0x52aa bytes) 000153 1744882648.477425 read 0x0073f3 thru 0x00f350 (0x7f5e bytes) 000154 1744882648.477991 mapread 0x01caf9 thru 0x0213b8 (0x48c0 bytes) 000155 1744882648.481340 write 0x02cbde thru 0x03ada2 (0xe1c5 bytes) 000156 1744882648.485216 fallocate 0x025748 thru 0x02a0f7 (0x49b0 bytes) 000157 1744882648.487290 write 0x0103f9 thru 0x010d7e (0x986 bytes) 000158 1744882648.489300 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xa46:0x0]: Invalid argument (22) 000159 1744882648.542243 read 0x028929 thru 0x0318d1 (0x8fa9 bytes) 000161 1744882648.545256 read 0x020f9c thru 0x02c3a2 (0xb407 bytes) 000163 1744882648.548228 mapwrite 0x03f4c1 thru 0x03ffff (0xb3f bytes) 000164 1744882648.558993 trunc from 0x040000 to 0x036032 (0x9fcd bytes) 000165 1744882648.563614 punch from 0x01b917 to 0x01d5e7 (0x1cd1 bytes) 000167 1744882648.566388 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xa46:0x0]: Invalid argument (22) 000168 1744882648.596923 trunc from 0x036033 to 0x014416 (0x21c1c bytes) 000170 1744882648.602473 trunc from 0x0135b7 to 0x014416 (0xe60 bytes) 000171 1744882648.604177 trunc from 0x014417 to 0x0189ca (0x45b4 bytes) 000172 1744882648.608880 trunc from 0x01412e to 0x0189ca (0x489d bytes) 000174 1744882648.611521 punch from 0x00b7a6 to 0x00f67e (0x3ed9 bytes) 000175 1744882648.613222 trunc from 0x0189cb to 0x015aa5 (0x2f25 bytes) 000177 1744882648.618379 punch from 0x00750c to 0x01588f (0xe384 bytes) 000179 1744882648.620769 mapwrite 0x0330ed thru 0x03ffff (0xcf13 bytes) 000180 1744882648.636067 punch from 0x01e79b to 0x027916 (0x917c bytes) 000182 1744882648.637988 punch from 0x0249fd to 0x028337 (0x393b bytes) 000183 1744882648.639241 punch from 0x027363 to 0x02e7f4 (0x7492 bytes) 000185 1744882648.641023 mapwrite 0x019de6 thru 0x029cb7 (0xfed2 bytes) 000186 1744882648.651357 trunc from 0x001565 to 0x001799 (0x235 bytes) 000188 1744882648.653070 mapread 0x02b29f thru 0x0302e8 (0x504a bytes) 000189 1744882648.655236 fallocate 0x003795 thru 0x009ca5 (0x6511 bytes) 000190 1744882648.657002 fallocate 0x0139ae thru 0x021ad8 (0xe12b bytes) 000191 1744882648.658384 punch from 0x03388e to 0x03845a (0x4bcd bytes) 000192 1744882648.660935 read 0x016dca thru 0x01dd5e (0x6f95 bytes) 000193 1744882648.661277 mapwrite 0x01ed8e thru 0x0234e0 (0x4753 bytes) 000194 1744882648.666625 mapread 0x01472d thru 0x01dd69 (0x963d bytes) 000195 1744882648.667362 mapread 0x01c97c thru 0x024f7d (0x8602 bytes) 000196 1744882648.667921 fallocate 0x000615 thru 0x003f39 (0x3925 bytes) 000198 1744882648.670164 mapread 0x02d7a7 thru 0x02e340 (0xb9a bytes) 000199 1744882648.670593 punch from 0x0027be to 0x008035 (0x5878 bytes) 000200 1744882648.672077 punch from 0x0187cd to 0x0223f7 (0x9c2b bytes) 000201 1744882648.673373 trunc from 0x02a4bd to 0x02b209 (0xd4d bytes) 000202 1744882648.674703 trunc from 0x00f424 to 0x01a8f4 (0xb4d1 bytes) 000203 1744882648.676178 fallocate 0x00f025 thru 0x0108c2 (0x189e bytes) 000204 1744882648.677395 punch from 0x004bcb to 0x012df7 (0xe22d bytes) 000205 1744882648.678608 mapread 0x01fd75 thru 0x02ab08 (0xad94 bytes) 000206 1744882648.683485 write 0x038edf thru 0x03ffff (0x7121 bytes) 000207 1744882648.685764 mapwrite 0x03607a thru 0x03ffff (0x9f86 bytes) 000209 1744882648.691285 mapwrite 0x0099c1 thru 0x010877 (0x6eb7 bytes) 000211 1744882648.698486 mapwrite 0x0366bf thru 0x03ffff (0x9941 bytes) 000212 1744882648.703328 mapwrite 0x03202f thru 0x038942 (0x6914 bytes) 000213 1744882648.707484 read 0x0180b3 thru 0x019ba0 (0x1aee bytes) 000214 1744882648.707912 trunc from 0x040000 to 0x010052 (0x2ffad bytes) 000215 1744882648.711864 write 0x024c93 thru 0x02a7bf (0x5b2d bytes) 000216 1744882648.712505 read 0x01ed66 thru 0x023d37 (0x4fd2 bytes) 000217 1744882648.714746 trunc from 0x02867d to 0x028f8b (0x90f bytes) 000219 1744882648.718584 read 0x009866 thru 0x0168ea (0xd085 bytes) 000220 1744882648.718962 mapwrite 0x03d1ee thru 0x03ffff (0x2e12 bytes) 000222 1744882648.730584 write 0x0201e1 thru 0x02abb8 (0xa9d8 bytes) 000223 1744882648.732894 MIRROR_EXTEND to 2 mirrors 000224 1744882648.851467 mapread 0x001fb6 thru 0x01190b (0xf956 bytes) 000225 1744882648.857924 punch from 0x025061 to 0x02e857 (0x97f7 bytes) 000227 1744882648.868117 mapwrite 0x01fe1d thru 0x029151 (0x9335 bytes) 000228 1744882648.879994 MIRROR_EXTEND to 3 mirrors 000229 1744882649.008594 fallocate 0x01f71f thru 0x028006 (0x88e8 bytes) 000230 1744882649.017408 write 0x029679 thru 0x02d2e7 (0x3c6f bytes) 000231 1744882649.021311 mapwrite 0x02d21e thru 0x02e6c4 (0x14a7 bytes) 000232 1744882649.028529 trunc from 0x040000 to 0x00212a (0x3ded5 bytes) 000233 1744882649.037547 mapread 0x00079d thru 0x00212a (0x198e bytes) 000234 1744882649.040528 read 0x0002a5 thru 0x00212a (0x1e86 bytes) 000235 1744882649.040874 mapwrite 0x036f9b thru 0x03ffff (0x9065 bytes) 000236 1744882649.054535 punch from 0x00939c to 0x01791e (0xe583 bytes) 000237 1744882649.055930 trunc from 0x040000 to 0x0261c7 (0x19e38 bytes) 000238 1744882649.059302 write 0x012547 thru 0x01c331 (0x9deb bytes) 000239 1744882649.062921 fallocate 0x02f6d7 thru 0x03e133 (0xea5d bytes) 000240 1744882649.064372 trunc from 0x003e18 to 0x010462 (0xc64b bytes) 000241 1744882649.065725 trunc from 0x012760 to 0x01f0dc (0xc97d bytes) 000242 1744882649.069445 punch from 0x01ddee to 0x024581 (0x6794 bytes) 000243 1744882649.070733 trunc from 0x03cfd3 to 0x03e133 (0x1161 bytes) 000244 1744882649.072044 fallocate 0x02c583 thru 0x03b2bc (0xed3a bytes) 000245 1744882649.073388 write 0x0086fc thru 0x00bcd4 (0x35d9 bytes) 000246 1744882649.077054 MIRROR_EXTEND to 4 mirrors 000247 1744882649.198919 punch from 0x016f33 to 0x01749a (0x568 bytes) 000248 1744882649.209615 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xa46:0x0]: Invalid argument (22) 000250 1744882649.259993 fallocate 0x036648 thru 0x03b230 (0x4be9 bytes) 000251 1744882649.261338 read 0x014ba3 thru 0x01750b (0x2969 bytes) 000252 1744882649.263038 write 0x033d7c thru 0x0349a3 (0xc28 bytes) 000253 1744882649.266364 mapwrite 0x01de7b thru 0x02b300 (0xd486 bytes) 000254 1744882649.275939 trunc from 0x00a24e to 0x00b2f4 (0x10a7 bytes) 000255 1744882649.277271 trunc from 0x019274 to 0x0249ad (0xb73a bytes) 000258 1744882649.279315 trunc from 0x03e134 to 0x014f98 (0x2919b bytes) 000259 1744882649.282754 fallocate 0x0229d8 thru 0x02f4d9 (0xcb02 bytes) 000260 1744882649.285552 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xa46:0x0]: Invalid argument (22) 000261 1744882649.329987 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xa46:0x0]: Invalid argument (22) 000262 1744882649.359080 trunc from 0x024d26 to 0x02a2ec (0x55c7 bytes) 000263 1744882649.361637 MIRROR_EXTEND to 2 mirrors 000264 1744882649.467501 trunc from 0x02464b to 0x02f4d9 (0xae8f bytes) 000265 1744882649.476534 read 0x01ca27 thru 0x0276cf (0xaca9 bytes) 000266 1744882649.479256 read 0x02088d thru 0x02381a (0x2f8e bytes) 000267 1744882649.479704 write 0x023f7e thru 0x0295a6 (0x5629 bytes) 000268 1744882649.483594 punch from 0x0058ea to 0x00dfa7 (0x86be bytes) 000269 1744882649.485224 fallocate 0x01f09d thru 0x02c83d (0xd7a1 bytes) 000270 1744882649.486750 trunc from 0x00e472 to 0x00fc03 (0x1792 bytes) 000271 1744882649.488020 write 0x033621 thru 0x03e433 (0xae13 bytes) 000272 1744882649.489095 punch from 0x00099e to 0x00dd29 (0xd38c bytes) 000273 1744882649.490796 fallocate 0x00300c thru 0x004833 (0x1828 bytes) 000274 1744882649.492545 mapread 0x001c97 thru 0x0086db (0x6a45 bytes) 000275 1744882649.502923 fallocate 0x012fe6 thru 0x01766b (0x4686 bytes) 000276 1744882649.504903 fallocate 0x023a8f thru 0x03356e (0xfae0 bytes) 000277 1744882649.506541 trunc from 0x004bcb to 0x011e26 (0xd25c bytes) 000278 1744882649.509297 MIRROR_RESYNC 2 mirrors 000279 1744882649.630202 fallocate 0x00070c thru 0x001441 (0xd36 bytes) 000280 1744882649.639494 MIRROR_RESYNC 2 mirrors 000281 1744882649.735180 trunc from 0x00e203 to 0x017364 (0x9162 bytes) 000282 1744882649.743535 punch from 0x0266ad to 0x02ac9f (0x45f3 bytes) 000283 1744882649.745244 trunc from 0x0113bd to 0x018b3c (0x7780 bytes) 000284 1744882649.746725 read 0x038854 thru 0x03e433 (0x5be0 bytes) 000286 1744882649.750562 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xa46:0x0]: Invalid argument (22) 000287 1744882649.779530 trunc from 0x03e434 to 0x000865 (0x3dbce bytes) 000289 1744882649.788684 fallocate 0x004a1c thru 0x0076ce (0x2cb3 bytes) 000290 1744882649.790171 read 0x005df5 thru 0x0076ce (0x18da bytes) 000291 1744882649.791768 mapread 0x000f3f thru 0x0076ce (0x6790 bytes) 000293 1744882649.796885 trunc from 0x0076cf to 0x012423 (0xad55 bytes) 000294 1744882649.800880 write 0x026fe3 thru 0x02ae56 (0x3e74 bytes) 000295 1744882649.801609 mapwrite 0x02b31f thru 0x036737 (0xb419 bytes) 000296 1744882649.816718 read 0x02f96b thru 0x036737 (0x6dcd bytes) 000297 1744882649.817158 fallocate 0x0118de thru 0x01ed64 (0xd487 bytes) 000298 1744882649.818721 trunc from 0x036738 to 0x02f7d5 (0x6f62 bytes) 000299 1744882649.823194 mapwrite 0x038368 thru 0x03ffff (0x7c98 bytes) 000300 1744882649.837428 mapread 0x01e422 thru 0x01ee00 (0x9df bytes) 000301 1744882649.839240 MIRROR_EXTEND to 2 mirrors 000302 1744882649.943908 read 0x00cd3a thru 0x012e82 (0x6149 bytes) 000303 1744882649.946239 mapread 0x017f52 thru 0x026a94 (0xeb43 bytes) 000304 1744882649.952794 write 0x02e600 thru 0x034ce5 (0x66e6 bytes) 000305 1744882649.962489 punch from 0x00200b to 0x011cee (0xfce4 bytes) 000306 1744882649.963889 mapwrite 0x03b226 thru 0x03ffff (0x4dda bytes) 000307 1744882649.971202 read 0x03272d thru 0x03ffff (0xd8d3 bytes) 000309 1744882649.973504 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xa46:0x0]: Invalid argument (22) 000310 1744882650.004584 fallocate 0x038e9a thru 0x03e0b4 (0x521b bytes) 000312 1744882650.006396 fallocate 0x0114e3 thru 0x017a41 (0x655f bytes) 000313 1744882650.007916 write 0x032e83 thru 0x03ffff (0xd17d bytes) 000314 1744882650.010858 mapwrite 0x0007b0 thru 0x0026c4 (0x1f15 bytes) 000315 1744882650.018035 punch from 0x01f741 to 0x02e222 (0xeae2 bytes) 000316 1744882650.019565 read 0x00450f thru 0x008171 (0x3c63 bytes) 000318 1744882650.021402 write 0x03a0d9 thru 0x03bdd2 (0x1cfa bytes) 000319 1744882650.021856 mapwrite 0x00e79d thru 0x013b14 (0x5378 bytes) 000320 1744882650.028045 fallocate 0x016d3f thru 0x01c3c5 (0x5687 bytes) 000322 1744882650.029904 trunc from 0x040000 to 0x00a2b9 (0x35d46 bytes) 000323 1744882650.037447 write 0x02e5e1 thru 0x03cbce (0xe5ee bytes) 000324 1744882650.039549 MIRROR_EXTEND to 2 mirrors 000325 1744882650.141488 read 0x01b202 thru 0x01ba9e (0x89d bytes) 000326 1744882650.142947 mapwrite 0x00c2f0 thru 0x01bbab (0xf8bc bytes) 000327 1744882650.173971 trunc from 0x03779f to 0x03cbce (0x5430 bytes) 000328 1744882650.175380 write 0x024dae thru 0x0321a5 (0xd3f8 bytes) 000330 1744882650.180178 write 0x012baf thru 0x01cfd4 (0xa426 bytes) 000331 1744882650.180870 write 0x01c784 thru 0x01ec45 (0x24c2 bytes) 000333 1744882650.181545 fallocate 0x000508 thru 0x00bd80 (0xb879 bytes) 000334 1744882650.183270 trunc from 0x03cbcf to 0x032eef (0x9cdf bytes) 000335 1744882650.187997 trunc from 0x021b27 to 0x02387b (0x1d55 bytes) 000336 1744882650.190636 MIRROR_EXTEND to 3 mirrors 000337 1744882650.309665 punch from 0x00a74b to 0x00e27d (0x3b33 bytes) 000338 1744882650.317529 read 0x030497 thru 0x032eef (0x2a59 bytes) 000339 1744882650.319378 write 0x031d39 thru 0x036f9d (0x5265 bytes) 000340 1744882650.321004 read 0x025019 thru 0x03261f (0xd607 bytes) 000342 1744882650.324319 fallocate 0x0034fe thru 0x012429 (0xef2c bytes) 000343 1744882650.325899 write 0x00c942 thru 0x01381a (0x6ed9 bytes) 000344 1744882650.326567 trunc from 0x00752f to 0x0078c7 (0x399 bytes) 000345 1744882650.327851 fallocate 0x03dc52 thru 0x03dd49 (0xf8 bytes) 000346 1744882650.329198 read 0x00864f thru 0x014cbd (0xc66f bytes) 000347 1744882650.330738 MIRROR_EXTEND to 4 mirrors 000348 1744882650.440287 punch from 0x03af73 to 0x03dd49 (0x2dd7 bytes) 000349 1744882650.448167 mapwrite 0x004884 thru 0x00582e (0xfab bytes) 000350 1744882650.454723 mapread 0x035bf2 thru 0x038cba (0x30c9 bytes) 000351 1744882650.456958 trunc from 0x039340 to 0x03dd49 (0x4a0a bytes) 000352 1744882650.459558 MIRROR_RESYNC 4 mirrors 000354 1744882650.632694 write 0x025503 thru 0x02a442 (0x4f40 bytes) 000355 1744882650.642516 trunc from 0x03cd51 to 0x03dd49 (0xff9 bytes) 000356 1744882650.644015 trunc from 0x015906 to 0x0190de (0x37d9 bytes) 000357 1744882650.645502 punch from 0x011f23 to 0x015816 (0x38f4 bytes) 000358 1744882650.647112 read 0x014981 thru 0x01b9d8 (0x7058 bytes) 000359 1744882650.649719 write 0x013341 thru 0x018bbf (0x587f bytes) 000360 1744882650.652341 mapread 0x02d691 thru 0x0398a7 (0xc217 bytes) 000361 1744882650.657153 punch from 0x0171d8 to 0x01b062 (0x3e8b bytes) 000362 1744882650.662858 MIRROR_EXTEND to 5 mirrors 000363 1744882650.778643 write 0x030e50 thru 0x038bb4 (0x7d65 bytes) 000364 1744882650.788785 read 0x00fa55 thru 0x01021e (0x7ca bytes) 000365 1744882650.791382 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xa46:0x0]: Invalid argument (22) 000367 1744882650.842385 mapwrite 0x012c32 thru 0x01e45a (0xb829 bytes) 000368 1744882650.856722 write 0x031aa2 thru 0x036729 (0x4c88 bytes) 000369 1744882650.861522 fallocate 0x010827 thru 0x015b92 (0x536c bytes) 000370 1744882650.863023 read 0x0014f8 thru 0x009c19 (0x8722 bytes) 000371 1744882650.865288 write 0x012a01 thru 0x020051 (0xd651 bytes) 000372 1744882650.866091 fallocate 0x0082bb thru 0x00e3fd (0x6143 bytes) 000373 1744882650.867765 mapwrite 0x023a1a thru 0x03366c (0xfc53 bytes) 000375 1744882650.877214 trunc from 0x03dd4a to 0x001193 (0x3cbb6 bytes) 000376 1744882650.886652 read 0x0001c1 thru 0x001193 (0xfd3 bytes) 000377 1744882650.888294 punch from 0x001053 to 0x001193 (0x141 bytes) 000378 1744882650.889962 mapread 0x00005e thru 0x001193 (0x1136 bytes) 000379 1744882650.891708 fallocate 0x0008d9 thru 0x00d510 (0xcc38 bytes) 000380 1744882650.893261 trunc from 0x00002f to 0x001193 (0x1165 bytes) 000381 1744882650.894499 mapread 0x000b8f thru 0x001193 (0x605 bytes) 000382 1744882650.897180 fallocate 0x035405 thru 0x03b252 (0x5e4e bytes) 000383 1744882650.898775 trunc from 0x03b253 to 0x016932 (0x24920 bytes) 000384 1744882650.903433 trunc from 0x00de13 to 0x00f230 (0x141e bytes) 000385 1744882650.904923 read 0x001d54 thru 0x00737e (0x562b bytes) 000386 1744882650.907663 mapwrite 0x033ef6 thru 0x036dd4 (0x2edf bytes) 000387 1744882650.918923 trunc from 0x01f823 to 0x0292ad (0x9a8b bytes) 000388 1744882650.920408 punch from 0x0228dc to 0x02c24d (0x9972 bytes) 000389 1744882650.922149 trunc from 0x036dd5 to 0x02e6ac (0x8728 bytes) 000391 1744882650.926062 fallocate 0x004f28 thru 0x00a8bb (0x5994 bytes) 000393 1744882650.927620 trunc from 0x02e6ad to 0x00cdec (0x218c0 bytes) 000394 1744882650.931548 write 0x000682 thru 0x008e19 (0x8798 bytes) 000395 1744882650.932361 mapwrite 0x03111e thru 0x0327d9 (0x16bc bytes) 000396 1744882650.943763 read 0x029ae4 thru 0x03190f (0x7e2c bytes) 000397 1744882650.946753 fallocate 0x03b7b2 thru 0x03ffff (0x484e bytes) 000398 1744882650.948382 punch from 0x01f87f to 0x0225db (0x2d5d bytes) 000399 1744882650.950862 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xa46:0x0]: Invalid argument (22) 000400 1744882650.998175 punch from 0x0255cf to 0x02be5c (0x688e bytes) 000401 1744882650.999874 write 0x010f03 thru 0x01e6c9 (0xd7c7 bytes) 000402 1744882651.004084 trunc from 0x040000 to 0x0206e0 (0x1f91f bytes) 000403 1744882651.009077 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xa46:0x0]: Invalid argument (22) 000404 1744882651.058650 fallocate 0x02b814 thru 0x0366f4 (0xaee1 bytes) 000405 1744882651.060506 read 0x0327d8 thru 0x0366f4 (0x3f1d bytes) 000406 1744882651.064067 MIRROR_EXTEND to 3 mirrors 000407 1744882651.171441 read 0x01f147 thru 0x027213 (0x80cd bytes) 000408 1744882651.175120 trunc from 0x0006b3 to 0x00eca9 (0xe5f7 bytes) 000409 1744882651.183628 write 0x037487 thru 0x03ade2 (0x395c bytes) 000410 1744882651.185231 trunc from 0x00b1f6 to 0x015ddf (0xabea bytes) 000412 1744882651.187193 read 0x03092e thru 0x0352a1 (0x4974 bytes) 000413 1744882651.190502 fallocate 0x03b377 thru 0x03ffff (0x4c89 bytes) 000414 1744882651.191994 punch from 0x01ceaa to 0x025601 (0x8758 bytes) 000415 1744882651.193294 trunc from 0x03ade3 to 0x015569 (0x25879 bytes) 000416 1744882651.203722 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xa46:0x0]: Invalid argument (22) 000417 1744882651.251189 trunc from 0x01556a to 0x0198ee (0x4385 bytes) 000418 1744882651.255331 read 0x007105 thru 0x01346f (0xc36b bytes) 000419 1744882651.257714 punch from 0x001534 to 0x00b3f8 (0x9ec5 bytes) 000420 1744882651.259040 punch from 0x01490f to 0x0198ee (0x4fe0 bytes) 000421 1744882651.260414 read 0x00b634 thru 0x015efe (0xa8cb bytes) 000423 1744882651.264412 MIRROR_EXTEND to 3 mirrors 000424 1744882651.367552 read 0x00a0e9 thru 0x0198ee (0xf806 bytes) 000425 1744882651.372430 MIRROR_RESYNC 3 mirrors 000426 1744882651.465411 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xa46:0x0]: Invalid argument (22) 000427 1744882651.518334 MIRROR_RESYNC 2 mirrors 000428 1744882651.552338 mapread 0x007a46 thru 0x016e60 (0xf41b bytes) 000429 1744882651.558536 trunc from 0x0198ef to 0x02dd32 (0x14444 bytes) 000430 1744882651.573856 trunc from 0x02dd33 to 0x011829 (0x1c509 bytes) 000431 1744882651.578393 mapwrite 0x002195 thru 0x00bf24 (0x9d90 bytes) 000432 1744882651.587579 trunc from 0x01182a to 0x039ad8 (0x282af bytes) 000433 1744882651.592171 trunc from 0x028386 to 0x02ed2d (0x69a8 bytes) 000434 1744882651.593619 punch from 0x0273a5 to 0x02fb01 (0x875d bytes) 000435 1744882651.594873 trunc from 0x00c639 to 0x0145a5 (0x7f6d bytes) 000436 1744882651.597976 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xa46:0x0]: Invalid argument (22) 000437 1744882651.628309 trunc from 0x039ad9 to 0x01307d (0x26a5b bytes) 000438 1744882651.632443 punch from 0x0093d9 to 0x01307d (0x9ca5 bytes) 000440 1744882651.634266 trunc from 0x00ebfe to 0x00f840 (0xc43 bytes) 000441 1744882651.635565 mapwrite 0x03750e thru 0x03ffff (0x8af2 bytes) 000442 1744882651.649911 trunc from 0x039b6f to 0x03fdfd (0x628f bytes) 000443 1744882651.651393 trunc from 0x040000 to 0x023f5e (0x1c0a1 bytes) 000444 1744882651.654888 read 0x00cbff thru 0x00f78c (0x2b8e bytes) 000445 1744882651.657645 MIRROR_EXTEND to 2 mirrors 000446 1744882651.761387 write 0x00ca07 thru 0x014e44 (0x843e bytes) 000447 1744882651.771761 trunc from 0x023f5f to 0x03d4c7 (0x19569 bytes) 000449 1744882651.779990 write 0x00cd52 thru 0x018ab3 (0xbd62 bytes) 000450 1744882651.781784 fallocate 0x00003c thru 0x00e562 (0xe527 bytes) 000451 1744882651.783333 punch from 0x039467 to 0x03d4c7 (0x4061 bytes) 000452 1744882651.784503 fallocate 0x03d60c thru 0x03dcff (0x6f4 bytes) 000453 1744882651.787188 MIRROR_EXTEND to 3 mirrors 000454 1744882651.899711 mapwrite 0x017aa9 thru 0x01b733 (0x3c8b bytes) 000455 1744882651.934396 trunc from 0x03d4c8 to 0x02d765 (0xfd62 bytes) 000457 1744882651.944174 fallocate 0x024b4b thru 0x034714 (0xfbca bytes) 000458 1744882651.945692 trunc from 0x01fd59 to 0x021507 (0x17af bytes) 000459 1744882651.947149 mapwrite 0x00ab28 thru 0x017e81 (0xd35a bytes) 000460 1744882651.956226 write 0x0040ac thru 0x010d61 (0xccb6 bytes) 000461 1744882651.956988 fallocate 0x02c60e thru 0x03a7bf (0xe1b2 bytes) 000462 1744882651.958371 mapwrite 0x01c1de thru 0x01e507 (0x232a bytes) 000463 1744882651.963297 mapread 0x00445b thru 0x00bf79 (0x7b1f bytes) 000464 1744882651.963713 trunc from 0x02d766 to 0x00b15e (0x22607 bytes) 000466 1744882651.968293 mapread 0x005cbf thru 0x00b15e (0x54a0 bytes) 000467 1744882651.970175 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xa46:0x0]: Invalid argument (22) 000468 1744882652.021197 trunc from 0x004610 to 0x00b15e (0x6b4f bytes) 000469 1744882652.023020 fallocate 0x03d00c thru 0x03ffff (0x2ff4 bytes) 000470 1744882652.026114 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xa46:0x0]: Invalid argument (22) 000471 1744882652.056006 mapwrite 0x02258a thru 0x02499d (0x2414 bytes) 000472 1744882652.064958 trunc from 0x040000 to 0x004ad2 (0x3b52d bytes) 000473 1744882652.070336 punch from 0x003a44 to 0x004ad2 (0x108f bytes) 000474 1744882652.071506 mapwrite 0x00c18c thru 0x015544 (0x93b9 bytes) 000475 1744882652.083487 fallocate 0x022b8a thru 0x030d5d (0xe1d4 bytes) 000476 1744882652.084867 mapwrite 0x011d34 thru 0x01c94a (0xac17 bytes) 000478 1744882652.096374 fallocate 0x038232 thru 0x0388be (0x68d bytes) 000479 1744882652.097885 mapread 0x019556 thru 0x01c94a (0x33f5 bytes) 000480 1744882652.098415 trunc from 0x00d811 to 0x014168 (0x6958 bytes) 000481 1744882652.100013 punch from 0x013d1a to 0x01c94a (0x8c31 bytes) 000482 1744882652.101421 read 0x015126 thru 0x01c94a (0x7825 bytes) 000483 1744882652.103699 mapwrite 0x03b4d1 thru 0x03c82e (0x135e bytes) 000485 1744882652.115808 MIRROR_EXTEND to 2 mirrors 000486 1744882652.233154 write 0x008451 thru 0x00abc0 (0x2770 bytes) 000487 1744882652.244149 punch from 0x008fb4 to 0x00ea52 (0x5a9f bytes) 000488 1744882652.247358 read 0x039a68 thru 0x03c82e (0x2dc7 bytes) 000489 1744882652.250496 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xa46:0x0]: Invalid argument (22) 000490 1744882652.283052 trunc from 0x03c82f to 0x02270f (0x1a11f bytes) 000491 1744882652.291555 mapwrite 0x01941d thru 0x01b2d6 (0x1eba bytes) 000492 1744882652.300461 mapread 0x009b85 thru 0x012cdf (0x915b bytes) 000495 1744882652.305638 trunc from 0x022710 to 0x01f419 (0x32f6 bytes) 000496 1744882652.310573 fallocate 0x02e353 thru 0x0365e4 (0x8292 bytes) 000497 1744882652.312260 trunc from 0x01f0b7 to 0x01f419 (0x363 bytes) 000498 1744882652.313695 mapwrite 0x01251a thru 0x016063 (0x3b4a bytes) 000499 1744882652.319342 fallocate 0x02edd1 thru 0x038a8c (0x9cbc bytes) 000500 1744882652.321063 trunc from 0x038a8d to 0x008e1b (0x2fc71 bytes) 000501 1744882652.325246 mapwrite 0x012772 thru 0x01a07e (0x790d bytes) 000502 1744882652.338826 mapwrite 0x03eb64 thru 0x03ffff (0x149c bytes) 000503 1744882652.349282 mapwrite 0x006bbb thru 0x007d77 (0x11bd bytes) 000504 1744882652.353064 mapread 0x006a4b thru 0x00ee19 (0x83cf bytes) 000505 1744882652.354793 read 0x037eb3 thru 0x039388 (0x14d6 bytes) 000506 1744882652.355193 mapread 0x02aeb7 thru 0x0335ea (0x8734 bytes) 000507 1744882652.359651 fallocate 0x016ec3 thru 0x01eff6 (0x8134 bytes) 000508 1744882652.361159 write 0x02e482 thru 0x03a32d (0xbeac bytes) 000509 1744882652.361871 trunc from 0x02b861 to 0x0384f5 (0xcc95 bytes) 000510 1744882652.365282 read 0x0037f4 thru 0x007082 (0x388f bytes) 000511 1744882652.366865 MIRROR_EXTEND to 2 mirrors 000512 1744882652.483519 punch from 0x01972b to 0x01b116 (0x19ec bytes) 000513 1744882652.491033 fallocate 0x0203e3 thru 0x02e20f (0xde2d bytes) 000514 1744882652.493617 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xa46:0x0]: Invalid argument (22) 000515 1744882652.523448 fallocate 0x03b4a0 thru 0x03ffff (0x4b60 bytes) 000517 1744882652.525436 trunc from 0x0288aa to 0x02ba77 (0x31ce bytes) 000518 1744882652.526991 punch from 0x02da8f to 0x0327ee (0x4d60 bytes) 000519 1744882652.528309 mapread 0x023ea0 thru 0x02f0b2 (0xb213 bytes) 000520 1744882652.537303 read 0x023990 thru 0x026d35 (0x33a6 bytes) 000521 1744882652.537667 mapread 0x02a9a4 thru 0x02c549 (0x1ba6 bytes) 000522 1744882652.538062 mapwrite 0x017c5d thru 0x0199f2 (0x1d96 bytes) 000523 1744882652.544949 punch from 0x026be9 to 0x02ad29 (0x4141 bytes) 000524 1744882652.546367 trunc from 0x040000 to 0x008e31 (0x371ce bytes) 000525 1744882652.555091 fallocate 0x02fb7d thru 0x03f06c (0xf4f0 bytes) 000526 1744882652.556369 write 0x010b78 thru 0x014a53 (0x3edc bytes) 000527 1744882652.557000 trunc from 0x00a5cc to 0x00c4c5 (0x1efa bytes) 000528 1744882652.558178 trunc from 0x00b021 to 0x00b96d (0x94d bytes) 000529 1744882652.559345 punch from 0x00948c to 0x00dc92 (0x4807 bytes) 000531 1744882652.561005 trunc from 0x0023fa to 0x0098cf (0x74d6 bytes) 000532 1744882652.562327 mapread 0x013d9b thru 0x014a53 (0xcb9 bytes) 000533 1744882652.562758 read 0x0019df thru 0x00f0c3 (0xd6e5 bytes) 000535 1744882652.566568 fallocate 0x002d48 thru 0x009eac (0x7165 bytes) 000536 1744882652.568040 mapread 0x01416e thru 0x014a53 (0x8e6 bytes) 000537 1744882652.568637 trunc from 0x012354 to 0x01364d (0x12fa bytes) 000538 1744882652.573602 write 0x02a503 thru 0x02d308 (0x2e06 bytes) 000540 1744882652.575685 trunc from 0x02d309 to 0x0364d2 (0x91ca bytes) 000541 1744882652.579329 write 0x011794 thru 0x01f76e (0xdfdb bytes) 000543 1744882652.582974 trunc from 0x032eba to 0x0364d2 (0x3619 bytes) 000544 1744882652.584293 mapwrite 0x014006 thru 0x019675 (0x5670 bytes) 000545 1744882652.590602 fallocate 0x0276e8 thru 0x033c29 (0xc542 bytes) 000546 1744882652.592369 punch from 0x001f91 to 0x0041a6 (0x2216 bytes) 000547 1744882652.593723 mapwrite 0x0147fb thru 0x022876 (0xe07c bytes) 000548 1744882652.603051 write 0x01409d thru 0x01ed88 (0xacec bytes) 000549 1744882652.604077 read 0x0043ed thru 0x009ab2 (0x56c6 bytes) 000550 1744882652.606029 punch from 0x02cc4b to 0x02f1e8 (0x259e bytes) 000552 1744882652.608023 trunc from 0x004210 to 0x0056b1 (0x14a2 bytes) 000553 1744882652.609354 fallocate 0x0093e1 thru 0x01229d (0x8ebd bytes) 000554 1744882652.610747 mapread 0x01877d thru 0x0210f9 (0x897d bytes) 000555 1744882652.611248 mapread 0x01f2bd thru 0x028eeb (0x9c2f bytes) 000556 1744882652.611720 punch from 0x031138 to 0x0364d2 (0x539b bytes) 000558 1744882652.613716 punch from 0x0275d6 to 0x028502 (0xf2d bytes) 000559 1744882652.616533 read 0x006c49 thru 0x0107f0 (0x9ba8 bytes) 000560 1744882652.617058 mapread 0x020c5e thru 0x025206 (0x45a9 bytes) 000561 1744882652.617695 punch from 0x000584 to 0x00728f (0x6d0c bytes) 000562 1744882652.619576 trunc from 0x0364d3 to 0x016bf1 (0x1f8e1 bytes) 000563 1744882652.624153 punch from 0x01138d to 0x016bf1 (0x5865 bytes) 000564 1744882652.627376 punch from 0x01488f to 0x016bf1 (0x2363 bytes) 000565 1744882652.628787 trunc from 0x010d13 to 0x016bf1 (0x5edf bytes) 000566 1744882652.630448 trunc from 0x016988 to 0x016bf1 (0x26a bytes) 000567 1744882652.631797 trunc from 0x016bf2 to 0x02b638 (0x14a47 bytes) 000568 1744882652.635224 fallocate 0x002bac thru 0x00f1d7 (0xc62c bytes) 000569 1744882652.636429 mapwrite 0x0069bf thru 0x0084c6 (0x1b08 bytes) 000570 1744882652.643292 trunc from 0x004b76 to 0x013749 (0xebd4 bytes) 000571 1744882652.645959 MIRROR_EXTEND to 2 mirrors 000573 1744882652.753791 punch from 0x022c2a to 0x02b638 (0x8a0f bytes) 000574 1744882652.761174 mapwrite 0x032703 thru 0x034bb0 (0x24ae bytes) 000575 1744882652.779446 mapwrite 0x02f8c5 thru 0x03e7ae (0xeeea bytes) 000577 1744882652.793475 mapwrite 0x0214b5 thru 0x028216 (0x6d62 bytes) 000578 1744882652.801645 fallocate 0x0194e6 thru 0x01967e (0x199 bytes) 000580 1744882652.803679 trunc from 0x00c0da to 0x010ec9 (0x4df0 bytes) 000581 1744882652.806883 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xa46:0x0]: Invalid argument (22) 000582 1744882652.849224 write 0x008740 thru 0x011e9f (0x9760 bytes) 000583 1744882652.853597 fallocate 0x000acd thru 0x00219d (0x16d1 bytes) 000584 1744882652.855555 write 0x00bb77 thru 0x015a83 (0x9f0d bytes) 000585 1744882652.857790 write 0x009baf thru 0x00bc79 (0x20cb bytes) 000586 1744882652.858482 mapwrite 0x00b11c thru 0x01753f (0xc424 bytes) 000587 1744882652.867892 punch from 0x008ae6 to 0x0139ca (0xaee5 bytes) 000588 1744882652.870036 read 0x022ded thru 0x0254da (0x26ee bytes) 000589 1744882652.872132 read 0x03d95d thru 0x03e7ae (0xe52 bytes) 000591 1744882652.876046 punch from 0x030740 to 0x0394dd (0x8d9e bytes) 000592 1744882652.877963 read 0x03bf07 thru 0x03e7ae (0x28a8 bytes) 000593 1744882652.878375 fallocate 0x03074a thru 0x030cb4 (0x56b bytes) 000594 1744882652.879876 punch from 0x0254ff to 0x02d930 (0x8432 bytes) 000595 1744882652.881547 mapread 0x02b2fc thru 0x0372b8 (0xbfbd bytes) 000596 1744882652.885077 fallocate 0x00cbab thru 0x01a8eb (0xdd41 bytes) 000597 1744882652.887038 read 0x008903 thru 0x0142a6 (0xb9a4 bytes) 000598 1744882652.887343 fallocate 0x03db15 thru 0x03ffff (0x24eb bytes) 000600 1744882652.888993 trunc from 0x022298 to 0x026364 (0x40cd bytes) 000602 1744882652.890845 mapwrite 0x01519a thru 0x0168d7 (0x173e bytes) 000603 1744882652.896298 trunc from 0x040000 to 0x00d1b3 (0x32e4c bytes) 000604 1744882652.901140 trunc from 0x00a2a4 to 0x00d1b3 (0x2f10 bytes) 000605 1744882652.902669 punch from 0x00a368 to 0x00d1b3 (0x2e4c bytes) 000607 1744882652.904285 mapwrite 0x03206e thru 0x036727 (0x46ba bytes) 000608 1744882652.916335 fallocate 0x00c3cc thru 0x016d33 (0xa968 bytes) 000610 1744882652.918584 read 0x00cc60 thru 0x01af2d (0xe2ce bytes) 000611 1744882652.919120 trunc from 0x02810c to 0x02faf0 (0x79e5 bytes) 000612 1744882652.920516 mapread 0x00a8b2 thru 0x0118a6 (0x6ff5 bytes) 000613 1744882652.921198 mapwrite 0x001b1f thru 0x008ef0 (0x73d2 bytes) 000614 1744882652.926742 trunc from 0x0044a7 to 0x007849 (0x33a3 bytes) 000615 1744882652.928288 mapread 0x01771b thru 0x0273fa (0xfce0 bytes) 000616 1744882652.928907 fallocate 0x036ce4 thru 0x03ffff (0x931c bytes) 000617 1744882652.931675 MIRROR_EXTEND to 2 mirrors 000619 1744882653.065868 trunc from 0x040000 to 0x0018b8 (0x3e747 bytes) 000620 1744882653.076670 read 0x000ed0 thru 0x0018b8 (0x9e9 bytes) 000621 1744882653.079496 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xa46:0x0]: Invalid argument (22) 000622 1744882653.109476 trunc from 0x000d84 to 0x0018b8 (0xb35 bytes) 000623 1744882653.110959 write 0x005ac9 thru 0x00d8b9 (0x7df1 bytes) 000624 1744882653.113869 MIRROR_EXTEND to 2 mirrors 000625 1744882653.212299 write 0x0100fe thru 0x0149b6 (0x48b9 bytes) 000626 1744882653.221606 write 0x020a59 thru 0x0286fa (0x7ca2 bytes) 000627 1744882653.222616 mapwrite 0x014cf2 thru 0x015f73 (0x1282 bytes) 000629 1744882653.230266 fallocate 0x0138ba thru 0x0224b7 (0xebfe bytes) 000630 1744882653.231864 fallocate 0x02fb99 thru 0x035af0 (0x5f58 bytes) 000631 1744882653.233437 read 0x025558 thru 0x032690 (0xd139 bytes) 000632 1744882653.236021 read 0x027686 thru 0x02d007 (0x5982 bytes) 000633 1744882653.238096 MIRROR_RESYNC 2 mirrors 000634 1744882653.346789 mapread 0x012109 thru 0x013bed (0x1ae5 bytes) 000635 1744882653.349352 trunc from 0x030bde to 0x035af0 (0x4f13 bytes) 000636 1744882653.356489 write 0x016466 thru 0x01a90a (0x44a5 bytes) 000637 1744882653.359122 punch from 0x00b661 to 0x013bfa (0x859a bytes) 000638 1744882653.360609 write 0x0122ed thru 0x015c4f (0x3963 bytes) 000640 1744882653.363056 mapwrite 0x027cc5 thru 0x036afd (0xee39 bytes) 000641 1744882653.381612 read 0x00722b thru 0x013ed5 (0xccab bytes) 000642 1744882653.383345 trunc from 0x036afe to 0x000c0f (0x35eee bytes) 000643 1744882653.387162 punch from 0x0002be to 0x000c0f (0x952 bytes) 000644 1744882653.389293 mapread 0x000b33 thru 0x000c0f (0xdd bytes) 000645 1744882653.391186 read 0x000731 thru 0x000c0f (0x4df bytes) 000646 1744882653.391451 mapwrite 0x03416d thru 0x037563 (0x33f7 bytes) 000647 1744882653.403459 MIRROR_EXTEND to 3 mirrors 000648 1744882653.501027 trunc from 0x034fe9 to 0x037563 (0x257b bytes) 000649 1744882653.509662 fallocate 0x0329cb thru 0x03ffff (0xd635 bytes) 000650 1744882653.511385 read 0x00c878 thru 0x018bdf (0xc368 bytes) 000651 1744882653.513727 fallocate 0x00532f thru 0x00cdb3 (0x7a85 bytes) 000653 1744882653.515562 trunc from 0x0110af to 0x01d397 (0xc2e9 bytes) 000654 1744882653.517056 punch from 0x02b3f3 to 0x039108 (0xdd16 bytes) 000655 1744882653.518464 punch from 0x000520 to 0x002c70 (0x2751 bytes) 000656 1744882653.519871 mapread 0x039f41 thru 0x03ec38 (0x4cf8 bytes) 000657 1744882653.524134 mapwrite 0x028921 thru 0x029d89 (0x1469 bytes) 000658 1744882653.532935 MIRROR_EXTEND to 4 mirrors 000659 1744882653.653103 mapread 0x02a0b3 thru 0x02e1aa (0x40f8 bytes) 000660 1744882653.660867 trunc from 0x009c4f to 0x00abfc (0xfae bytes) 000661 1744882653.669447 punch from 0x022a72 to 0x02acd9 (0x8268 bytes) 000662 1744882653.671017 mapwrite 0x03a465 thru 0x03b354 (0xef0 bytes) 000665 1744882653.678689 write 0x003851 thru 0x00ec4e (0xb3fe bytes) 000666 1744882653.682843 trunc from 0x040000 to 0x01ea10 (0x215ef bytes) 000667 1744882653.690925 mapwrite 0x00b355 thru 0x01312f (0x7ddb bytes) 000668 1744882653.698853 trunc from 0x00d42e to 0x01a985 (0xd558 bytes) 000669 1744882653.701258 MIRROR_RESYNC 4 mirrors 000670 1744882653.868163 read 0x00afa8 thru 0x015bb9 (0xac12 bytes) 000671 1744882653.870460 punch from 0x00adf9 to 0x00c625 (0x182d bytes) 000672 1744882653.878543 punch from 0x00ef5f to 0x0119ed (0x2a8f bytes) 000673 1744882653.879824 fallocate 0x03d347 thru 0x03ffff (0x2cb9 bytes) 000674 1744882653.881197 trunc from 0x017ed0 to 0x01ea10 (0x6b41 bytes) 000675 1744882653.882675 mapwrite 0x003223 thru 0x0070a7 (0x3e85 bytes) 000676 1744882653.892440 fallocate 0x035238 thru 0x03ffff (0xadc8 bytes) 000678 1744882653.894510 trunc from 0x01701b to 0x0260bd (0xf0a3 bytes) 000679 1744882653.897539 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xa46:0x0]: Invalid argument (22) 000681 1744882653.946604 mapread 0x006ccf thru 0x0163e0 (0xf712 bytes) 000682 1744882653.953328 mapwrite 0x02ac30 thru 0x0381da (0xd5ab bytes) 000684 1744882653.964560 punch from 0x03a620 to 0x03ffff (0x59e0 bytes) 000685 1744882653.966409 mapwrite 0x029d0e thru 0x02d399 (0x368c bytes) 000686 1744882653.971331 write 0x009164 thru 0x00ca43 (0x38e0 bytes) 000687 1744882653.973099 punch from 0x027088 to 0x03534a (0xe2c3 bytes) 000688 1744882653.974657 trunc from 0x040000 to 0x019f98 (0x26067 bytes) 000689 1744882653.982264 fallocate 0x007717 thru 0x017310 (0xfbfa bytes) 000690 1744882653.983643 mapwrite 0x021df7 thru 0x0278db (0x5ae5 bytes) 000691 1744882653.995866 punch from 0x026b71 to 0x0278db (0xd6b bytes) 000692 1744882653.997489 mapwrite 0x0026e2 thru 0x003884 (0x11a3 bytes) 000693 1744882654.003587 read 0x00f6cb thru 0x01a6be (0xaff4 bytes) 000695 1744882654.004335 trunc from 0x021535 to 0x0278db (0x63a7 bytes) 000696 1744882654.005743 mapread 0x00a6d1 thru 0x00e636 (0x3f66 bytes) 000697 1744882654.007627 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xa46:0x0]: Invalid argument (22) 000698 1744882654.058093 mapwrite 0x01085b thru 0x01fec0 (0xf666 bytes) 000699 1744882654.072650 read 0x0266fc thru 0x0278db (0x11e0 bytes) 000700 1744882654.076201 MIRROR_RESYNC 2 mirrors 000701 1744882654.179468 trunc from 0x0019ae to 0x00793b (0x5f8e bytes) 000702 1744882654.187318 write 0x017619 thru 0x023f5c (0xc944 bytes) 000703 1744882654.190114 trunc from 0x00decb to 0x01230f (0x4445 bytes) 000704 1744882654.191477 punch from 0x00939b to 0x0129fa (0x9660 bytes) 000705 1744882654.192909 fallocate 0x01872f thru 0x0272df (0xebb1 bytes) 000706 1744882654.194505 mapread 0x0263cc thru 0x0278db (0x1510 bytes) 000707 1744882654.196605 fallocate 0x015936 thru 0x020fd9 (0xb6a4 bytes) 000708 1744882654.198238 read 0x019d5b thru 0x0278db (0xdb81 bytes) 000709 1744882654.198676 trunc from 0x0278dc to 0x02675a (0x1181 bytes) 000710 1744882654.207480 punch from 0x00e3e8 to 0x01594e (0x7567 bytes) 000711 1744882654.209210 fallocate 0x039c70 thru 0x03fe43 (0x61d4 bytes) 000712 1744882654.210698 mapread 0x014eb2 thru 0x02398d (0xeadc bytes) 000713 1744882654.213199 trunc from 0x02675b to 0x01ea2b (0x7d2f bytes) 000714 1744882654.217053 mapwrite 0x00bfb5 thru 0x0168aa (0xa8f6 bytes) 000715 1744882654.226836 mapwrite 0x015cc4 thru 0x01e9c7 (0x8d04 bytes) 000716 1744882654.232170 mapwrite 0x0217d0 thru 0x02d68c (0xbebd bytes) 000717 1744882654.244878 mapread 0x018cb3 thru 0x01901b (0x369 bytes) 000718 1744882654.245372 mapwrite 0x001758 thru 0x009bb0 (0x8459 bytes) 000719 1744882654.252719 MIRROR_RESYNC 2 mirrors 000720 1744882654.347269 mapwrite 0x02182d thru 0x022bb5 (0x1389 bytes) 000721 1744882654.375274 trunc from 0x011350 to 0x01604a (0x4cfb bytes) 000722 1744882654.376789 read 0x029918 thru 0x02d68c (0x3d75 bytes) 000723 1744882654.378650 fallocate 0x00f507 thru 0x00f903 (0x3fd bytes) 000724 1744882654.380138 trunc from 0x02d68d to 0x011797 (0x1bef5 bytes) 000725 1744882654.388584 trunc from 0x011798 to 0x02539c (0x13c05 bytes) 000726 1744882654.391990 read 0x002bbc thru 0x00b376 (0x87bb bytes) 000727 1744882654.396425 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xa46:0x0]: Invalid argument (22) 000728 1744882654.429905 fallocate 0x03b188 thru 0x03de1a (0x2c93 bytes) 000729 1744882654.431585 mapread 0x017a99 thru 0x01c16e (0x46d6 bytes) 000730 1744882654.435922 fallocate 0x0117db thru 0x020b38 (0xf35e bytes) 000731 1744882654.437299 trunc from 0x02539d to 0x02e86a (0x94ce bytes) 000732 1744882654.442202 write 0x03a9c6 thru 0x03bad6 (0x1111 bytes) 000733 1744882654.442723 write 0x00536b thru 0x013739 (0xe3cf bytes) 000735 1744882654.446978 MIRROR_EXTEND to 2 mirrors 000736 1744882654.548170 read 0x01bc63 thru 0x01cb0f (0xead bytes) 000738 1744882654.551151 MIRROR_EXTEND to 3 mirrors 000739 1744882654.663874 trunc from 0x03a69e to 0x03bad6 (0x1439 bytes) 000740 1744882654.673097 write 0x00d365 thru 0x0138af (0x654b bytes) 000741 1744882654.677194 mapread 0x003334 thru 0x004b11 (0x17de bytes) 000742 1744882654.679758 trunc from 0x03bad7 to 0x01ebc0 (0x1cf16 bytes) 000743 1744882654.688610 trunc from 0x01c1e0 to 0x01ebc0 (0x29e1 bytes) 000744 1744882654.689912 write 0x0395cf thru 0x03ffff (0x6a31 bytes) 000745 1744882654.690720 write 0x032887 thru 0x03f83d (0xcfb7 bytes) 000746 1744882654.695024 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xa46:0x0]: Invalid argument (22) 000748 1744882654.748377 mapread 0x01383d thru 0x01944e (0x5c12 bytes) 000749 1744882654.753060 punch from 0x01797b to 0x0184a3 (0xb29 bytes) 000750 1744882654.754460 read 0x03c2ff thru 0x03ffff (0x3d01 bytes) 000751 1744882654.756079 mapread 0x03725c thru 0x038fdb (0x1d80 bytes) 000752 1744882654.758607 trunc from 0x01c85f to 0x029471 (0xcc13 bytes) 000753 1744882654.760173 trunc from 0x0053d2 to 0x0098ac (0x44db bytes) 000754 1744882654.761624 punch from 0x03c59a to 0x03ffff (0x3a66 bytes) 000755 1744882654.763214 fallocate 0x003812 thru 0x004e2d (0x161c bytes) 000756 1744882654.765967 MIRROR_RESYNC 2 mirrors 000757 1744882654.903745 write 0x035bfc thru 0x03ffff (0xa404 bytes) 000758 1744882654.913508 trunc from 0x040000 to 0x021e21 (0x1e1de bytes) 000759 1744882654.921250 trunc from 0x001393 to 0x00b7a0 (0xa40e bytes) 000760 1744882654.922803 mapwrite 0x0121c1 thru 0x01ccd8 (0xab18 bytes) 000761 1744882654.933790 trunc from 0x014ed6 to 0x01c49d (0x75c8 bytes) 000762 1744882654.935216 trunc from 0x019057 to 0x021e21 (0x8dcb bytes) 000763 1744882654.936568 punch from 0x0121da to 0x01d109 (0xaf30 bytes) 000764 1744882654.938123 fallocate 0x011481 thru 0x0181f3 (0x6d73 bytes) 000765 1744882654.939648 mapread 0x006797 thru 0x0067e5 (0x4f bytes) 000766 1744882654.941963 trunc from 0x021e22 to 0x001c09 (0x20218 bytes) 000767 1744882654.946778 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xa46:0x0]: Invalid argument (22) 000768 1744882654.979514 trunc from 0x0019d7 to 0x001c09 (0x233 bytes) 000769 1744882654.981218 mapwrite 0x00f7f1 thru 0x014b72 (0x5382 bytes) 000770 1744882654.998893 trunc from 0x014b73 to 0x02ff2f (0x1b3bd bytes) 000771 1744882655.003487 punch from 0x001847 to 0x00b56f (0x9d29 bytes) 000772 1744882655.005115 read 0x01669e thru 0x01f738 (0x909b bytes) 000773 1744882655.008089 punch from 0x00f6d1 to 0x00fe0f (0x73f bytes) 000774 1744882655.009490 write 0x032b8e thru 0x03ffff (0xd472 bytes) 000775 1744882655.010660 trunc from 0x040000 to 0x034e1f (0xb1e0 bytes) 000778 1744882655.015401 mapread 0x01f9bc thru 0x02df98 (0xe5dd bytes) 000779 1744882655.020562 mapread 0x00d473 thru 0x0153b4 (0x7f42 bytes) 000780 1744882655.023093 write 0x007a2a thru 0x0178e2 (0xfeb9 bytes) 000782 1744882655.026311 trunc from 0x034e20 to 0x022222 (0x12bfd bytes) 000783 1744882655.030334 write 0x00d023 thru 0x013b77 (0x6b55 bytes) 000785 1744882655.031479 punch from 0x011c1f to 0x01338b (0x176d bytes) 000786 1744882655.035375 punch from 0x0104dc to 0x013c4c (0x3771 bytes) 000787 1744882655.036716 mapwrite 0x00a59a thru 0x00b858 (0x12bf bytes) 000788 1744882655.041263 trunc from 0x022223 to 0x019bcc (0x8656 bytes) 000789 1744882655.044980 write 0x03f4bf thru 0x03ffff (0xb41 bytes) 000790 1744882655.045596 mapwrite 0x037dfb thru 0x03ffff (0x8205 bytes) 000791 1744882655.055168 mapwrite 0x02b519 thru 0x037802 (0xc2ea bytes) 000793 1744882655.064033 mapread 0x015565 thru 0x016647 (0x10e3 bytes) 000794 1744882655.065699 MIRROR_EXTEND to 2 mirrors 000795 1744882655.181544 trunc from 0x040000 to 0x03a034 (0x5fcb bytes) 000796 1744882655.196625 mapwrite 0x019d20 thru 0x028bf7 (0xeed8 bytes) 000797 1744882655.211529 fallocate 0x004a21 thru 0x00c703 (0x7ce3 bytes) 000799 1744882655.213915 trunc from 0x03a035 to 0x021137 (0x18efd bytes) 000800 1744882655.219343 read 0x00820d thru 0x0136c4 (0xb4b8 bytes) 000801 1744882655.221911 mapread 0x0007f1 thru 0x00a49f (0x9caf bytes) 000802 1744882655.225430 trunc from 0x021138 to 0x01407d (0xd0ba bytes) 000803 1744882655.229290 mapwrite 0x0253bb thru 0x033fc7 (0xec0d bytes) 000804 1744882655.244790 fallocate 0x0335dc thru 0x03c276 (0x8c9b bytes) 000805 1744882655.246524 mapread 0x015c27 thru 0x015fa0 (0x37a bytes) 000806 1744882655.249143 mapread 0x01aa79 thru 0x027fdf (0xd567 bytes) 000807 1744882655.249757 write 0x0122ae thru 0x0193ee (0x7141 bytes) 000808 1744882655.250460 punch from 0x002215 to 0x003d11 (0x1afd bytes) 000809 1744882655.251847 write 0x019981 thru 0x01a294 (0x914 bytes) 000810 1744882655.252348 trunc from 0x03c277 to 0x006177 (0x360ff bytes) 000811 1744882655.256678 fallocate 0x03282b thru 0x037894 (0x506a bytes) 000812 1744882655.258346 trunc from 0x00184b to 0x002891 (0x1047 bytes) 000813 1744882655.260046 write 0x023ed7 thru 0x0333e7 (0xf511 bytes) 000814 1744882655.264510 write 0x01bae2 thru 0x0245da (0x8af9 bytes) 000815 1744882655.267241 read 0x025a7a thru 0x02d06e (0x75f5 bytes) 000816 1744882655.267743 mapread 0x01cabe thru 0x02c4fd (0xfa40 bytes) 000817 1744882655.268469 write 0x03f3d6 thru 0x03ffff (0xc2a bytes) 000818 1744882655.269014 mapwrite 0x03b3e1 thru 0x03ffff (0x4c1f bytes) 000819 1744882655.279319 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xa46:0x0]: Invalid argument (22) 000820 1744882655.316578 mapread 0x000099 thru 0x010059 (0xffc1 bytes) 000821 1744882655.324399 mapread 0x009fc7 thru 0x014cbe (0xacf8 bytes) 000822 1744882655.325060 punch from 0x00ab25 to 0x0191aa (0xe686 bytes) 000824 1744882655.327920 write 0x02c2fb thru 0x02f913 (0x3619 bytes) 000825 1744882655.332217 trunc from 0x037a02 to 0x03c502 (0x4b01 bytes) 000826 1744882655.333998 fallocate 0x01da35 thru 0x0264e0 (0x8aac bytes) 000827 1744882655.335368 fallocate 0x007e16 thru 0x00c756 (0x4941 bytes) 000829 1744882655.337492 mapread 0x00fed3 thru 0x01d136 (0xd264 bytes) 000830 1744882655.342085 trunc from 0x01d00e to 0x0245bf (0x75b2 bytes) 000831 1744882655.343901 read 0x003f3d thru 0x0108ba (0xc97e bytes) 000832 1744882655.345508 trunc from 0x000e9a to 0x0100f1 (0xf258 bytes) 000833 1744882655.347408 write 0x000cd3 thru 0x00489d (0x3bcb bytes) 000835 1744882655.350915 punch from 0x02fc4b to 0x037eae (0x8264 bytes) 000838 1744882655.356049 mapread 0x03bb3d thru 0x03ffff (0x44c3 bytes) 000839 1744882655.358280 mapwrite 0x0324b5 thru 0x03ffff (0xdb4b bytes) 000840 1744882655.365776 read 0x02ccff thru 0x0388b7 (0xbbb9 bytes) 000841 1744882655.366264 mapread 0x01d547 thru 0x021be2 (0x469c bytes) 000843 1744882655.369214 fallocate 0x03e4dd thru 0x03ffff (0x1b23 bytes) 000844 1744882655.370559 mapread 0x01114b thru 0x01d7e8 (0xc69e bytes) 000845 1744882655.371109 punch from 0x0379b8 to 0x03ffff (0x8648 bytes) 000846 1744882655.372474 trunc from 0x01d889 to 0x02a838 (0xcfb0 bytes) 000847 1744882655.374049 trunc from 0x040000 to 0x0302ec (0xfd13 bytes) 000848 1744882655.378109 fallocate 0x00a850 thru 0x00e7af (0x3f60 bytes) 000849 1744882655.379907 trunc from 0x0302ed to 0x03e253 (0xdf67 bytes) 000850 1744882655.383962 trunc from 0x03920a to 0x03e253 (0x504a bytes) 000851 1744882655.385498 punch from 0x014c0d to 0x015054 (0x448 bytes) 000852 1744882655.387948 MIRROR_EXTEND to 2 mirrors 000853 1744882655.501758 fallocate 0x012961 thru 0x02026c (0xd90c bytes) 000854 1744882655.509581 write 0x00b470 thru 0x00d04a (0x1bdb bytes) 000856 1744882655.514424 MIRROR_RESYNC 2 mirrors 000857 1744882655.615234 trunc from 0x032ed2 to 0x03e253 (0xb382 bytes) 000859 1744882655.624207 mapread 0x038b3d thru 0x03e253 (0x5717 bytes) 000860 1744882655.629144 mapread 0x0010c4 thru 0x002a8a (0x19c7 bytes) 000861 1744882655.631692 trunc from 0x00042c to 0x003fe7 (0x3bbc bytes) 000862 1744882655.633501 trunc from 0x014015 to 0x0206f0 (0xc6dc bytes) 000863 1744882655.635098 write 0x039f0e thru 0x03c1e7 (0x22da bytes) 000864 1744882655.635794 trunc from 0x02921a to 0x033f20 (0xad07 bytes) 000865 1744882655.637383 trunc from 0x02e3f0 to 0x033b11 (0x5722 bytes) 000866 1744882655.638797 write 0x02a6e1 thru 0x02d4fa (0x2e1a bytes) 000867 1744882655.642302 fallocate 0x030db5 thru 0x03aeeb (0xa137 bytes) 000869 1744882655.644215 read 0x000b03 thru 0x00871b (0x7c19 bytes) 000870 1744882655.645873 trunc from 0x001311 to 0x0075d0 (0x62c0 bytes) 000871 1744882655.649799 MIRROR_RESYNC 2 mirrors 000872 1744882655.772526 mapwrite 0x00c2a5 thru 0x0167fc (0xa558 bytes) 000874 1744882655.809594 trunc from 0x03e254 to 0x01b255 (0x22ffe bytes) 000875 1744882655.820509 read 0x011892 thru 0x01b255 (0x99c4 bytes) 000876 1744882655.821014 trunc from 0x01b256 to 0x006571 (0x14ce4 bytes) 000877 1744882655.826384 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xa46:0x0]: Invalid argument (22) 000878 1744882655.861519 mapwrite 0x000dda thru 0x006cd7 (0x5efe bytes) 000879 1744882655.877689 punch from 0x003f1f to 0x006cd7 (0x2db9 bytes) 000880 1744882655.880040 mapread 0x0003bb thru 0x004e00 (0x4a46 bytes) 000881 1744882655.882254 mapread 0x0034a4 thru 0x006cd7 (0x3834 bytes) 000882 1744882655.882708 fallocate 0x01ccc3 thru 0x025347 (0x8685 bytes) 000883 1744882655.884459 write 0x004b48 thru 0x00eed9 (0xa392 bytes) 000884 1744882655.886739 read 0x0130ed thru 0x01f68c (0xc5a0 bytes) 000885 1744882655.889383 trunc from 0x025348 to 0x01d7ed (0x7b5a bytes) 000886 1744882655.893169 trunc from 0x01d7ee to 0x0268c8 (0x90db bytes) 000887 1744882655.896692 fallocate 0x01d1ce thru 0x021866 (0x4699 bytes) 000888 1744882655.898450 mapwrite 0x0089a9 thru 0x011564 (0x8bbc bytes) 000889 1744882655.908335 trunc from 0x0268c9 to 0x025d39 (0xb8f bytes) 000890 1744882655.912735 trunc from 0x008d12 to 0x0169cb (0xdcba bytes) 000891 1744882655.914290 punch from 0x00994b to 0x00e89d (0x4f53 bytes) 000892 1744882655.915513 punch from 0x0184de to 0x025581 (0xd0a4 bytes) 000894 1744882655.917531 trunc from 0x025d3a to 0x01b098 (0xaca1 bytes) 000896 1744882655.921945 trunc from 0x0015b5 to 0x00bce4 (0xa730 bytes) 000897 1744882655.924008 mapwrite 0x03987c thru 0x03ffff (0x6784 bytes) 000898 1744882655.938073 punch from 0x029cd2 to 0x02b8d6 (0x1c05 bytes) 000900 1744882655.940328 trunc from 0x040000 to 0x03ba39 (0x45c6 bytes) 000901 1744882655.944658 punch from 0x02fce6 to 0x039963 (0x9c7e bytes) 000902 1744882655.946353 read 0x0285a3 thru 0x035d3c (0xd79a bytes) 000903 1744882655.950126 mapwrite 0x01029c thru 0x017e5b (0x7bc0 bytes) 000904 1744882655.958653 MIRROR_EXTEND to 2 mirrors 000905 1744882656.096527 trunc from 0x003a98 to 0x009987 (0x5ef0 bytes) 000906 1744882656.107170 MIRROR_RESYNC 2 mirrors 000908 1744882656.235339 trunc from 0x019045 to 0x01fcc0 (0x6c7c bytes) 000909 1744882656.243803 punch from 0x01e711 to 0x02239c (0x3c8c bytes) 000910 1744882656.245228 trunc from 0x03ba3a to 0x03d087 (0x164e bytes) 000912 1744882656.249794 fallocate 0x015647 thru 0x016d8c (0x1746 bytes) 000913 1744882656.251391 trunc from 0x020cf7 to 0x028efc (0x8206 bytes) 000914 1744882656.252873 read 0x026004 thru 0x033e9c (0xde99 bytes) 000916 1744882656.255590 mapwrite 0x03044f thru 0x03ffff (0xfbb1 bytes) 000917 1744882656.271927 trunc from 0x0137a4 to 0x02264e (0xeeab bytes) 000918 1744882656.273695 read 0x0199f0 thru 0x01babd (0x20ce bytes) 000919 1744882656.278086 MIRROR_RESYNC 2 mirrors 000920 1744882656.379216 MIRROR_RESYNC 2 mirrors 000921 1744882656.418236 read 0x01af2c thru 0x02863e (0xd713 bytes) 000922 1744882656.421231 trunc from 0x01c17f to 0x0232ed (0x716f bytes) 000923 1744882656.430784 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xa46:0x0]: Invalid argument (22) 000924 1744882656.467268 mapread 0x0164b0 thru 0x025d89 (0xf8da bytes) 000925 1744882656.474588 fallocate 0x0055c3 thru 0x00be0f (0x684d bytes) 000926 1744882656.476389 read 0x034f14 thru 0x03ffff (0xb0ec bytes) 000927 1744882656.479873 fallocate 0x0241a5 thru 0x026053 (0x1eaf bytes) 000928 1744882656.481658 mapwrite 0x0396f9 thru 0x03ffff (0x6907 bytes) 000929 1744882656.489595 punch from 0x0165ab to 0x021d7a (0xb7d0 bytes) 000930 1744882656.491404 mapread 0x01cdf4 thru 0x0299f4 (0xcc01 bytes) 000931 1744882656.494547 fallocate 0x018691 thru 0x0202c1 (0x7c31 bytes) 000932 1744882656.496245 read 0x008517 thru 0x013c71 (0xb75b bytes) 000933 1744882656.499262 mapread 0x002528 thru 0x00de9e (0xb977 bytes) 000934 1744882656.499868 fallocate 0x035692 thru 0x037e55 (0x27c4 bytes) 000935 1744882656.501405 mapread 0x0149e2 thru 0x01bfdf (0x75fe bytes) 000936 1744882656.501977 trunc from 0x0327f5 to 0x038920 (0x612c bytes) 000938 1744882656.503643 read 0x0344eb thru 0x03fd99 (0xb8af bytes) 000939 1744882656.505977 trunc from 0x040000 to 0x0055d1 (0x3aa2e bytes) 000940 1744882656.515314 trunc from 0x0055d2 to 0x01e582 (0x18fb1 bytes) 000942 1744882656.518893 trunc from 0x01d3b1 to 0x01e582 (0x11d2 bytes) 000943 1744882656.520232 fallocate 0x03e0e9 thru 0x03ffff (0x1f17 bytes) 000944 1744882656.521757 fallocate 0x03c19f thru 0x03ffff (0x3e61 bytes) 000945 1744882656.524661 MIRROR_EXTEND to 2 mirrors 000946 1744882656.639108 MIRROR_EXTEND to 3 mirrors 000947 1744882656.754756 punch from 0x02324b to 0x025e30 (0x2be6 bytes) 000948 1744882656.763721 mapread 0x014cc2 thru 0x01b595 (0x68d4 bytes) 000949 1744882656.768195 trunc from 0x040000 to 0x01960d (0x269f2 bytes) 000950 1744882656.778855 mapwrite 0x03e685 thru 0x03ffff (0x197b bytes) 000951 1744882656.790274 fallocate 0x01227a thru 0x018e55 (0x6bdc bytes) 000952 1744882656.792347 read 0x015bfc thru 0x018021 (0x2426 bytes) 000953 1744882656.792853 mapread 0x00a9c0 thru 0x019b5f (0xf1a0 bytes) 000954 1744882656.796686 write 0x01ce23 thru 0x029cc9 (0xcea7 bytes) 000955 1744882656.802086 fallocate 0x01fdd5 thru 0x02d982 (0xdbae bytes) 000956 1744882656.805984 MIRROR_EXTEND to 4 mirrors 000957 1744882656.921625 trunc from 0x040000 to 0x01e0d9 (0x21f26 bytes) 000959 1744882656.935494 mapread 0x013808 thru 0x01e0d9 (0xa8d2 bytes) 000960 1744882656.941600 punch from 0x002606 to 0x00a426 (0x7e21 bytes) 000961 1744882656.944236 MIRROR_RESYNC 4 mirrors 000962 1744882657.092676 write 0x03d511 thru 0x03ffff (0x2aef bytes) 000963 1744882657.101068 read 0x0218e5 thru 0x025eba (0x45d6 bytes) 000965 1744882657.105062 MIRROR_EXTEND to 5 mirrors 000966 1744882657.242584 MIRROR_EXTEND to 6 mirrors 000968 1744882657.376969 trunc from 0x040000 to 0x02850a (0x17af5 bytes) 000969 1744882657.394572 mapwrite 0x00cf4a thru 0x016ee8 (0x9f9f bytes) 000970 1744882657.408460 read 0x0186a7 thru 0x01e66b (0x5fc5 bytes) 000971 1744882657.413071 read 0x01dcc9 thru 0x0217a7 (0x3adf bytes) 000972 1744882657.416750 trunc from 0x0014b9 to 0x0082a8 (0x6df0 bytes) 000973 1744882657.418937 punch from 0x019016 to 0x01da1f (0x4a0a bytes) 000974 1744882657.420884 mapread 0x01efe1 thru 0x02850a (0x952a bytes) 000975 1744882657.421511 fallocate 0x00894c thru 0x014beb (0xc2a0 bytes) 000976 1744882657.423289 mapwrite 0x003b88 thru 0x003ba3 (0x1c bytes) 000977 1744882657.429230 mapread 0x025d20 thru 0x02850a (0x27eb bytes) 000978 1744882657.429891 fallocate 0x011f2a thru 0x01a9b3 (0x8a8a bytes) 000979 1744882657.431804 read 0x024e76 thru 0x02850a (0x3695 bytes) 000980 1744882657.433666 MIRROR_RESYNC 6 mirrors 000981 1744882657.765364 trunc from 0x0253b0 to 0x027bc7 (0x2818 bytes) 000982 1744882657.773546 trunc from 0x02850b to 0x020485 (0x8085 bytes) 000983 1744882657.781735 mapwrite 0x00a405 thru 0x0151ab (0xada7 bytes) 000984 1744882657.795415 write 0x03f192 thru 0x03ffff (0xe6e bytes) 000985 1744882657.796100 write 0x03ed43 thru 0x03ffff (0x12bd bytes) 000986 1744882657.799551 trunc from 0x040000 to 0x01a183 (0x25e7c bytes) 000987 1744882657.804169 trunc from 0x01a184 to 0x032bf6 (0x18a73 bytes) 000989 1744882657.808232 read 0x0021a6 thru 0x00633a (0x4195 bytes) 000991 1744882657.810304 fallocate 0x0349d2 thru 0x03ffff (0xb62e bytes) 000992 1744882657.811927 read 0x01a43f thru 0x022154 (0x7d16 bytes) 000993 1744882657.814952 mapread 0x021e52 thru 0x02ce6e (0xb01d bytes) 000994 1744882657.815543 punch from 0x0037b7 to 0x0044b6 (0xd00 bytes) 000995 1744882657.817342 mapwrite 0x03742b thru 0x03ffff (0x8bd5 bytes) 000996 1744882657.830267 trunc from 0x040000 to 0x000236 (0x3fdc9 bytes) 000997 1744882657.834470 trunc from 0x00010c to 0x000236 (0x12b bytes) 000998 1744882657.837078 trunc from 0x000237 to 0x02405c (0x23e26 bytes) 000999 1744882657.841326 trunc from 0x02405d to 0x019524 (0xab38 bytes) 001000 1744882657.844977 punch from 0x00e1f2 to 0x00fa72 (0x1881 bytes) All operations completed A-OK! PASS 70a (12s) == sanity-flr test 71: check mirror extend parent fid ==== 05:37:39 (1744882659) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000404:0xa88:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (3s) == sanity-flr test 200a: stress test ===================== 05:37:41 (1744882661) Starting client: oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre2 Starting client: oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7557184 .. Extending file size to 5448032 .. failed Extending file size to 6956064 .. Extending file size to 9197696 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8207392 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8385024 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8458080 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8683232 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9212224 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 4184 4185 4187 4188 4191 192.168.201.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg124-client.virtnet /mnt/lustre2 (opts:) 192.168.201.124@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg124-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (63s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 05:38:44 (1744882724) Starting client: oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre2 Starting client: oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7014976 .. Extending file size to 3856480 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8707936 .. failed Extending file size to 8824224 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 8928768 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9336640 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9023520 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9114656 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9412672 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 8911 8912 8914 8915 8918 failed 192.168.201.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg124-client.virtnet /mnt/lustre2 (opts:) 192.168.201.124@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg124-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 05:39:47 (1744882787) Starting client: oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2704 1284984 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3284 1284404 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 14996 3592024 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 23064 3583956 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38060 7175980 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2704 1284984 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 3284 1284404 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 14996 3592024 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 23064 3583956 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 38060 7175980 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0396649 s, 52.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0578847 s, 36.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg124-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 05:39:52 (1744882792) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0338382 s, 62.0 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 05:39:54 (1744882794) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x37:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 0x20002 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa0:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 0x20002 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa0:0x0] } lcme_id: 0x30001 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x7f:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa1:0x0] } PASS 203 (1s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 05:39:55 (1744882795) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0222163 s, 94.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xdf0:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xdf0:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xdf1:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 05:39:57 (1744882797) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0240166 s, 87.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xdf1:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xdf2:0x0] } lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xdf2:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xdf1:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xdf2:0x0] } lcme_id: 0x10003 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xdf2:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xdf3:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 05:39:59 (1744882799) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0314083 s, 66.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x80:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa2:0x0] } lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x80:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa2:0x0] } lcme_id: 0x20006 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa3:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 05:40:22 (1744882822) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg124-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0205414 s, 51.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xdf3:0x0] } lcme_id: 0x10005 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xdf5:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xdf4:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xdf3:0x0] } lcme_id: 0x10005 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xdf5:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xdf4:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xdf4:0x0] } PASS 204d (35s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 05:40:57 (1744882857) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg124-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x81:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x82:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0262576 s, 79.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x81:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x82:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa4:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x81:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x82:0x0] } lcme_id: 0x20005 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa4:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (32s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 05:41:29 (1744882889) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg124-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg124-server: Pool lustre.test_204f created Adding targets to pool oleg124-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg124-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.027578 s, 76.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x83:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa5:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x83:0x0] } lcme_id: 0x10002 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 0x20003 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa5:0x0] } lcme_id: 0x20004 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg124-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg124-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg124-server: Pool lustre.test_204f destroyed PASS 204f (43s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 05:42:12 (1744882932) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 05:42:14 (1744882934) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 05:42:16 (1744882936) oleg124-server: Pool lustre.test_206 created Adding targets to pool oleg124-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg124-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg124-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg124-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg124-server: Pool lustre.test_206 destroyed PASS 206 (12s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 05:42:28 (1744882948) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0221558 s, 47.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.010682 s, 98.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x537307db Mirror 3: 0x537307db PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 05:42:30 (1744882950) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 05:42:31 (1744882951) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 05:42:32 (1744882952) Starting client: oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0139342 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x16b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1e20000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1e20000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1e20000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x10f0000 Read this (4 bytes): BBBB 192.168.201.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg124-client.virtnet /mnt/lustre2 (opts:) PASS 209a (14s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 05:42:46 (1744882966) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.0026351 s, 3.1 MB/s the very first read read with warm cache PASS 209b (1s) == sanity-flr test 210a: handle broken mirrored lovea ==== 05:42:47 (1744882967) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00947043 s, 111 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 0x10001 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x56 lmm_fid: [0x200000405:0x56:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 0x20001 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x56 lmm_fid: [0x200000405:0x56:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611222 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-17 05:42:48.000000000 -0400 Modify: 2025-04-17 05:42:48.000000000 -0400 Change: 2025-04-17 05:42:48.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 05:42:49 (1744882969) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00845386 s, 124 MB/s oleg124-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg124-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg124-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg124-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg124-server: O/280000401/d14/8e: File not found by ext2_lookup PASS 210b (9s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 05:42:58 (1744882978) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0349208 s, 1.2 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00674666 s, 607 kB/s size after second write -rw-r--r-- 1 root root 4096 Apr 17 05:42 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5b:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Apr 17 05:42 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 695 sec ============ 05:43:00 (1744882980) === sanity-flr: start cleanup 05:43:01 (1744882981) === === sanity-flr: finish cleanup 05:43:03 (1744882983) ===