-----============= acceptance-small: sanity-flr ============----- Fri May 10 08:07:55 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 6 201 44c === sanity-flr: start setup 08:07:59 (1715342879) === oleg119-client.virtnet: executing check_config_client /mnt/lustre oleg119-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg119-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b6495800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6495800.idle_timeout=debug disable quota as required oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 08:08:06 (1715342886) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f7465] debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0a: lfs mirror create with -N option == 08:08:07 (1715342887) PASS 0a (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 08:08:14 (1715342894) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 08:08:18 (1715342898) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0d: lfs mirror extend with -N option == 08:08:21 (1715342901) PASS 0d (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 08:08:28 (1715342908) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 08:08:31 (1715342911) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0g: lfs mirror create flags support === 08:08:34 (1715342914) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b6495800.stats=clear osc.lustre-OST0001-osc-ffff8800b6495800.stats=clear PASS 0g (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 08:08:38 (1715342918) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 08:08:42 (1715342922) PASS 0j (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 08:08:46 (1715342926) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 1: create components with setstripe options ========================================================== 08:08:49 (1715342929) lfs mirror mirror: cannot get UNLOCK lease, ext 4: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 2: create components from existing files ========================================================== 08:08:54 (1715342934) PASS 2 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 3: create components from files located on different MDTs ========================================================== 08:08:58 (1715342938) PASS 3 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 08:09:03 (1715342943) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 08:09:07 (1715342947) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 7: Create mirror with DoM component === 08:09:12 (1715342952) PASS 7 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 08:09:16 (1715342956) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.386156 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0439267 s, 23.9 MB/s PASS 21 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 08:09:21 (1715342961) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.0476388 s, 176 MB/s PASS 22 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 08:09:26 (1715342966) Stopping /mnt/lustre-ost1 (opts:) on oleg119-server oleg119-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6495800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6495800.ost_server_uuid in DISCONN state after 3 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg119-server: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg119-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg119-server oleg119-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6495800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6495800.ost_server_uuid in DISCONN state after 0 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg119-server: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg119-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (26s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 08:09:53 (1715342993) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB) copied, 0.0845954 s, 74.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg119-server oleg119-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6495800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6495800.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg119-server: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg119-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (9s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33a: read can choose available mirror to read ========================================================== 08:10:03 (1715343003) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg119-server oleg119-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6495800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6495800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg119-server: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg119-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg119-server oleg119-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6495800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6495800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg119-server: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg119-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (14s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 08:10:19 (1715343019) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.695519 s, 75.4 MB/s -rw-r--r-- 1 root root 52428800 May 10 08:10 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.179s user 0m0.001s sys 0m0.094s Stopping /mnt/lustre-ost1 (opts:) on oleg119-server oleg119-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6495800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6495800.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost2)... real 0m0.203s user 0m0.001s sys 0m0.098s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg119-server: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg119-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg119-server oleg119-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6495800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6495800.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost1)... real 0m0.197s user 0m0.000s sys 0m0.091s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg119-server: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg119-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg119-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b6495800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6495800.ost_server_uuid in FULL state after 0 sec PASS 33b (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 08:10:44 (1715343044) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 08:10:46 (1715343046) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 08:10:49 (1715343049) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 35: allow to write to mirrored files == 08:10:51 (1715343051) PASS 35 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36a: write to mirrored files ========== 08:10:54 (1715343054) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.116932 s, 108 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 08:10:58 (1715343058) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112895 s, 92.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00799266 s, 131 MB/s PASS 36b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 08:11:01 (1715343061) PASS 36c (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 08:11:05 (1715343065) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000790017 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000608151 s, 1.7 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000592824 s, 1.7 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000651374 s, 1.6 MB/s flv=10 olv=10 PASS 36d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 37: mirror I/O API verification ======= 08:11:08 (1715343068) create test files with size 15744807 .. 56e2c767d9b378eed11f574e5bdc3744 - cf656a65c22a38e00ff01282b31ae08c - fd2722b46f6e0cb6607193606df1a2a0 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg119-client.virtnet /mnt/lustre (opts:) Starting client: oleg119-client.virtnet: -o user_xattr,flock oleg119-server@tcp:/lustre /mnt/lustre PASS 37 (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 38: resync ============================ 08:11:14 (1715343074) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 08:11:18 (1715343078) PASS 39 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 08:11:20 (1715343080) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0117259 s, 89.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x37:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x38:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3a:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107482 s, 97.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 41: lfs mirror resync check =========== 08:11:24 (1715343084) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0172973 s, 121 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0732333 s, 57.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0288859 s, 72.6 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 42: lfs mirror verify ================= 08:11:27 (1715343087) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43a: mirror pick on write ============= 08:11:29 (1715343089) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 08:11:32 (1715343092) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00792919 s, 132 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44a: lfs mirror split check =========== 08:11:35 (1715343095) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0647985 s, 48.5 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 44a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44b: mirror split does not create stale file ========================================================== 08:11:38 (1715343098) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x42:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x46:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 08:11:42 (1715343102) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0583122 s, 180 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg119-client.virtnet /mnt/lustre (opts:) Starting client: oleg119-client.virtnet: -o user_xattr,flock oleg119-server@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 08:11:45 (1715343105) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800a27ee000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0816396 s, 128 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg119-client.virtnet /mnt/lustre (opts:) Starting client: oleg119-client.virtnet: -o user_xattr,flock oleg119-server@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff88012a58e000.xattr_cache=1 PASS 44e (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 08:11:50 (1715343110) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (6s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 46: Verify setstripe --copy option ==== 08:11:57 (1715343117) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 47: Verify mirror obj alloc =========== 08:12:04 (1715343124) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 48: Verify snapshot mirror ============ 08:12:06 (1715343126) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0590681 s, 53.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0444003 s, 70.8 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 03023af53e39c8b1c92ca24e1dba5dcb - mirror 1 checksum: 13033eb213696b1a1470607921613df0 - mirror 2 checksum: 03023af53e39c8b1c92ca24e1dba5dcb - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri May 10 08:12:06 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri May 10 08:12:06 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x53:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 03023af53e39c8b1c92ca24e1dba5dcb - mirror 1 checksum: 13033eb213696b1a1470607921613df0 - mirror 2 checksum: 13033eb213696b1a1470607921613df0 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri May 10 08:12:06 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri May 10 08:12:06 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x53:0x0] } PASS 48 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 49a: FIEMAP upon FLR file ============= 08:12:09 (1715343129) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00771833 s, 136 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 525312.. 526335: 1024: 0001: net,eof 1: 0.. 1023: 531456.. 532479: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50A: mirror split update layout generation ========================================================== 08:12:12 (1715343132) Failing mds1 on oleg119-server Stopping /mnt/lustre-mds1 (opts:) on oleg119-server 08:12:19 (1715343139) shut down Failover mds1 to oleg119-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0000 08:12:32 (1715343152) targets are mounted 08:12:32 (1715343152) facet_failover done oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113688 s, 92.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } PASS 50A (27s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 08:12:41 (1715343161) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0074684 s, 2.7 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00381433 s, 5.4 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50b: mirror rsync handles sparseness == 08:12:44 (1715343164) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00789266 s, 2.6 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00253006 s, 8.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 08:12:48 (1715343168) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00191077 s, 8.6 MB/s PASS 50c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50d: mirror rsync keep holes ========== 08:12:51 (1715343171) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00766802 s, 2.7 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 08:12:54 (1715343174) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00321172 s, 1.3 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0138575 s, 296 kB/s PASS 60a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 08:12:59 (1715343179) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 60b (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 08:13:05 (1715343185) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 08:13:26 (1715343206) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 08:13:45 (1715343225) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 08:14:05 (1715343245) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70: mirror create and split race ====== 08:14:10 (1715343250) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 30519 mirror split pid 30520 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4716: 30519 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4716: 30520 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70a: flr mode fsx test ================ 08:15:14 (1715343314) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 3314 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fallocating to largest ever: 0x11803 000001 1715343314.530588 fallocate 0x0102ce thru 0x011802 (0x1535 bytes) 000003 1715343314.533925 read 0x00d776 thru 0x0108fe (0x3189 bytes) 000006 1715343314.536846 read 0x00eb59 thru 0x011802 (0x2caa bytes) 000007 1715343314.539164 mapwrite 0x001cd7 thru 0x00844c (0x6776 bytes) 000009 1715343314.546190 punch from 0x00bb50 to 0x011802 (0x5cb3 bytes) 000010 1715343314.547487 punch from 0x00e290 to 0x011802 (0x3573 bytes) fallocating to largest ever: 0x139f8 000012 1715343314.548974 fallocate 0x00eac6 thru 0x0139f7 (0x4f32 bytes) truncating to largest ever: 0x3ee63 000014 1715343314.550579 trunc from 0x011803 to 0x03ee62 (0x2d660 bytes) 000015 1715343314.554649 fallocate 0x0233ac thru 0x03212e (0xed83 bytes) 000018 1715343314.556455 trunc from 0x03ee63 to 0x01ab79 (0x242e9 bytes) 000020 1715343314.559943 mapread 0x01a508 thru 0x01ab79 (0x672 bytes) 000021 1715343314.562540 read 0x0038f5 thru 0x0044ca (0xbd6 bytes) 000022 1715343314.562797 trunc from 0x01ab7a to 0x000790 (0x1a3e9 bytes) 000023 1715343314.566326 mapread 0x0005d7 thru 0x000790 (0x1ba bytes) 000025 1715343314.567307 read 0x000181 thru 0x000790 (0x610 bytes) 000026 1715343314.567731 fallocate 0x00ac68 thru 0x014963 (0x9cfc bytes) 000027 1715343314.569318 mapwrite 0x01a4fd thru 0x0247a6 (0xa2aa bytes) 000029 1715343314.582347 read 0x022ec2 thru 0x0247a6 (0x18e5 bytes) 000031 1715343314.583443 read 0x004065 thru 0x004cd2 (0xc6e bytes) 000032 1715343314.583789 mapwrite 0x01dde3 thru 0x02b8be (0xdadc bytes) 000033 1715343314.595547 trunc from 0x02b8bf to 0x03c14b (0x1088d bytes) 000035 1715343314.599892 read 0x006d3e thru 0x015a24 (0xece7 bytes) 000036 1715343314.601580 MIRROR_EXTEND to 2 mirrors 000037 1715343314.705337 read 0x01e553 thru 0x02599e (0x744c bytes) 000038 1715343314.709298 fallocate 0x017eed thru 0x0277d2 (0xf8e6 bytes) 000039 1715343314.718434 mapread 0x004146 thru 0x00f4b9 (0xb374 bytes) 000040 1715343314.725131 mapwrite 0x009dc4 thru 0x011df9 (0x8036 bytes) 000041 1715343314.730866 trunc from 0x03c14c to 0x024075 (0x180d6 bytes) 000042 1715343314.739989 read 0x015a7a thru 0x01c27d (0x6804 bytes) 000043 1715343314.742296 punch from 0x01970a to 0x024075 (0xa96c bytes) 000044 1715343314.743960 trunc from 0x024076 to 0x005bdb (0x1e49a bytes) 000045 1715343314.747724 fallocate 0x01d185 thru 0x0277f5 (0xa671 bytes) 000046 1715343314.749193 trunc from 0x0277f6 to 0x01fddc (0x7a19 bytes) 000047 1715343314.752957 mapwrite 0x03e153 thru 0x03ffff (0x1ead bytes) 000048 1715343314.762472 read 0x00927f thru 0x00b47a (0x21fc bytes) 000049 1715343314.765243 read 0x021875 thru 0x029d3c (0x84c8 bytes) 000050 1715343314.765549 fallocate 0x0085b1 thru 0x00faf2 (0x7542 bytes) 000051 1715343314.767020 mapread 0x024c90 thru 0x0295bb (0x492c bytes) 000052 1715343314.768632 MIRROR_EXTEND to 3 mirrors 000053 1715343314.863604 mapread 0x02b8a0 thru 0x0338de (0x803f bytes) 000054 1715343314.869370 write 0x0064d6 thru 0x0068fe (0x429 bytes) 000055 1715343314.881601 trunc from 0x040000 to 0x000e05 (0x3f1fa bytes) 000056 1715343314.895698 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000060 1715343314.978205 trunc from 0x000e06 to 0x034e96 (0x34091 bytes) 000061 1715343314.987740 mapwrite 0x038eb2 thru 0x03c7aa (0x38f9 bytes) 000062 1715343315.013881 fallocate 0x015c14 thru 0x017aa3 (0x1e90 bytes) 000063 1715343315.016808 trunc from 0x03c7ab to 0x005a64 (0x36d46 bytes) 000068 1715343315.028112 MIRROR_EXTEND to 3 mirrors 000069 1715343315.219258 mapwrite 0x02641b thru 0x02f5d5 (0x91bb bytes) 000070 1715343315.259772 write 0x01b044 thru 0x02142b (0x63e8 bytes) 000071 1715343315.264648 mapwrite 0x0071eb thru 0x012fb5 (0xbdcb bytes) 000072 1715343315.281372 mapread 0x00a49d thru 0x01830e (0xde72 bytes) 000073 1715343315.282331 mapwrite 0x0353e4 thru 0x0384f2 (0x310f bytes) 000074 1715343315.304440 write 0x028a07 thru 0x0308d9 (0x7ed3 bytes) 000075 1715343315.306320 mapread 0x018b87 thru 0x01e9b5 (0x5e2f bytes) 000076 1715343315.309852 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000078 1715343315.397973 mapwrite 0x001960 thru 0x004e54 (0x34f5 bytes) 000079 1715343315.414029 mapread 0x03140e thru 0x036c31 (0x5824 bytes) 000080 1715343315.417875 fallocate 0x027c78 thru 0x030dd0 (0x9159 bytes) 000081 1715343315.420211 mapread 0x00ddad thru 0x00e912 (0xb66 bytes) 000082 1715343315.420952 mapread 0x013852 thru 0x02175e (0xdf0d bytes) 000083 1715343315.430595 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000084 1715343315.479266 punch from 0x0112ae to 0x0169ce (0x5721 bytes) 000085 1715343315.482406 read 0x0104e9 thru 0x01d823 (0xd33b bytes) 000086 1715343315.486868 write 0x00dbab thru 0x015624 (0x7a7a bytes) 000087 1715343315.492303 read 0x0188c5 thru 0x028518 (0xfc54 bytes) 000088 1715343315.497347 mapread 0x02964e thru 0x02c83f (0x31f2 bytes) 000089 1715343315.498775 write 0x0273f0 thru 0x027663 (0x274 bytes) 000090 1715343315.500026 mapwrite 0x02b3d4 thru 0x02ba94 (0x6c1 bytes) 000091 1715343315.508383 trunc from 0x0384f3 to 0x02e18f (0xa363 bytes) 000092 1715343315.516560 fallocate 0x0178e3 thru 0x01ed0c (0x742a bytes) 000093 1715343315.519763 mapread 0x024dfc thru 0x0290a9 (0x42ae bytes) fallocating to largest ever: 0x40000 000094 1715343315.520687 fallocate 0x03c7d9 thru 0x03ffff (0x3827 bytes) 000095 1715343315.523483 write 0x00cc85 thru 0x0188a0 (0xbc1c bytes) 000096 1715343315.524837 mapwrite 0x02f24b thru 0x03a826 (0xb5dc bytes) 000099 1715343315.561661 trunc from 0x040000 to 0x0362ac (0x9d53 bytes) 000100 1715343315.569100 mapwrite 0x02cf3c thru 0x0398cb (0xc990 bytes) 000101 1715343315.594680 read 0x00d1ac thru 0x015ed0 (0x8d25 bytes) 000102 1715343315.595270 write 0x00d0ad thru 0x01977e (0xc6d2 bytes) 000103 1715343315.596490 mapread 0x01053c thru 0x0127a0 (0x2265 bytes) 000104 1715343315.597265 trunc from 0x0398cc to 0x02f9d1 (0x9efa bytes) 000105 1715343315.604452 write 0x03fd38 thru 0x03ffff (0x2c8 bytes) 000106 1715343315.605273 fallocate 0x008dc3 thru 0x00e5d9 (0x5817 bytes) 000107 1715343315.608394 trunc from 0x040000 to 0x02475e (0x1b8a1 bytes) 000108 1715343315.615924 fallocate 0x006a98 thru 0x0159bd (0xef26 bytes) 000109 1715343315.619162 trunc from 0x02475f to 0x00d0f0 (0x1766e bytes) 000110 1715343315.626586 write 0x006ed4 thru 0x00a488 (0x35b5 bytes) 000111 1715343315.627544 mapwrite 0x023104 thru 0x025e43 (0x2d40 bytes) 000112 1715343315.649092 trunc from 0x025e44 to 0x037621 (0x117de bytes) 000113 1715343315.656804 punch from 0x022cab to 0x026724 (0x3a7a bytes) 000119 1715343315.662803 read 0x00a1d8 thru 0x017973 (0xd79c bytes) 000120 1715343315.663408 fallocate 0x03cf28 thru 0x03ffff (0x30d8 bytes) 000121 1715343315.666661 mapread 0x03f915 thru 0x03ffff (0x6eb bytes) 000122 1715343315.672298 write 0x026bca thru 0x027dee (0x1225 bytes) 000123 1715343315.673135 fallocate 0x028914 thru 0x030a19 (0x8106 bytes) 000124 1715343315.675792 write 0x006aad thru 0x006ccf (0x223 bytes) 000125 1715343315.676537 fallocate 0x036751 thru 0x03ffff (0x98af bytes) 000127 1715343315.679996 mapwrite 0x000205 thru 0x00ae1d (0xac19 bytes) 000129 1715343315.693678 mapwrite 0x03cc8a thru 0x03ffff (0x3376 bytes) 000130 1715343315.700921 read 0x01a6eb thru 0x021dc8 (0x76de bytes) 000131 1715343315.701621 mapwrite 0x03114d thru 0x0322d9 (0x118d bytes) 000135 1715343315.709989 punch from 0x01f63b to 0x02a772 (0xb138 bytes) 000137 1715343315.713958 punch from 0x010829 to 0x01a7bb (0x9f93 bytes) 000138 1715343315.716554 mapread 0x028f85 thru 0x029a46 (0xac2 bytes) 000139 1715343315.721190 trunc from 0x040000 to 0x031465 (0xeb9a bytes) 000140 1715343315.728612 fallocate 0x017340 thru 0x02471e (0xd3df bytes) 000141 1715343315.731782 read 0x01f15d thru 0x028bf8 (0x9a9c bytes) 000142 1715343315.732767 mapwrite 0x0071e9 thru 0x009b43 (0x295b bytes) 000143 1715343315.742202 trunc from 0x031466 to 0x0288d9 (0x8b8c bytes) 000144 1715343315.751092 MIRROR_EXTEND to 2 mirrors 000145 1715343315.918649 trunc from 0x0288da to 0x02b432 (0x2b59 bytes) 000147 1715343315.942376 mapread 0x01892c thru 0x027d31 (0xf406 bytes) 000149 1715343315.953027 trunc from 0x02b433 to 0x039b88 (0xe756 bytes) 000150 1715343315.960865 write 0x03014e thru 0x0323a0 (0x2253 bytes) 000152 1715343315.965163 trunc from 0x039b89 to 0x031703 (0x8485 bytes) 000153 1715343315.971083 write 0x02ab23 thru 0x02d31c (0x27fa bytes) 000154 1715343315.973665 MIRROR_EXTEND to 3 mirrors 000155 1715343316.157493 read 0x02fcc2 thru 0x031703 (0x1a42 bytes) 000157 1715343316.160621 mapread 0x0087ef thru 0x00928f (0xaa1 bytes) 000158 1715343316.167743 MIRROR_EXTEND to 4 mirrors 000160 1715343316.399467 read 0x0232e7 thru 0x02c167 (0x8e81 bytes) 000162 1715343316.413361 MIRROR_EXTEND to 5 mirrors 000163 1715343316.611230 trunc from 0x031704 to 0x022105 (0xf5fe bytes) 000165 1715343316.631088 read 0x01c459 thru 0x022105 (0x5cad bytes) 000166 1715343316.634692 fallocate 0x01e5c2 thru 0x02a5bf (0xbffe bytes) 000168 1715343316.637102 fallocate 0x026e89 thru 0x0325f5 (0xb76d bytes) 000170 1715343316.639581 fallocate 0x02933a thru 0x035921 (0xc5e8 bytes) 000172 1715343316.642073 trunc from 0x022106 to 0x036d35 (0x14c30 bytes) 000173 1715343316.649523 mapread 0x02b8de thru 0x02facb (0x41ee bytes) 000174 1715343316.658594 MIRROR_RESYNC 5 mirrors 000175 1715343316.951961 mapwrite 0x02712a thru 0x030dd7 (0x9cae bytes) 000176 1715343317.000391 read 0x0024e1 thru 0x002ce1 (0x801 bytes) 000177 1715343317.003411 mapwrite 0x0112c4 thru 0x013965 (0x26a2 bytes) 000178 1715343317.014649 write 0x005b98 thru 0x0094e3 (0x394c bytes) 000179 1715343317.021304 MIRROR_RESYNC 5 mirrors 000180 1715343317.393547 mapread 0x006a39 thru 0x00d033 (0x65fb bytes) 000181 1715343317.402159 write 0x0347ec thru 0x0392d4 (0x4ae9 bytes) 000182 1715343317.420624 trunc from 0x0392d5 to 0x029909 (0xf9cb bytes) 000183 1715343317.437846 mapread 0x00947f thru 0x00a136 (0xcb8 bytes) 000184 1715343317.442182 fallocate 0x0135bb thru 0x02284f (0xf295 bytes) 000185 1715343317.445297 punch from 0x013dd9 to 0x01a61d (0x6845 bytes) 000186 1715343317.447839 punch from 0x0116a9 to 0x014f6f (0x38c7 bytes) 000187 1715343317.450622 read 0x01ae10 thru 0x021ea7 (0x7098 bytes) 000188 1715343317.454732 trunc from 0x02990a to 0x0369b1 (0xd0a8 bytes) 000189 1715343317.462748 mapread 0x00dcd7 thru 0x01bce5 (0xe00f bytes) 000191 1715343317.470990 read 0x010a88 thru 0x01cccd (0xc246 bytes) 000192 1715343317.471590 mapwrite 0x016cc1 thru 0x0258d1 (0xec11 bytes) 000193 1715343317.491170 read 0x034fe1 thru 0x0369b1 (0x19d1 bytes) 000194 1715343317.491829 fallocate 0x0070c8 thru 0x00f370 (0x82a9 bytes) 000195 1715343317.497636 MIRROR_EXTEND to 6 mirrors 000197 1715343317.726195 punch from 0x00bcc1 to 0x00fbca (0x3f0a bytes) 000198 1715343317.743150 write 0x035f39 thru 0x03ffff (0xa0c7 bytes) 000199 1715343317.748760 punch from 0x01e3e4 to 0x02dc05 (0xf822 bytes) 000200 1715343317.751636 mapread 0x024757 thru 0x02c650 (0x7efa bytes) 000202 1715343317.762136 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000205 1715343317.846271 punch from 0x027b06 to 0x02dbed (0x60e8 bytes) 000206 1715343317.850152 MIRROR_RESYNC 5 mirrors 000209 1715343318.307076 MIRROR_EXTEND to 6 mirrors 000211 1715343318.568688 punch from 0x038658 to 0x03ffff (0x79a8 bytes) 000212 1715343318.586241 fallocate 0x0024c2 thru 0x009a95 (0x75d4 bytes) 000213 1715343318.589409 read 0x029850 thru 0x034e79 (0xb62a bytes) 000214 1715343318.593834 mapwrite 0x014292 thru 0x01a185 (0x5ef4 bytes) 000216 1715343318.614703 mapread 0x009562 thru 0x00b7a9 (0x2248 bytes) 000217 1715343318.619084 fallocate 0x000d30 thru 0x00ff9a (0xf26b bytes) 000218 1715343318.622031 punch from 0x005794 to 0x00ac4c (0x54b9 bytes) 000219 1715343318.627174 MIRROR_EXTEND to 7 mirrors 000220 1715343318.869263 fallocate 0x01d4e0 thru 0x028b86 (0xb6a7 bytes) 000221 1715343318.886420 read 0x01854e thru 0x01ecb0 (0x6763 bytes) 000222 1715343318.890901 trunc from 0x040000 to 0x03c9a3 (0x365c bytes) 000223 1715343318.909974 fallocate 0x023343 thru 0x0319fd (0xe6bb bytes) 000225 1715343318.913363 fallocate 0x007067 thru 0x012519 (0xb4b3 bytes) 000226 1715343318.919611 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000228 1715343319.003613 fallocate 0x01d75e thru 0x023515 (0x5db8 bytes) 000229 1715343319.006558 punch from 0x0116cf to 0x0183ef (0x6d21 bytes) 000230 1715343319.009604 fallocate 0x01bad8 thru 0x02909f (0xd5c8 bytes) 000231 1715343319.015611 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000232 1715343319.095772 fallocate 0x038419 thru 0x03c1be (0x3da6 bytes) 000233 1715343319.098679 mapwrite 0x02b1c6 thru 0x030763 (0x559e bytes) 000234 1715343319.117907 mapwrite 0x00e98c thru 0x01baa4 (0xd119 bytes) 000235 1715343319.135821 mapread 0x0035b0 thru 0x0075e9 (0x403a bytes) 000236 1715343319.139829 trunc from 0x03c9a4 to 0x02224e (0x1a755 bytes) 000237 1715343319.150111 trunc from 0x02224f to 0x00a11b (0x18133 bytes) 000238 1715343319.157579 trunc from 0x00a11c to 0x00333f (0x6ddc bytes) 000242 1715343319.166346 mapwrite 0x014b32 thru 0x01a255 (0x5724 bytes) 000243 1715343319.187730 trunc from 0x01a256 to 0x01140a (0x8e4b bytes) 000245 1715343319.195589 mapread 0x000a19 thru 0x00aad1 (0xa0b9 bytes) 000246 1715343319.202211 mapread 0x001358 thru 0x003e4d (0x2af6 bytes) 000247 1715343319.205176 MIRROR_EXTEND to 6 mirrors 000248 1715343319.406826 fallocate 0x03e07d thru 0x03ffff (0x1f83 bytes) 000249 1715343319.421404 punch from 0x0262bd to 0x02c2dd (0x6021 bytes) 000250 1715343319.423391 punch from 0x016755 to 0x025ce5 (0xf591 bytes) 000251 1715343319.425332 fallocate 0x02445e thru 0x02b14b (0x6cee bytes) 000252 1715343319.427751 punch from 0x00365f to 0x00e160 (0xab02 bytes) 000253 1715343319.430146 write 0x02d4cc thru 0x02f45e (0x1f93 bytes) 000254 1715343319.439150 MIRROR_RESYNC 6 mirrors 000255 1715343319.761561 MIRROR_EXTEND to 7 mirrors 000256 1715343320.005379 MIRROR_EXTEND to 8 mirrors 000257 1715343320.202709 read 0x036350 thru 0x03e3e3 (0x8094 bytes) 000258 1715343320.206389 mapwrite 0x01069f thru 0x01ef96 (0xe8f8 bytes) 000259 1715343320.252602 fallocate 0x01495b thru 0x018273 (0x3919 bytes) 000260 1715343320.254712 read 0x01b08b thru 0x023db1 (0x8d27 bytes) 000261 1715343320.259334 trunc from 0x040000 to 0x0198e7 (0x26718 bytes) 000262 1715343320.271488 read 0x00caed thru 0x0198e7 (0xcdfb bytes) 000263 1715343320.272147 mapread 0x0125b3 thru 0x0198e7 (0x7335 bytes) 000264 1715343320.272999 punch from 0x00db38 to 0x00facf (0x1f98 bytes) 000265 1715343320.275690 trunc from 0x0198e8 to 0x01a931 (0x104a bytes) 000266 1715343320.282906 trunc from 0x01a932 to 0x00c8cb (0xe066 bytes) 000267 1715343320.290567 trunc from 0x00c8cc to 0x01a529 (0xdc5e bytes) 000268 1715343320.297796 punch from 0x00c267 to 0x01a3e1 (0xe17b bytes) 000269 1715343320.302748 MIRROR_EXTEND to 9 mirrors 000270 1715343320.488752 mapwrite 0x014458 thru 0x01ea0c (0xa5b5 bytes) 000271 1715343320.526101 fallocate 0x0069b7 thru 0x00ff38 (0x9582 bytes) 000272 1715343320.528119 mapwrite 0x008036 thru 0x00b90e (0x38d9 bytes) 000273 1715343320.536528 mapread 0x00449e thru 0x01323a (0xed9d bytes) 000274 1715343320.537114 fallocate 0x03ad30 thru 0x03ffff (0x52d0 bytes) 000275 1715343320.539038 mapwrite 0x028ecf thru 0x035cc0 (0xcdf2 bytes) 000276 1715343320.562533 mapwrite 0x008880 thru 0x00d27e (0x49ff bytes) 000277 1715343320.569482 punch from 0x03530f to 0x035cc0 (0x9b2 bytes) 000279 1715343320.571519 write 0x003b34 thru 0x00e8c9 (0xad96 bytes) 000280 1715343320.572273 fallocate 0x036f79 thru 0x03ffff (0x9087 bytes) 000281 1715343320.574304 mapread 0x0192b1 thru 0x020bcd (0x791d bytes) 000282 1715343320.578379 MIRROR_EXTEND to 10 mirrors 000283 1715343320.726370 trunc from 0x035cc1 to 0x01abc6 (0x1b0fa bytes) 000284 1715343320.749130 mapwrite 0x022bd1 thru 0x02ffa8 (0xd3d8 bytes) 000286 1715343320.778895 mapread 0x015ab1 thru 0x018e5f (0x33af bytes) 000287 1715343320.783419 fallocate 0x00952c thru 0x016f3a (0xda0f bytes) 000288 1715343320.786041 read 0x016095 thru 0x01e6ea (0x8656 bytes) 000289 1715343320.789318 MIRROR_RESYNC 10 mirrors 000290 1715343321.294704 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000293 1715343321.432236 mapwrite 0x03cc0c thru 0x03def7 (0x12ec bytes) 000294 1715343321.469855 punch from 0x02dfc1 to 0x03035b (0x239b bytes) 000296 1715343321.475348 MIRROR_EXTEND to 10 mirrors 000297 1715343321.673055 punch from 0x007a29 to 0x009101 (0x16d9 bytes) 000299 1715343321.689836 trunc from 0x03def8 to 0x03cfef (0xf08 bytes) 000300 1715343321.708783 punch from 0x029ee4 to 0x0308cb (0x69e8 bytes) 000301 1715343321.711970 punch from 0x0301ca to 0x03cfef (0xce26 bytes) 000302 1715343321.715031 mapwrite 0x000981 thru 0x004e57 (0x44d7 bytes) 000303 1715343321.736894 mapwrite 0x033612 thru 0x037e30 (0x481f bytes) 000304 1715343321.749311 punch from 0x038ab0 to 0x03c852 (0x3da3 bytes) 000306 1715343321.752533 read 0x0275b9 thru 0x02e10f (0x6b57 bytes) 000307 1715343321.756758 trunc from 0x03cff0 to 0x017da3 (0x2524c bytes) 000308 1715343321.764615 mapwrite 0x027593 thru 0x02b005 (0x3a73 bytes) 000309 1715343321.785796 mapwrite 0x004b1f thru 0x008080 (0x3562 bytes) 000310 1715343321.795121 fallocate 0x036d84 thru 0x03a293 (0x3510 bytes) 000311 1715343321.797895 write 0x00d56e thru 0x01397a (0x640d bytes) 000313 1715343321.801338 write 0x01d060 thru 0x023102 (0x60a3 bytes) 000314 1715343321.806036 punch from 0x02b67c to 0x03798a (0xc30f bytes) 000315 1715343321.809166 punch from 0x0145c8 to 0x023699 (0xf0d2 bytes) 000316 1715343321.816165 trunc from 0x03a294 to 0x032836 (0x7a5d bytes) 000318 1715343321.823711 read 0x031467 thru 0x032836 (0x13d0 bytes) 000319 1715343321.828964 punch from 0x01aab2 to 0x0244c5 (0x9a14 bytes) 000320 1715343321.831757 write 0x0065ef thru 0x0107bd (0xa1cf bytes) 000321 1715343321.832918 punch from 0x008a00 to 0x0168d1 (0xded2 bytes) 000322 1715343321.839734 mapwrite 0x0297c9 thru 0x038b42 (0xf37a bytes) 000323 1715343321.866061 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000326 1715343321.953063 mapread 0x005e65 thru 0x01017f (0xa31b bytes) 000329 1715343321.965153 punch from 0x022227 to 0x02b0dd (0x8eb7 bytes) 000330 1715343321.967892 mapread 0x01703d thru 0x0256fc (0xe6c0 bytes) 000331 1715343321.975873 punch from 0x00796e to 0x00e765 (0x6df8 bytes) 000332 1715343321.978960 write 0x0349dc thru 0x035d8e (0x13b3 bytes) 000333 1715343321.985775 write 0x02995b thru 0x02adc2 (0x1468 bytes) 000334 1715343321.986722 punch from 0x00d1c7 to 0x01b84a (0xe684 bytes) 000335 1715343321.989374 punch from 0x028608 to 0x03319a (0xab93 bytes) 000336 1715343321.995522 mapread 0x01bc33 thru 0x01ef8a (0x3358 bytes) 000337 1715343321.999856 fallocate 0x037a74 thru 0x03c6bd (0x4c4a bytes) 000338 1715343322.002973 read 0x0193dd thru 0x023513 (0xa137 bytes) 000339 1715343322.003892 read 0x034c84 thru 0x0363f4 (0x1771 bytes) 000340 1715343322.004741 punch from 0x01876e to 0x01903f (0x8d2 bytes) 000341 1715343322.007604 trunc from 0x038b43 to 0x032db7 (0x5d8b bytes) 000342 1715343322.015283 write 0x021293 thru 0x022e55 (0x1bc3 bytes) 000343 1715343322.019279 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000344 1715343322.104407 punch from 0x02c057 to 0x032db7 (0x6d61 bytes) 000345 1715343322.109129 MIRROR_EXTEND to 9 mirrors 000346 1715343322.304366 punch from 0x0025a6 to 0x00f798 (0xd1f3 bytes) 000347 1715343322.320816 trunc from 0x032db8 to 0x02307d (0xfd3a bytes) 000348 1715343322.333995 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000349 1715343322.420312 mapwrite 0x01fab7 thru 0x02abbf (0xb109 bytes) 000350 1715343322.452896 trunc from 0x02abc0 to 0x03ebb6 (0x13ff7 bytes) 000351 1715343322.460055 mapread 0x030dff thru 0x03ebb6 (0xddb8 bytes) 000352 1715343322.468783 mapwrite 0x00f19c thru 0x01b8de (0xc743 bytes) 000353 1715343322.488062 punch from 0x03c63d to 0x03d11d (0xae1 bytes) 000354 1715343322.490819 read 0x033eb0 thru 0x03c23e (0x838f bytes) 000355 1715343322.493717 mapread 0x0203df thru 0x02eef4 (0xeb16 bytes) 000356 1715343322.497336 mapread 0x03556e thru 0x038702 (0x3195 bytes) 000357 1715343322.498209 write 0x00eb25 thru 0x012312 (0x37ee bytes) 000358 1715343322.499309 trunc from 0x03ebb7 to 0x011cae (0x2cf08 bytes) 000359 1715343322.506974 write 0x0005ae thru 0x003d50 (0x37a3 bytes) 000360 1715343322.511591 punch from 0x001613 to 0x004f8b (0x3979 bytes) 000361 1715343322.519810 trunc from 0x011caf to 0x0221b2 (0x10504 bytes) 000362 1715343322.527083 fallocate 0x038b56 thru 0x03e45e (0x5909 bytes) 000364 1715343322.530581 punch from 0x01e998 to 0x0221b2 (0x381b bytes) 000365 1715343322.533373 fallocate 0x03c9b0 thru 0x03ccb4 (0x305 bytes) 000366 1715343322.536135 read 0x01c8cc thru 0x0221b2 (0x58e7 bytes) 000367 1715343322.541128 mapwrite 0x03a164 thru 0x03ffff (0x5e9c bytes) 000368 1715343322.564315 mapread 0x028425 thru 0x030094 (0x7c70 bytes) 000369 1715343322.565202 fallocate 0x014269 thru 0x01e388 (0xa120 bytes) 000370 1715343322.568090 fallocate 0x022999 thru 0x026232 (0x389a bytes) 000372 1715343322.571588 mapread 0x00f453 thru 0x01141e (0x1fcc bytes) 000373 1715343322.572561 read 0x0030be thru 0x00fcc7 (0xcc0a bytes) 000374 1715343322.575483 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000375 1715343322.663017 punch from 0x00cb93 to 0x0158ce (0x8d3c bytes) 000376 1715343322.668304 MIRROR_RESYNC 7 mirrors 000378 1715343322.994624 mapwrite 0x02c713 thru 0x02ff66 (0x3854 bytes) 000379 1715343323.028398 mapwrite 0x031f2b thru 0x0334c1 (0x1597 bytes) 000380 1715343323.033703 fallocate 0x023c2c thru 0x03361e (0xf9f3 bytes) 000381 1715343323.035599 punch from 0x00f799 to 0x012c74 (0x34dc bytes) 000382 1715343323.037780 trunc from 0x040000 to 0x00f15d (0x30ea2 bytes) 000384 1715343323.054587 read 0x00ad53 thru 0x00f15d (0x440b bytes) 000386 1715343323.060226 MIRROR_RESYNC 7 mirrors 000388 1715343323.355323 trunc from 0x00f15e to 0x02a192 (0x1b035 bytes) 000389 1715343323.380703 trunc from 0x02a193 to 0x035ad0 (0xb93e bytes) 000390 1715343323.389993 mapwrite 0x027203 thru 0x035e7f (0xec7d bytes) 000391 1715343323.422403 write 0x028009 thru 0x02cd0e (0x4d06 bytes) 000392 1715343323.423559 write 0x017330 thru 0x024aea (0xd7bb bytes) 000393 1715343323.428900 mapread 0x010f72 thru 0x01164b (0x6da bytes) 000394 1715343323.429666 punch from 0x035a4d to 0x035e7f (0x433 bytes) 000395 1715343323.432165 read 0x029bb7 thru 0x035e7f (0xc2c9 bytes) 000396 1715343323.435031 mapwrite 0x012a9d thru 0x015873 (0x2dd7 bytes) 000397 1715343323.443096 mapwrite 0x00b760 thru 0x01318f (0x7a30 bytes) 000398 1715343323.455281 trunc from 0x035e80 to 0x0132ad (0x22bd2 bytes) 000400 1715343323.463360 write 0x018713 thru 0x020270 (0x7b5e bytes) 000403 1715343323.465468 read 0x0047d7 thru 0x006986 (0x21b0 bytes) 000404 1715343323.466100 fallocate 0x015752 thru 0x0170ac (0x195b bytes) 000405 1715343323.468966 punch from 0x0014de to 0x00f951 (0xe474 bytes) 000406 1715343323.472009 write 0x017e45 thru 0x027be2 (0xfd9e bytes) 000407 1715343323.476971 write 0x0093e4 thru 0x013b54 (0xa771 bytes) 000408 1715343323.478122 punch from 0x017165 to 0x022353 (0xb1ef bytes) 000409 1715343323.484946 read 0x01507d thru 0x01d9d3 (0x8957 bytes) 000410 1715343323.489267 write 0x035e38 thru 0x03f1fa (0x93c3 bytes) 000411 1715343323.490842 fallocate 0x02de94 thru 0x02efa3 (0x1110 bytes) 000413 1715343323.494042 punch from 0x032c13 to 0x03d4e3 (0xa8d1 bytes) 000414 1715343323.501102 write 0x023e1b thru 0x032dac (0xef92 bytes) 000415 1715343323.507341 MIRROR_RESYNC 7 mirrors 000416 1715343323.907805 MIRROR_RESYNC 7 mirrors 000417 1715343323.984938 trunc from 0x03f1fb to 0x03d3ed (0x1e0d bytes) 000418 1715343324.003470 punch from 0x01a81b to 0x025bcc (0xb3b2 bytes) 000419 1715343324.005104 write 0x03afd0 thru 0x03d099 (0x20ca bytes) 000421 1715343324.009480 mapread 0x028e31 thru 0x02ae90 (0x2060 bytes) 000422 1715343324.012980 trunc from 0x03d3ee to 0x000e5a (0x3c593 bytes) 000423 1715343324.021106 punch from 0x000132 to 0x000e5a (0xd29 bytes) 000424 1715343324.023614 fallocate 0x0240e7 thru 0x024b93 (0xaad bytes) 000426 1715343324.029195 MIRROR_EXTEND to 8 mirrors 000427 1715343324.181440 read 0x000b64 thru 0x000e5a (0x2f7 bytes) 000428 1715343324.185253 mapread 0x00051f thru 0x000e5a (0x93c bytes) 000429 1715343324.192330 punch from 0x000bd1 to 0x000e5a (0x28a bytes) 000430 1715343324.210392 mapwrite 0x024945 thru 0x033db1 (0xf46d bytes) 000431 1715343324.245459 write 0x00f667 thru 0x0179f6 (0x8390 bytes) 000432 1715343324.250375 write 0x006c15 thru 0x00a3d1 (0x37bd bytes) 000435 1715343324.255330 fallocate 0x0009a8 thru 0x001363 (0x9bc bytes) 000438 1715343324.259103 fallocate 0x013268 thru 0x019493 (0x622c bytes) 000439 1715343324.261874 punch from 0x00e6ec to 0x017537 (0x8e4c bytes) 000440 1715343324.269288 mapread 0x00f94e thru 0x017ec3 (0x8576 bytes) 000442 1715343324.277149 fallocate 0x03f76a thru 0x03ffff (0x896 bytes) 000443 1715343324.279946 trunc from 0x033db2 to 0x0121a6 (0x21c0b bytes) 000444 1715343324.287443 mapwrite 0x015b16 thru 0x01c580 (0x6a6b bytes) 000445 1715343324.309195 mapread 0x01a9e3 thru 0x01c580 (0x1b9e bytes) 000447 1715343324.310596 write 0x0306e5 thru 0x03e5f9 (0xdf15 bytes) 000448 1715343324.312569 mapread 0x020230 thru 0x02fcea (0xfabb bytes) 000449 1715343324.321094 write 0x013a58 thru 0x022b9e (0xf147 bytes) 000450 1715343324.324942 MIRROR_RESYNC 8 mirrors 000451 1715343324.935590 MIRROR_EXTEND to 9 mirrors 000452 1715343325.197920 mapwrite 0x002ca8 thru 0x00ce6f (0xa1c8 bytes) 000453 1715343325.250154 MIRROR_EXTEND to 10 mirrors 000454 1715343325.480359 MIRROR_RESYNC 10 mirrors 000455 1715343326.035227 write 0x0322d3 thru 0x036bbe (0x48ec bytes) 000456 1715343326.054388 trunc from 0x03e5fa to 0x00ce0e (0x317eb bytes) 000457 1715343326.063897 write 0x03d2ae thru 0x03da16 (0x769 bytes) 000458 1715343326.067394 MIRROR_RESYNC 10 mirrors 000459 1715343326.503109 mapwrite 0x00b301 thru 0x0147aa (0x94aa bytes) 000460 1715343326.537473 read 0x01035a thru 0x013329 (0x2fd0 bytes) 000462 1715343326.538598 mapwrite 0x03fffb thru 0x03ffff (0x5 bytes) 000463 1715343326.551479 mapread 0x032f88 thru 0x03d077 (0xa0f0 bytes) 000464 1715343326.554485 write 0x022fcc thru 0x027b73 (0x4ba8 bytes) 000465 1715343326.557057 punch from 0x039a0f to 0x03ffff (0x65f1 bytes) 000466 1715343326.558491 fallocate 0x01c52f thru 0x02be24 (0xf8f6 bytes) 000467 1715343326.560008 read 0x0279c3 thru 0x02e411 (0x6a4f bytes) 000468 1715343326.562708 mapread 0x010760 thru 0x01568f (0x4f30 bytes) 000469 1715343326.564616 MIRROR_EXTEND to 11 mirrors 000470 1715343326.718185 mapread 0x03fce9 thru 0x03ffff (0x317 bytes) 000471 1715343326.725737 mapread 0x03824b thru 0x0395d1 (0x1387 bytes) 000472 1715343326.735156 mapwrite 0x0388f1 thru 0x03ffff (0x770f bytes) 000474 1715343326.775727 mapwrite 0x02df99 thru 0x030f80 (0x2fe8 bytes) 000475 1715343326.785817 read 0x01149d thru 0x015f75 (0x4ad9 bytes) 000476 1715343326.788949 fallocate 0x003b62 thru 0x00d747 (0x9be6 bytes) 000477 1715343326.791043 fallocate 0x01b860 thru 0x01e354 (0x2af5 bytes) 000478 1715343326.793043 mapread 0x00a66e thru 0x01520e (0xaba1 bytes) 000479 1715343326.796320 trunc from 0x040000 to 0x0182b2 (0x27d4d bytes) 000480 1715343326.810231 trunc from 0x0182b3 to 0x0075d6 (0x10cdc bytes) 000481 1715343326.815779 punch from 0x001c65 to 0x0075d6 (0x5972 bytes) 000482 1715343326.817563 punch from 0x002f07 to 0x005b61 (0x2c5b bytes) 000483 1715343326.819124 mapread 0x0005cc thru 0x0015f9 (0x102e bytes) 000484 1715343326.823622 mapread 0x000e83 thru 0x001a46 (0xbc4 bytes) 000485 1715343326.824000 mapwrite 0x0303a6 thru 0x03e20b (0xde66 bytes) 000487 1715343326.840420 mapwrite 0x001759 thru 0x00d7b2 (0xc05a bytes) 000488 1715343326.849237 write 0x02d0f4 thru 0x035e87 (0x8d94 bytes) 000489 1715343326.849833 fallocate 0x009934 thru 0x016aa3 (0xd170 bytes) 000490 1715343326.851254 mapwrite 0x0339bb thru 0x037e32 (0x4478 bytes) 000491 1715343326.855644 punch from 0x018b2d to 0x0237d4 (0xaca8 bytes) 000492 1715343326.858439 MIRROR_RESYNC 11 mirrors 000493 1715343327.190197 read 0x038add thru 0x03e20b (0x572f bytes) 000494 1715343327.192807 read 0x025fd3 thru 0x03282e (0xc85c bytes) 000495 1715343327.196525 read 0x00e596 thru 0x01d827 (0xf292 bytes) 000496 1715343327.198148 MIRROR_EXTEND to 12 mirrors 000498 1715343327.385203 mapwrite 0x01ac29 thru 0x01f806 (0x4bde bytes) 000499 1715343327.422020 read 0x01a8ff thru 0x023931 (0x9033 bytes) 000501 1715343327.423746 read 0x032da0 thru 0x03c91d (0x9b7e bytes) 000502 1715343327.429264 write 0x017a3b thru 0x022263 (0xa829 bytes) 000503 1715343327.432697 trunc from 0x03e20c to 0x03edcd (0xbc2 bytes) 000504 1715343327.441317 trunc from 0x03edce to 0x0069d3 (0x383fa bytes) 000505 1715343327.448024 write 0x0180b4 thru 0x01aa4b (0x2998 bytes) 000506 1715343327.450985 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000507 1715343327.515426 punch from 0x007072 to 0x011957 (0xa8e6 bytes) 000509 1715343327.519862 MIRROR_RESYNC 11 mirrors 000510 1715343327.889485 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000511 1715343327.986640 read 0x010292 thru 0x012c76 (0x29e5 bytes) 000512 1715343327.989241 punch from 0x0089ea to 0x009388 (0x99f bytes) 000514 1715343328.001773 punch from 0x014592 to 0x014697 (0x106 bytes) 000515 1715343328.003806 trunc from 0x01aa4c to 0x00ff81 (0xaaca bytes) 000517 1715343328.017350 write 0x0389e3 thru 0x03ffff (0x761d bytes) 000518 1715343328.019193 mapread 0x027b00 thru 0x0341eb (0xc6ec bytes) 000519 1715343328.028268 punch from 0x000b3b to 0x00d2c2 (0xc788 bytes) 000520 1715343328.032225 MIRROR_RESYNC 10 mirrors 000521 1715343328.294587 fallocate 0x005594 thru 0x00fcea (0xa757 bytes) 000522 1715343328.305448 mapwrite 0x03e0c5 thru 0x03ffff (0x1f3b bytes) 000523 1715343328.313876 punch from 0x023f4b to 0x02da53 (0x9b09 bytes) 000524 1715343328.315746 read 0x0074b6 thru 0x011f50 (0xaa9b bytes) 000525 1715343328.318670 trunc from 0x040000 to 0x00d4fb (0x32b04 bytes) 000526 1715343328.332033 mapread 0x007947 thru 0x00d4fb (0x5bb5 bytes) 000527 1715343328.332895 mapwrite 0x029823 thru 0x035712 (0xbef0 bytes) 000528 1715343328.351412 trunc from 0x035713 to 0x0017a7 (0x33f6b bytes) 000529 1715343328.356758 mapwrite 0x0099a2 thru 0x0131fd (0x985c bytes) 000531 1715343328.374092 read 0x011186 thru 0x0131fd (0x2078 bytes) 000532 1715343328.376419 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000535 1715343328.443894 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000536 1715343328.507224 fallocate 0x016019 thru 0x0235a6 (0xd58e bytes) 000537 1715343328.509990 trunc from 0x0131fe to 0x03aad4 (0x278d7 bytes) 000538 1715343328.517374 mapread 0x00a8c4 thru 0x0191a9 (0xe8e6 bytes) 000539 1715343328.527916 mapwrite 0x010bfd thru 0x01e638 (0xda3c bytes) 000540 1715343328.538052 mapwrite 0x00fa4c thru 0x015fe4 (0x6599 bytes) 000542 1715343328.546037 fallocate 0x0025f8 thru 0x009c65 (0x766e bytes) 000545 1715343328.549090 mapread 0x0100dc thru 0x01b923 (0xb848 bytes) 000546 1715343328.550102 punch from 0x00833a to 0x008da1 (0xa68 bytes) 000547 1715343328.554456 MIRROR_RESYNC 8 mirrors 000548 1715343328.938904 write 0x02755b thru 0x02a902 (0x33a8 bytes) 000549 1715343328.953901 punch from 0x02c14a to 0x02c5db (0x492 bytes) 000550 1715343328.955901 read 0x0250c3 thru 0x0262b7 (0x11f5 bytes) 000551 1715343328.958356 punch from 0x01aefe to 0x029e17 (0xef1a bytes) 000552 1715343328.962289 mapwrite 0x002874 thru 0x0050ad (0x283a bytes) 000554 1715343328.971395 trunc from 0x03aad5 to 0x0360db (0x49f9 bytes) 000555 1715343328.983736 write 0x03773c thru 0x03e7dc (0x70a1 bytes) 000556 1715343328.985006 read 0x02abeb thru 0x02fd89 (0x519f bytes) 000558 1715343328.990041 mapread 0x0349e7 thru 0x037771 (0x2d8b bytes) 000559 1715343328.990903 punch from 0x02452e to 0x033d05 (0xf7d8 bytes) 000561 1715343328.995840 MIRROR_EXTEND to 9 mirrors 000562 1715343329.175227 punch from 0x01686d to 0x01b8f6 (0x508a bytes) 000563 1715343329.187739 punch from 0x026186 to 0x0307a6 (0xa621 bytes) 000564 1715343329.190230 read 0x020967 thru 0x02aee9 (0xa583 bytes) 000565 1715343329.194033 fallocate 0x03a4a8 thru 0x03ded7 (0x3a30 bytes) 000567 1715343329.196964 punch from 0x007423 to 0x00e2e6 (0x6ec4 bytes) 000568 1715343329.199308 write 0x02181a thru 0x022511 (0xcf8 bytes) 000569 1715343329.201986 mapwrite 0x001c23 thru 0x003780 (0x1b5e bytes) 000570 1715343329.217244 punch from 0x030e58 to 0x03e7dc (0xd985 bytes) 000571 1715343329.219428 punch from 0x02fe9b to 0x03a88f (0xa9f5 bytes) 000572 1715343329.221547 mapwrite 0x03024a thru 0x03c2bf (0xc076 bytes) 000573 1715343329.237306 mapread 0x0094ae thru 0x00ecc6 (0x5819 bytes) 000574 1715343329.240464 MIRROR_RESYNC 9 mirrors 000575 1715343329.790625 punch from 0x0149ff to 0x014c0c (0x20e bytes) 000576 1715343329.800399 fallocate 0x036ef9 thru 0x03ffff (0x9107 bytes) 000578 1715343329.802514 mapwrite 0x015f54 thru 0x019485 (0x3532 bytes) 000579 1715343329.816299 trunc from 0x040000 to 0x00375f (0x3c8a0 bytes) 000581 1715343329.828396 read 0x001ebf thru 0x00375f (0x18a1 bytes) 000582 1715343329.830969 trunc from 0x003760 to 0x02cc7f (0x29520 bytes) 000584 1715343329.839357 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000585 1715343329.912161 mapread 0x01ced1 thru 0x02bb11 (0xec41 bytes) 000586 1715343329.924978 fallocate 0x035f30 thru 0x03ffff (0xa0d0 bytes) 000587 1715343329.927836 mapread 0x03192b thru 0x039795 (0x7e6b bytes) 000589 1715343329.936091 mapread 0x02a61c thru 0x030259 (0x5c3e bytes) 000590 1715343329.936886 punch from 0x015026 to 0x022f46 (0xdf21 bytes) 000591 1715343329.939744 punch from 0x0259fa to 0x02a28a (0x4891 bytes) 000592 1715343329.942254 mapread 0x03e360 thru 0x03ffff (0x1ca0 bytes) 000593 1715343329.943020 fallocate 0x03729b thru 0x03f96c (0x86d2 bytes) 000594 1715343329.945537 punch from 0x038a81 to 0x03b106 (0x2686 bytes) 000595 1715343329.948268 trunc from 0x040000 to 0x001e01 (0x3e1fe bytes) 000596 1715343329.958114 trunc from 0x001e02 to 0x034dd6 (0x32fd5 bytes) 000597 1715343329.964945 read 0x01b301 thru 0x01df4d (0x2c4d bytes) 000599 1715343329.968351 mapwrite 0x02db67 thru 0x03ccb5 (0xf14f bytes) 000600 1715343329.997179 trunc from 0x03ccb6 to 0x0365b8 (0x66fd bytes) 000601 1715343330.004404 mapread 0x0052ab thru 0x011e75 (0xcbcb bytes) 000604 1715343330.013713 fallocate 0x01a2c1 thru 0x027698 (0xd3d8 bytes) 000605 1715343330.016718 mapread 0x034300 thru 0x0365b8 (0x22b9 bytes) 000606 1715343330.017431 punch from 0x007863 to 0x00f870 (0x800e bytes) 000607 1715343330.022625 MIRROR_EXTEND to 9 mirrors 000609 1715343330.233089 read 0x02ef5f thru 0x02f5ae (0x650 bytes) 000610 1715343330.242261 MIRROR_EXTEND to 10 mirrors 000611 1715343330.446240 mapwrite 0x03d6b8 thru 0x03ffff (0x2948 bytes) 000612 1715343330.483330 fallocate 0x018f67 thru 0x01b087 (0x2121 bytes) 000613 1715343330.486773 mapread 0x027d67 thru 0x02d0fa (0x5394 bytes) 000614 1715343330.491928 write 0x028c04 thru 0x0302ab (0x76a8 bytes) 000615 1715343330.493154 read 0x00ee58 thru 0x01805d (0x9206 bytes) 000616 1715343330.497132 fallocate 0x03f71b thru 0x03ffff (0x8e5 bytes) 000617 1715343330.502227 MIRROR_RESYNC 10 mirrors 000619 1715343331.028998 read 0x000102 thru 0x00136d (0x126c bytes) 000620 1715343331.034056 MIRROR_EXTEND to 11 mirrors 000622 1715343331.291490 fallocate 0x02dd74 thru 0x035621 (0x78ae bytes) 000623 1715343331.308741 mapwrite 0x016d62 thru 0x01c9af (0x5c4e bytes) 000624 1715343331.328012 punch from 0x00ba65 to 0x01525f (0x97fb bytes) 000625 1715343331.330631 read 0x036cf9 thru 0x03ffff (0x9307 bytes) 000626 1715343331.334595 mapread 0x000358 thru 0x007e67 (0x7b10 bytes) 000627 1715343331.341359 write 0x01dd36 thru 0x02c5cc (0xe897 bytes) 000628 1715343331.347589 write 0x00aaea thru 0x012723 (0x7c3a bytes) 000629 1715343331.350976 punch from 0x02cd15 to 0x02fa7e (0x2d6a bytes) 000630 1715343331.358165 read 0x034bbf thru 0x03695f (0x1da1 bytes) 000631 1715343331.362467 mapwrite 0x01bf3a thru 0x01d5a0 (0x1667 bytes) 000632 1715343331.369641 mapread 0x01cd26 thru 0x025db1 (0x908c bytes) 000633 1715343331.370418 fallocate 0x00b455 thru 0x00bc1c (0x7c8 bytes) 000634 1715343331.373316 mapread 0x037ea2 thru 0x03ffff (0x815e bytes) 000635 1715343331.374140 write 0x01748a thru 0x026087 (0xebfe bytes) 000636 1715343331.375684 read 0x02c12f thru 0x032ff2 (0x6ec4 bytes) 000637 1715343331.378826 MIRROR_EXTEND to 12 mirrors 000638 1715343331.551669 write 0x017bb3 thru 0x018847 (0xc95 bytes) 000643 1715343331.562081 mapwrite 0x0016d2 thru 0x0061e8 (0x4b17 bytes) 000644 1715343331.569285 punch from 0x035ce6 to 0x0368a4 (0xbbf bytes) 000645 1715343331.571611 MIRROR_RESYNC 12 mirrors 000646 1715343332.283855 MIRROR_SPLIT mirror 8 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000647 1715343332.440116 MIRROR_RESYNC 11 mirrors 000649 1715343332.555145 mapwrite 0x0028f0 thru 0x0071c4 (0x48d5 bytes) 000650 1715343332.601798 mapread 0x03c22e thru 0x03d212 (0xfe5 bytes) 000651 1715343332.606228 punch from 0x03580d to 0x03e4ea (0x8cde bytes) 000652 1715343332.609248 write 0x03df3a thru 0x03ece3 (0xdaa bytes) 000653 1715343332.615990 fallocate 0x02980f thru 0x03609f (0xc891 bytes) 000655 1715343332.619188 mapread 0x0347d7 thru 0x03eefb (0xa725 bytes) 000656 1715343332.625949 MIRROR_EXTEND to 12 mirrors 000657 1715343332.857145 trunc from 0x040000 to 0x010817 (0x2f7e8 bytes) 000658 1715343332.886566 trunc from 0x010818 to 0x011110 (0x8f9 bytes) 000660 1715343332.896377 write 0x000b04 thru 0x006acb (0x5fc8 bytes) 000661 1715343332.901680 trunc from 0x011111 to 0x02b732 (0x1a622 bytes) 000662 1715343332.908988 mapwrite 0x023a99 thru 0x0324ec (0xea54 bytes) 000663 1715343332.938922 read 0x00d058 thru 0x012d3f (0x5ce8 bytes) 000664 1715343332.943273 fallocate 0x03f010 thru 0x03ffff (0xff0 bytes) 000665 1715343332.946125 trunc from 0x0324ed to 0x039ec8 (0x79dc bytes) 000666 1715343332.953135 mapwrite 0x03b73a thru 0x03ffff (0x48c6 bytes) 000667 1715343332.974626 read 0x01c1c8 thru 0x025df2 (0x9c2b bytes) 000668 1715343332.979223 punch from 0x021352 to 0x02dcf0 (0xc99f bytes) 000669 1715343332.982022 write 0x02e945 thru 0x0303b4 (0x1a70 bytes) 000670 1715343332.982886 mapwrite 0x03cefe thru 0x03ffff (0x3102 bytes) 000671 1715343332.991871 mapwrite 0x010a9a thru 0x01d8af (0xce16 bytes) 000672 1715343333.002886 punch from 0x00c91e to 0x00f3b9 (0x2a9c bytes) 000673 1715343333.005444 mapwrite 0x005c5e thru 0x011c2f (0xbfd2 bytes) 000674 1715343333.025217 fallocate 0x00d892 thru 0x015f9f (0x870e bytes) 000675 1715343333.027949 fallocate 0x003e24 thru 0x00f72a (0xb907 bytes) 000676 1715343333.030548 mapread 0x01c4c2 thru 0x022f6d (0x6aac bytes) 000677 1715343333.035570 mapwrite 0x0079cf thru 0x013f5d (0xc58f bytes) 000678 1715343333.048642 MIRROR_RESYNC 12 mirrors 000679 1715343333.730155 MIRROR_SPLIT mirror 2 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000680 1715343333.879079 fallocate 0x03c13d thru 0x03e1ce (0x2092 bytes) 000681 1715343333.896209 write 0x0065a9 thru 0x0066bc (0x114 bytes) 000682 1715343333.903269 MIRROR_EXTEND to 12 mirrors 000683 1715343334.140797 fallocate 0x00d135 thru 0x01bb87 (0xea53 bytes) 000684 1715343334.158268 punch from 0x02b478 to 0x02b6b7 (0x240 bytes) 000685 1715343334.160999 read 0x00e8f2 thru 0x011a15 (0x3124 bytes) 000689 1715343334.166011 read 0x02df56 thru 0x039bd2 (0xbc7d bytes) 000690 1715343334.173351 fallocate 0x02b6b9 thru 0x02d54d (0x1e95 bytes) 000691 1715343334.176359 mapwrite 0x004351 thru 0x00594e (0x15fe bytes) 000692 1715343334.185100 trunc from 0x040000 to 0x03ab9a (0x5465 bytes) 000693 1715343334.196485 punch from 0x0181d9 to 0x01dfa5 (0x5dcd bytes) 000694 1715343334.199267 trunc from 0x03ab9b to 0x027209 (0x13991 bytes) 000695 1715343334.209474 MIRROR_RESYNC 12 mirrors 000699 1715343334.884012 write 0x001f02 thru 0x00a1f5 (0x82f4 bytes) 000700 1715343334.903673 fallocate 0x0089a4 thru 0x00eea7 (0x6504 bytes) 000702 1715343334.907025 write 0x0364d8 thru 0x03b46f (0x4f98 bytes) 000703 1715343334.908185 fallocate 0x00d64d thru 0x00eb0e (0x14c2 bytes) 000704 1715343334.911279 read 0x01f0ce thru 0x029f12 (0xae45 bytes) 000705 1715343334.915425 punch from 0x03225a to 0x03577d (0x3524 bytes) 000706 1715343334.917895 read 0x00be6e thru 0x011850 (0x59e3 bytes) 000707 1715343334.922359 fallocate 0x033d83 thru 0x0345b6 (0x834 bytes) 000709 1715343334.925467 fallocate 0x015435 thru 0x023de6 (0xe9b2 bytes) 000713 1715343334.929345 fallocate 0x02bfe5 thru 0x03b278 (0xf294 bytes) 000714 1715343334.934364 MIRROR_RESYNC 12 mirrors 000716 1715343335.503399 trunc from 0x03b470 to 0x033cdb (0x7794 bytes) 000717 1715343335.525421 MIRROR_RESYNC 12 mirrors 000718 1715343336.061272 mapwrite 0x006c16 thru 0x01051b (0x9906 bytes) 000719 1715343336.110807 mapread 0x023b7f thru 0x02a40f (0x6891 bytes) 000720 1715343336.115633 punch from 0x024b2f to 0x024fe6 (0x4b8 bytes) 000722 1715343336.118580 mapwrite 0x008b1f thru 0x0101eb (0x76cd bytes) 000724 1715343336.128629 mapread 0x02a93c thru 0x032b8e (0x8253 bytes) 000725 1715343336.134890 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000726 1715343336.218591 read 0x01ee05 thru 0x022279 (0x3475 bytes) 000727 1715343336.222709 fallocate 0x01d7e7 thru 0x0226f7 (0x4f11 bytes) 000728 1715343336.225527 fallocate 0x01291f thru 0x0197bd (0x6e9f bytes) 000729 1715343336.228186 fallocate 0x03b0ab thru 0x03e64d (0x35a3 bytes) 000730 1715343336.230827 read 0x001e58 thru 0x005530 (0x36d9 bytes) 000731 1715343336.235934 mapwrite 0x03da26 thru 0x03ffff (0x25da bytes) 000732 1715343336.258349 read 0x0104ef thru 0x0128fb (0x240d bytes) 000734 1715343336.259344 trunc from 0x040000 to 0x023685 (0x1c97a bytes) 000735 1715343336.266974 mapwrite 0x03d27d thru 0x03ffff (0x2d83 bytes) 000736 1715343336.287120 read 0x00a52d thru 0x0180f9 (0xdbcd bytes) 000737 1715343336.287928 fallocate 0x00a8ed thru 0x011b7e (0x7292 bytes) 000738 1715343336.290935 mapread 0x03c102 thru 0x03ffff (0x3efe bytes) 000739 1715343336.291833 write 0x012b6c thru 0x01d30c (0xa7a1 bytes) 000740 1715343336.293160 read 0x034df8 thru 0x03ffff (0xb208 bytes) 000741 1715343336.293762 mapread 0x035b07 thru 0x03ffff (0xa4f9 bytes) 000743 1715343336.294937 mapwrite 0x00ab47 thru 0x019e14 (0xf2ce bytes) 000744 1715343336.305898 punch from 0x03c8cb to 0x03d9fd (0x1133 bytes) 000746 1715343336.311161 MIRROR_EXTEND to 12 mirrors 000748 1715343336.537474 mapread 0x0078b6 thru 0x01702c (0xf777 bytes) 000749 1715343336.560900 MIRROR_RESYNC 12 mirrors 000750 1715343337.115980 trunc from 0x040000 to 0x031472 (0xeb8d bytes) 000751 1715343337.145199 fallocate 0x006a0b thru 0x00caf0 (0x60e6 bytes) 000752 1715343337.148074 write 0x02f3cf thru 0x032754 (0x3386 bytes) 000753 1715343337.151482 write 0x00e58b thru 0x018adf (0xa555 bytes) 000754 1715343337.156901 mapread 0x00debb thru 0x0128c6 (0x4a0c bytes) 000755 1715343337.161931 mapwrite 0x02b745 thru 0x02e10b (0x29c7 bytes) 000756 1715343337.177020 mapread 0x009b41 thru 0x009ed6 (0x396 bytes) 000757 1715343337.180346 MIRROR_EXTEND to 13 mirrors 000758 1715343337.388994 write 0x01fce5 thru 0x02cf0f (0xd22b bytes) 000761 1715343337.410374 mapread 0x01016b thru 0x01f0e5 (0xef7b bytes) 000762 1715343337.418335 fallocate 0x03e8a7 thru 0x03ffff (0x1759 bytes) 000764 1715343337.421688 write 0x01a8f1 thru 0x01c047 (0x1757 bytes) 000765 1715343337.422481 mapread 0x020d73 thru 0x02bd20 (0xafae bytes) 000766 1715343337.423277 read 0x01681e thru 0x016b86 (0x369 bytes) 000768 1715343337.424201 mapread 0x01bbfb thru 0x026835 (0xac3b bytes) 000769 1715343337.424977 trunc from 0x040000 to 0x03ceee (0x3111 bytes) 000770 1715343337.440933 mapread 0x0267d4 thru 0x02d34f (0x6b7c bytes) 000771 1715343337.445477 mapread 0x02f76b thru 0x03ceee (0xd784 bytes) 000772 1715343337.452351 MIRROR_EXTEND to 14 mirrors 000773 1715343337.658120 read 0x01d317 thru 0x027f77 (0xac61 bytes) 000774 1715343337.667716 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000775 1715343337.771171 mapread 0x0329ee thru 0x0362d9 (0x38ec bytes) 000777 1715343337.787899 punch from 0x02101b to 0x02610a (0x50f0 bytes) 000778 1715343337.799873 fallocate 0x03b96f thru 0x03ffff (0x4691 bytes) 000779 1715343337.801793 mapwrite 0x03c4fb thru 0x03ffff (0x3b05 bytes) 000780 1715343337.823222 mapread 0x00a4cc thru 0x013b89 (0x96be bytes) 000781 1715343337.828774 read 0x0302df thru 0x0341e8 (0x3f0a bytes) 000782 1715343337.832877 MIRROR_SPLIT mirror 12 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000783 1715343337.909382 trunc from 0x040000 to 0x005648 (0x3a9b7 bytes) 000784 1715343337.919904 write 0x031f1c thru 0x036cf1 (0x4dd6 bytes) 000785 1715343337.921244 trunc from 0x036cf2 to 0x009a14 (0x2d2dd bytes) 000786 1715343337.930565 MIRROR_RESYNC 12 mirrors 000787 1715343338.442051 MIRROR_RESYNC 12 mirrors 000788 1715343338.531263 trunc from 0x009a15 to 0x0247a2 (0x1ad8e bytes) 000789 1715343338.550372 fallocate 0x025422 thru 0x025a7d (0x65c bytes) 000790 1715343338.553374 punch from 0x00f651 to 0x01a40e (0xadbe bytes) 000791 1715343338.556032 mapwrite 0x02d578 thru 0x0376f7 (0xa180 bytes) 000792 1715343338.589337 mapwrite 0x03baba thru 0x03ffff (0x4546 bytes) 000793 1715343338.609241 punch from 0x0396e7 to 0x03ffff (0x6919 bytes) 000795 1715343338.612561 trunc from 0x040000 to 0x02e5b4 (0x11a4b bytes) 000796 1715343338.619725 fallocate 0x038378 thru 0x03e8d7 (0x6560 bytes) 000797 1715343338.622605 mapread 0x02557c thru 0x02956d (0x3ff2 bytes) 000798 1715343338.626863 mapwrite 0x02da29 thru 0x035ba8 (0x8180 bytes) 000800 1715343338.649184 fallocate 0x00f78e thru 0x018385 (0x8bf8 bytes) 000801 1715343338.652160 mapread 0x02669a thru 0x02c85e (0x61c5 bytes) 000802 1715343338.652952 punch from 0x00297e to 0x00bf16 (0x9599 bytes) 000803 1715343338.655540 read 0x003c69 thru 0x011089 (0xd421 bytes) 000805 1715343338.660560 fallocate 0x0051be thru 0x009916 (0x4759 bytes) 000807 1715343338.664386 punch from 0x0121f7 to 0x022049 (0xfe53 bytes) 000808 1715343338.667053 write 0x031366 thru 0x034d35 (0x39d0 bytes) 000809 1715343338.670316 MIRROR_RESYNC 12 mirrors 000810 1715343339.371126 trunc from 0x035ba9 to 0x0018a2 (0x34306 bytes) 000811 1715343339.393246 MIRROR_RESYNC 12 mirrors 000812 1715343339.659520 MIRROR_RESYNC 12 mirrors 000814 1715343339.718736 mapread 0x000ecb thru 0x0018a2 (0x9d8 bytes) 000815 1715343339.722414 MIRROR_EXTEND to 13 mirrors 000816 1715343339.898050 punch from 0x00164f to 0x0018a2 (0x254 bytes) 000817 1715343339.915545 trunc from 0x0018a3 to 0x027260 (0x259be bytes) 000818 1715343339.926879 punch from 0x00ebde to 0x01d7e7 (0xec0a bytes) 000819 1715343339.929417 mapread 0x01ce92 thru 0x02044c (0x35bb bytes) 000820 1715343339.938394 mapwrite 0x02f1db thru 0x03c284 (0xd0aa bytes) 000821 1715343339.967232 write 0x0047ad thru 0x008a98 (0x42ec bytes) 000822 1715343339.972114 read 0x011d02 thru 0x0163e9 (0x46e8 bytes) 000823 1715343339.976065 fallocate 0x003d92 thru 0x01320e (0xf47d bytes) 000824 1715343339.979130 read 0x03b3d5 thru 0x03c284 (0xeb0 bytes) 000825 1715343339.980002 trunc from 0x03c285 to 0x00effc (0x2d288 bytes) 000826 1715343339.987672 trunc from 0x00effd to 0x023e82 (0x14e86 bytes) 000827 1715343339.995177 mapread 0x00e13f thru 0x017a4a (0x990c bytes) 000829 1715343340.001075 fallocate 0x037af0 thru 0x037d69 (0x27a bytes) 000830 1715343340.004229 trunc from 0x023e83 to 0x03474a (0x108c8 bytes) 000831 1715343340.011397 mapwrite 0x01de6d thru 0x02d97c (0xfb10 bytes) 000833 1715343340.030248 trunc from 0x03474b to 0x02be5b (0x88ef bytes) 000834 1715343340.037874 read 0x01cd60 thru 0x021baf (0x4e50 bytes) 000835 1715343340.038693 punch from 0x0240fb to 0x029c71 (0x5b77 bytes) 000836 1715343340.041599 trunc from 0x02be5c to 0x022c87 (0x91d4 bytes) 000837 1715343340.048382 punch from 0x009f42 to 0x00b447 (0x1506 bytes) 000838 1715343340.051304 mapread 0x00a7ba thru 0x011379 (0x6bc0 bytes) 000839 1715343340.057154 MIRROR_EXTEND to 14 mirrors 000840 1715343340.270981 mapwrite 0x02381a thru 0x025f22 (0x2709 bytes) 000841 1715343340.311459 trunc from 0x025f23 to 0x0232f3 (0x2c2f bytes) 000842 1715343340.320991 mapread 0x014724 thru 0x01ecf0 (0xa5cd bytes) 000843 1715343340.329532 punch from 0x0152e4 to 0x017cb9 (0x29d6 bytes) 000844 1715343340.332197 fallocate 0x030bf4 thru 0x03ce47 (0xc254 bytes) 000845 1715343340.335185 read 0x02538b thru 0x02b80e (0x6484 bytes) 000846 1715343340.339304 write 0x0185e7 thru 0x02399f (0xb3b9 bytes) 000847 1715343340.340646 fallocate 0x005ca8 thru 0x010cd5 (0xb02e bytes) 000848 1715343340.343784 write 0x01ce28 thru 0x023383 (0x655c bytes) 000850 1715343340.345348 trunc from 0x03ce48 to 0x03e633 (0x17ec bytes) 000851 1715343340.352698 fallocate 0x008de7 thru 0x017486 (0xe6a0 bytes) 000852 1715343340.355433 write 0x00ce0c thru 0x018c3c (0xbe31 bytes) 000853 1715343340.362672 MIRROR_EXTEND to 15 mirrors 000855 1715343340.586298 fallocate 0x00a6e4 thru 0x00d358 (0x2c75 bytes) 000857 1715343340.606203 write 0x000d42 thru 0x0064f9 (0x57b8 bytes) 000858 1715343340.613011 mapwrite 0x03f4a0 thru 0x03ffff (0xb60 bytes) 000860 1715343340.642293 punch from 0x01260c to 0x020ec9 (0xe8be bytes) 000861 1715343340.645273 mapread 0x011cd2 thru 0x01e6d8 (0xca07 bytes) 000862 1715343340.653262 mapwrite 0x00bfe3 thru 0x00d38a (0x13a8 bytes) 000863 1715343340.664820 write 0x015c83 thru 0x017d04 (0x2082 bytes) 000866 1715343340.667173 punch from 0x02c011 to 0x038f17 (0xcf07 bytes) 000867 1715343340.670003 mapwrite 0x021595 thru 0x026dc0 (0x582c bytes) 000868 1715343340.679694 mapwrite 0x022554 thru 0x025cc3 (0x3770 bytes) 000869 1715343340.686537 fallocate 0x013f32 thru 0x01d018 (0x90e7 bytes) 000870 1715343340.689259 fallocate 0x02837d thru 0x033e31 (0xbab5 bytes) 000871 1715343340.691989 mapwrite 0x03702b thru 0x03a51c (0x34f2 bytes) 000872 1715343340.703994 fallocate 0x0059c9 thru 0x0101dc (0xa814 bytes) 000873 1715343340.707095 write 0x035e62 thru 0x0363b3 (0x552 bytes) 000874 1715343340.708084 mapread 0x00d8d1 thru 0x00e4b2 (0xbe2 bytes) 000875 1715343340.711251 MIRROR_RESYNC 15 mirrors 000878 1715343341.659209 trunc from 0x040000 to 0x03da24 (0x25db bytes) 000880 1715343341.681562 mapwrite 0x034381 thru 0x039758 (0x53d8 bytes) 000881 1715343341.701908 write 0x020cec thru 0x02d668 (0xc97d bytes) 000883 1715343341.710045 punch from 0x005cd4 to 0x0090cb (0x33f8 bytes) 000885 1715343341.713439 mapread 0x03213f thru 0x03c9d8 (0xa89a bytes) 000886 1715343341.714309 write 0x001b5c thru 0x00fca3 (0xe148 bytes) 000887 1715343341.719476 mapwrite 0x01810a thru 0x01cac4 (0x49bb bytes) 000888 1715343341.733943 write 0x021d3c thru 0x024d37 (0x2ffc bytes) 000889 1715343341.734864 mapread 0x0107b1 thru 0x016e08 (0x6658 bytes) 000890 1715343341.740214 MIRROR_EXTEND to 16 mirrors 000891 1715343341.954713 write 0x030f76 thru 0x03154d (0x5d8 bytes) 000892 1715343341.978633 mapread 0x022ca9 thru 0x02790a (0x4c62 bytes) 000893 1715343341.983732 punch from 0x00608d to 0x0128a3 (0xc817 bytes) 000895 1715343341.987095 write 0x03c560 thru 0x03ffff (0x3aa0 bytes) 000896 1715343341.991904 mapwrite 0x033603 thru 0x03ffff (0xc9fd bytes) 000897 1715343342.007727 write 0x000f01 thru 0x00ce03 (0xbf03 bytes) 000898 1715343342.012525 punch from 0x035d82 to 0x03fc27 (0x9ea6 bytes) 000899 1715343342.015403 mapwrite 0x02fe0e thru 0x03294b (0x2b3e bytes) 000900 1715343342.024815 mapwrite 0x02c97d thru 0x039b05 (0xd189 bytes) 000901 1715343342.039427 mapread 0x038c2e thru 0x03ff0a (0x72dd bytes) 000902 1715343342.040010 fallocate 0x03a74e thru 0x03ffff (0x58b2 bytes) 000904 1715343342.042627 punch from 0x0364b4 to 0x03ffff (0x9b4c bytes) 000905 1715343342.047839 MIRROR_SPLIT mirror 11 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000906 1715343342.136438 mapwrite 0x026ea8 thru 0x02bc0f (0x4d68 bytes) 000907 1715343342.155309 read 0x0287fa thru 0x032f30 (0xa737 bytes) 000908 1715343342.158734 mapread 0x034f4b thru 0x036f10 (0x1fc6 bytes) 000909 1715343342.163373 trunc from 0x040000 to 0x03acbf (0x5340 bytes) 000910 1715343342.180438 punch from 0x01b13b to 0x023e84 (0x8d4a bytes) 000911 1715343342.183399 mapread 0x032da9 thru 0x03acbf (0x7f17 bytes) 000912 1715343342.187403 MIRROR_EXTEND to 16 mirrors 000913 1715343342.421076 trunc from 0x03acc0 to 0x009fc3 (0x30cfc bytes) 000916 1715343342.453387 fallocate 0x029f1d thru 0x02f733 (0x5817 bytes) 000917 1715343342.458621 MIRROR_RESYNC 16 mirrors 000918 1715343343.106924 punch from 0x028fe1 to 0x02f733 (0x6753 bytes) 000919 1715343343.122359 trunc from 0x02f734 to 0x021b67 (0xdbcc bytes) 000920 1715343343.136954 trunc from 0x021b68 to 0x030f3e (0xf3d7 bytes) 000921 1715343343.142683 punch from 0x023d01 to 0x027614 (0x3914 bytes) 000922 1715343343.145442 write 0x038cff thru 0x03ffff (0x7301 bytes) 000924 1715343343.147185 mapread 0x034bb7 thru 0x03b6f4 (0x6b3e bytes) 000925 1715343343.154370 trunc from 0x040000 to 0x025a26 (0x1a5d9 bytes) 000926 1715343343.161897 fallocate 0x019c4f thru 0x022c1a (0x8fcc bytes) 000929 1715343343.165480 trunc from 0x025a27 to 0x0336e1 (0xdcbb bytes) 000930 1715343343.172440 mapwrite 0x014090 thru 0x02325b (0xf1cc bytes) 000932 1715343343.191976 mapwrite 0x023423 thru 0x0247c6 (0x13a4 bytes) 000933 1715343343.198318 trunc from 0x0336e2 to 0x01c835 (0x16eac bytes) 000935 1715343343.205965 write 0x017e1f thru 0x02615c (0xe33e bytes) 000937 1715343343.209957 MIRROR_RESYNC 16 mirrors 000938 1715343343.861215 MIRROR_RESYNC 16 mirrors 000940 1715343343.962799 write 0x032d83 thru 0x032ec2 (0x140 bytes) 000941 1715343343.975242 MIRROR_RESYNC 16 mirrors 000942 1715343344.720907 mapread 0x006271 thru 0x010d1e (0xaaae bytes) 000943 1715343344.729427 read 0x00c564 thru 0x016b3c (0xa5d9 bytes) 000944 1715343344.732927 punch from 0x021fc8 to 0x02f7d1 (0xd80a bytes) 000945 1715343344.749727 trunc from 0x032ec3 to 0x03cb49 (0x9c87 bytes) 000947 1715343344.758007 punch from 0x007635 to 0x011418 (0x9de4 bytes) 000949 1715343344.760832 punch from 0x005097 to 0x00f6b2 (0xa61c bytes) 000950 1715343344.763127 mapwrite 0x00fbfb thru 0x01e407 (0xe80d bytes) 000951 1715343344.778962 mapwrite 0x02b310 thru 0x0325ea (0x72db bytes) 000954 1715343344.789974 write 0x0037eb thru 0x011c55 (0xe46b bytes) 000955 1715343344.795808 mapread 0x035c42 thru 0x03cb49 (0x6f08 bytes) 000956 1715343344.799818 mapwrite 0x02a308 thru 0x037ae1 (0xd7da bytes) 000957 1715343344.815485 MIRROR_SPLIT mirror 6 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000958 1715343344.901665 read 0x02572c thru 0x025a5f (0x334 bytes) 000960 1715343344.904356 trunc from 0x03cb4a to 0x034771 (0x83d8 bytes) 000961 1715343344.913367 write 0x03b921 thru 0x03ffff (0x46df bytes) 000963 1715343344.914713 punch from 0x0264b8 to 0x02bdf4 (0x593d bytes) 000964 1715343344.917359 mapread 0x03c0db thru 0x03ffff (0x3f25 bytes) 000966 1715343344.918681 trunc from 0x040000 to 0x00ba2c (0x345d3 bytes) 000967 1715343344.925734 read 0x008159 thru 0x00ba2c (0x38d4 bytes) 000968 1715343344.929567 write 0x0076c7 thru 0x011f77 (0xa8b1 bytes) 000970 1715343344.931278 fallocate 0x015e93 thru 0x024f64 (0xf0d2 bytes) 000971 1715343344.934175 read 0x01f271 thru 0x024f64 (0x5cf4 bytes) 000972 1715343344.938876 mapwrite 0x00739c thru 0x008c6f (0x18d4 bytes) 000973 1715343344.949114 mapread 0x011659 thru 0x013181 (0x1b29 bytes) 000974 1715343344.950019 mapwrite 0x00f309 thru 0x014812 (0x550a bytes) 000975 1715343344.960872 MIRROR_RESYNC 15 mirrors 000976 1715343345.295359 mapread 0x01e14d thru 0x02249d (0x4351 bytes) 000977 1715343345.299164 mapwrite 0x02d505 thru 0x035505 (0x8001 bytes) 000978 1715343345.316751 write 0x009e80 thru 0x016460 (0xc5e1 bytes) 000979 1715343345.321231 MIRROR_SPLIT mirror 11 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000980 1715343345.367658 punch from 0x00f9aa to 0x012199 (0x27f0 bytes) 000982 1715343345.369240 write 0x01e7c5 thru 0x01f2f6 (0xb32 bytes) 000983 1715343345.372557 fallocate 0x039363 thru 0x03ffff (0x6c9d bytes) 000984 1715343345.373802 fallocate 0x014e27 thru 0x015f29 (0x1103 bytes) 000985 1715343345.375040 read 0x0183e9 thru 0x020a11 (0x8629 bytes) 000986 1715343345.376887 punch from 0x01732c to 0x0251c5 (0xde9a bytes) 000988 1715343345.379808 mapwrite 0x01104b thru 0x01b64c (0xa602 bytes) 000989 1715343345.389927 MIRROR_EXTEND to 15 mirrors 000990 1715343345.498300 read 0x004c5d thru 0x00c1fc (0x75a0 bytes) 000992 1715343345.505367 write 0x03ce4a thru 0x03ffff (0x31b6 bytes) 000994 1715343345.521327 write 0x037bf2 thru 0x03ffff (0x840e bytes) 000995 1715343345.524445 punch from 0x02ea10 to 0x0333b8 (0x49a9 bytes) 000996 1715343345.527094 mapwrite 0x03175a thru 0x03d11f (0xb9c6 bytes) 000997 1715343345.542483 mapread 0x003df9 thru 0x010879 (0xca81 bytes) 000999 1715343345.552201 MIRROR_RESYNC 15 mirrors 001000 1715343346.131010 MIRROR_SPLIT mirror 11 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) All operations completed A-OK! PASS 70a (33s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200a: stress test ===================== 08:15:49 (1715343349) Starting client: oleg119-client.virtnet: -o user_xattr,flock oleg119-server@tcp:/lustre /mnt/lustre2 Starting client: oleg119-client.virtnet: -o user_xattr,flock oleg119-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 613824 .. Extending file size to 6630720 .. Extending file size to 6937952 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8674176 .. Extending file size to 7781664 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8607424 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8688544 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9308416 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8759936 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8845792 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Waiting 7819 7820 7821 7823 7825 192.168.201.119@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg119-client.virtnet /mnt/lustre2 (opts:) 192.168.201.119@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg119-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (67s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 08:16:58 (1715343418) Starting client: oleg119-client.virtnet: -o user_xattr,flock oleg119-server@tcp:/lustre /mnt/lustre2 Starting client: oleg119-client.virtnet: -o user_xattr,flock oleg119-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6743488 .. Extending file size to 8487552 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7432160 .. failed Extending file size to 7739520 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8989088 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9019648 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8282080 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8948800 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9207200 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8992256 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 10519 10520 10522 10523 10525 192.168.201.119@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg119-client.virtnet /mnt/lustre2 (opts:) 192.168.201.119@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg119-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 08:18:04 (1715343484) Starting client: oleg119-client.virtnet: -o user_xattr,flock oleg119-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2912 1284776 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2680 1285008 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 15732 3591288 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 21676 3585344 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37408 7176632 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2912 1284776 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2680 1285008 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 15732 3591288 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 21676 3585344 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37408 7176632 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0296299 s, 70.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0442366 s, 47.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.119@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg119-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 08:18:10 (1715343490) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0320249 s, 65.5 MB/s PASS 202 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 203: mirror file preserve mirror ID === 08:18:15 (1715343495) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc8:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc8:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa8:0x0] } PASS 203 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 08:18:18 (1715343498) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0380533 s, 55.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xa1c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xa1c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xa1d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 08:18:23 (1715343503) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0452341 s, 46.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaa:0x0] } PASS 204b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 08:18:27 (1715343507) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0299825 s, 69.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } PASS 204c (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 08:18:54 (1715343534) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg119-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.01985 s, 52.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcd:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcd:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xce:0x0] } PASS 204d (35s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 08:19:31 (1715343571) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg119-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0368417 s, 56.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (37s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 08:20:11 (1715343611) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg119-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool oleg119-server: Pool lustre.test_204f created Adding targets to pool oleg119-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg119-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0425649 s, 49.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xa1d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xa1e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xa1d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xa1e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg119-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg119-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg119-server: Pool lustre.test_204f destroyed PASS 204f (52s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 205: lfs mirror extend to set prefer and nocompr flags ========================================================== 08:21:05 (1715343665) PASS 205 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 08:21:08 (1715343668) oleg119-server: Pool lustre.test_206 created Adding targets to pool oleg119-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg119-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg119-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg119-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg119-server: Pool lustre.test_206 destroyed PASS 206 (11s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 08:21:20 (1715343680) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0161287 s, 65.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00887287 s, 118 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xd11a8e8e Mirror 3: 0xd11a8e8e PASS 207 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 08:21:24 (1715343684) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 08:21:26 (1715343686) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 08:21:28 (1715343688) Starting client: oleg119-client.virtnet: -o user_xattr,flock oleg119-server@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.00669034 s, 0.6 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1730000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xd70000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xd70000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xd70000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1b90000 Read this (4 bytes): BBBB 192.168.201.119@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg119-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 08:21:42 (1715343702) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00261456 s, 3.1 MB/s the very first read read with warm cache PASS 209b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210a: handle broken mirrored lovea ==== 08:21:45 (1715343705) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0166473 s, 63.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x99 lmm_fid: [0x200000405:0x99:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x99 lmm_fid: [0x200000405:0x99:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb9:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611289 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-05-10 08:21:46.000000000 -0400 Modify: 2024-05-10 08:21:46.000000000 -0400 Change: 2024-05-10 08:21:46.000000000 -0400 Birth: - PASS 210a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 08:21:50 (1715343710) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0193291 s, 54.2 MB/s oleg119-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg119-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg119-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg119-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg119-server: O/280000401/d26/da: File not found by ext2_lookup PASS 210b (15s) debug_raw_pointers=0 debug_raw_pointers=0 == sanity-flr test complete, duration 849 sec ============ 08:22:06 (1715343726) === sanity-flr: start cleanup 08:22:06 (1715343726) === === sanity-flr: finish cleanup 08:22:08 (1715343728) ===