-----============= acceptance-small: sanity-flr ============----- Mon May 13 04:00:23 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 6 201 44c === sanity-flr: start setup 04:00:28 (1715587228) === oleg329-client.virtnet: executing check_config_client /mnt/lustre oleg329-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg329-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012ab0b000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012ab0b000.idle_timeout=debug disable quota as required oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 04:00:36 (1715587236) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f7484] debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0a: lfs mirror create with -N option == 04:00:37 (1715587237) PASS 0a (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 04:00:45 (1715587245) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 04:00:48 (1715587248) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0d: lfs mirror extend with -N option == 04:00:51 (1715587251) PASS 0d (6s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 04:01:00 (1715587260) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 04:01:03 (1715587263) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0g: lfs mirror create flags support === 04:01:06 (1715587266) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88012ab0b000.stats=clear osc.lustre-OST0001-osc-ffff88012ab0b000.stats=clear PASS 0g (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 04:01:11 (1715587271) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 04:01:16 (1715587276) PASS 0j (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 04:01:21 (1715587281) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 1: create components with setstripe options ========================================================== 04:01:24 (1715587284) lfs mirror mirror: cannot get UNLOCK lease, ext 4: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 2: create components from existing files ========================================================== 04:01:28 (1715587288) PASS 2 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 3: create components from files located on different MDTs ========================================================== 04:01:32 (1715587292) PASS 3 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 04:01:36 (1715587296) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 04:01:40 (1715587300) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 7: Create mirror with DoM component === 04:01:45 (1715587305) PASS 7 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 04:01:48 (1715587308) 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.534393 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0365319 s, 28.7 MB/s PASS 21 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 04:01:53 (1715587313) 15+0 records in 15+0 records out 15728640 bytes (16 MB) copied, 0.110699 s, 142 MB/s PASS 22 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 04:01:57 (1715587317) Stopping /mnt/lustre-ost1 (opts:) on oleg329-server oleg329-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012ab0b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012ab0b000.ost_server_uuid in DISCONN state after 3 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg329-server: oleg329-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg329-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg329-server oleg329-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012ab0b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012ab0b000.ost_server_uuid in DISCONN state after 3 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg329-server: oleg329-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg329-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (29s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 04:02:28 (1715587348) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB) copied, 0.107105 s, 58.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg329-server oleg329-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012ab0b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012ab0b000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg329-server: oleg329-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg329-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (12s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33a: read can choose available mirror to read ========================================================== 04:02:42 (1715587362) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg329-server oleg329-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012ab0b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012ab0b000.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg329-server: oleg329-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg329-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg329-server oleg329-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012ab0b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012ab0b000.ost_server_uuid in DISCONN state after 3 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg329-server: oleg329-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg329-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 04:03:07 (1715587387) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.842176 s, 62.3 MB/s -rw-r--r-- 1 root root 52428800 May 13 04:03 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.319s user 0m0.000s sys 0m0.114s Stopping /mnt/lustre-ost1 (opts:) on oleg329-server oleg329-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012ab0b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012ab0b000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.254s user 0m0.000s sys 0m0.117s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg329-server: oleg329-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg329-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg329-server oleg329-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012ab0b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012ab0b000.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost1)... real 0m0.251s user 0m0.000s sys 0m0.106s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg329-server: oleg329-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg329-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg329-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012ab0b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012ab0b000.ost_server_uuid in FULL state after 0 sec PASS 33b (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 04:03:33 (1715587413) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 04:03:36 (1715587416) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 04:03:38 (1715587418) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 35: allow to write to mirrored files == 04:03:41 (1715587421) PASS 35 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36a: write to mirrored files ========== 04:03:44 (1715587424) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.138809 s, 90.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 04:03:49 (1715587429) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100218 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108164 s, 96.9 MB/s PASS 36b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 04:03:53 (1715587433) PASS 36c (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 04:03:57 (1715587437) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000597793 s, 1.7 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000659808 s, 1.6 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000696478 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000976053 s, 1.0 MB/s flv=10 olv=10 PASS 36d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 37: mirror I/O API verification ======= 04:04:01 (1715587441) create test files with size 15741165 .. a9f0399e6e7939775abb792fd4469210 - aea33c4e5dc0e8f6b824a9ec15096b40 - 0b5d45de419de1b3b677e02a2b575052 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg329-client.virtnet /mnt/lustre (opts:) Starting client: oleg329-client.virtnet: -o user_xattr,flock oleg329-server@tcp:/lustre /mnt/lustre PASS 37 (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 38: resync ============================ 04:04:07 (1715587447) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 04:04:12 (1715587452) PASS 39 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 04:04:15 (1715587455) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0141573 s, 74.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3e:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3c:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106905 s, 98.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 41: lfs mirror resync check =========== 04:04:18 (1715587458) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0208448 s, 101 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.083146 s, 50.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0307059 s, 68.3 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 42: lfs mirror verify ================= 04:04:22 (1715587462) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43a: mirror pick on write ============= 04:04:25 (1715587465) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 04:04:27 (1715587467) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00978912 s, 107 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44a: lfs mirror split check =========== 04:04:31 (1715587471) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0977589 s, 32.2 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 44a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44b: mirror split does not create stale file ========================================================== 04:04:35 (1715587475) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x49:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x48:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 04:04:39 (1715587479) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0763962 s, 137 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg329-client.virtnet /mnt/lustre (opts:) Starting client: oleg329-client.virtnet: -o user_xattr,flock oleg329-server@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 04:04:44 (1715587484) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800b6f2d800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0736778 s, 142 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg329-client.virtnet /mnt/lustre (opts:) Starting client: oleg329-client.virtnet: -o user_xattr,flock oleg329-server@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800aff63800.xattr_cache=1 PASS 44e (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 04:04:49 (1715587489) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (7s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 46: Verify setstripe --copy option ==== 04:04:58 (1715587498) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 47: Verify mirror obj alloc =========== 04:05:05 (1715587505) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 48: Verify snapshot mirror ============ 04:05:07 (1715587507) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0657536 s, 47.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0519498 s, 60.6 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: b299fe8c498a6707207568a1ca70c06f - mirror 1 checksum: 5476a216d07793c6363cec800ac0c3f5 - mirror 2 checksum: b299fe8c498a6707207568a1ca70c06f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon May 13 04:05:08 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon May 13 04:05:08 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: b299fe8c498a6707207568a1ca70c06f - mirror 1 checksum: 5476a216d07793c6363cec800ac0c3f5 - mirror 2 checksum: 5476a216d07793c6363cec800ac0c3f5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon May 13 04:05:08 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon May 13 04:05:08 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } PASS 48 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 49a: FIEMAP upon FLR file ============= 04:05:11 (1715587511) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109167 s, 96.1 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 529408.. 530431: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50A: mirror split update layout generation ========================================================== 04:05:14 (1715587514) Failing mds1 on oleg329-server Stopping /mnt/lustre-mds1 (opts:) on oleg329-server 04:05:21 (1715587521) shut down Failover mds1 to oleg329-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg329-server: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-MDT0000 04:05:34 (1715587534) targets are mounted 04:05:34 (1715587534) facet_failover done oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102879 s, 102 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } PASS 50A (27s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 04:05:43 (1715587543) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00771933 s, 2.7 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00383039 s, 5.3 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50b: mirror rsync handles sparseness == 04:05:47 (1715587547) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0101226 s, 2.0 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00433331 s, 4.7 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 04:05:50 (1715587550) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00268278 s, 6.1 MB/s PASS 50c (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50d: mirror rsync keep holes ========== 04:05:54 (1715587554) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.010833 s, 1.9 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 04:05:57 (1715587557) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00159074 s, 2.6 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00692619 s, 591 kB/s PASS 60a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 04:06:00 (1715587560) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 60b (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 04:06:05 (1715587565) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 04:06:25 (1715587585) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 04:06:44 (1715587604) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (16s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 04:07:02 (1715587622) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70: mirror create and split race ====== 04:07:05 (1715587625) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 30590 mirror split pid 30591 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7325: 30591 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4716: 30590 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) PASS 70 (62s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70a: flr mode fsx test ================ 04:08:08 (1715587688) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 7689 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read 000002 1715587689.335465 write 0x031c27 thru 0x03ffff (0xe3d9 bytes) 000003 1715587689.336629 mapread 0x00ee07 thru 0x01bc37 (0xce31 bytes) truncating to largest ever: 0x3a8d3 000004 1715587689.343166 trunc from 0x040000 to 0x03a8d2 (0x572d bytes) 000005 1715587689.347320 read 0x0336aa thru 0x03a8d2 (0x7229 bytes) 000006 1715587689.347758 read 0x033640 thru 0x033a88 (0x449 bytes) 000008 1715587689.348566 write 0x01040c thru 0x016b9c (0x6791 bytes) 000010 1715587689.349467 read 0x032d02 thru 0x03714f (0x444e bytes) 000011 1715587689.349848 read 0x03490c thru 0x03a8d2 (0x5fc7 bytes) 000012 1715587689.350185 fallocate 0x020b18 thru 0x02dda2 (0xd28b bytes) 000013 1715587689.352058 fallocate 0x01dc7c thru 0x024c52 (0x6fd7 bytes) 000016 1715587689.354416 mapwrite 0x03a3b2 thru 0x03ffff (0x5c4e bytes) 000017 1715587689.368905 mapread 0x0199e3 thru 0x01cde6 (0x3404 bytes) 000018 1715587689.369509 mapwrite 0x0105b8 thru 0x010cc0 (0x709 bytes) 000019 1715587689.373422 fallocate 0x023a5d thru 0x026659 (0x2bfd bytes) 000020 1715587689.374781 mapread 0x028b67 thru 0x0331f7 (0xa691 bytes) 000021 1715587689.375338 read 0x03574d thru 0x03ef53 (0x9807 bytes) 000023 1715587689.376039 mapread 0x037153 thru 0x03ffff (0x8ead bytes) 000024 1715587689.376473 read 0x0399e6 thru 0x03c6db (0x2cf6 bytes) 000025 1715587689.376839 write 0x0104c7 thru 0x016248 (0x5d82 bytes) 000027 1715587689.378185 mapwrite 0x019f6c thru 0x0228ad (0x8942 bytes) 000029 1715587689.383595 write 0x038b69 thru 0x03ffff (0x7497 bytes) 000030 1715587689.384265 mapwrite 0x033603 thru 0x03a100 (0x6afe bytes) fallocating to largest ever: 0x40000 000033 1715587689.389498 fallocate 0x03aaa6 thru 0x03ffff (0x555a bytes) 000034 1715587689.390924 trunc from 0x040000 to 0x005ac6 (0x3a539 bytes) 000035 1715587689.394890 trunc from 0x005ac7 to 0x03c7d5 (0x36d0f bytes) 000036 1715587689.398415 trunc from 0x03c7d6 to 0x03ac9f (0x1b36 bytes) 000037 1715587689.402055 mapwrite 0x03afec thru 0x03c63c (0x1651 bytes) 000038 1715587689.415089 fallocate 0x01addd thru 0x0280d9 (0xd2fd bytes) 000039 1715587689.416961 trunc from 0x03c63d to 0x01aa41 (0x21bfb bytes) 000041 1715587689.421227 read 0x007e66 thru 0x00d55f (0x56fa bytes) 000042 1715587689.421578 punch from 0x00dfab to 0x016f41 (0x8f97 bytes) 000044 1715587689.423742 trunc from 0x01aa42 to 0x0396ae (0x1ec6d bytes) 000045 1715587689.428414 mapwrite 0x0098bd thru 0x00a937 (0x107b bytes) 000046 1715587689.434236 trunc from 0x0396af to 0x0191ba (0x204f4 bytes) 000047 1715587689.438574 punch from 0x008f1a to 0x018277 (0xf35e bytes) 000048 1715587689.440524 mapwrite 0x03000b thru 0x036258 (0x624e bytes) 000049 1715587689.456626 read 0x017ce6 thru 0x01d5a8 (0x58c3 bytes) 000050 1715587689.457110 fallocate 0x0116a6 thru 0x0208bd (0xf218 bytes) 000051 1715587689.459239 trunc from 0x036259 to 0x0114f6 (0x24d62 bytes) 000052 1715587689.463479 fallocate 0x0102a1 thru 0x01215f (0x1ebf bytes) 000053 1715587689.465377 trunc from 0x012160 to 0x0377c7 (0x25668 bytes) 000054 1715587689.469841 punch from 0x01cbe8 to 0x0249f5 (0x7e0e bytes) 000057 1715587689.472133 mapread 0x02ce9f thru 0x0377c7 (0xa929 bytes) 000058 1715587689.477064 punch from 0x00f308 to 0x01f181 (0xfe7a bytes) 000060 1715587689.479631 write 0x006ca5 thru 0x008d81 (0x20dd bytes) 000061 1715587689.480274 write 0x0272d7 thru 0x0305e2 (0x930c bytes) 000062 1715587689.481017 mapwrite 0x030f27 thru 0x037bfc (0x6cd6 bytes) 000063 1715587689.494027 trunc from 0x037bfd to 0x0363dc (0x1820 bytes) 000065 1715587689.498100 mapread 0x02d25f thru 0x03044c (0x31ee bytes) 000067 1715587689.498960 write 0x02dac6 thru 0x03160e (0x3b49 bytes) 000070 1715587689.500044 write 0x01ec0b thru 0x02ac6b (0xc061 bytes) 000071 1715587689.502416 fallocate 0x036827 thru 0x03ffff (0x97d9 bytes) 000072 1715587689.504376 mapwrite 0x0176b7 thru 0x018eb6 (0x1800 bytes) 000074 1715587689.514297 mapwrite 0x0223f0 thru 0x02a868 (0x8479 bytes) 000075 1715587689.521414 write 0x022568 thru 0x031135 (0xebce bytes) 000076 1715587689.522527 fallocate 0x00d35b thru 0x01634d (0x8ff3 bytes) 000077 1715587689.524645 write 0x037055 thru 0x0399d5 (0x2981 bytes) 000078 1715587689.525347 mapread 0x0079a7 thru 0x008ab1 (0x110b bytes) 000079 1715587689.525926 punch from 0x0120d6 to 0x01679c (0x46c7 bytes) 000080 1715587689.527948 trunc from 0x0399d6 to 0x00ef94 (0x2aa41 bytes) 000082 1715587689.533912 mapwrite 0x03eff7 thru 0x03ffff (0x1009 bytes) 000083 1715587689.547742 punch from 0x013886 to 0x014bf0 (0x136b bytes) 000086 1715587689.550132 mapwrite 0x0378dc thru 0x03ffff (0x8724 bytes) 000088 1715587689.557079 read 0x02a89a thru 0x037c1b (0xd382 bytes) 000089 1715587689.560971 fallocate 0x017032 thru 0x0253dc (0xe3ab bytes) 000090 1715587689.562893 mapwrite 0x0335da thru 0x036354 (0x2d7b bytes) 000091 1715587689.566833 trunc from 0x040000 to 0x03b884 (0x477b bytes) 000092 1715587689.570556 punch from 0x005feb to 0x015260 (0xf276 bytes) 000093 1715587689.571907 trunc from 0x03b885 to 0x038e94 (0x29f0 bytes) 000094 1715587689.576035 punch from 0x030a66 to 0x032b28 (0x20c3 bytes) 000095 1715587689.577834 fallocate 0x03b0d0 thru 0x03ffff (0x4f30 bytes) 000096 1715587689.580163 write 0x0147df thru 0x01cd99 (0x85bb bytes) 000097 1715587689.582494 mapwrite 0x01733c thru 0x022f52 (0xbc17 bytes) 000098 1715587689.590746 mapwrite 0x032a31 thru 0x037859 (0x4e29 bytes) 000100 1715587689.598462 mapwrite 0x0326d8 thru 0x03fa7e (0xd3a7 bytes) 000101 1715587689.616780 trunc from 0x040000 to 0x01ea0a (0x215f5 bytes) 000104 1715587689.622053 mapread 0x002014 thru 0x00a3d7 (0x83c4 bytes) 000105 1715587689.628775 trunc from 0x01ea0b to 0x01121c (0xd7ee bytes) 000107 1715587689.633879 mapwrite 0x02d5b5 thru 0x0358f8 (0x8344 bytes) 000111 1715587689.659129 read 0x026f6b thru 0x0358f8 (0xe98e bytes) 000112 1715587689.664789 MIRROR_EXTEND to 2 mirrors 000113 1715587689.771024 punch from 0x009280 to 0x009cab (0xa2c bytes) 000114 1715587689.784492 fallocate 0x01d189 thru 0x02208f (0x4f07 bytes) 000115 1715587689.786689 mapwrite 0x024fe2 thru 0x031b50 (0xcb6f bytes) 000118 1715587689.805809 punch from 0x02e566 to 0x0358f8 (0x7393 bytes) 000120 1715587689.808854 mapread 0x0070a0 thru 0x007e5f (0xdc0 bytes) 000121 1715587689.812542 trunc from 0x0358f9 to 0x03fe52 (0xa55a bytes) 000123 1715587689.825567 mapwrite 0x010c69 thru 0x013156 (0x24ee bytes) 000124 1715587689.834227 read 0x02dc8e thru 0x039daf (0xc122 bytes) 000125 1715587689.836991 trunc from 0x03fe53 to 0x02580f (0x1a643 bytes) 000127 1715587689.843997 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000129 1715587689.891715 write 0x03b8d2 thru 0x03ffff (0x472e bytes) 000130 1715587689.894361 read 0x02bc67 thru 0x033387 (0x7721 bytes) 000131 1715587689.898215 read 0x0016ef thru 0x0030af (0x19c1 bytes) 000132 1715587689.903010 trunc from 0x040000 to 0x00d58c (0x32a73 bytes) 000133 1715587689.909357 read 0x006f4b thru 0x00d58c (0x6642 bytes) 000134 1715587689.909775 write 0x016d94 thru 0x02030f (0x957c bytes) 000136 1715587689.911038 read 0x004cdb thru 0x00da9e (0x8dc4 bytes) 000137 1715587689.911331 mapwrite 0x02c563 thru 0x030e00 (0x489e bytes) 000138 1715587689.928486 write 0x013069 thru 0x014019 (0xfb1 bytes) 000139 1715587689.929116 mapwrite 0x012994 thru 0x019d5c (0x73c9 bytes) 000140 1715587689.935172 fallocate 0x03d689 thru 0x03ffff (0x2977 bytes) 000141 1715587689.936994 write 0x001625 thru 0x0052e4 (0x3cc0 bytes) 000142 1715587689.938888 MIRROR_EXTEND to 2 mirrors 000143 1715587690.074904 trunc from 0x040000 to 0x01a487 (0x25b78 bytes) 000144 1715587690.087005 mapwrite 0x037b18 thru 0x03ffff (0x84e8 bytes) 000146 1715587690.107011 write 0x0254ed thru 0x02c4bb (0x6fcf bytes) 000148 1715587690.111344 mapread 0x022ace thru 0x030309 (0xd83c bytes) 000149 1715587690.114104 punch from 0x010dcb to 0x01c42d (0xb663 bytes) 000150 1715587690.116624 mapread 0x018fb1 thru 0x01a1a2 (0x11f2 bytes) 000151 1715587690.119715 trunc from 0x040000 to 0x02f535 (0x10aca bytes) 000152 1715587690.127925 read 0x02f2d1 thru 0x02f535 (0x265 bytes) 000153 1715587690.128346 punch from 0x01bceb to 0x01fd6e (0x4084 bytes) 000154 1715587690.132403 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000155 1715587690.171587 write 0x03d258 thru 0x03ffff (0x2da8 bytes) 000156 1715587690.173562 read 0x0338a0 thru 0x0371ca (0x392b bytes) 000157 1715587690.176210 fallocate 0x00e850 thru 0x00ed23 (0x4d4 bytes) 000158 1715587690.178167 trunc from 0x040000 to 0x02374e (0x1c8b1 bytes) 000160 1715587690.183131 fallocate 0x001d14 thru 0x00c148 (0xa435 bytes) 000162 1715587690.185167 trunc from 0x02374f to 0x023ec3 (0x775 bytes) 000163 1715587690.188796 write 0x011a36 thru 0x01bf1d (0xa4e8 bytes) 000164 1715587690.191710 mapread 0x00b151 thru 0x00c074 (0xf24 bytes) 000165 1715587690.194213 write 0x00b2a6 thru 0x01528b (0x9fe6 bytes) 000166 1715587690.194974 trunc from 0x023ec4 to 0x002f4e (0x20f75 bytes) 000167 1715587690.199557 mapread 0x000b76 thru 0x002f4e (0x23d9 bytes) 000168 1715587690.201486 fallocate 0x024c5a thru 0x027eaf (0x3256 bytes) 000169 1715587690.203268 write 0x00b769 thru 0x00ed98 (0x3630 bytes) 000171 1715587690.204299 punch from 0x005ee7 to 0x00e6d9 (0x87f3 bytes) 000173 1715587690.207364 fallocate 0x03133a thru 0x03d50f (0xc1d6 bytes) 000175 1715587690.209102 trunc from 0x03d510 to 0x026985 (0x16b8a bytes) 000176 1715587690.212569 punch from 0x01e661 to 0x026985 (0x8325 bytes) 000177 1715587690.213846 mapwrite 0x01b9da thru 0x021fda (0x6601 bytes) 000178 1715587690.224126 read 0x01c419 thru 0x026985 (0xa56d bytes) 000181 1715587690.227794 mapwrite 0x039265 thru 0x039d81 (0xb1d bytes) 000182 1715587690.240540 write 0x007037 thru 0x0092ea (0x22b4 bytes) 000186 1715587690.242305 mapread 0x029c82 thru 0x02f65c (0x59db bytes) 000187 1715587690.242935 mapwrite 0x00bf43 thru 0x012979 (0x6a37 bytes) 000188 1715587690.251205 mapread 0x00a059 thru 0x017ebb (0xde63 bytes) 000189 1715587690.251882 write 0x023f50 thru 0x02c89a (0x894b bytes) 000194 1715587690.253928 trunc from 0x039d82 to 0x012ab6 (0x272cb bytes) 000195 1715587690.259716 write 0x01e85e thru 0x01e919 (0xbc bytes) 000196 1715587690.260249 mapwrite 0x00d2b4 thru 0x01b27d (0xdfca bytes) 000197 1715587690.280886 punch from 0x01500d to 0x01e725 (0x9719 bytes) 000198 1715587690.282777 punch from 0x01c24d to 0x01c545 (0x2f9 bytes) 000201 1715587690.284702 mapread 0x00c6cf thru 0x0142fd (0x7c2f bytes) 000202 1715587690.285189 fallocate 0x005fa0 thru 0x00f7f7 (0x9858 bytes) 000203 1715587690.286574 fallocate 0x0167fa thru 0x024e03 (0xe60a bytes) 000206 1715587690.288578 trunc from 0x01e91a to 0x03207c (0x13763 bytes) 000207 1715587690.292667 punch from 0x00435c to 0x00ecfe (0xa9a3 bytes) 000208 1715587690.294309 mapwrite 0x006bc9 thru 0x013970 (0xcda8 bytes) 000210 1715587690.312441 read 0x009e05 thru 0x012ee8 (0x90e4 bytes) 000211 1715587690.312773 trunc from 0x03207d to 0x00f14e (0x22f2e bytes) 000212 1715587690.317110 write 0x02a151 thru 0x032362 (0x8212 bytes) 000213 1715587690.318075 fallocate 0x00adbc thru 0x011e87 (0x70cc bytes) 000214 1715587690.320173 read 0x029f35 thru 0x0314eb (0x75b7 bytes) 000215 1715587690.323372 mapwrite 0x03f7c1 thru 0x03ffff (0x83f bytes) 000216 1715587690.338387 write 0x00a870 thru 0x00b7b8 (0xf49 bytes) 000217 1715587690.339107 read 0x0200b3 thru 0x02ee9e (0xedec bytes) 000218 1715587690.339555 trunc from 0x040000 to 0x029404 (0x16bfb bytes) 000219 1715587690.344980 read 0x007ab6 thru 0x00ddad (0x62f8 bytes) 000220 1715587690.345415 trunc from 0x029405 to 0x01eebe (0xa546 bytes) 000221 1715587690.350009 mapwrite 0x001093 thru 0x00c87a (0xb7e8 bytes) 000222 1715587690.358831 trunc from 0x01eebf to 0x028690 (0x97d2 bytes) 000224 1715587690.365205 fallocate 0x01bcbf thru 0x02738e (0xb6d0 bytes) 000225 1715587690.367490 mapread 0x004524 thru 0x00eac9 (0xa5a6 bytes) 000226 1715587690.368088 mapwrite 0x013f38 thru 0x01ba07 (0x7ad0 bytes) 000227 1715587690.378004 punch from 0x012f76 to 0x01c2b9 (0x9344 bytes) 000228 1715587690.379592 trunc from 0x028691 to 0x01c131 (0xc55f bytes) 000230 1715587690.384527 punch from 0x00351b to 0x00c076 (0x8b5c bytes) 000232 1715587690.386886 trunc from 0x01c132 to 0x002da8 (0x19389 bytes) 000233 1715587690.391942 trunc from 0x002da9 to 0x019f26 (0x1717e bytes) 000237 1715587690.397266 mapread 0x0103b0 thru 0x012fb9 (0x2c0a bytes) 000239 1715587690.400685 trunc from 0x019f27 to 0x0122b9 (0x7c6d bytes) 000240 1715587690.405141 trunc from 0x0122ba to 0x001d2c (0x1058d bytes) 000242 1715587690.409683 mapwrite 0x002507 thru 0x010ecb (0xe9c5 bytes) 000243 1715587690.440717 mapwrite 0x039090 thru 0x03ffff (0x6f70 bytes) 000244 1715587690.454427 mapwrite 0x03b923 thru 0x03ffff (0x46dd bytes) 000245 1715587690.459109 punch from 0x01840a to 0x026a1a (0xe611 bytes) 000246 1715587690.460575 read 0x029f72 thru 0x02ca74 (0x2b03 bytes) 000249 1715587690.461653 trunc from 0x040000 to 0x00e127 (0x31ed8 bytes) 000250 1715587690.466361 mapread 0x00b402 thru 0x00e127 (0x2d26 bytes) 000251 1715587690.466956 trunc from 0x00e128 to 0x006c83 (0x74a4 bytes) 000252 1715587690.471391 trunc from 0x006c84 to 0x0107ae (0x9b2b bytes) 000254 1715587690.476138 mapwrite 0x029f76 thru 0x0345a8 (0xa633 bytes) 000255 1715587690.491587 mapread 0x032dfe thru 0x0345a8 (0x17ab bytes) 000256 1715587690.492223 mapread 0x029106 thru 0x02fdb2 (0x6cad bytes) 000258 1715587690.493024 read 0x011bcd thru 0x01b73d (0x9b71 bytes) 000260 1715587690.493589 mapread 0x013296 thru 0x01f3ec (0xc157 bytes) 000262 1715587690.494375 mapread 0x01c80e thru 0x0249b7 (0x81aa bytes) 000263 1715587690.494727 trunc from 0x0345a9 to 0x0391db (0x4c33 bytes) 000264 1715587690.499358 mapread 0x02dfa5 thru 0x0391db (0xb237 bytes) 000265 1715587690.502584 mapwrite 0x03afe8 thru 0x03ffff (0x5018 bytes) 000266 1715587690.515293 trunc from 0x040000 to 0x014c8e (0x2b371 bytes) 000267 1715587690.519265 fallocate 0x0018bc thru 0x00e259 (0xc99e bytes) 000268 1715587690.520887 mapwrite 0x00649f thru 0x0084c9 (0x202b bytes) 000269 1715587690.525883 mapread 0x000281 thru 0x007969 (0x76e9 bytes) 000270 1715587690.527753 MIRROR_EXTEND to 2 mirrors 000271 1715587690.626582 read 0x009286 thru 0x0109f0 (0x776b bytes) 000274 1715587690.630265 MIRROR_EXTEND to 3 mirrors 000275 1715587690.748928 mapwrite 0x00a8fa thru 0x00fe0a (0x5511 bytes) 000276 1715587690.780979 mapwrite 0x0056f3 thru 0x00591c (0x22a bytes) 000277 1715587690.785008 punch from 0x0116eb to 0x014c8e (0x35a4 bytes) 000280 1715587690.787257 mapwrite 0x00974f thru 0x00c625 (0x2ed7 bytes) 000281 1715587690.792348 MIRROR_EXTEND to 4 mirrors 000282 1715587690.905564 MIRROR_EXTEND to 5 mirrors 000285 1715587691.020071 punch from 0x00fd6e to 0x014c8e (0x4f21 bytes) 000286 1715587691.029378 write 0x01a60c thru 0x024f47 (0xa93c bytes) 000287 1715587691.031243 mapread 0x019d2a thru 0x024f47 (0xb21e bytes) 000288 1715587691.032962 mapwrite 0x019f82 thru 0x01ee5d (0x4edc bytes) 000290 1715587691.038852 mapwrite 0x00ec1c thru 0x01c83a (0xdc1f bytes) 000291 1715587691.049410 fallocate 0x0278ba thru 0x031a68 (0xa1af bytes) 000292 1715587691.051542 mapwrite 0x01fbf8 thru 0x0269e1 (0x6dea bytes) 000293 1715587691.071338 punch from 0x01039d to 0x017f58 (0x7bbc bytes) 000294 1715587691.073336 punch from 0x00a12f to 0x013c2d (0x9aff bytes) 000295 1715587691.075012 mapread 0x014810 thru 0x01f949 (0xb13a bytes) 000296 1715587691.077913 punch from 0x020d34 to 0x0269e1 (0x5cae bytes) 000297 1715587691.081381 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000298 1715587691.132638 read 0x0015c2 thru 0x004002 (0x2a41 bytes) 000299 1715587691.134621 fallocate 0x0392c8 thru 0x03ffff (0x6d38 bytes) 000300 1715587691.136725 mapread 0x002b4e thru 0x0119b1 (0xee64 bytes) 000301 1715587691.144067 mapwrite 0x01a35e thru 0x021168 (0x6e0b bytes) 000302 1715587691.151762 mapwrite 0x0334a8 thru 0x03ffff (0xcb58 bytes) 000303 1715587691.166167 trunc from 0x040000 to 0x025da6 (0x1a259 bytes) 000305 1715587691.170079 write 0x029124 thru 0x02c5d8 (0x34b5 bytes) 000307 1715587691.170797 mapread 0x02a1eb thru 0x02bdc8 (0x1bde bytes) 000308 1715587691.171156 mapread 0x00e425 thru 0x01b3a7 (0xcf83 bytes) 000309 1715587691.171515 mapwrite 0x02d837 thru 0x0339d6 (0x61a0 bytes) 000311 1715587691.184273 trunc from 0x0339d7 to 0x02123e (0x12798 bytes) 000312 1715587691.188946 mapread 0x000746 thru 0x0038f0 (0x31ab bytes) 000313 1715587691.191571 write 0x000a01 thru 0x00ffe9 (0xf5e9 bytes) 000315 1715587691.193090 fallocate 0x038ab8 thru 0x03ffff (0x7548 bytes) 000316 1715587691.196557 MIRROR_RESYNC 4 mirrors 000318 1715587691.350649 MIRROR_RESYNC 4 mirrors 000320 1715587691.395135 punch from 0x006c02 to 0x01638c (0xf78b bytes) 000321 1715587691.406016 read 0x00d007 thru 0x01573c (0x8736 bytes) 000323 1715587691.409120 mapwrite 0x00c0b0 thru 0x010da6 (0x4cf7 bytes) 000324 1715587691.419293 trunc from 0x02123f to 0x020dc2 (0x47c bytes) 000325 1715587691.430374 mapread 0x01e186 thru 0x020dc2 (0x2c3d bytes) 000329 1715587691.435580 mapwrite 0x027b21 thru 0x03106b (0x954b bytes) 000330 1715587691.451129 fallocate 0x00f23f thru 0x01c2d5 (0xd097 bytes) 000331 1715587691.454520 MIRROR_EXTEND to 5 mirrors 000332 1715587691.579864 MIRROR_RESYNC 5 mirrors 000333 1715587691.800925 mapread 0x00acd0 thru 0x010646 (0x5977 bytes) 000334 1715587691.806113 trunc from 0x03106c to 0x02d303 (0x3d68 bytes) 000335 1715587691.819475 fallocate 0x00bfc2 thru 0x018b6a (0xcba9 bytes) 000336 1715587691.821069 mapwrite 0x010406 thru 0x01e89d (0xe498 bytes) 000338 1715587691.835404 mapread 0x0255f9 thru 0x028ab4 (0x34bc bytes) 000339 1715587691.839882 MIRROR_RESYNC 5 mirrors 000340 1715587692.047372 punch from 0x00de02 to 0x01a480 (0xc67f bytes) 000341 1715587692.056475 punch from 0x0273fb to 0x02d303 (0x5f09 bytes) 000342 1715587692.058104 mapread 0x014a1e thru 0x016d2a (0x230d bytes) 000343 1715587692.063812 trunc from 0x02d304 to 0x022790 (0xab73 bytes) 000345 1715587692.074424 mapread 0x01d336 thru 0x01e13f (0xe0a bytes) 000347 1715587692.077321 mapwrite 0x03693a thru 0x03ffff (0x96c6 bytes) 000348 1715587692.093045 read 0x02811a thru 0x02b952 (0x3839 bytes) 000349 1715587692.095066 fallocate 0x005fda thru 0x00ba58 (0x5a7f bytes) 000350 1715587692.096679 mapwrite 0x002e61 thru 0x00ac72 (0x7e12 bytes) 000351 1715587692.105543 punch from 0x03511a to 0x03a9ab (0x5892 bytes) 000352 1715587692.107201 mapread 0x01f584 thru 0x020c9e (0x171b bytes) 000353 1715587692.107588 mapwrite 0x0123c9 thru 0x018c69 (0x68a1 bytes) 000354 1715587692.112205 trunc from 0x040000 to 0x03ea51 (0x15ae bytes) 000355 1715587692.116204 punch from 0x004733 to 0x0141cb (0xfa99 bytes) 000356 1715587692.117728 write 0x031888 thru 0x03307a (0x17f3 bytes) 000357 1715587692.118368 mapwrite 0x024060 thru 0x032ce6 (0xec87 bytes) 000359 1715587692.129848 trunc from 0x03ea52 to 0x039151 (0x5900 bytes) 000360 1715587692.134428 trunc from 0x039152 to 0x01743d (0x21d14 bytes) 000361 1715587692.138688 mapread 0x00e4e4 thru 0x01743d (0x8f5a bytes) 000362 1715587692.141341 mapwrite 0x02f463 thru 0x034b37 (0x56d5 bytes) 000363 1715587692.154985 write 0x02d36c thru 0x0398bb (0xc550 bytes) 000364 1715587692.155916 write 0x03ba37 thru 0x03ffff (0x45c9 bytes) 000365 1715587692.156395 punch from 0x016af7 to 0x017ea9 (0x13b3 bytes) 000366 1715587692.158271 punch from 0x024455 to 0x0299ee (0x559a bytes) 000368 1715587692.160182 fallocate 0x01be3c thru 0x02973a (0xd8ff bytes) 000369 1715587692.162217 fallocate 0x038c97 thru 0x03f081 (0x63eb bytes) 000370 1715587692.164559 trunc from 0x040000 to 0x020e79 (0x1f186 bytes) 000371 1715587692.169317 fallocate 0x00d9e7 thru 0x0189b8 (0xafd2 bytes) 000372 1715587692.171138 trunc from 0x020e7a to 0x0344c4 (0x1364b bytes) 000375 1715587692.176440 write 0x02a385 thru 0x02bab5 (0x1731 bytes) 000376 1715587692.182223 MIRROR_RESYNC 5 mirrors 000377 1715587692.471039 fallocate 0x01bf8b thru 0x02389d (0x7913 bytes) 000379 1715587692.479252 mapread 0x006b5c thru 0x0116de (0xab83 bytes) 000380 1715587692.486947 MIRROR_EXTEND to 6 mirrors 000382 1715587692.649419 read 0x029ec4 thru 0x02bfb7 (0x20f4 bytes) 000383 1715587692.653967 write 0x03315b thru 0x037394 (0x423a bytes) 000385 1715587692.670678 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000386 1715587692.738012 MIRROR_EXTEND to 6 mirrors 000388 1715587692.909586 mapread 0x019d1e thru 0x01c1b1 (0x2494 bytes) 000389 1715587692.921817 read 0x01466e thru 0x017e2c (0x37bf bytes) 000390 1715587692.924340 mapwrite 0x02b6f5 thru 0x035139 (0x9a45 bytes) 000391 1715587692.965346 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000392 1715587693.028763 mapread 0x00b139 thru 0x00c365 (0x122d bytes) 000393 1715587693.032952 mapread 0x017b07 thru 0x022eaa (0xb3a4 bytes) 000395 1715587693.042542 read 0x000eda thru 0x0015c5 (0x6ec bytes) 000396 1715587693.044854 mapread 0x02f568 thru 0x037394 (0x7e2d bytes) 000397 1715587693.051427 mapread 0x021bce thru 0x0260a6 (0x44d9 bytes) 000399 1715587693.052315 write 0x03cea6 thru 0x03ffff (0x315a bytes) 000400 1715587693.054856 read 0x00f2a1 thru 0x01733b (0x809b bytes) 000401 1715587693.058168 mapread 0x037dd0 thru 0x03851c (0x74d bytes) 000402 1715587693.058769 mapread 0x019da7 thru 0x024fb9 (0xb213 bytes) 000403 1715587693.059281 fallocate 0x0279e3 thru 0x03784d (0xfe6b bytes) 000405 1715587693.063354 punch from 0x005da1 to 0x00e148 (0x83a8 bytes) 000406 1715587693.065828 write 0x00c524 thru 0x012cd7 (0x67b4 bytes) 000407 1715587693.069304 mapwrite 0x011632 thru 0x012993 (0x1362 bytes) 000408 1715587693.078303 trunc from 0x040000 to 0x036851 (0x97ae bytes) 000409 1715587693.090445 trunc from 0x036852 to 0x0199c8 (0x1ce89 bytes) 000411 1715587693.096554 mapwrite 0x0279f6 thru 0x02e3c9 (0x69d4 bytes) 000414 1715587693.124647 MIRROR_EXTEND to 6 mirrors 000415 1715587693.302836 trunc from 0x02e3ca to 0x011c1d (0x1c7ac bytes) 000417 1715587693.326504 fallocate 0x03c46a thru 0x03ffff (0x3b96 bytes) 000418 1715587693.330100 write 0x00c293 thru 0x017bf7 (0xb965 bytes) 000419 1715587693.334156 mapwrite 0x02b8dc thru 0x0324fc (0x6c21 bytes) 000420 1715587693.345820 punch from 0x029dc6 to 0x034fef (0xb22a bytes) 000421 1715587693.349424 MIRROR_EXTEND to 7 mirrors 000422 1715587693.508215 read 0x03b605 thru 0x03ffff (0x49fb bytes) 000423 1715587693.512580 read 0x00b236 thru 0x01ad11 (0xfadc bytes) 000425 1715587693.520136 fallocate 0x00f4ed thru 0x017750 (0x8264 bytes) 000426 1715587693.531528 punch from 0x02c8fc to 0x0396fc (0xce01 bytes) 000427 1715587693.533225 trunc from 0x040000 to 0x0134ba (0x2cb45 bytes) 000428 1715587693.545470 mapread 0x005311 thru 0x0134ba (0xe1aa bytes) 000429 1715587693.568074 MIRROR_RESYNC 7 mirrors 000430 1715587693.819479 fallocate 0x0085d7 thru 0x0109a0 (0x83ca bytes) 000431 1715587693.831489 trunc from 0x0134bb to 0x027d83 (0x148c9 bytes) 000432 1715587693.843929 fallocate 0x02b64f thru 0x032b51 (0x7503 bytes) 000433 1715587693.846048 read 0x032a49 thru 0x032b51 (0x109 bytes) 000434 1715587693.848030 write 0x0387db thru 0x03ffff (0x7825 bytes) 000435 1715587693.848907 punch from 0x01c4a3 to 0x0240af (0x7c0d bytes) 000436 1715587693.850985 trunc from 0x040000 to 0x0234bd (0x1cb42 bytes) 000439 1715587693.855985 trunc from 0x0234be to 0x021990 (0x1b2d bytes) 000440 1715587693.859901 mapread 0x010062 thru 0x01e7b9 (0xe758 bytes) 000441 1715587693.866490 read 0x014274 thru 0x021990 (0xd71d bytes) 000442 1715587693.866935 write 0x0008db thru 0x0081ba (0x78e0 bytes) 000443 1715587693.869392 write 0x002b5c thru 0x003490 (0x935 bytes) 000444 1715587693.871106 MIRROR_RESYNC 7 mirrors 000445 1715587694.117737 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000446 1715587694.213828 read 0x021657 thru 0x021990 (0x33a bytes) 000447 1715587694.216765 trunc from 0x021991 to 0x0031f7 (0x1e799 bytes) 000448 1715587694.236401 MIRROR_EXTEND to 7 mirrors 000449 1715587694.344293 MIRROR_RESYNC 7 mirrors 000451 1715587694.481645 MIRROR_RESYNC 7 mirrors 000452 1715587694.527512 read 0x00254b thru 0x0031f7 (0xcad bytes) 000453 1715587694.529473 trunc from 0x0031f8 to 0x02c25b (0x29064 bytes) 000454 1715587694.543863 read 0x01b259 thru 0x025369 (0xa111 bytes) 000455 1715587694.547524 read 0x006128 thru 0x00ee07 (0x8ce0 bytes) 000459 1715587694.548840 mapwrite 0x01fc2d thru 0x02733b (0x770f bytes) 000460 1715587694.557157 trunc from 0x02c25c to 0x0057f2 (0x26a69 bytes) 000462 1715587694.562683 punch from 0x0019f0 to 0x0057f2 (0x3e03 bytes) 000463 1715587694.564350 punch from 0x00279e to 0x0057f2 (0x3055 bytes) 000464 1715587694.566067 trunc from 0x0057f3 to 0x000751 (0x50a1 bytes) 000466 1715587694.570013 write 0x001bd2 thru 0x00fccc (0xe0fb bytes) 000467 1715587694.570828 mapread 0x00eed7 thru 0x00fccc (0xdf6 bytes) 000468 1715587694.571243 fallocate 0x0222c8 thru 0x026f9f (0x4cd8 bytes) 000469 1715587694.573348 mapwrite 0x00e5f3 thru 0x018093 (0x9aa1 bytes) 000470 1715587694.587937 read 0x0008a5 thru 0x00539e (0x4afa bytes) 000471 1715587694.588348 punch from 0x001ec3 to 0x004536 (0x2674 bytes) 000472 1715587694.590168 write 0x025284 thru 0x033915 (0xe692 bytes) 000473 1715587694.591378 punch from 0x023d1e to 0x02e657 (0xa93a bytes) 000474 1715587694.595892 trunc from 0x033916 to 0x031b80 (0x1d95 bytes) 000475 1715587694.599936 punch from 0x026bf1 to 0x02e940 (0x7d50 bytes) 000477 1715587694.602311 mapread 0x00087f thru 0x009df4 (0x9576 bytes) 000478 1715587694.605861 fallocate 0x016ba3 thru 0x024bf3 (0xe051 bytes) 000479 1715587694.607555 punch from 0x0244ae to 0x030255 (0xbda8 bytes) 000480 1715587694.610443 MIRROR_EXTEND to 8 mirrors 000481 1715587694.760918 MIRROR_EXTEND to 9 mirrors 000483 1715587694.896397 trunc from 0x031b81 to 0x01dce3 (0x13e9d bytes) 000484 1715587694.915862 trunc from 0x01dce4 to 0x03cbf3 (0x1ef10 bytes) 000487 1715587694.922082 mapwrite 0x00b43d thru 0x014eb0 (0x9a74 bytes) 000488 1715587694.940055 read 0x021401 thru 0x02f8db (0xe4db bytes) 000489 1715587694.943918 mapwrite 0x03bbfe thru 0x03ffff (0x4402 bytes) 000490 1715587694.959900 MIRROR_RESYNC 9 mirrors 000491 1715587695.308598 mapread 0x00f26b thru 0x019f63 (0xacf9 bytes) 000492 1715587695.316965 punch from 0x026c94 to 0x02f5ca (0x8937 bytes) 000493 1715587695.328708 punch from 0x01c2e1 to 0x024ea3 (0x8bc3 bytes) 000494 1715587695.330336 write 0x0166c6 thru 0x020142 (0x9a7d bytes) 000495 1715587695.336952 write 0x000526 thru 0x007c85 (0x7760 bytes) 000496 1715587695.342519 MIRROR_RESYNC 9 mirrors 000497 1715587695.622693 trunc from 0x040000 to 0x021961 (0x1e69e bytes) 000498 1715587695.633661 read 0x00bf2d thru 0x01629c (0xa370 bytes) 000499 1715587695.638142 MIRROR_RESYNC 9 mirrors 000501 1715587695.968224 mapread 0x017ae5 thru 0x021961 (0x9e7d bytes) 000502 1715587695.974311 mapread 0x008ad8 thru 0x0163c4 (0xd8ed bytes) 000504 1715587695.979328 punch from 0x013303 to 0x01d6af (0xa3ad bytes) 000505 1715587695.994237 write 0x02655b thru 0x02687a (0x320 bytes) 000506 1715587695.995228 mapwrite 0x03d336 thru 0x03ffff (0x2cca bytes) 000507 1715587696.018990 MIRROR_RESYNC 9 mirrors 000508 1715587696.456645 write 0x00197e thru 0x00714a (0x57cd bytes) 000509 1715587696.468548 punch from 0x036c91 to 0x037db0 (0x1120 bytes) 000510 1715587696.469955 write 0x030c08 thru 0x03931d (0x8716 bytes) 000511 1715587696.473842 fallocate 0x02a2cb thru 0x036e0c (0xcb42 bytes) 000513 1715587696.477654 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000514 1715587696.535433 punch from 0x03a1a7 to 0x03ffff (0x5e59 bytes) 000516 1715587696.537337 punch from 0x02cc6d to 0x02e884 (0x1c18 bytes) 000518 1715587696.538906 mapwrite 0x03501c thru 0x03715a (0x213f bytes) 000519 1715587696.549569 mapwrite 0x00b0da thru 0x00f5e7 (0x450e bytes) 000520 1715587696.558633 punch from 0x008c71 to 0x00e1b6 (0x5546 bytes) 000521 1715587696.560930 mapwrite 0x03c10b thru 0x03ffff (0x3ef5 bytes) 000522 1715587696.565409 read 0x01e9db thru 0x023893 (0x4eb9 bytes) 000523 1715587696.568205 mapwrite 0x0093e2 thru 0x00f780 (0x639f bytes) 000524 1715587696.578037 trunc from 0x040000 to 0x03062b (0xf9d4 bytes) 000525 1715587696.589171 read 0x00995f thru 0x011dc4 (0x8466 bytes) 000526 1715587696.589575 punch from 0x025707 to 0x029704 (0x3ffe bytes) 000527 1715587696.591200 punch from 0x019bf2 to 0x021c53 (0x8062 bytes) 000528 1715587696.593186 fallocate 0x017f06 thru 0x01d1c7 (0x52c2 bytes) 000529 1715587696.595477 mapread 0x01132d thru 0x01a837 (0x950b bytes) 000530 1715587696.598665 mapread 0x000c19 thru 0x00edc1 (0xe1a9 bytes) 000531 1715587696.599165 mapwrite 0x02615f thru 0x02a474 (0x4316 bytes) 000532 1715587696.608217 read 0x021fb1 thru 0x02adfc (0x8e4c bytes) 000533 1715587696.610635 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000534 1715587696.671360 mapread 0x00a4db thru 0x01472d (0xa253 bytes) 000535 1715587696.679667 trunc from 0x03062c to 0x002d30 (0x2d8fb bytes) 000536 1715587696.685977 write 0x00a4e6 thru 0x01016a (0x5c85 bytes) 000537 1715587696.686584 mapread 0x00a6aa thru 0x00ae9e (0x7f5 bytes) 000538 1715587696.687184 fallocate 0x01b564 thru 0x020b23 (0x55c0 bytes) 000540 1715587696.689359 mapwrite 0x00ab7a thru 0x00d703 (0x2b8a bytes) 000541 1715587696.696210 write 0x02556a thru 0x034eb2 (0xf949 bytes) 000542 1715587696.697207 write 0x01d1ab thru 0x01fd02 (0x2b58 bytes) 000543 1715587696.699857 read 0x01399d thru 0x019d24 (0x6388 bytes) 000544 1715587696.701686 punch from 0x034998 to 0x034eb2 (0x51b bytes) 000545 1715587696.706188 mapread 0x032d7d thru 0x034eb2 (0x2136 bytes) 000546 1715587696.708226 fallocate 0x02bf0b thru 0x02d493 (0x1589 bytes) 000547 1715587696.709692 fallocate 0x01ebfd thru 0x02e296 (0xf69a bytes) 000548 1715587696.711135 trunc from 0x034eb3 to 0x017a3c (0x1d476 bytes) 000549 1715587696.718040 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000550 1715587696.775416 mapread 0x01023f thru 0x017a3c (0x77fe bytes) 000551 1715587696.782406 punch from 0x00200a to 0x008e59 (0x6e50 bytes) 000552 1715587696.784069 mapwrite 0x02dacc thru 0x032ee9 (0x541e bytes) 000553 1715587696.799765 read 0x02b3b2 thru 0x032ee9 (0x7b38 bytes) 000554 1715587696.800209 mapread 0x024780 thru 0x030234 (0xbab5 bytes) 000555 1715587696.803058 mapwrite 0x01ca15 thru 0x01cb58 (0x144 bytes) 000556 1715587696.809265 read 0x008007 thru 0x015273 (0xd26d bytes) 000557 1715587696.812312 fallocate 0x00a183 thru 0x018cfa (0xeb78 bytes) 000558 1715587696.814390 write 0x01a8df thru 0x02048c (0x5bae bytes) 000559 1715587696.815179 fallocate 0x00608e thru 0x011729 (0xb69c bytes) 000560 1715587696.817346 mapwrite 0x03b031 thru 0x03ffff (0x4fcf bytes) 000561 1715587696.831086 read 0x014e30 thru 0x0170ee (0x22bf bytes) 000562 1715587696.832931 MIRROR_EXTEND to 7 mirrors 000563 1715587696.985598 mapread 0x01a562 thru 0x01d495 (0x2f34 bytes) 000565 1715587696.996525 punch from 0x010c18 to 0x017c21 (0x700a bytes) 000566 1715587697.008636 mapread 0x01d577 thru 0x01ee67 (0x18f1 bytes) 000567 1715587697.011918 read 0x018d98 thru 0x01bb25 (0x2d8e bytes) 000568 1715587697.013638 MIRROR_RESYNC 7 mirrors 000569 1715587697.539788 trunc from 0x040000 to 0x014f64 (0x2b09b bytes) 000570 1715587697.556061 read 0x002dc8 thru 0x0082c6 (0x54ff bytes) 000571 1715587697.561523 MIRROR_RESYNC 7 mirrors 000572 1715587697.872426 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000573 1715587697.954096 read 0x013b63 thru 0x014f64 (0x1402 bytes) 000575 1715587697.956230 fallocate 0x01d588 thru 0x02d3e6 (0xfe5f bytes) 000576 1715587697.968846 MIRROR_RESYNC 6 mirrors 000577 1715587698.259366 read 0x011ba3 thru 0x0169c6 (0x4e24 bytes) 000578 1715587698.263999 punch from 0x010f09 to 0x014b2a (0x3c22 bytes) 000579 1715587698.279711 MIRROR_EXTEND to 7 mirrors 000580 1715587698.460957 trunc from 0x02d3e7 to 0x0022db (0x2b10b bytes) 000584 1715587698.479023 mapread 0x001e75 thru 0x0022db (0x467 bytes) 000586 1715587698.484795 MIRROR_EXTEND to 8 mirrors 000587 1715587698.586215 read 0x0009e4 thru 0x0022db (0x18f8 bytes) 000588 1715587698.590408 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000589 1715587698.650590 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000590 1715587698.712394 punch from 0x001d78 to 0x0022db (0x564 bytes) 000592 1715587698.718092 trunc from 0x0022dc to 0x01b931 (0x19656 bytes) 000593 1715587698.730726 mapwrite 0x0205c7 thru 0x024c80 (0x46ba bytes) 000594 1715587698.749177 mapread 0x0219bb thru 0x024c80 (0x32c6 bytes) 000595 1715587698.749902 read 0x00ea1c thru 0x0184d9 (0x9abe bytes) 000596 1715587698.753441 punch from 0x01bbd2 to 0x024c80 (0x90af bytes) 000597 1715587698.755510 mapwrite 0x00bbec thru 0x00fcfe (0x4113 bytes) 000598 1715587698.765452 read 0x0120fc thru 0x01c3e9 (0xa2ee bytes) 000599 1715587698.769868 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000600 1715587698.817970 trunc from 0x024c81 to 0x002048 (0x22c38 bytes) 000601 1715587698.823351 write 0x00f070 thru 0x010f53 (0x1ee4 bytes) 000602 1715587698.824010 mapwrite 0x00c58b thru 0x0113a0 (0x4e16 bytes) 000603 1715587698.839979 fallocate 0x039f08 thru 0x03ffff (0x60f8 bytes) 000606 1715587698.842974 fallocate 0x037f7e thru 0x03ffff (0x8082 bytes) 000607 1715587698.844937 fallocate 0x03afe5 thru 0x03ffff (0x501b bytes) 000608 1715587698.846575 read 0x0232a9 thru 0x02ba0f (0x8767 bytes) 000609 1715587698.849636 punch from 0x02c9e5 to 0x0325d1 (0x5bed bytes) 000610 1715587698.851196 fallocate 0x005c4d thru 0x012984 (0xcd38 bytes) 000611 1715587698.852811 trunc from 0x040000 to 0x005aa1 (0x3a55e bytes) 000612 1715587698.857106 fallocate 0x01b64c thru 0x0253c4 (0x9d79 bytes) 000613 1715587698.858778 fallocate 0x00066f thru 0x00db47 (0xd4d9 bytes) 000615 1715587698.860509 mapwrite 0x00e645 thru 0x01731d (0x8cd9 bytes) 000616 1715587698.876646 trunc from 0x01731e to 0x012bb6 (0x4767 bytes) 000618 1715587698.881261 write 0x00dd31 thru 0x010b06 (0x2dd6 bytes) 000619 1715587698.881978 fallocate 0x0386b8 thru 0x03d50b (0x4e54 bytes) 000620 1715587698.884390 write 0x01b82c thru 0x01bd07 (0x4dc bytes) 000621 1715587698.885101 mapwrite 0x034d19 thru 0x03c0b7 (0x739f bytes) 000622 1715587698.899801 write 0x017027 thru 0x025f07 (0xeee1 bytes) 000623 1715587698.904842 mapwrite 0x01524b thru 0x0196cc (0x4482 bytes) 000624 1715587698.911086 write 0x0114ce thru 0x0198ee (0x8421 bytes) 000625 1715587698.911988 trunc from 0x03c0b8 to 0x02a15a (0x11f5d bytes) 000626 1715587698.917545 trunc from 0x02a15b to 0x01022a (0x19f30 bytes) 000627 1715587698.923490 trunc from 0x01022b to 0x01f581 (0xf357 bytes) 000628 1715587698.929269 fallocate 0x03f5e3 thru 0x03ffff (0xa1d bytes) 000629 1715587698.931786 trunc from 0x040000 to 0x01fc33 (0x203cc bytes) 000630 1715587698.935947 punch from 0x01dd2c to 0x01eabc (0xd91 bytes) 000631 1715587698.937419 punch from 0x00daad to 0x019967 (0xbebb bytes) 000632 1715587698.939393 fallocate 0x022671 thru 0x02a1ab (0x7b3b bytes) 000636 1715587698.941537 read 0x0180df thru 0x01b24a (0x316c bytes) 000637 1715587698.945010 mapwrite 0x00dabe thru 0x00ff52 (0x2495 bytes) 000639 1715587698.952842 mapread 0x0058ae thru 0x0147a5 (0xeef8 bytes) 000640 1715587698.953532 mapwrite 0x00280e thru 0x005973 (0x3166 bytes) 000641 1715587698.961420 write 0x0010ee thru 0x001670 (0x583 bytes) 000642 1715587698.962037 write 0x03d2e9 thru 0x03d687 (0x39f bytes) 000643 1715587698.962630 punch from 0x00a608 to 0x017a70 (0xd469 bytes) 000644 1715587698.964942 mapwrite 0x015b8e thru 0x023d86 (0xe1f9 bytes) 000645 1715587698.979029 write 0x0063d7 thru 0x0135f9 (0xd223 bytes) 000646 1715587698.981675 mapwrite 0x011da8 thru 0x01f0bf (0xd318 bytes) 000647 1715587698.991403 MIRROR_EXTEND to 6 mirrors 000648 1715587699.116714 write 0x034f4e thru 0x03ffff (0xb0b2 bytes) 000649 1715587699.129447 mapread 0x029c42 thru 0x02d58d (0x394c bytes) 000650 1715587699.135823 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000651 1715587699.189977 trunc from 0x040000 to 0x02ffa0 (0x1005f bytes) 000652 1715587699.195445 read 0x0190ae thru 0x01e3f1 (0x5344 bytes) 000653 1715587699.198138 write 0x01a7e4 thru 0x020c44 (0x6461 bytes) 000654 1715587699.200254 fallocate 0x028470 thru 0x02d982 (0x5513 bytes) 000655 1715587699.202360 mapwrite 0x0360b7 thru 0x03ffff (0x9f49 bytes) 000656 1715587699.220704 mapread 0x032fdf thru 0x03ffff (0xd021 bytes) 000657 1715587699.221326 write 0x004fa1 thru 0x00e803 (0x9863 bytes) 000659 1715587699.225492 read 0x02d557 thru 0x0394de (0xbf88 bytes) 000661 1715587699.229306 trunc from 0x040000 to 0x035ae0 (0xa51f bytes) 000662 1715587699.234552 mapwrite 0x039913 thru 0x03ffff (0x66ed bytes) 000663 1715587699.255648 read 0x03fba2 thru 0x03ffff (0x45e bytes) 000664 1715587699.256089 mapwrite 0x039adc thru 0x03ffff (0x6524 bytes) 000665 1715587699.262281 punch from 0x021ffc to 0x0302ee (0xe2f3 bytes) 000666 1715587699.264933 trunc from 0x040000 to 0x0276ee (0x18911 bytes) 000667 1715587699.270260 read 0x02159a thru 0x0276ee (0x6155 bytes) 000668 1715587699.273515 read 0x008988 thru 0x0101cc (0x7845 bytes) 000669 1715587699.273895 read 0x011608 thru 0x0165e9 (0x4fe2 bytes) 000671 1715587699.274423 mapwrite 0x002d9f thru 0x010419 (0xd67b bytes) 000673 1715587699.285583 MIRROR_RESYNC 5 mirrors 000674 1715587699.437015 read 0x01c02e thru 0x02595d (0x9930 bytes) 000675 1715587699.440229 trunc from 0x0276ef to 0x02a3c8 (0x2cda bytes) 000676 1715587699.456273 mapread 0x01af30 thru 0x029d70 (0xee41 bytes) 000677 1715587699.466013 MIRROR_RESYNC 5 mirrors 000678 1715587699.615506 write 0x00db89 thru 0x011c9d (0x4115 bytes) 000679 1715587699.625897 trunc from 0x02a3c9 to 0x00e495 (0x1bf33 bytes) 000680 1715587699.634781 punch from 0x00071d to 0x00b6ac (0xaf90 bytes) 000681 1715587699.636322 fallocate 0x033764 thru 0x03ffff (0xc89c bytes) 000682 1715587699.638233 fallocate 0x02430f thru 0x02857a (0x426c bytes) 000683 1715587699.639859 punch from 0x02446f to 0x027233 (0x2dc5 bytes) 000685 1715587699.641577 trunc from 0x040000 to 0x00dafb (0x32504 bytes) 000688 1715587699.646643 punch from 0x009d92 to 0x00d3e8 (0x3657 bytes) 000690 1715587699.650363 fallocate 0x02fa04 thru 0x03d5e3 (0xdbe0 bytes) 000691 1715587699.652162 fallocate 0x0258cc thru 0x02bd88 (0x64bd bytes) 000693 1715587699.654174 mapread 0x034354 thru 0x03d5e3 (0x9290 bytes) 000694 1715587699.660569 mapwrite 0x019e30 thru 0x01a43c (0x60d bytes) 000695 1715587699.668002 write 0x039829 thru 0x03ffff (0x67d7 bytes) 000696 1715587699.668771 read 0x0190ec thru 0x0218d2 (0x87e7 bytes) 000697 1715587699.669044 mapwrite 0x036146 thru 0x03ffff (0x9eba bytes) 000698 1715587699.674799 fallocate 0x020ff9 thru 0x028ab5 (0x7abd bytes) 000699 1715587699.676362 mapwrite 0x030429 thru 0x03aa36 (0xa60e bytes) 000700 1715587699.682208 trunc from 0x040000 to 0x01e81d (0x217e2 bytes) 000701 1715587699.686174 write 0x02729a thru 0x031a75 (0xa7dc bytes) 000702 1715587699.687171 mapread 0x00e148 thru 0x012954 (0x480d bytes) 000703 1715587699.689703 write 0x031098 thru 0x036499 (0x5402 bytes) 000704 1715587699.690455 fallocate 0x029b2f thru 0x030310 (0x67e2 bytes) 000705 1715587699.693212 trunc from 0x03649a to 0x02f351 (0x7148 bytes) 000706 1715587699.698369 trunc from 0x02f352 to 0x01d209 (0x12148 bytes) 000708 1715587699.703233 mapread 0x0106e4 thru 0x01d209 (0xcb26 bytes) 000709 1715587699.705439 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000710 1715587699.761307 mapread 0x00786c thru 0x00f981 (0x8116 bytes) 000711 1715587699.768857 MIRROR_EXTEND to 5 mirrors 000712 1715587699.890769 mapwrite 0x0083de thru 0x0148fe (0xc521 bytes) 000713 1715587699.926389 trunc from 0x01d20a to 0x00ac01 (0x12608 bytes) 000714 1715587699.937097 trunc from 0x00ac02 to 0x01db67 (0x12f66 bytes) 000715 1715587699.942371 punch from 0x019bfc to 0x01db67 (0x3f6c bytes) 000716 1715587699.943978 fallocate 0x02c0ce thru 0x02cdc3 (0xcf6 bytes) 000717 1715587699.945400 read 0x00522b thru 0x00952a (0x4300 bytes) 000719 1715587699.946199 read 0x00f97c thru 0x0161a4 (0x6829 bytes) 000721 1715587699.949729 mapread 0x007b70 thru 0x017757 (0xfbe8 bytes) 000722 1715587699.950678 punch from 0x00d76d to 0x00f193 (0x1a27 bytes) 000723 1715587699.952507 fallocate 0x02d833 thru 0x03d5e1 (0xfdaf bytes) 000724 1715587699.955947 MIRROR_RESYNC 5 mirrors 000725 1715587700.150705 write 0x0237c8 thru 0x02ee7f (0xb6b8 bytes) 000726 1715587700.160321 mapwrite 0x00adec thru 0x015807 (0xaa1c bytes) 000727 1715587700.174490 write 0x03bc3a thru 0x03ffff (0x43c6 bytes) 000728 1715587700.175208 write 0x02171f thru 0x02348b (0x1d6d bytes) 000729 1715587700.177522 trunc from 0x040000 to 0x007beb (0x38414 bytes) 000730 1715587700.182896 punch from 0x000158 to 0x002a32 (0x28db bytes) 000731 1715587700.184445 punch from 0x0060fc to 0x007beb (0x1af0 bytes) 000733 1715587700.186384 mapwrite 0x027307 thru 0x033735 (0xc42f bytes) 000734 1715587700.203812 MIRROR_EXTEND to 6 mirrors 000735 1715587700.335961 write 0x030f2e thru 0x036a75 (0x5b48 bytes) 000737 1715587700.347774 mapread 0x02237e thru 0x0321f3 (0xfe76 bytes) 000739 1715587700.353299 punch from 0x01c34f to 0x02756b (0xb21d bytes) 000740 1715587700.356625 MIRROR_RESYNC 6 mirrors 000741 1715587700.539549 MIRROR_EXTEND to 7 mirrors 000742 1715587700.679294 read 0x0004ab thru 0x009d92 (0x98e8 bytes) 000744 1715587700.682341 trunc from 0x036a76 to 0x013db6 (0x22cbf bytes) 000746 1715587700.703913 trunc from 0x013db7 to 0x03f368 (0x2b5b2 bytes) 000748 1715587700.711399 mapwrite 0x0378ee thru 0x03ad60 (0x3473 bytes) 000749 1715587700.726204 MIRROR_EXTEND to 8 mirrors 000750 1715587700.860282 write 0x012f71 thru 0x01f322 (0xc3b2 bytes) 000751 1715587700.873440 mapread 0x031722 thru 0x03f14f (0xda2e bytes) 000752 1715587700.878773 mapwrite 0x03c04d thru 0x03ffff (0x3fb3 bytes) 000753 1715587700.897393 MIRROR_RESYNC 8 mirrors 000754 1715587701.224202 MIRROR_EXTEND to 9 mirrors 000755 1715587701.374851 mapread 0x028773 thru 0x034587 (0xbe15 bytes) 000757 1715587701.381837 write 0x0094d3 thru 0x00a0c3 (0xbf1 bytes) 000759 1715587701.395292 mapread 0x00af40 thru 0x00b72f (0x7f0 bytes) 000761 1715587701.395971 fallocate 0x00dedb thru 0x01569a (0x77c0 bytes) 000762 1715587701.397793 trunc from 0x040000 to 0x02be17 (0x141e8 bytes) 000763 1715587701.402459 mapread 0x01b61f thru 0x026884 (0xb266 bytes) 000764 1715587701.406445 trunc from 0x02be18 to 0x005031 (0x26de6 bytes) 000766 1715587701.412543 MIRROR_RESYNC 9 mirrors 000768 1715587701.613925 write 0x034c2e thru 0x03f9cb (0xad9e bytes) 000771 1715587701.625706 punch from 0x019a8b to 0x0217b8 (0x7d2e bytes) 000774 1715587701.627574 read 0x0367c8 thru 0x03f9cb (0x9204 bytes) 000775 1715587701.628026 punch from 0x033efe to 0x03b734 (0x7837 bytes) 000776 1715587701.631663 mapread 0x00b38a thru 0x019096 (0xdd0d bytes) 000777 1715587701.637903 read 0x03467b thru 0x03f9cb (0xb351 bytes) 000778 1715587701.640821 fallocate 0x011176 thru 0x01bc97 (0xab22 bytes) 000780 1715587701.642729 punch from 0x01db9d to 0x0223a1 (0x4805 bytes) 000782 1715587701.644465 fallocate 0x03e145 thru 0x03ffff (0x1ebb bytes) 000783 1715587701.647662 MIRROR_RESYNC 9 mirrors 000784 1715587701.840149 trunc from 0x040000 to 0x036262 (0x9d9d bytes) 000785 1715587701.855833 fallocate 0x00aee8 thru 0x00fdcd (0x4ee6 bytes) 000786 1715587701.857965 trunc from 0x036263 to 0x01f460 (0x16e02 bytes) 000788 1715587701.863210 fallocate 0x028561 thru 0x032666 (0xa106 bytes) 000789 1715587701.867098 MIRROR_RESYNC 9 mirrors 000791 1715587702.136780 fallocate 0x025845 thru 0x02bd8b (0x6547 bytes) 000793 1715587702.149888 mapread 0x00bfdb thru 0x00ca7f (0xaa5 bytes) 000794 1715587702.155886 MIRROR_EXTEND to 10 mirrors 000795 1715587702.278884 fallocate 0x00f91e thru 0x019ec1 (0xa5a4 bytes) 000796 1715587702.291764 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000797 1715587702.347118 MIRROR_EXTEND to 10 mirrors 000800 1715587702.472949 fallocate 0x03c9c2 thru 0x03ffff (0x363e bytes) 000801 1715587702.483496 read 0x013507 thru 0x022d7e (0xf878 bytes) 000802 1715587702.486367 mapwrite 0x034f9a thru 0x03e834 (0x989b bytes) 000803 1715587702.503886 MIRROR_EXTEND to 11 mirrors 000804 1715587702.625269 mapread 0x020a57 thru 0x0214f9 (0xaa3 bytes) 000805 1715587702.632377 mapread 0x0016dc thru 0x00a0de (0x8a03 bytes) 000807 1715587702.643430 fallocate 0x0105bd thru 0x01f075 (0xeab9 bytes) 000808 1715587702.655842 punch from 0x01399d to 0x01cd24 (0x9388 bytes) 000809 1715587702.657470 write 0x030c5c thru 0x03e41b (0xd7c0 bytes) 000810 1715587702.663297 punch from 0x01b4c5 to 0x022b60 (0x769c bytes) 000812 1715587702.667136 MIRROR_EXTEND to 12 mirrors 000816 1715587702.807627 mapread 0x016ffb thru 0x020e48 (0x9e4e bytes) 000818 1715587702.818481 trunc from 0x040000 to 0x00f061 (0x30f9e bytes) 000821 1715587702.834150 write 0x02f8a7 thru 0x0351bd (0x5917 bytes) 000823 1715587702.835632 trunc from 0x0351be to 0x02357c (0x11c41 bytes) 000824 1715587702.841977 write 0x037c33 thru 0x03ffff (0x83cd bytes) 000825 1715587702.842950 trunc from 0x040000 to 0x013c2a (0x2c3d5 bytes) 000827 1715587702.847477 write 0x036898 thru 0x03ffff (0x9768 bytes) 000828 1715587702.849880 MIRROR_RESYNC 12 mirrors 000829 1715587703.223000 mapread 0x014cdb thru 0x01d1a8 (0x84ce bytes) 000830 1715587703.228545 read 0x035b76 thru 0x03ffff (0xa48a bytes) 000831 1715587703.230909 mapread 0x03acff thru 0x03c307 (0x1609 bytes) 000832 1715587703.231285 write 0x001caa thru 0x00b2db (0x9632 bytes) 000833 1715587703.244965 trunc from 0x040000 to 0x03f176 (0xe89 bytes) 000835 1715587703.250691 punch from 0x00a451 to 0x00cdfe (0x29ae bytes) 000838 1715587703.255270 write 0x015162 thru 0x01e9af (0x984e bytes) 000839 1715587703.258675 write 0x01cf3a thru 0x02901d (0xc0e4 bytes) 000840 1715587703.260645 punch from 0x0190db to 0x021840 (0x8766 bytes) 000841 1715587703.264571 punch from 0x00879b to 0x00dcdf (0x5545 bytes) 000842 1715587703.266047 punch from 0x021cbe to 0x0241de (0x2521 bytes) 000843 1715587703.267176 read 0x03e2b6 thru 0x03f176 (0xec1 bytes) 000844 1715587703.270305 write 0x03284c thru 0x034937 (0x20ec bytes) 000845 1715587703.270895 trunc from 0x03f177 to 0x00d970 (0x31806 bytes) 000846 1715587703.275214 punch from 0x002215 to 0x00d970 (0xb75c bytes) 000848 1715587703.277088 fallocate 0x01c41a thru 0x021ee9 (0x5ad0 bytes) 000851 1715587703.279331 read 0x00035d thru 0x00ab34 (0xa7d8 bytes) 000853 1715587703.282650 trunc from 0x021eea to 0x011a63 (0x10486 bytes) 000854 1715587703.287830 MIRROR_SPLIT mirror 9 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000855 1715587703.338595 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000856 1715587703.385480 read 0x00d50a thru 0x011a63 (0x455a bytes) 000858 1715587703.388598 punch from 0x0089e8 to 0x011a63 (0x907c bytes) 000859 1715587703.390898 read 0x00f5d7 thru 0x011a63 (0x248d bytes) 000860 1715587703.394366 fallocate 0x025855 thru 0x02cfeb (0x7797 bytes) 000861 1715587703.396287 write 0x005c37 thru 0x00b053 (0x541d bytes) 000862 1715587703.398638 mapread 0x001964 thru 0x00c598 (0xac35 bytes) 000863 1715587703.399351 punch from 0x00e05e to 0x011a63 (0x3a06 bytes) 000865 1715587703.402061 mapwrite 0x03b51c thru 0x03ffff (0x4ae4 bytes) 000866 1715587703.419328 punch from 0x01291e to 0x01ad9a (0x847d bytes) 000867 1715587703.422175 write 0x032983 thru 0x03b0e6 (0x8764 bytes) 000870 1715587703.423549 mapwrite 0x03b54e thru 0x03ffff (0x4ab2 bytes) 000871 1715587703.429496 write 0x0367f5 thru 0x03da69 (0x7275 bytes) 000873 1715587703.430781 trunc from 0x040000 to 0x029fa4 (0x1605b bytes) 000874 1715587703.435941 write 0x02fa72 thru 0x03a95a (0xaee9 bytes) 000875 1715587703.436756 punch from 0x002990 to 0x00ee62 (0xc4d3 bytes) 000876 1715587703.439890 MIRROR_EXTEND to 11 mirrors 000878 1715587703.556632 read 0x034acd thru 0x03904c (0x4580 bytes) 000879 1715587703.560707 fallocate 0x00b2a6 thru 0x018893 (0xd5ee bytes) 000883 1715587703.570495 write 0x0211e1 thru 0x02d966 (0xc786 bytes) 000884 1715587703.575672 write 0x018318 thru 0x02404c (0xbd35 bytes) 000885 1715587703.580796 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000886 1715587703.643934 read 0x01c376 thru 0x027b0a (0xb795 bytes) 000887 1715587703.647653 fallocate 0x02473c thru 0x0295a9 (0x4e6e bytes) 000888 1715587703.649746 mapread 0x01506e thru 0x02035f (0xb2f2 bytes) 000890 1715587703.655905 mapwrite 0x02eb10 thru 0x031555 (0x2a46 bytes) 000891 1715587703.665745 trunc from 0x03a95b to 0x0254b2 (0x154a8 bytes) 000892 1715587703.676753 mapread 0x01cf3f thru 0x0249b4 (0x7a76 bytes) 000894 1715587703.677676 punch from 0x014d20 to 0x01bff0 (0x72d1 bytes) 000895 1715587703.679458 mapread 0x01bda9 thru 0x021dde (0x6036 bytes) 000896 1715587703.682928 read 0x01856c thru 0x01a162 (0x1bf7 bytes) 000898 1715587703.683482 trunc from 0x0254b3 to 0x03f9a1 (0x1a4ef bytes) 000899 1715587703.688425 fallocate 0x02dbba thru 0x03d503 (0xf94a bytes) 000901 1715587703.690526 trunc from 0x03f9a2 to 0x01648e (0x29513 bytes) 000902 1715587703.694101 fallocate 0x015210 thru 0x01ab7d (0x596e bytes) 000905 1715587703.696076 fallocate 0x022d76 thru 0x02f3fe (0xc689 bytes) 000906 1715587703.697626 mapwrite 0x0159f1 thru 0x0204a8 (0xaab8 bytes) 000910 1715587703.710870 punch from 0x006372 to 0x013035 (0xccc4 bytes) 000911 1715587703.712552 mapread 0x015465 thru 0x020421 (0xafbd bytes) 000912 1715587703.714492 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000913 1715587703.765231 read 0x00892e thru 0x0152ba (0xc98d bytes) 000914 1715587703.767645 trunc from 0x0204a9 to 0x03e8cb (0x1e423 bytes) 000915 1715587703.772236 punch from 0x02084a to 0x028bdf (0x8396 bytes) 000918 1715587703.774298 read 0x00144f thru 0x0048fe (0x34b0 bytes) 000919 1715587703.780080 MIRROR_RESYNC 9 mirrors 000920 1715587704.157212 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000921 1715587704.257702 write 0x011c4d thru 0x01735d (0x5711 bytes) 000922 1715587704.272471 trunc from 0x03e8cc to 0x00165a (0x3d271 bytes) 000923 1715587704.277961 mapread 0x000c8b thru 0x00165a (0x9d0 bytes) 000924 1715587704.280945 write 0x02a7ec thru 0x02ef5f (0x4774 bytes) 000926 1715587704.282093 fallocate 0x01ba1b thru 0x02b989 (0xff6f bytes) 000928 1715587704.284156 mapwrite 0x0334ac thru 0x03b13a (0x7c8f bytes) 000929 1715587704.297958 mapwrite 0x03f39d thru 0x03ffff (0xc63 bytes) 000931 1715587704.308081 fallocate 0x015e2d thru 0x01cb56 (0x6d2a bytes) 000932 1715587704.310014 mapread 0x02c4ea thru 0x035e2d (0x9944 bytes) 000933 1715587704.310738 write 0x00ca98 thru 0x016ef7 (0xa460 bytes) 000934 1715587704.315091 fallocate 0x025fcf thru 0x034be6 (0xec18 bytes) 000935 1715587704.319704 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000936 1715587704.377088 MIRROR_EXTEND to 8 mirrors 000937 1715587704.528996 fallocate 0x028b17 thru 0x02a4e0 (0x19ca bytes) 000938 1715587704.537395 trunc from 0x040000 to 0x02f182 (0x10e7d bytes) 000940 1715587704.543648 fallocate 0x03b042 thru 0x03ffff (0x4fbe bytes) 000942 1715587704.545824 punch from 0x004045 to 0x00cb61 (0x8b1d bytes) 000943 1715587704.547709 fallocate 0x014b73 thru 0x01e975 (0x9e03 bytes) 000944 1715587704.549291 mapwrite 0x003328 thru 0x008f46 (0x5c1f bytes) 000945 1715587704.564822 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000947 1715587704.616053 trunc from 0x02f183 to 0x008c57 (0x2652b bytes) 000948 1715587704.622881 MIRROR_EXTEND to 8 mirrors 000950 1715587704.741080 MIRROR_RESYNC 8 mirrors 000951 1715587704.954512 read 0x004ed8 thru 0x008024 (0x314d bytes) 000952 1715587704.956659 mapread 0x00229e thru 0x006533 (0x4296 bytes) 000953 1715587704.959474 mapread 0x0009f7 thru 0x005a4c (0x5056 bytes) 000954 1715587704.963420 fallocate 0x005658 thru 0x009c73 (0x461c bytes) 000957 1715587704.974314 read 0x004e38 thru 0x009c73 (0x4e3c bytes) 000958 1715587704.977170 punch from 0x0040c0 to 0x009c73 (0x5bb4 bytes) 000959 1715587704.979136 write 0x02c3d4 thru 0x034342 (0x7f6f bytes) 000960 1715587704.982039 MIRROR_EXTEND to 9 mirrors 000961 1715587705.118120 read 0x029175 thru 0x029638 (0x4c4 bytes) 000962 1715587705.121392 punch from 0x025531 to 0x02d32e (0x7dfe bytes) 000964 1715587705.131089 trunc from 0x034343 to 0x03c59e (0x825c bytes) 000965 1715587705.136028 fallocate 0x020bf0 thru 0x02cf3d (0xc34e bytes) 000966 1715587705.137466 trunc from 0x03c59f to 0x027070 (0x1552e bytes) 000967 1715587705.142223 MIRROR_EXTEND to 10 mirrors 000968 1715587705.268564 fallocate 0x01738c thru 0x01e499 (0x710e bytes) 000969 1715587705.278654 write 0x014830 thru 0x01cadc (0x82ad bytes) 000970 1715587705.284794 write 0x03acde thru 0x03ffff (0x5322 bytes) 000971 1715587705.287208 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000972 1715587705.342494 write 0x03e5d5 thru 0x03ffff (0x1a2b bytes) 000974 1715587705.345418 read 0x02ca2b thru 0x034df8 (0x83ce bytes) 000976 1715587705.348862 fallocate 0x006c1d thru 0x00c935 (0x5d19 bytes) 000977 1715587705.350775 fallocate 0x02114b thru 0x029ada (0x8990 bytes) 000978 1715587705.352542 mapread 0x034b35 thru 0x03ffff (0xb4cb bytes) 000980 1715587705.358121 MIRROR_EXTEND to 10 mirrors 000981 1715587705.486332 punch from 0x019bc8 to 0x01cd46 (0x317f bytes) 000982 1715587705.495893 punch from 0x0356e6 to 0x03ffff (0xa91a bytes) 000983 1715587705.498504 MIRROR_EXTEND to 11 mirrors 000985 1715587705.630848 trunc from 0x040000 to 0x033c6b (0xc394 bytes) 000986 1715587705.646176 trunc from 0x033c6c to 0x01cbe2 (0x17089 bytes) 000988 1715587705.652081 punch from 0x006bf1 to 0x008cdf (0x20ef bytes) 000989 1715587705.653720 write 0x0120be thru 0x01d5d0 (0xb513 bytes) 000990 1715587705.655762 read 0x00093d thru 0x0081cc (0x7890 bytes) 000991 1715587705.658030 mapread 0x012661 thru 0x01d5d0 (0xaf70 bytes) 000992 1715587705.660257 MIRROR_RESYNC 11 mirrors 000993 1715587706.175846 read 0x00f98a thru 0x01444e (0x4ac5 bytes) 000994 1715587706.177807 punch from 0x01ae5c to 0x01d5d0 (0x2775 bytes) 000995 1715587706.186360 punch from 0x00b184 to 0x011aab (0x6928 bytes) 000996 1715587706.188246 mapread 0x011ae0 thru 0x01bb50 (0xa071 bytes) 000997 1715587706.194092 fallocate 0x039d61 thru 0x03ffff (0x629f bytes) 000998 1715587706.195527 read 0x01fb1a thru 0x02b666 (0xbb4d bytes) 000999 1715587706.198477 write 0x01ca41 thru 0x025052 (0x8612 bytes) 001000 1715587706.200410 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) All operations completed A-OK! PASS 70a (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200a: stress test ===================== 04:08:28 (1715587708) Starting client: oleg329-client.virtnet: -o user_xattr,flock oleg329-server@tcp:/lustre /mnt/lustre2 Starting client: oleg329-client.virtnet: -o user_xattr,flock oleg329-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 210816 .. Extending file size to 5382176 .. failed Extending file size to 1241216 .. Extending file size to 7690336 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8395584 .. Extending file size to 7235488 .. Extending file size to 7852416 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8770272 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8834880 .. failed Extending file size to 8902432 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9166368 .. Extending file size to 9188128 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9316768 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 11760 11761 11762 11764 11766 done 192.168.203.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg329-client.virtnet /mnt/lustre2 (opts:) 192.168.203.129@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg329-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 04:09:34 (1715587774) Starting client: oleg329-client.virtnet: -o user_xattr,flock oleg329-server@tcp:/lustre /mnt/lustre2 Starting client: oleg329-client.virtnet: -o user_xattr,flock oleg329-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4363872 .. Extending file size to 1547328 .. done Extending file size to 6572832 .. Extending file size to 2334816 .. Extending file size to 6604096 .. Extending file size to 8057408 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6343904 .. Extending file size to 6600096 .. Extending file size to 7197248 .. Extending file size to 8864128 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7353440 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9353888 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9017824 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9148000 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 15312 15313 15314 15316 15318 done 192.168.203.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg329-client.virtnet /mnt/lustre2 (opts:) 192.168.203.129@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg329-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 04:10:39 (1715587839) Starting client: oleg329-client.virtnet: -o user_xattr,flock oleg329-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2512 1285176 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3308 1284380 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 21936 3585084 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 15968 3591052 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37904 7176136 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2512 1285176 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 3308 1284380 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 21936 3585084 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 15968 3591052 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37904 7176136 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0312138 s, 67.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0489871 s, 42.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg329-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 04:10:45 (1715587845) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0275606 s, 76.1 MB/s PASS 202 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 203: mirror file preserve mirror ID === 04:10:48 (1715587848) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } PASS 203 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 04:10:51 (1715587851) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0201436 s, 104 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xead:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xead:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xeae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 04:10:54 (1715587854) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0224777 s, 93.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xeae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xeaf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xeaf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xeae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xeaf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xeaf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xeb0:0x0] } PASS 204b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 04:10:58 (1715587858) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0170215 s, 123 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xeb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xeb1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xeb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xeb1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xeb2:0x0] } PASS 204c (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 04:11:22 (1715587882) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg329-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0130267 s, 80.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } PASS 204d (31s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 04:11:55 (1715587915) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg329-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0190141 s, 110 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (29s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 04:12:25 (1715587945) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg329-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool oleg329-server: Pool lustre.test_204f created Adding targets to pool oleg329-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg329-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0186122 s, 113 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg329-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg329-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg329-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (42s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 205: lfs mirror extend to set prefer and nocompr flags ========================================================== 04:13:09 (1715587989) PASS 205 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 04:13:12 (1715587992) oleg329-server: Pool lustre.test_206 created Adding targets to pool oleg329-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg329-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg329-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg329-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg329-server: Pool lustre.test_206 destroyed PASS 206 (14s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 04:13:28 (1715588008) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0157691 s, 66.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104236 s, 101 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x6ac4a5f Mirror 3: 0x6ac4a5f PASS 207 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 04:13:31 (1715588011) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 04:13:33 (1715588013) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 04:13:35 (1715588015) Starting client: oleg329-client.virtnet: -o user_xattr,flock oleg329-server@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.00755608 s, 0.5 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1870000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2240000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2240000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2240000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xc50000 Read this (4 bytes): BBBB 192.168.203.129@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg329-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 04:13:50 (1715588030) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00314309 s, 2.6 MB/s the very first read read with warm cache PASS 209b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210a: handle broken mirrored lovea ==== 04:13:53 (1715588033) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00964216 s, 109 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7e lmm_fid: [0x200000405:0x7e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7e lmm_fid: [0x200000405:0x7e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb8:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611262 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-05-13 04:13:53.000000000 -0400 Modify: 2024-05-13 04:13:53.000000000 -0400 Change: 2024-05-13 04:13:54.000000000 -0400 Birth: - PASS 210a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 04:13:57 (1715588037) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101 s, 104 MB/s oleg329-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg329-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg329-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg329-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg329-server: O/280000401/d25/b9: File not found by ext2_lookup PASS 210b (10s) debug_raw_pointers=0 debug_raw_pointers=0 == sanity-flr test complete, duration 824 sec ============ 04:14:07 (1715588047) === sanity-flr: start cleanup 04:14:07 (1715588047) === === sanity-flr: finish cleanup 04:14:08 (1715588048) ===