-----============= acceptance-small: sanity-flr ============----- Fri May 10 08:07:36 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 6 201 44c 49a === sanity-flr: start setup 08:07:40 (1715342860) === oleg205-client.virtnet: executing check_config_client /mnt/lustre oleg205-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg205-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b4174000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b4174000.idle_timeout=debug disable quota as required oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 08:07:47 (1715342867) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f6869] debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0a: lfs mirror create with -N option == 08:07:48 (1715342868) PASS 0a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 08:07:53 (1715342873) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 08:07:56 (1715342876) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0d: lfs mirror extend with -N option == 08:07:58 (1715342878) PASS 0d (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 08:08:04 (1715342884) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 08:08:06 (1715342886) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0g: lfs mirror create flags support === 08:08:09 (1715342889) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b4174000.stats=clear osc.lustre-OST0001-osc-ffff8800b4174000.stats=clear PASS 0g (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 08:08:12 (1715342892) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 08:08:16 (1715342896) PASS 0j (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 08:08:20 (1715342900) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 1: create components with setstripe options ========================================================== 08:08:22 (1715342902) lfs mirror mirror: cannot get UNLOCK lease, ext 4: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 2: create components from existing files ========================================================== 08:08:26 (1715342906) PASS 2 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 3: create components from files located on different MDTs ========================================================== 08:08:30 (1715342910) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 08:08:32 (1715342912) lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 08:08:36 (1715342916) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (1s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 7: Create mirror with DoM component === 08:08:40 (1715342920) PASS 7 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 08:08:43 (1715342923) 13+0 records in 13+0 records out 13631488 bytes (14 MB) copied, 0.491747 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0419685 s, 25.0 MB/s PASS 21 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 08:08:48 (1715342928) 19+0 records in 19+0 records out 19922944 bytes (20 MB) copied, 0.118601 s, 168 MB/s PASS 22 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 08:08:52 (1715342932) Stopping /mnt/lustre-ost1 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b4174000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b4174000.ost_server_uuid in DISCONN state after 1 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg205-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b4174000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b4174000.ost_server_uuid in DISCONN state after 3 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg205-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 08:09:17 (1715342957) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0354753 s, 59.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b4174000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b4174000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg205-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (9s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33a: read can choose available mirror to read ========================================================== 08:09:27 (1715342967) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b4174000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b4174000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg205-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b4174000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b4174000.ost_server_uuid in DISCONN state after 4 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg205-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 08:09:48 (1715342988) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.839655 s, 62.4 MB/s -rw-r--r-- 1 root root 52428800 May 10 08:09 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.222s user 0m0.000s sys 0m0.165s Stopping /mnt/lustre-ost1 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b4174000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b4174000.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... real 0m0.186s user 0m0.000s sys 0m0.139s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg205-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b4174000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b4174000.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost1)... real 0m0.210s user 0m0.000s sys 0m0.170s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg205-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg205-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b4174000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b4174000.ost_server_uuid in FULL state after 0 sec PASS 33b (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 08:10:14 (1715343014) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 08:10:16 (1715343016) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 08:10:18 (1715343018) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 35: allow to write to mirrored files == 08:10:20 (1715343020) PASS 35 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36a: write to mirrored files ========== 08:10:23 (1715343023) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.120571 s, 104 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 08:10:27 (1715343027) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00719939 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00782431 s, 134 MB/s PASS 36b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 08:10:30 (1715343030) PASS 36c (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 08:10:34 (1715343034) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000688443 s, 1.5 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000458268 s, 2.2 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000689052 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000756708 s, 1.4 MB/s flv=10 olv=10 PASS 36d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 37: mirror I/O API verification ======= 08:10:38 (1715343038) create test files with size 15753202 .. 8a54a1bc52ff666d8c5e3df2f85abe66 - 3ab93b19f4dd8c6c45f9de8111989217 - 4902dd5c9b3ec0bf82e424b9883ff05e - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg205-client.virtnet /mnt/lustre (opts:) Starting client: oleg205-client.virtnet: -o user_xattr,flock oleg205-server@tcp:/lustre /mnt/lustre PASS 37 (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 38: resync ============================ 08:10:44 (1715343044) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 08:10:48 (1715343048) PASS 39 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 08:10:52 (1715343052) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0247454 s, 42.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x47:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00865102 s, 121 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x48:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 41: lfs mirror resync check =========== 08:10:55 (1715343055) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0241889 s, 86.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0752937 s, 55.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0414336 s, 50.6 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 42: lfs mirror verify ================= 08:10:59 (1715343059) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43a: mirror pick on write ============= 08:11:01 (1715343061) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 08:11:04 (1715343064) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0080695 s, 130 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44a: lfs mirror split check =========== 08:11:07 (1715343067) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44b: mirror split does not create stale file ========================================================== 08:11:09 (1715343069) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x50:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 08:11:12 (1715343072) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0606905 s, 173 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg205-client.virtnet /mnt/lustre (opts:) Starting client: oleg205-client.virtnet: -o user_xattr,flock oleg205-server@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 08:11:16 (1715343076) llite.lustre-ffff8800aa560000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0741652 s, 141 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg205-client.virtnet /mnt/lustre (opts:) Starting client: oleg205-client.virtnet: -o user_xattr,flock oleg205-server@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800b4177000.xattr_cache=1 PASS 44e (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 08:11:21 (1715343081) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (6s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 46: Verify setstripe --copy option ==== 08:11:29 (1715343089) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 47: Verify mirror obj alloc =========== 08:11:35 (1715343095) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 48: Verify snapshot mirror ============ 08:11:38 (1715343098) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0511761 s, 61.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.048604 s, 64.7 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 9c2822067f0ee9cf0525dd0c4a60fa87 - mirror 1 checksum: 2f7c40aac6d092dee75b8cb9e9b5fbf6 - mirror 2 checksum: 9c2822067f0ee9cf0525dd0c4a60fa87 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri May 10 08:11:40 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri May 10 08:11:40 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 9c2822067f0ee9cf0525dd0c4a60fa87 - mirror 1 checksum: 2f7c40aac6d092dee75b8cb9e9b5fbf6 - mirror 2 checksum: 2f7c40aac6d092dee75b8cb9e9b5fbf6 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri May 10 08:11:40 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri May 10 08:11:40 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } PASS 48 (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50A: mirror split update layout generation ========================================================== 08:11:42 (1715343102) Failing mds1 on oleg205-server Stopping /mnt/lustre-mds1 (opts:) on oleg205-server 08:11:49 (1715343109) shut down Failover mds1 to oleg205-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-MDT0000 08:12:01 (1715343121) targets are mounted 08:12:01 (1715343121) facet_failover done oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00654981 s, 160 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x14 lmm_fid: [0x200000404:0x14:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 50A (29s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 08:12:12 (1715343132) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50b: mirror rsync handles sparseness == 08:12:15 (1715343135) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 08:12:17 (1715343137) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00215779 s, 7.6 MB/s PASS 50c (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50d: mirror rsync keep holes ========== 08:12:20 (1715343140) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 08:12:23 (1715343143) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 08:12:25 (1715343145) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 60b (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 08:12:30 (1715343150) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 08:12:50 (1715343170) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 08:13:08 (1715343188) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 08:13:27 (1715343207) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70: mirror create and split race ====== 08:13:30 (1715343210) mirror create pid 26682 mirror split pid 26683 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4716: 26682 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4716: 26683 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70a: flr mode fsx test ================ 08:14:33 (1715343273) Seed set to 3274 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000003 1715343274.085304 MIRROR_EXTEND to 2 mirrors skipping zero size read 000008 1715343274.154351 mapwrite 0x013ae8 thru 0x01b432 (0x794b bytes) 000010 1715343274.203658 mapwrite 0x022cab thru 0x02573a (0x2a90 bytes) truncating to largest ever: 0x1db9e 000013 1715343274.226093 trunc from 0x02573b to 0x01db9d (0x7b9d bytes) 000014 1715343274.230224 mapwrite 0x03f2c8 thru 0x03ffff (0xd38 bytes) 000015 1715343274.249968 mapwrite 0x004b18 thru 0x00b370 (0x6859 bytes) truncating to largest ever: 0x220b1 000016 1715343274.268731 trunc from 0x040000 to 0x0220b0 (0x1df4f bytes) 000017 1715343274.273103 trunc from 0x0220b1 to 0x000c4c (0x21464 bytes) 000020 1715343274.277427 read 0x0002bb thru 0x000c4c (0x992 bytes) 000022 1715343274.279228 mapwrite 0x009a87 thru 0x0158ae (0xbe28 bytes) 000023 1715343274.304095 read 0x0155ea thru 0x0158ae (0x2c5 bytes) 000024 1715343274.304595 mapread 0x00011b thru 0x00f772 (0xf658 bytes) 000028 1715343274.307252 read 0x00606a thru 0x00fee8 (0x9e7f bytes) 000029 1715343274.307519 write 0x0277cf thru 0x02e430 (0x6c62 bytes) truncating to largest ever: 0x2f225 000031 1715343274.308316 trunc from 0x02e431 to 0x02f224 (0xdf4 bytes) 000032 1715343274.311979 trunc from 0x02f225 to 0x012ab9 (0x1c76b bytes) 000033 1715343274.315731 mapread 0x00fb06 thru 0x012ab9 (0x2fb4 bytes) truncating to largest ever: 0x349b3 000034 1715343274.316403 trunc from 0x012aba to 0x0349b2 (0x21ef9 bytes) 000036 1715343274.319882 read 0x018a70 thru 0x0233a2 (0xa933 bytes) 000040 1715343274.324083 write 0x03e4a0 thru 0x03ffff (0x1b60 bytes) 000042 1715343274.324731 write 0x02d48b thru 0x02f8be (0x2434 bytes) 000045 1715343274.325569 read 0x035356 thru 0x03ea06 (0x96b1 bytes) 000046 1715343274.327862 read 0x036bee thru 0x03e965 (0x7d78 bytes) 000048 1715343274.328323 write 0x031347 thru 0x03ffff (0xecb9 bytes) 000049 1715343274.328937 write 0x01bde6 thru 0x01cba0 (0xdbb bytes) 000050 1715343274.329349 mapwrite 0x03232f thru 0x0363fa (0x40cc bytes) 000052 1715343274.349294 MIRROR_RESYNC 2 mirrors 000055 1715343274.444518 read 0x01aeb7 thru 0x01d4ad (0x25f7 bytes) 000056 1715343274.446413 read 0x014eca thru 0x0233b2 (0xe4e9 bytes) 000057 1715343274.450540 trunc from 0x040000 to 0x0197c0 (0x2683f bytes) 000059 1715343274.472375 write 0x0091f1 thru 0x01537f (0xc18f bytes) 000060 1715343274.475483 write 0x037f9e thru 0x03ffff (0x8062 bytes) 000063 1715343274.476660 mapwrite 0x0016af thru 0x009720 (0x8072 bytes) 000065 1715343274.502230 write 0x022535 thru 0x0243c7 (0x1e93 bytes) 000066 1715343274.505244 mapwrite 0x03e0ff thru 0x03ffff (0x1f01 bytes) 000067 1715343274.520487 read 0x013116 thru 0x01f459 (0xc344 bytes) 000069 1715343274.525011 mapread 0x009c21 thru 0x017b37 (0xdf17 bytes) 000070 1715343274.525538 mapread 0x03da9d thru 0x03ffff (0x2563 bytes) 000071 1715343274.525923 read 0x01691f thru 0x018911 (0x1ff3 bytes) 000076 1715343274.527309 mapwrite 0x0206c5 thru 0x0272c1 (0x6bfd bytes) 000078 1715343274.545490 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000079 1715343274.580296 read 0x025200 thru 0x02bcc2 (0x6ac3 bytes) 000084 1715343274.584315 mapwrite 0x02cdf8 thru 0x02d24a (0x453 bytes) 000085 1715343274.601224 read 0x0385b2 thru 0x03ffff (0x7a4e bytes) 000087 1715343274.607000 mapread 0x01ca5e thru 0x026331 (0x98d4 bytes) 000088 1715343274.609143 MIRROR_EXTEND to 2 mirrors 000089 1715343274.721970 trunc from 0x040000 to 0x030c76 (0xf389 bytes) 000092 1715343274.742495 write 0x03eefc thru 0x03ffff (0x1104 bytes) 000095 1715343274.743509 write 0x002480 thru 0x00ff8c (0xdb0d bytes) 000097 1715343274.746380 mapwrite 0x01af87 thru 0x0207d5 (0x584f bytes) 000100 1715343274.770020 MIRROR_EXTEND to 3 mirrors 000102 1715343274.877099 mapread 0x011464 thru 0x01f946 (0xe4e3 bytes) 000103 1715343274.885146 mapread 0x02889f thru 0x02a535 (0x1c97 bytes) 000104 1715343274.887793 mapwrite 0x016ee7 thru 0x01de44 (0x6f5e bytes) 000106 1715343274.933175 mapread 0x0051db thru 0x00e1db (0x9001 bytes) 000107 1715343274.939419 MIRROR_RESYNC 3 mirrors 000109 1715343275.062032 trunc from 0x040000 to 0x017ea4 (0x2815b bytes) 000110 1715343275.079720 mapread 0x001ef9 thru 0x011b00 (0xfc08 bytes) 000111 1715343275.086370 mapread 0x010e70 thru 0x017ea4 (0x7035 bytes) 000113 1715343275.087446 mapwrite 0x03c065 thru 0x03ffff (0x3f9b bytes) 000115 1715343275.111558 mapwrite 0x01852b thru 0x0232dc (0xadb2 bytes) 000117 1715343275.136204 read 0x00c511 thru 0x0111c0 (0x4cb0 bytes) 000118 1715343275.136650 mapwrite 0x0235c2 thru 0x027e69 (0x48a8 bytes) 000120 1715343275.153070 read 0x03e932 thru 0x03ffff (0x16ce bytes) 000121 1715343275.153364 trunc from 0x040000 to 0x00fff6 (0x30009 bytes) 000122 1715343275.158237 mapwrite 0x01adbb thru 0x022663 (0x78a9 bytes) 000123 1715343275.184506 mapwrite 0x01c986 thru 0x02a351 (0xd9cc bytes) 000124 1715343275.211675 read 0x01ae51 thru 0x0267bc (0xb96c bytes) 000127 1715343275.212749 mapread 0x01a944 thru 0x025a5c (0xb119 bytes) 000128 1715343275.213275 write 0x00c188 thru 0x016501 (0xa37a bytes) 000129 1715343275.213980 mapread 0x00cc22 thru 0x01c338 (0xf717 bytes) 000132 1715343275.215174 write 0x002f76 thru 0x0037e0 (0x86b bytes) 000133 1715343275.215751 trunc from 0x02a352 to 0x0135e4 (0x16d6d bytes) 000134 1715343275.220208 mapread 0x00862b thru 0x012ccf (0xa6a5 bytes) 000135 1715343275.220675 mapread 0x010693 thru 0x0135e4 (0x2f52 bytes) 000136 1715343275.221089 trunc from 0x0135e5 to 0x00378f (0xfe55 bytes) 000139 1715343275.225911 mapread 0x001022 thru 0x003241 (0x2220 bytes) 000141 1715343275.228258 MIRROR_EXTEND to 4 mirrors 000142 1715343275.334273 mapwrite 0x029a16 thru 0x029b59 (0x144 bytes) 000144 1715343275.368376 trunc from 0x029b5a to 0x0165c7 (0x13592 bytes) 000147 1715343275.374438 trunc from 0x0165c8 to 0x02f90e (0x19347 bytes) 000148 1715343275.378364 write 0x034007 thru 0x037f1f (0x3f19 bytes) 000149 1715343275.379156 mapwrite 0x03cc37 thru 0x03ffff (0x33c9 bytes) 000150 1715343275.402510 trunc from 0x040000 to 0x02706a (0x18f95 bytes) 000151 1715343275.406558 trunc from 0x02706b to 0x01fe21 (0x7249 bytes) 000153 1715343275.410091 trunc from 0x01fe22 to 0x01ace2 (0x513f bytes) 000155 1715343275.414403 write 0x0364e9 thru 0x03f1a2 (0x8cba bytes) 000157 1715343275.415575 mapwrite 0x00be4e thru 0x018d42 (0xcef5 bytes) 000158 1715343275.439324 mapwrite 0x0003ed thru 0x00ffb2 (0xfbc6 bytes) 000161 1715343275.457547 mapread 0x0077e6 thru 0x00d5a8 (0x5dc3 bytes) 000162 1715343275.459387 MIRROR_EXTEND to 5 mirrors 000163 1715343275.580695 trunc from 0x03f1a3 to 0x00acce (0x344d4 bytes) 000164 1715343275.600924 MIRROR_EXTEND to 6 mirrors 000165 1715343275.721221 mapread 0x007669 thru 0x00acce (0x3666 bytes) 000166 1715343275.726227 trunc from 0x00accf to 0x014975 (0x9ca7 bytes) 000167 1715343275.743780 trunc from 0x014976 to 0x01af71 (0x65fc bytes) 000168 1715343275.749110 mapwrite 0x01fd99 thru 0x02135e (0x15c6 bytes) 000169 1715343275.769723 read 0x01fb53 thru 0x02135e (0x180c bytes) 000170 1715343275.771257 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000173 1715343275.821068 write 0x00067d thru 0x00a1c4 (0x9b48 bytes) 000175 1715343275.824857 mapread 0x006053 thru 0x0117ad (0xb75b bytes) 000177 1715343275.830042 mapread 0x01c16c thru 0x02135e (0x51f3 bytes) 000178 1715343275.833413 mapread 0x004e9f thru 0x01349b (0xe5fd bytes) 000179 1715343275.834162 write 0x0386a0 thru 0x03dce5 (0x5646 bytes) 000182 1715343275.835902 trunc from 0x03dce6 to 0x0058f4 (0x383f1 bytes) 000183 1715343275.841145 mapread 0x005872 thru 0x0058f4 (0x83 bytes) 000185 1715343275.841847 mapread 0x003a98 thru 0x0058f4 (0x1e5d bytes) 000186 1715343275.842266 mapwrite 0x019350 thru 0x024fc4 (0xbc75 bytes) 000189 1715343275.871612 mapread 0x007c93 thru 0x012fd6 (0xb344 bytes) 000190 1715343275.875716 mapwrite 0x0206c0 thru 0x02d974 (0xd2b5 bytes) 000191 1715343275.899712 read 0x026cba thru 0x02d974 (0x6cbb bytes) 000193 1715343275.900615 write 0x03fed0 thru 0x03ffff (0x130 bytes) truncating to largest ever: 0x3a0c9 000194 1715343275.904979 trunc from 0x040000 to 0x03a0c8 (0x5f37 bytes) 000198 1715343275.910461 mapread 0x0157c7 thru 0x01c489 (0x6cc3 bytes) truncating to largest ever: 0x3a4bb 000200 1715343275.911602 trunc from 0x03a0c9 to 0x03a4ba (0x3f2 bytes) 000203 1715343275.916023 mapread 0x02000e thru 0x021c7d (0x1c70 bytes) 000205 1715343275.916945 read 0x02f5ca thru 0x03a4ba (0xaef1 bytes) 000207 1715343275.920243 write 0x01d25d thru 0x0292fb (0xc09f bytes) 000208 1715343275.920984 trunc from 0x03a4bb to 0x026534 (0x13f86 bytes) 000210 1715343275.926265 mapread 0x0149ef thru 0x01f95b (0xaf6d bytes) 000212 1715343275.927364 mapread 0x012afc thru 0x01a7eb (0x7cf0 bytes) 000214 1715343275.928119 read 0x005d78 thru 0x011a90 (0xbd19 bytes) 000215 1715343275.929895 MIRROR_EXTEND to 6 mirrors 000217 1715343276.058721 trunc from 0x026535 to 0x01753f (0xeff5 bytes) 000221 1715343276.077684 mapread 0x00188e thru 0x00f010 (0xd783 bytes) 000222 1715343276.086510 MIRROR_RESYNC 6 mirrors 000224 1715343276.290514 mapwrite 0x023db7 thru 0x02c200 (0x844a bytes) 000225 1715343276.335328 read 0x013912 thru 0x023889 (0xff78 bytes) 000227 1715343276.340279 MIRROR_EXTEND to 7 mirrors 000228 1715343276.455827 mapwrite 0x01a9f2 thru 0x024b28 (0xa137 bytes) 000229 1715343276.507810 mapread 0x01d66b thru 0x028100 (0xaa96 bytes) 000231 1715343276.510138 MIRROR_EXTEND to 8 mirrors 000232 1715343276.627127 read 0x00427d thru 0x004809 (0x58d bytes) 000233 1715343276.629987 write 0x029aef thru 0x02edc5 (0x52d7 bytes) 000235 1715343276.648958 trunc from 0x02edc6 to 0x039bd5 (0xae10 bytes) 000238 1715343276.660837 mapwrite 0x023e58 thru 0x026938 (0x2ae1 bytes) 000239 1715343276.683166 read 0x019bfc thru 0x01c0cd (0x24d2 bytes) 000240 1715343276.687482 mapwrite 0x00e6f3 thru 0x010a9b (0x23a9 bytes) 000241 1715343276.702976 trunc from 0x039bd6 to 0x035284 (0x4951 bytes) 000242 1715343276.707656 mapread 0x0160a7 thru 0x019187 (0x30e1 bytes) 000243 1715343276.708487 trunc from 0x035285 to 0x0376e7 (0x2463 bytes) 000244 1715343276.712252 read 0x01e574 thru 0x020c72 (0x26ff bytes) 000246 1715343276.712965 read 0x0013cf thru 0x00a039 (0x8c6b bytes) 000247 1715343276.714680 MIRROR_RESYNC 8 mirrors 000248 1715343276.964695 mapread 0x02b3f1 thru 0x0359d5 (0xa5e5 bytes) 000253 1715343276.972302 read 0x0131a4 thru 0x01428a (0x10e7 bytes) 000256 1715343276.974508 mapwrite 0x014af8 thru 0x01f165 (0xa66e bytes) 000258 1715343277.027428 MIRROR_RESYNC 8 mirrors 000259 1715343277.264332 trunc from 0x0376e8 to 0x027b95 (0xfb52 bytes) 000261 1715343277.282747 read 0x00ba59 thru 0x01b6dc (0xfc84 bytes) 000262 1715343277.286214 mapwrite 0x00e417 thru 0x018fad (0xab97 bytes) 000264 1715343277.307706 write 0x003ded thru 0x00a7cc (0x69e0 bytes) 000267 1715343277.312500 MIRROR_EXTEND to 9 mirrors 000268 1715343277.444266 write 0x03be9c thru 0x03ffff (0x4164 bytes) 000269 1715343277.460423 mapwrite 0x004372 thru 0x0137d9 (0xf468 bytes) 000272 1715343277.486078 trunc from 0x040000 to 0x001979 (0x3e686 bytes) 000273 1715343277.496210 read 0x0018df thru 0x001979 (0x9b bytes) 000274 1715343277.499283 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000275 1715343277.557917 MIRROR_EXTEND to 9 mirrors 000276 1715343277.654056 MIRROR_RESYNC 9 mirrors 000278 1715343277.865865 trunc from 0x00197a to 0x007638 (0x5cbf bytes) 000282 1715343277.887198 mapwrite 0x0153ea thru 0x01b46a (0x6081 bytes) 000283 1715343277.914734 read 0x011c4a thru 0x0139e1 (0x1d98 bytes) 000286 1715343277.917419 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000287 1715343277.970340 trunc from 0x01b46b to 0x0292a2 (0xde38 bytes) 000290 1715343277.975557 trunc from 0x0292a3 to 0x00a920 (0x1e982 bytes) 000298 1715343277.981513 write 0x004572 thru 0x005696 (0x1125 bytes) 000299 1715343277.986193 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000300 1715343278.035780 read 0x001339 thru 0x009dc4 (0x8a8c bytes) 000301 1715343278.038554 mapread 0x009bd7 thru 0x00a920 (0xd4a bytes) 000305 1715343278.041681 mapread 0x00455d thru 0x00a920 (0x63c4 bytes) 000306 1715343278.042378 mapread 0x000422 thru 0x0039c5 (0x35a4 bytes) 000309 1715343278.044704 trunc from 0x00a921 to 0x03178a (0x26e6a bytes) 000313 1715343278.049502 read 0x0042d7 thru 0x013b18 (0xf842 bytes) 000314 1715343278.053282 write 0x02cc3d thru 0x032042 (0x5406 bytes) 000315 1715343278.055664 MIRROR_RESYNC 7 mirrors 000317 1715343278.270294 MIRROR_RESYNC 7 mirrors 000318 1715343278.318574 mapread 0x02f934 thru 0x032042 (0x270f bytes) 000319 1715343278.324321 MIRROR_RESYNC 7 mirrors 000320 1715343278.375788 write 0x00db0e thru 0x014abf (0x6fb2 bytes) 000321 1715343278.393755 mapread 0x02a763 thru 0x02b850 (0x10ee bytes) 000322 1715343278.396878 write 0x00a7c4 thru 0x019566 (0xeda3 bytes) 000323 1715343278.400918 mapwrite 0x020578 thru 0x02dd4b (0xd7d4 bytes) 000325 1715343278.422369 read 0x01c9c0 thru 0x02b6b3 (0xecf4 bytes) 000328 1715343278.423484 mapwrite 0x002af1 thru 0x00697a (0x3e8a bytes) 000330 1715343278.439842 mapwrite 0x03af3a thru 0x03ffff (0x50c6 bytes) 000331 1715343278.465372 trunc from 0x040000 to 0x00fed7 (0x30128 bytes) 000332 1715343278.469217 trunc from 0x00fed8 to 0x011c5f (0x1d88 bytes) 000333 1715343278.472475 read 0x00fc59 thru 0x011709 (0x1ab1 bytes) 000335 1715343278.476262 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000336 1715343278.523808 mapwrite 0x004b8b thru 0x004ed1 (0x347 bytes) 000337 1715343278.540925 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000338 1715343278.586293 trunc from 0x011c60 to 0x031583 (0x1f924 bytes) 000339 1715343278.590212 mapwrite 0x03c4d8 thru 0x03ffff (0x3b28 bytes) 000341 1715343278.613750 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000345 1715343278.661109 trunc from 0x040000 to 0x010cfd (0x2f302 bytes) 000347 1715343278.664948 trunc from 0x010cfe to 0x013b43 (0x2e46 bytes) 000348 1715343278.668118 trunc from 0x013b44 to 0x01f753 (0xbc10 bytes) 000349 1715343278.671111 mapwrite 0x0241c3 thru 0x02a6a0 (0x64de bytes) 000350 1715343278.695413 trunc from 0x02a6a1 to 0x00c008 (0x1e698 bytes) 000351 1715343278.699480 write 0x003434 thru 0x00bdbf (0x898c bytes) 000352 1715343278.702468 read 0x00917c thru 0x00c008 (0x2e8d bytes) 000354 1715343278.703370 mapwrite 0x033d80 thru 0x0359ef (0x1c70 bytes) 000356 1715343278.726709 read 0x019718 thru 0x01d742 (0x402b bytes) 000359 1715343278.731311 mapwrite 0x004751 thru 0x010ba9 (0xc459 bytes) 000360 1715343278.748048 read 0x01e75c thru 0x01ed10 (0x5b5 bytes) 000361 1715343278.748357 read 0x001531 thru 0x00b690 (0xa160 bytes) 000362 1715343278.748792 mapread 0x0060b6 thru 0x008424 (0x236f bytes) 000365 1715343278.749819 mapwrite 0x001165 thru 0x00fc13 (0xeaaf bytes) 000366 1715343278.768968 MIRROR_EXTEND to 5 mirrors 000367 1715343278.877960 read 0x01c22a thru 0x024a04 (0x87db bytes) 000368 1715343278.881320 mapread 0x027664 thru 0x02ca92 (0x542f bytes) 000369 1715343278.887706 trunc from 0x0359f0 to 0x0164d6 (0x1f519 bytes) 000370 1715343278.910331 MIRROR_RESYNC 5 mirrors 000372 1715343279.075146 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000375 1715343279.133409 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000377 1715343279.189893 MIRROR_RESYNC 3 mirrors 000378 1715343279.225618 read 0x010d81 thru 0x011f7c (0x11fc bytes) 000383 1715343279.228431 read 0x00c07e thru 0x0164d6 (0xa459 bytes) 000384 1715343279.231081 mapread 0x00ab8a thru 0x00d8ca (0x2d41 bytes) truncating to largest ever: 0x3c155 000386 1715343279.231772 trunc from 0x0164d7 to 0x03c154 (0x25c7e bytes) 000387 1715343279.251082 MIRROR_RESYNC 3 mirrors 000388 1715343279.379614 mapwrite 0x023827 thru 0x02cb2a (0x9304 bytes) 000390 1715343279.428626 mapread 0x007770 thru 0x00c2f8 (0x4b89 bytes) 000391 1715343279.431625 trunc from 0x03c155 to 0x008ba0 (0x335b4 bytes) 000393 1715343279.437525 MIRROR_EXTEND to 4 mirrors 000394 1715343279.538275 mapwrite 0x0096ab thru 0x00e390 (0x4ce6 bytes) 000395 1715343279.578129 MIRROR_EXTEND to 5 mirrors 000396 1715343279.677134 read 0x00590d thru 0x00de61 (0x8555 bytes) 000397 1715343279.680235 write 0x02dca1 thru 0x02fedf (0x223f bytes) 000398 1715343279.694275 trunc from 0x02fee0 to 0x03340f (0x3530 bytes) 000400 1715343279.702468 trunc from 0x033410 to 0x02ab63 (0x88ac bytes) 000401 1715343279.706089 write 0x03e043 thru 0x03fee4 (0x1ea2 bytes) 000405 1715343279.707200 mapread 0x0203cb thru 0x02c3a6 (0xbfdc bytes) 000406 1715343279.714426 read 0x00ea6f thru 0x01a7e7 (0xbd79 bytes) 000408 1715343279.717917 trunc from 0x03fee5 to 0x00e57d (0x31967 bytes) 000410 1715343279.721834 write 0x02dde9 thru 0x03b1f5 (0xd40d bytes) 000413 1715343279.724190 MIRROR_EXTEND to 6 mirrors 000415 1715343279.828280 mapread 0x0388ee thru 0x03b1f5 (0x2908 bytes) 000417 1715343279.832655 write 0x02672c thru 0x028728 (0x1ffd bytes) 000419 1715343279.852996 MIRROR_EXTEND to 7 mirrors 000420 1715343279.971652 read 0x027413 thru 0x0301ee (0x8ddc bytes) 000425 1715343279.980296 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000426 1715343280.033929 write 0x018064 thru 0x027fb2 (0xff4f bytes) 000428 1715343280.053228 trunc from 0x03b1f6 to 0x021258 (0x19f9d bytes) 000429 1715343280.062537 write 0x0204e2 thru 0x029b01 (0x9620 bytes) truncating to largest ever: 0x3e1ce 000431 1715343280.063559 trunc from 0x029b02 to 0x03e1cd (0x146cc bytes) 000432 1715343280.067389 mapwrite 0x013fd8 thru 0x014bbc (0xbe5 bytes) 000433 1715343280.088516 MIRROR_EXTEND to 7 mirrors 000435 1715343280.211641 trunc from 0x03e1ce to 0x004aa1 (0x3972c bytes) 000436 1715343280.229400 read 0x0026a3 thru 0x004aa1 (0x23ff bytes) 000437 1715343280.231185 write 0x02f7bd thru 0x0382c0 (0x8b04 bytes) 000438 1715343280.232282 mapwrite 0x032d08 thru 0x03ffff (0xd2f8 bytes) 000443 1715343280.261531 trunc from 0x040000 to 0x0125fb (0x2da04 bytes) 000445 1715343280.266387 read 0x012024 thru 0x0125fb (0x5d8 bytes) 000446 1715343280.269472 mapwrite 0x028dce thru 0x038b2d (0xfd60 bytes) 000447 1715343280.293899 read 0x02b47a thru 0x02b9a1 (0x528 bytes) 000448 1715343280.294210 write 0x00fb78 thru 0x01c35e (0xc7e7 bytes) 000451 1715343280.296619 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000452 1715343280.349450 mapread 0x01677a thru 0x01ef7a (0x8801 bytes) 000453 1715343280.354482 mapwrite 0x03c6e1 thru 0x03ffff (0x391f bytes) 000454 1715343280.374327 read 0x020c71 thru 0x02a528 (0x98b8 bytes) 000456 1715343280.377652 trunc from 0x040000 to 0x036905 (0x96fa bytes) 000458 1715343280.381820 trunc from 0x036906 to 0x028552 (0xe3b3 bytes) 000459 1715343280.385394 trunc from 0x028553 to 0x02008b (0x84c7 bytes) 000461 1715343280.389575 trunc from 0x02008c to 0x034a5e (0x149d3 bytes) 000464 1715343280.393528 trunc from 0x034a5f to 0x00e71d (0x26341 bytes) 000465 1715343280.397157 read 0x0007f1 thru 0x009e48 (0x9658 bytes) 000466 1715343280.400105 mapwrite 0x00439f thru 0x01139e (0xd000 bytes) 000469 1715343280.422255 trunc from 0x01139f to 0x00aa5a (0x6944 bytes) 000470 1715343280.426247 mapwrite 0x029a33 thru 0x02ba28 (0x1ff6 bytes) 000472 1715343280.450828 MIRROR_RESYNC 6 mirrors 000473 1715343280.643978 write 0x004854 thru 0x00698d (0x213a bytes) 000474 1715343280.660041 mapread 0x01e02e thru 0x025aeb (0x7abe bytes) 000476 1715343280.664837 MIRROR_EXTEND to 7 mirrors 000479 1715343280.776371 read 0x021b88 thru 0x02ba28 (0x9ea1 bytes) 000481 1715343280.780508 read 0x01af6e thru 0x01d703 (0x2796 bytes) 000482 1715343280.784162 write 0x013f7b thru 0x0171c9 (0x324f bytes) 000483 1715343280.803296 MIRROR_RESYNC 7 mirrors 000484 1715343280.996280 write 0x024803 thru 0x028ce0 (0x44de bytes) 000486 1715343281.011933 read 0x027990 thru 0x02ba28 (0x4099 bytes) 000487 1715343281.013828 read 0x018dce thru 0x020cdb (0x7f0e bytes) 000488 1715343281.017161 mapwrite 0x016e75 thru 0x0184b3 (0x163f bytes) 000489 1715343281.033499 trunc from 0x02ba29 to 0x01fd1e (0xbd0a bytes) 000490 1715343281.042752 trunc from 0x01fd1f to 0x01100d (0xed11 bytes) 000492 1715343281.046728 write 0x0304d5 thru 0x0351eb (0x4d17 bytes) 000494 1715343281.047893 trunc from 0x0351ec to 0x005a03 (0x2f7e8 bytes) 000495 1715343281.051215 write 0x03b31e thru 0x03ffff (0x4ce2 bytes) 000496 1715343281.051754 mapwrite 0x03e2ae thru 0x03ffff (0x1d52 bytes) 000497 1715343281.066175 mapwrite 0x00d6ad thru 0x015a3d (0x8391 bytes) 000498 1715343281.093460 write 0x0332e4 thru 0x03ffff (0xcd1c bytes) 000499 1715343281.096114 MIRROR_RESYNC 7 mirrors 000501 1715343281.273950 mapwrite 0x026d3e thru 0x031373 (0xa636 bytes) 000502 1715343281.322795 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000503 1715343281.367375 mapwrite 0x01ce54 thru 0x024200 (0x73ad bytes) 000504 1715343281.389242 trunc from 0x040000 to 0x0328ec (0xd713 bytes) 000507 1715343281.397305 read 0x015233 thru 0x017126 (0x1ef4 bytes) 000508 1715343281.398973 trunc from 0x0328ed to 0x019712 (0x191da bytes) 000509 1715343281.402959 mapread 0x008259 thru 0x0170a2 (0xee4a bytes) 000511 1715343281.407670 read 0x00b5be thru 0x0174a7 (0xbeea bytes) 000516 1715343281.409135 write 0x039dde thru 0x03eb8d (0x4db0 bytes) 000517 1715343281.409855 mapread 0x02dd4d thru 0x03384a (0x5afe bytes) 000518 1715343281.413232 mapread 0x0194e5 thru 0x023cfa (0xa816 bytes) 000523 1715343281.415427 write 0x0388ae thru 0x03ffff (0x7752 bytes) 000524 1715343281.417732 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000526 1715343281.464811 mapwrite 0x034d34 thru 0x03f948 (0xac15 bytes) 000527 1715343281.488121 mapwrite 0x01ae6f thru 0x026124 (0xb2b6 bytes) 000530 1715343281.510594 mapwrite 0x00d799 thru 0x012450 (0x4cb8 bytes) 000538 1715343281.527501 write 0x01f481 thru 0x0215f4 (0x2174 bytes) 000539 1715343281.527988 trunc from 0x040000 to 0x0008d3 (0x3f72c bytes) 000540 1715343281.532670 write 0x01df04 thru 0x02d75d (0xf85a bytes) 000541 1715343281.535079 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000543 1715343281.581879 mapwrite 0x0106bc thru 0x018a03 (0x8348 bytes) 000544 1715343281.604641 write 0x0196c8 thru 0x029002 (0xf93b bytes) 000546 1715343281.606882 mapwrite 0x00e83f thru 0x014db1 (0x6573 bytes) 000547 1715343281.622180 mapwrite 0x00f9b4 thru 0x017a69 (0x80b6 bytes) 000550 1715343281.635721 mapread 0x017d02 thru 0x018cf3 (0xff2 bytes) 000551 1715343281.636122 write 0x01949f thru 0x024f6b (0xbacd bytes) 000553 1715343281.636934 mapread 0x004b86 thru 0x00b624 (0x6a9f bytes) 000556 1715343281.639910 mapread 0x0101ca thru 0x0191a6 (0x8fdd bytes) 000560 1715343281.642153 MIRROR_RESYNC 4 mirrors 000561 1715343281.783106 mapwrite 0x036da8 thru 0x03bb03 (0x4d5c bytes) 000563 1715343281.822821 write 0x007daa thru 0x00c2ec (0x4543 bytes) 000564 1715343281.825979 read 0x032be8 thru 0x0370d5 (0x44ee bytes) 000565 1715343281.827814 MIRROR_RESYNC 4 mirrors 000567 1715343281.962466 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000570 1715343282.020154 mapread 0x01c29b thru 0x024103 (0x7e69 bytes) 000573 1715343282.026173 mapread 0x02cdce thru 0x03114a (0x437d bytes) 000574 1715343282.029328 mapwrite 0x03b303 thru 0x03ffff (0x4cfd bytes) 000578 1715343282.070143 read 0x01c96b thru 0x024d38 (0x83ce bytes) 000579 1715343282.072746 write 0x035e7d thru 0x03e63f (0x87c3 bytes) 000580 1715343282.074777 MIRROR_RESYNC 3 mirrors 000581 1715343282.182430 write 0x009c88 thru 0x00f392 (0x570b bytes) 000582 1715343282.198707 mapread 0x03f4a1 thru 0x03ffff (0xb5f bytes) 000584 1715343282.201367 read 0x007296 thru 0x008b47 (0x18b2 bytes) 000586 1715343282.203207 trunc from 0x040000 to 0x030bff (0xf400 bytes) 000587 1715343282.211448 write 0x03e84a thru 0x03ffff (0x17b6 bytes) 000590 1715343282.213789 MIRROR_RESYNC 3 mirrors 000591 1715343282.321488 mapread 0x01d557 thru 0x01dd0b (0x7b5 bytes) 000592 1715343282.323397 mapread 0x026dfa thru 0x02a7e1 (0x39e8 bytes) 000593 1715343282.327244 mapwrite 0x02fcc7 thru 0x032000 (0x233a bytes) 000594 1715343282.370173 MIRROR_EXTEND to 4 mirrors 000597 1715343282.473793 trunc from 0x040000 to 0x0222bb (0x1dd44 bytes) 000599 1715343282.493274 write 0x007dd0 thru 0x012c7a (0xaeab bytes) 000600 1715343282.497744 MIRROR_RESYNC 4 mirrors 000601 1715343282.633189 trunc from 0x0222bc to 0x021753 (0xb68 bytes) 000606 1715343282.653886 read 0x00b8a7 thru 0x016279 (0xa9d3 bytes) 000608 1715343282.656487 trunc from 0x021754 to 0x004d90 (0x1c9c3 bytes) 000609 1715343282.661447 write 0x031794 thru 0x03930d (0x7b7a bytes) 000610 1715343282.662313 write 0x023334 thru 0x02ced7 (0x9ba4 bytes) 000611 1715343282.664738 trunc from 0x03930e to 0x03d08b (0x3d7e bytes) 000618 1715343282.670718 write 0x03fd9e thru 0x03ffff (0x262 bytes) 000619 1715343282.672724 MIRROR_EXTEND to 5 mirrors 000622 1715343282.801312 read 0x02df92 thru 0x03b62a (0xd699 bytes) 000623 1715343282.804799 mapwrite 0x0382ae thru 0x03e3b1 (0x6104 bytes) 000625 1715343282.855530 write 0x026a51 thru 0x029507 (0x2ab7 bytes) 000626 1715343282.859932 read 0x0227cf thru 0x02cc5a (0xa48c bytes) 000627 1715343282.860360 trunc from 0x040000 to 0x012ad6 (0x2d529 bytes) 000628 1715343282.870054 mapwrite 0x03c6d7 thru 0x03ffff (0x3929 bytes) 000632 1715343282.896181 mapread 0x031f97 thru 0x03d0ee (0xb158 bytes) 000633 1715343282.896934 trunc from 0x040000 to 0x039eaa (0x6155 bytes) 000637 1715343282.902199 mapread 0x00e1a5 thru 0x0166ba (0x8516 bytes) 000638 1715343282.902951 write 0x0265c1 thru 0x027d5f (0x179f bytes) 000641 1715343282.904102 read 0x0109c8 thru 0x011ddb (0x1414 bytes) 000642 1715343282.904407 mapwrite 0x02dd47 thru 0x039232 (0xb4ec bytes) 000645 1715343282.921588 trunc from 0x039eab to 0x03e197 (0x42ed bytes) 000646 1715343282.925855 write 0x014530 thru 0x0234fb (0xefcc bytes) 000648 1715343282.927408 mapwrite 0x019898 thru 0x01bcde (0x2447 bytes) 000649 1715343282.945461 MIRROR_RESYNC 5 mirrors 000650 1715343283.131720 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000653 1715343283.196792 write 0x000018 thru 0x007eb0 (0x7e99 bytes) 000654 1715343283.216597 mapread 0x010b24 thru 0x01806a (0x7547 bytes) 000655 1715343283.221168 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000656 1715343283.274330 trunc from 0x03e198 to 0x01e96c (0x1f82b bytes) 000657 1715343283.279062 trunc from 0x01e96d to 0x0129c6 (0xbfa6 bytes) 000658 1715343283.283066 read 0x010873 thru 0x0129c6 (0x2154 bytes) 000659 1715343283.284838 write 0x01a56e thru 0x01ae03 (0x896 bytes) 000662 1715343283.285786 trunc from 0x01ae04 to 0x030fd1 (0x161ce bytes) 000663 1715343283.289702 read 0x006c8d thru 0x01510a (0xe47e bytes) 000664 1715343283.296150 MIRROR_EXTEND to 4 mirrors 000665 1715343283.403542 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000666 1715343283.459662 read 0x01e1bf thru 0x0208dc (0x271e bytes) 000668 1715343283.461854 write 0x00fe7e thru 0x01e076 (0xe1f9 bytes) 000672 1715343283.482070 MIRROR_RESYNC 3 mirrors truncating to largest ever: 0x3f4ff 000674 1715343283.611024 trunc from 0x030fd2 to 0x03f4fe (0xe52d bytes) 000676 1715343283.632610 read 0x03d69d thru 0x03f4fe (0x1e62 bytes) 000677 1715343283.634273 read 0x013926 thru 0x01c1b7 (0x8892 bytes) 000678 1715343283.638071 write 0x0368bb thru 0x03e5a4 (0x7cea bytes) 000679 1715343283.638890 read 0x03a30a thru 0x03f4fe (0x51f5 bytes) 000680 1715343283.639335 mapwrite 0x009ab8 thru 0x00fcd6 (0x621f bytes) 000681 1715343283.656784 trunc from 0x03f4ff to 0x011c0a (0x2d8f4 bytes) 000683 1715343283.662292 mapread 0x00cf7b thru 0x011c0a (0x4c90 bytes) 000685 1715343283.662991 mapread 0x002087 thru 0x005185 (0x30ff bytes) 000686 1715343283.663428 mapwrite 0x029c22 thru 0x035b6c (0xbf4b bytes) 000687 1715343283.689082 write 0x03af20 thru 0x03ffff (0x50e0 bytes) 000691 1715343283.690972 write 0x019860 thru 0x019905 (0xa6 bytes) 000692 1715343283.691414 mapwrite 0x019599 thru 0x019eb0 (0x918 bytes) 000695 1715343283.704797 mapwrite 0x01f7f4 thru 0x020957 (0x1164 bytes) 000700 1715343283.718884 write 0x014276 thru 0x016183 (0x1f0e bytes) 000701 1715343283.719355 write 0x03c8d3 thru 0x03ffff (0x372d bytes) 000702 1715343283.719790 mapread 0x0130af thru 0x02000b (0xcf5d bytes) 000703 1715343283.720435 mapwrite 0x02947f thru 0x02adae (0x1930 bytes) 000704 1715343283.735827 write 0x0137f6 thru 0x014731 (0xf3c bytes) 000705 1715343283.736383 read 0x0232af thru 0x029b2c (0x687e bytes) 000709 1715343283.737302 trunc from 0x040000 to 0x0020a0 (0x3df5f bytes) 000711 1715343283.744213 MIRROR_RESYNC 3 mirrors 000712 1715343283.842975 read 0x000c28 thru 0x0020a0 (0x1479 bytes) 000713 1715343283.845216 MIRROR_EXTEND to 4 mirrors 000714 1715343283.932754 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000716 1715343283.982946 trunc from 0x0020a1 to 0x034dbd (0x32d1d bytes) 000721 1715343284.004035 trunc from 0x034dbe to 0x000a54 (0x34369 bytes) 000723 1715343284.008881 mapread 0x000088 thru 0x000a54 (0x9cd bytes) 000725 1715343284.011209 mapwrite 0x030f1e thru 0x037874 (0x6957 bytes) 000729 1715343284.037442 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000731 1715343284.083089 write 0x01b7e4 thru 0x024222 (0x8a3f bytes) 000732 1715343284.085730 mapwrite 0x035273 thru 0x038f53 (0x3ce1 bytes) 000733 1715343284.105706 write 0x01e160 thru 0x027294 (0x9135 bytes) 000734 1715343284.107585 read 0x003b4c thru 0x00fee8 (0xc39d bytes) 000735 1715343284.111956 MIRROR_EXTEND to 3 mirrors 000736 1715343284.216079 trunc from 0x038f54 to 0x00fdd5 (0x2917e bytes) 000738 1715343284.237669 read 0x00b838 thru 0x00fdd5 (0x459e bytes) 000739 1715343284.240349 read 0x00153f thru 0x009138 (0x7bfa bytes) 000740 1715343284.243007 read 0x0003c6 thru 0x000cb2 (0x8ed bytes) 000741 1715343284.243272 mapread 0x005f35 thru 0x00addf (0x4eab bytes) 000743 1715343284.243932 mapwrite 0x010c54 thru 0x0139fb (0x2da8 bytes) 000744 1715343284.265909 mapread 0x00f893 thru 0x0139fb (0x4169 bytes) 000745 1715343284.267920 MIRROR_EXTEND to 4 mirrors 000746 1715343284.361005 mapread 0x00521e thru 0x008b8b (0x396e bytes) 000748 1715343284.365914 read 0x00b0ea thru 0x00baf8 (0xa0f bytes) 000749 1715343284.366153 mapread 0x006752 thru 0x0115f3 (0xaea2 bytes) 000750 1715343284.367594 read 0x0019d9 thru 0x00da38 (0xc060 bytes) 000751 1715343284.367811 write 0x01aced thru 0x02a173 (0xf487 bytes) 000753 1715343284.383033 read 0x021390 thru 0x02a173 (0x8de4 bytes) 000755 1715343284.383929 read 0x0273c9 thru 0x02a173 (0x2dab bytes) 000756 1715343284.384376 mapwrite 0x01eec3 thru 0x02380c (0x494a bytes) 000758 1715343284.399558 trunc from 0x02a174 to 0x00f602 (0x1ab71 bytes) 000760 1715343284.409021 MIRROR_EXTEND to 5 mirrors 000763 1715343284.520529 write 0x00516b thru 0x00aea9 (0x5d3f bytes) 000765 1715343284.538666 mapwrite 0x01ca4f thru 0x01d547 (0xaf9 bytes) 000770 1715343284.566032 mapwrite 0x008577 thru 0x00a6cd (0x2157 bytes) 000771 1715343284.578369 read 0x015cff thru 0x01d547 (0x7849 bytes) 000772 1715343284.578837 write 0x01ea93 thru 0x026cc4 (0x8232 bytes) 000773 1715343284.579558 mapwrite 0x009d2f thru 0x0113d5 (0x76a7 bytes) 000774 1715343284.598149 read 0x00c911 thru 0x0145e7 (0x7cd7 bytes) 000775 1715343284.602110 MIRROR_RESYNC 5 mirrors 000776 1715343284.752392 mapread 0x001efc thru 0x00bbc3 (0x9cc8 bytes) 000777 1715343284.757789 write 0x00c6f6 thru 0x01bbea (0xf4f5 bytes) 000778 1715343284.775255 trunc from 0x026cc5 to 0x02a44c (0x3788 bytes) 000781 1715343284.784108 write 0x024d05 thru 0x02d29e (0x859a bytes) 000782 1715343284.786884 mapread 0x016317 thru 0x01cb5d (0x6847 bytes) 000783 1715343284.788592 read 0x010c5f thru 0x019e07 (0x91a9 bytes) 000785 1715343284.789439 read 0x010e3a thru 0x0158c0 (0x4a87 bytes) 000791 1715343284.792739 MIRROR_EXTEND to 6 mirrors 000793 1715343284.908601 trunc from 0x02d29f to 0x0101c5 (0x1d0d9 bytes) 000798 1715343284.930456 mapread 0x0059b9 thru 0x00832b (0x2973 bytes) 000799 1715343284.935264 read 0x008833 thru 0x00c4e0 (0x3cae bytes) 000800 1715343284.935565 read 0x00b99e thru 0x0101c5 (0x4828 bytes) 000801 1715343284.935902 trunc from 0x0101c6 to 0x0357c3 (0x255fe bytes) 000803 1715343284.941252 mapwrite 0x0207a0 thru 0x025757 (0x4fb8 bytes) 000804 1715343284.960373 mapread 0x02c0d7 thru 0x0357c3 (0x96ed bytes) 000811 1715343284.963324 MIRROR_EXTEND to 7 mirrors 000812 1715343285.082876 write 0x00e3b9 thru 0x0145a8 (0x61f0 bytes) 000814 1715343285.101873 MIRROR_RESYNC 7 mirrors 000815 1715343285.330344 mapwrite 0x0245a3 thru 0x02fdbe (0xb81c bytes) 000818 1715343285.381894 read 0x015439 thru 0x01b179 (0x5d41 bytes) 000819 1715343285.384396 mapread 0x007d1e thru 0x016308 (0xe5eb bytes) 000821 1715343285.390297 MIRROR_RESYNC 7 mirrors 000822 1715343285.599367 mapread 0x017fcf thru 0x023451 (0xb483 bytes) 000823 1715343285.607601 MIRROR_EXTEND to 8 mirrors 000824 1715343285.729396 trunc from 0x0357c4 to 0x02c7d5 (0x8fee bytes) 000825 1715343285.752515 mapread 0x009e96 thru 0x00a6b8 (0x823 bytes) 000826 1715343285.755940 trunc from 0x02c7d6 to 0x01cd24 (0xfab1 bytes) 000827 1715343285.761141 trunc from 0x01cd25 to 0x00ca85 (0x1029f bytes) 000828 1715343285.765020 mapread 0x007fc8 thru 0x00ca85 (0x4abe bytes) 000829 1715343285.767710 mapread 0x0069ba thru 0x00ca85 (0x60cc bytes) 000831 1715343285.768322 mapwrite 0x038b1a thru 0x03ffff (0x74e6 bytes) 000834 1715343285.792682 mapwrite 0x02f494 thru 0x03bbda (0xc747 bytes) 000836 1715343285.812689 mapwrite 0x0247b5 thru 0x0294f0 (0x4d3c bytes) 000837 1715343285.829005 read 0x002745 thru 0x006637 (0x3ef3 bytes) 000839 1715343285.832741 MIRROR_RESYNC 8 mirrors 000841 1715343286.034532 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000842 1715343286.114496 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000843 1715343286.188574 write 0x00cdf9 thru 0x01bbab (0xedb3 bytes) 000845 1715343286.209617 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000847 1715343286.263906 MIRROR_EXTEND to 6 mirrors 000849 1715343286.386844 read 0x01fd4e thru 0x0266c4 (0x6977 bytes) 000850 1715343286.390117 MIRROR_RESYNC 6 mirrors 000851 1715343286.551477 mapwrite 0x02dccf thru 0x030689 (0x29bb bytes) 000856 1715343286.596590 mapread 0x034589 thru 0x03ffff (0xba77 bytes) 000857 1715343286.600262 read 0x0178d0 thru 0x01bccd (0x43fe bytes) 000859 1715343286.604430 MIRROR_RESYNC 6 mirrors 000861 1715343286.768350 write 0x029ace thru 0x02aef7 (0x142a bytes) 000863 1715343286.786340 mapread 0x012366 thru 0x01f121 (0xcdbc bytes) 000864 1715343286.790947 trunc from 0x040000 to 0x030152 (0xfead bytes) 000865 1715343286.799669 mapread 0x012a4c thru 0x01eb4f (0xc104 bytes) 000868 1715343286.800815 mapwrite 0x035e7d thru 0x03ffff (0xa183 bytes) 000871 1715343286.824304 read 0x018d96 thru 0x01dfda (0x5245 bytes) 000877 1715343286.826672 mapread 0x002a81 thru 0x0064c8 (0x3a48 bytes) 000878 1715343286.829530 trunc from 0x040000 to 0x00af79 (0x35086 bytes) 000879 1715343286.833851 write 0x028f4a thru 0x02db19 (0x4bd0 bytes) 000880 1715343286.834404 write 0x02d4b3 thru 0x03843c (0xaf8a bytes) 000881 1715343286.835002 mapread 0x036251 thru 0x03843c (0x21ec bytes) 000882 1715343286.835379 read 0x013ef0 thru 0x01551c (0x162d bytes) 000883 1715343286.838819 trunc from 0x03843d to 0x002a40 (0x359fc bytes) 000884 1715343286.844195 MIRROR_RESYNC 6 mirrors 000886 1715343286.999362 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000890 1715343287.051467 mapread 0x000a0b thru 0x002a40 (0x2036 bytes) 000891 1715343287.055056 trunc from 0x002a41 to 0x01b90f (0x18ecf bytes) 000893 1715343287.075347 MIRROR_RESYNC 5 mirrors 000894 1715343287.232593 mapwrite 0x011224 thru 0x01162c (0x409 bytes) 000895 1715343287.277193 mapread 0x01375e thru 0x0155fa (0x1e9d bytes) 000896 1715343287.280419 write 0x013fe3 thru 0x0196b1 (0x56cf bytes) 000899 1715343287.281518 mapread 0x00edab thru 0x018cbd (0x9f13 bytes) 000901 1715343287.282555 write 0x029a57 thru 0x0392cd (0xf877 bytes) 000903 1715343287.285897 MIRROR_RESYNC 5 mirrors 000904 1715343287.446143 trunc from 0x0392ce to 0x008764 (0x30b69 bytes) 000905 1715343287.465277 mapread 0x00698e thru 0x008764 (0x1dd7 bytes) 000906 1715343287.470682 mapread 0x0054a3 thru 0x008764 (0x32c2 bytes) 000907 1715343287.471341 write 0x038efc thru 0x03c817 (0x391c bytes) 000908 1715343287.472256 mapwrite 0x035f24 thru 0x03ffff (0xa0dc bytes) 000909 1715343287.498560 trunc from 0x040000 to 0x033f6d (0xc092 bytes) 000910 1715343287.503258 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000911 1715343287.550123 read 0x009bad thru 0x00b056 (0x14aa bytes) 000913 1715343287.552322 mapwrite 0x010aef thru 0x01220f (0x1721 bytes) 000917 1715343287.571423 read 0x01ea65 thru 0x026c97 (0x8233 bytes) 000920 1715343287.575309 trunc from 0x033f6e to 0x02601e (0xdf4f bytes) 000921 1715343287.579123 mapread 0x003f54 thru 0x0111ec (0xd299 bytes) 000922 1715343287.579662 write 0x00d6ca thru 0x01c786 (0xf0bd bytes) 000923 1715343287.580560 write 0x00270f thru 0x007737 (0x5029 bytes) 000924 1715343287.581024 read 0x0059ff thru 0x00cf16 (0x7518 bytes) 000926 1715343287.581463 mapread 0x009c66 thru 0x00e541 (0x48dc bytes) 000928 1715343287.582091 mapread 0x01d971 thru 0x02601e (0x86ae bytes) 000930 1715343287.583817 MIRROR_EXTEND to 5 mirrors 000932 1715343287.698973 trunc from 0x02601f to 0x00a993 (0x1b68b bytes) 000933 1715343287.719109 mapwrite 0x0198de thru 0x020078 (0x679b bytes) 000938 1715343287.744552 read 0x001d4f thru 0x006688 (0x493a bytes) 000939 1715343287.746876 trunc from 0x020079 to 0x02fb66 (0xfaee bytes) 000940 1715343287.750951 write 0x00e5a7 thru 0x01aebe (0xc918 bytes) 000941 1715343287.754134 read 0x02454a thru 0x029047 (0x4afe bytes) 000942 1715343287.757030 write 0x0155a3 thru 0x01da0c (0x846a bytes) 000943 1715343287.757812 mapwrite 0x01a17f thru 0x026849 (0xc6cb bytes) 000944 1715343287.776434 mapread 0x02e6e7 thru 0x02fb66 (0x1480 bytes) 000945 1715343287.778241 MIRROR_EXTEND to 6 mirrors 000946 1715343287.876961 mapwrite 0x004346 thru 0x012d54 (0xea0f bytes) 000947 1715343287.924268 mapread 0x00138d thru 0x002df6 (0x1a6a bytes) 000949 1715343287.925306 read 0x02dec4 thru 0x02fb66 (0x1ca3 bytes) 000950 1715343287.928222 MIRROR_EXTEND to 7 mirrors 000951 1715343288.029236 mapwrite 0x02623c thru 0x02a5f9 (0x43be bytes) 000952 1715343288.080540 MIRROR_EXTEND to 8 mirrors 000954 1715343288.192461 mapwrite 0x023ce2 thru 0x0266f2 (0x2a11 bytes) 000955 1715343288.239024 write 0x02ef53 thru 0x03bd16 (0xcdc4 bytes) 000956 1715343288.242492 MIRROR_EXTEND to 9 mirrors 000957 1715343288.372051 mapread 0x024fd8 thru 0x02d66c (0x8695 bytes) 000962 1715343288.378094 mapread 0x005a3d thru 0x00f95a (0x9f1e bytes) 000963 1715343288.382600 mapwrite 0x02a2db thru 0x0341b9 (0x9edf bytes) 000964 1715343288.430203 mapread 0x018708 thru 0x02238c (0x9c85 bytes) 000965 1715343288.434407 mapread 0x007f39 thru 0x008384 (0x44c bytes) 000967 1715343288.437269 mapwrite 0x011349 thru 0x01f593 (0xe24b bytes) 000968 1715343288.455395 read 0x01b809 thru 0x01c3a9 (0xba1 bytes) 000970 1715343288.457195 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000973 1715343288.505333 read 0x01d7f4 thru 0x02c726 (0xef33 bytes) 000974 1715343288.507715 write 0x00c1e2 thru 0x00d17b (0xf9a bytes) 000975 1715343288.511156 MIRROR_RESYNC 8 mirrors 000976 1715343288.747145 read 0x01b9ff thru 0x020571 (0x4b73 bytes) 000977 1715343288.750199 MIRROR_EXTEND to 9 mirrors 000978 1715343288.874324 read 0x012e2e thru 0x021ca5 (0xee78 bytes) 000981 1715343288.876852 write 0x019763 thru 0x01fad9 (0x6377 bytes) 000982 1715343288.894383 read 0x00fb4f thru 0x018778 (0x8c2a bytes) 000984 1715343288.897990 write 0x02bee8 thru 0x03763a (0xb753 bytes) 000985 1715343288.898784 mapread 0x00886d thru 0x00a192 (0x1926 bytes) 000987 1715343288.899398 mapwrite 0x02d9dd thru 0x031ae2 (0x4106 bytes) 000988 1715343288.915359 trunc from 0x03bd17 to 0x0066c3 (0x35653 bytes) 000989 1715343288.924381 mapwrite 0x01949a thru 0x01e702 (0x5269 bytes) 000992 1715343288.950086 mapread 0x01ac35 thru 0x01e702 (0x3ace bytes) 000993 1715343288.950508 write 0x03b196 thru 0x03ffff (0x4e6a bytes) 000995 1715343288.951301 read 0x02a8c6 thru 0x03353a (0x8c75 bytes) 000996 1715343288.955526 mapread 0x038ed3 thru 0x03bb70 (0x2c9e bytes) 001000 1715343288.956995 trunc from 0x040000 to 0x020870 (0x1f78f bytes) All operations completed A-OK! PASS 70a (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200a: stress test ===================== 08:14:51 (1715343291) Starting client: oleg205-client.virtnet: -o user_xattr,flock oleg205-server@tcp:/lustre /mnt/lustre2 Starting client: oleg205-client.virtnet: -o user_xattr,flock oleg205-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5557120 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4616288 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6005824 .. Extending file size to 7226112 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8153120 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9148096 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8551264 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8578720 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8664928 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9173568 .. done Extending file size to 8931840 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9186240 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8957664 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 6597 6598 6599 6601 6603 failed 192.168.202.105@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg205-client.virtnet /mnt/lustre2 (opts:) 192.168.202.105@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg205-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 08:15:56 (1715343356) Starting client: oleg205-client.virtnet: -o user_xattr,flock oleg205-server@tcp:/lustre /mnt/lustre2 Starting client: oleg205-client.virtnet: -o user_xattr,flock oleg205-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6067392 .. Extending file size to 337664 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7278528 .. failed Extending file size to 7899712 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6107584 .. Extending file size to 7702464 .. Extending file size to 7977280 .. Extending file size to 8112320 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8237696 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8114432 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8376160 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8706080 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8966400 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9106464 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9228096 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 10054 10055 10056 10058 10060 192.168.202.105@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg205-client.virtnet /mnt/lustre2 (opts:) 192.168.202.105@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg205-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 08:17:01 (1715343421) Starting client: oleg205-client.virtnet: -o user_xattr,flock oleg205-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 5120 2203392 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 5120 2203392 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0290894 s, 72.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0689164 s, 30.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.105@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg205-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 08:17:07 (1715343427) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.025089 s, 83.6 MB/s PASS 202 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 203: mirror file preserve mirror ID === 08:17:10 (1715343430) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcbf:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcbf:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcc0:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0xcba:0x0] } PASS 203 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 08:17:13 (1715343433) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.023019 s, 91.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcc1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcbb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcbc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 08:17:17 (1715343437) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0297193 s, 70.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcc2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcc3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcbd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcc2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcc3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcbd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcbe:0x0] } PASS 204b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 08:17:20 (1715343440) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0299313 s, 70.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcc4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcbf:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcc4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcbf:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcc0:0x0] } PASS 204c (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 08:17:44 (1715343464) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg205-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0171275 s, 61.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcc5:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcc2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcc1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcc5:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcc2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcc1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcc6:0x0] } PASS 204d (44s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 08:18:30 (1715343510) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg205-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcc3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcc4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.022339 s, 93.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcc3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcc4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcc7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcc3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcc4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcc7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (39s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 08:19:11 (1715343551) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg205-server mds-ost sync done. Creating new pool oleg205-server: Pool lustre.test_204f created Adding targets to pool oleg205-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg205-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0334499 s, 62.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcc5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcc8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcc5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcc8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg205-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg205-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg205-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (52s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 205: lfs mirror extend to set prefer and nocompr flags ========================================================== 08:20:05 (1715343605) PASS 205 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 08:20:10 (1715343610) oleg205-server: Pool lustre.test_206 created Adding targets to pool oleg205-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg205-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Updated after 2s: want 'lustre-OST0000_UUID lustre-OST0001_UUID ' got 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg205-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg205-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg205-server: Pool lustre.test_206 destroyed PASS 206 (14s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 08:20:25 (1715343625) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0287068 s, 36.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0149623 s, 70.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x11888e77 Mirror 3: 0x11888e77 PASS 207 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 08:20:31 (1715343631) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 08:20:34 (1715343634) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 08:20:38 (1715343638) Starting client: oleg205-client.virtnet: -o user_xattr,flock oleg205-server@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0200284 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2290000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xbb0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xbb0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xbb0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x12e0000 Read this (4 bytes): BBBB 192.168.202.105@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg205-client.virtnet /mnt/lustre2 (opts:) PASS 209a (16s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 08:20:56 (1715343656) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.0054269 s, 1.5 MB/s the very first read read with warm cache PASS 209b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210a: handle broken mirrored lovea ==== 08:21:00 (1715343660) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101889 s, 103 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x6c5 lmm_fid: [0x200000404:0x6c5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x6c5 lmm_fid: [0x200000404:0x6c5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcce:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835653 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-05-10 08:21:01.000000000 -0400 Modify: 2024-05-10 08:21:01.000000000 -0400 Change: 2024-05-10 08:21:01.000000000 -0400 Birth: - PASS 210a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 08:21:05 (1715343665) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) debug_raw_pointers=0 debug_raw_pointers=0 == sanity-flr test complete, duration 811 sec ============ 08:21:07 (1715343667) === sanity-flr: start cleanup 08:21:07 (1715343667) === === sanity-flr: finish cleanup 08:21:08 (1715343668) ===