-----============= acceptance-small: sanity-flr ============----- Fri May 10 05:20:02 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 6 201 44c 49a === sanity-flr: start setup 05:20:08 (1715332808) === oleg253-client.virtnet: executing check_config_client /mnt/lustre oleg253-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg253-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012a5f5800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012a5f5800.idle_timeout=debug disable quota as required oleg253-server: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 05:20:16 (1715332816) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f6872] debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0a: lfs mirror create with -N option == 05:20:18 (1715332818) PASS 0a (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 05:20:26 (1715332826) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 05:20:29 (1715332829) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0d: lfs mirror extend with -N option == 05:20:32 (1715332832) PASS 0d (8s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 05:20:43 (1715332843) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 05:20:46 (1715332846) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0g: lfs mirror create flags support === 05:20:50 (1715332850) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88012a5f5800.stats=clear osc.lustre-OST0001-osc-ffff88012a5f5800.stats=clear PASS 0g (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 05:20:55 (1715332855) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 05:21:01 (1715332861) PASS 0j (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 05:21:05 (1715332865) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 1: create components with setstripe options ========================================================== 05:21:09 (1715332869) lfs mirror mirror: cannot get UNLOCK lease, ext 4: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 2: create components from existing files ========================================================== 05:21:14 (1715332874) PASS 2 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 3: create components from files located on different MDTs ========================================================== 05:21:18 (1715332878) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 05:21:21 (1715332881) lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 05:21:25 (1715332885) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 7: Create mirror with DoM component === 05:21:31 (1715332891) PASS 7 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 05:21:35 (1715332895) 11+0 records in 11+0 records out 11534336 bytes (12 MB) copied, 0.566217 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0867755 s, 12.1 MB/s PASS 21 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 05:21:40 (1715332900) 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.123603 s, 136 MB/s PASS 22 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 05:21:45 (1715332905) Stopping /mnt/lustre-ost1 (opts:) on oleg253-server oleg253-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a5f5800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a5f5800.ost_server_uuid in DISCONN state after 3 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg253-server: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg253-server: oleg253-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg253-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg253-server oleg253-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a5f5800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a5f5800.ost_server_uuid in DISCONN state after 2 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg253-server: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg253-server: oleg253-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg253-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (29s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 05:22:16 (1715332936) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB) copied, 0.111052 s, 56.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg253-server oleg253-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a5f5800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a5f5800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg253-server: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg253-server: oleg253-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg253-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (12s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33a: read can choose available mirror to read ========================================================== 05:22:30 (1715332950) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg253-server oleg253-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a5f5800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a5f5800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg253-server: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg253-server: oleg253-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg253-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg253-server oleg253-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a5f5800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a5f5800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg253-server: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg253-server: oleg253-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg253-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 05:22:55 (1715332975) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.772517 s, 67.9 MB/s -rw-r--r-- 1 root root 52428800 May 10 05:22 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.246s user 0m0.000s sys 0m0.122s Stopping /mnt/lustre-ost1 (opts:) on oleg253-server oleg253-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a5f5800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a5f5800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.222s user 0m0.000s sys 0m0.173s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg253-server: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg253-server: oleg253-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg253-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg253-server oleg253-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a5f5800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a5f5800.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost1)... real 0m0.313s user 0m0.000s sys 0m0.131s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg253-server: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg253-server: oleg253-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg253-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg253-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012a5f5800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a5f5800.ost_server_uuid in FULL state after 0 sec PASS 33b (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 05:23:21 (1715333001) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 05:23:24 (1715333004) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 05:23:27 (1715333007) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 35: allow to write to mirrored files == 05:23:30 (1715333010) PASS 35 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36a: write to mirrored files ========== 05:23:34 (1715333014) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.190227 s, 66.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 05:23:40 (1715333020) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00980105 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109504 s, 95.8 MB/s PASS 36b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 05:23:44 (1715333024) PASS 36c (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 05:23:49 (1715333029) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000678458 s, 1.5 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00153408 s, 668 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000842698 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00205299 s, 499 kB/s flv=10 olv=10 PASS 36d (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 37: mirror I/O API verification ======= 05:23:54 (1715333034) create test files with size 15750642 .. 5a186bdf8a7046d501cd828a9a651303 - af59d15520b0477cb02631472352e434 - c4d389150f85808ac3bb945825b73ed9 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg253-client.virtnet /mnt/lustre (opts:) Starting client: oleg253-client.virtnet: -o user_xattr,flock oleg253-server@tcp:/lustre /mnt/lustre PASS 37 (7s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 38: resync ============================ 05:24:04 (1715333044) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 05:24:10 (1715333050) PASS 39 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 05:24:14 (1715333054) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0541626 s, 19.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x44:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0170329 s, 61.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4b:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 41: lfs mirror resync check =========== 05:24:19 (1715333059) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0557586 s, 37.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.114963 s, 36.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0518727 s, 40.4 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 42: lfs mirror verify ================= 05:24:24 (1715333064) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43a: mirror pick on write ============= 05:24:27 (1715333067) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 05:24:29 (1715333069) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00870595 s, 120 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44a: lfs mirror split check =========== 05:24:33 (1715333073) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44b: mirror split does not create stale file ========================================================== 05:24:35 (1715333075) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4d:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (3s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 05:24:41 (1715333081) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0742823 s, 141 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg253-client.virtnet /mnt/lustre (opts:) Starting client: oleg253-client.virtnet: -o user_xattr,flock oleg253-server@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 05:24:46 (1715333086) llite.lustre-ffff88012bf82000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0812244 s, 129 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg253-client.virtnet /mnt/lustre (opts:) Starting client: oleg253-client.virtnet: -o user_xattr,flock oleg253-server@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800a257d800.xattr_cache=1 PASS 44e (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 05:24:52 (1715333092) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (9s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 46: Verify setstripe --copy option ==== 05:25:03 (1715333103) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (7s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 47: Verify mirror obj alloc =========== 05:25:12 (1715333112) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 48: Verify snapshot mirror ============ 05:25:14 (1715333114) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0808046 s, 38.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0503296 s, 62.5 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 154bd693b6baeca17863682a68d0f1b2 - mirror 1 checksum: a70df9bf2da4b57140c093c0920a5e7f - mirror 2 checksum: 154bd693b6baeca17863682a68d0f1b2 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri May 10 05:25:15 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri May 10 05:25:15 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 154bd693b6baeca17863682a68d0f1b2 - mirror 1 checksum: a70df9bf2da4b57140c093c0920a5e7f - mirror 2 checksum: a70df9bf2da4b57140c093c0920a5e7f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri May 10 05:25:15 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri May 10 05:25:15 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } PASS 48 (3s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50A: mirror split update layout generation ========================================================== 05:25:20 (1715333120) Failing mds1 on oleg253-server Stopping /mnt/lustre-mds1 (opts:) on oleg253-server 05:25:27 (1715333127) shut down Failover mds1 to oleg253-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg253-server: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-MDT0000 05:25:39 (1715333139) targets are mounted 05:25:39 (1715333139) facet_failover done oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.011056 s, 94.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x14 lmm_fid: [0x200000404:0x14:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } PASS 50A (27s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 05:25:49 (1715333149) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50b: mirror rsync handles sparseness == 05:25:52 (1715333152) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 05:25:55 (1715333155) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00248066 s, 6.6 MB/s PASS 50c (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50d: mirror rsync keep holes ========== 05:25:59 (1715333159) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 05:26:02 (1715333162) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 05:26:04 (1715333164) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 60b (7s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 05:26:13 (1715333173) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 05:26:33 (1715333193) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 05:26:52 (1715333212) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 05:27:11 (1715333231) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70: mirror create and split race ====== 05:27:15 (1715333235) mirror create pid 26724 mirror split pid 26725 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4716: 26724 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4716: 26725 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (61s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70a: flr mode fsx test ================ 05:28:18 (1715333298) Seed set to 3299 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1715333299.181057 mapwrite 0x00352e thru 0x009912 (0x63e5 bytes) 000002 1715333299.223615 write 0x02b666 thru 0x032c3d (0x75d8 bytes) 000003 1715333299.224442 mapwrite 0x0072ef thru 0x00ae8e (0x3ba0 bytes) 000005 1715333299.247523 mapread 0x0002de thru 0x00d75d (0xd480 bytes) 000007 1715333299.249982 read 0x0113dd thru 0x017ffc (0x6c20 bytes) 000008 1715333299.252501 mapread 0x0236c7 thru 0x0254ca (0x1e04 bytes) 000010 1715333299.255788 write 0x022e6b thru 0x030a8c (0xdc22 bytes) 000012 1715333299.257200 mapread 0x0001e8 thru 0x00fe72 (0xfc8b bytes) 000015 1715333299.258330 mapread 0x0175fc thru 0x023563 (0xbf68 bytes) truncating to largest ever: 0x18a2c 000016 1715333299.261193 trunc from 0x032c3e to 0x018a2b (0x1a212 bytes) 000019 1715333299.265873 write 0x028a6e thru 0x034603 (0xbb96 bytes) 000021 1715333299.267079 mapwrite 0x004a6c thru 0x014050 (0xf5e5 bytes) 000022 1715333299.289663 mapread 0x014c40 thru 0x0181a2 (0x3563 bytes) 000024 1715333299.290508 trunc from 0x034604 to 0x006c67 (0x2d99c bytes) 000026 1715333299.296123 read 0x001896 thru 0x006c67 (0x53d2 bytes) 000028 1715333299.296890 mapwrite 0x00ab28 thru 0x01850f (0xd9e8 bytes) 000030 1715333299.323265 trunc from 0x018510 to 0x00d88c (0xac83 bytes) 000031 1715333299.327471 write 0x015b67 thru 0x017c1c (0x20b6 bytes) 000033 1715333299.328240 read 0x00c7e7 thru 0x00c889 (0xa3 bytes) 000035 1715333299.328882 mapwrite 0x0090b3 thru 0x00eb4f (0x5a9d bytes) 000037 1715333299.349980 write 0x008650 thru 0x0110df (0x8a90 bytes) 000040 1715333299.353302 MIRROR_EXTEND to 2 mirrors 000041 1715333299.462173 mapwrite 0x00f9ef thru 0x010718 (0xd2a bytes) 000043 1715333299.506339 read 0x00ffbb thru 0x017c1c (0x7c62 bytes) 000044 1715333299.506773 mapread 0x00b03a thru 0x013641 (0x8608 bytes) 000047 1715333299.507976 mapread 0x00c8d2 thru 0x017c1c (0xb34b bytes) 000048 1715333299.508399 write 0x022365 thru 0x02d803 (0xb49f bytes) 000049 1715333299.509208 read 0x0161a2 thru 0x020615 (0xa474 bytes) 000051 1715333299.512635 read 0x0062bd thru 0x00f097 (0x8ddb bytes) 000052 1715333299.513095 mapread 0x017f3e thru 0x01930a (0x13cd bytes) 000053 1715333299.513628 read 0x01c6e8 thru 0x01fcd9 (0x35f2 bytes) 000054 1715333299.514041 read 0x0065fb thru 0x00f4f9 (0x8eff bytes) 000057 1715333299.515084 mapread 0x02cc97 thru 0x02d803 (0xb6d bytes) 000058 1715333299.515622 mapwrite 0x02e85b thru 0x03813e (0x98e4 bytes) 000059 1715333299.546616 MIRROR_RESYNC 2 mirrors 000060 1715333299.651245 write 0x00b87a thru 0x016b58 (0xb2df bytes) 000062 1715333299.671868 read 0x028f15 thru 0x03625b (0xd347 bytes) truncating to largest ever: 0x36d0b 000065 1715333299.675427 trunc from 0x03813f to 0x036d0a (0x1434 bytes) 000066 1715333299.685402 mapread 0x0242e5 thru 0x027294 (0x2fb0 bytes) 000067 1715333299.688696 trunc from 0x036d0b to 0x02ee15 (0x7ef5 bytes) 000069 1715333299.694334 read 0x003f03 thru 0x01385d (0xf95b bytes) 000070 1715333299.697472 mapread 0x00f766 thru 0x01438e (0x4c29 bytes) 000071 1715333299.697977 read 0x018bc4 thru 0x01ca75 (0x3eb2 bytes) 000072 1715333299.699938 MIRROR_RESYNC 2 mirrors 000073 1715333299.806320 MIRROR_EXTEND to 3 mirrors 000074 1715333299.926632 mapread 0x01873c thru 0x01b644 (0x2f09 bytes) 000075 1715333299.931424 trunc from 0x02ee16 to 0x02f83a (0xa25 bytes) 000076 1715333299.954506 mapwrite 0x01632f thru 0x01932c (0x2ffe bytes) 000079 1715333299.974616 write 0x02b6c4 thru 0x03207f (0x69bc bytes) 000081 1715333299.978773 mapwrite 0x00ad7f thru 0x01753f (0xc7c1 bytes) 000084 1715333300.006324 write 0x02437b thru 0x02ef10 (0xab96 bytes) 000087 1715333300.007660 mapread 0x020479 thru 0x026781 (0x6309 bytes) 000088 1715333300.010789 write 0x015d1d thru 0x0204d2 (0xa7b6 bytes) 000090 1715333300.011712 mapwrite 0x029fee thru 0x032d52 (0x8d65 bytes) 000091 1715333300.038542 trunc from 0x032d53 to 0x012d86 (0x1ffcc bytes) 000093 1715333300.043343 mapwrite 0x02ec43 thru 0x032448 (0x3806 bytes) 000094 1715333300.068297 trunc from 0x032449 to 0x01c194 (0x162b4 bytes) 000096 1715333300.072969 trunc from 0x01c195 to 0x003cbd (0x184d7 bytes) 000098 1715333300.076943 write 0x02f377 thru 0x030d90 (0x1a1a bytes) 000100 1715333300.077650 mapread 0x02b4dc thru 0x02d036 (0x1b5b bytes) 000101 1715333300.080606 mapread 0x02fb3e thru 0x030d90 (0x1253 bytes) 000102 1715333300.082230 MIRROR_RESYNC 3 mirrors 000105 1715333300.211883 trunc from 0x030d91 to 0x00e203 (0x22b8d bytes) 000106 1715333300.236239 mapread 0x0073b1 thru 0x00e203 (0x6e53 bytes) 000107 1715333300.242702 mapwrite 0x01f05d thru 0x02e028 (0xefcc bytes) 000108 1715333300.279180 MIRROR_RESYNC 3 mirrors 000109 1715333300.412337 write 0x035a35 thru 0x03ffff (0xa5cb bytes) 000111 1715333300.428480 mapwrite 0x00d63f thru 0x00e2b5 (0xc77 bytes) 000112 1715333300.450281 read 0x026dd4 thru 0x0277bd (0x9ea bytes) 000116 1715333300.455312 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000117 1715333300.518813 trunc from 0x040000 to 0x02caed (0x13512 bytes) 000119 1715333300.528071 write 0x03d3bb thru 0x03ffff (0x2c45 bytes) 000120 1715333300.530433 MIRROR_RESYNC 2 mirrors 000121 1715333300.638733 trunc from 0x040000 to 0x02b675 (0x1498a bytes) 000122 1715333300.661914 write 0x034cad thru 0x03ffff (0xb353 bytes) 000123 1715333300.662876 mapread 0x017222 thru 0x0176a8 (0x487 bytes) 000124 1715333300.665003 read 0x010847 thru 0x01a2f5 (0x9aaf bytes) 000126 1715333300.670491 MIRROR_EXTEND to 3 mirrors 000128 1715333300.799635 trunc from 0x040000 to 0x026cfc (0x19303 bytes) 000129 1715333300.824408 mapwrite 0x010deb thru 0x013d5e (0x2f74 bytes) 000131 1715333300.851244 mapwrite 0x018fd0 thru 0x01daee (0x4b1f bytes) 000133 1715333300.872933 mapwrite 0x03b42a thru 0x03ffff (0x4bd6 bytes) 000134 1715333300.897716 read 0x03869f thru 0x03ffff (0x7961 bytes) 000135 1715333300.898192 read 0x02825a thru 0x032c2d (0xa9d4 bytes) 000138 1715333300.904103 MIRROR_EXTEND to 4 mirrors 000139 1715333301.034756 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000140 1715333301.111822 read 0x02d405 thru 0x033be5 (0x67e1 bytes) 000141 1715333301.117353 mapread 0x0344d3 thru 0x0398e4 (0x5412 bytes) 000142 1715333301.126738 trunc from 0x040000 to 0x0222be (0x1dd41 bytes) 000143 1715333301.149030 trunc from 0x0222bf to 0x021739 (0xb85 bytes) 000146 1715333301.154494 mapread 0x01fad5 thru 0x021739 (0x1c65 bytes) 000147 1715333301.157793 mapwrite 0x00992f thru 0x009c92 (0x364 bytes) 000149 1715333301.181550 trunc from 0x02173a to 0x00a297 (0x174a2 bytes) 000152 1715333301.187569 mapread 0x0003f2 thru 0x0076fb (0x730a bytes) 000153 1715333301.191408 mapread 0x002d3d thru 0x00a297 (0x755b bytes) 000154 1715333301.193657 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000156 1715333301.249240 write 0x02b794 thru 0x031b74 (0x63e1 bytes) 000157 1715333301.249988 write 0x03218e thru 0x03bf29 (0x9d9c bytes) 000158 1715333301.250692 mapread 0x01ef04 thru 0x029668 (0xa765 bytes) 000161 1715333301.262851 MIRROR_EXTEND to 3 mirrors 000165 1715333301.410107 MIRROR_EXTEND to 4 mirrors 000166 1715333301.540079 MIRROR_EXTEND to 5 mirrors 000167 1715333301.684518 MIRROR_EXTEND to 6 mirrors 000168 1715333301.846304 read 0x009bd9 thru 0x0199bf (0xfde7 bytes) 000169 1715333301.851751 read 0x00c616 thru 0x0105ac (0x3f97 bytes) 000171 1715333301.856792 write 0x016ff0 thru 0x023f84 (0xcf95 bytes) 000172 1715333301.884139 read 0x0314d3 thru 0x031ab8 (0x5e6 bytes) 000173 1715333301.886236 MIRROR_EXTEND to 7 mirrors 000174 1715333302.002473 read 0x028ad3 thru 0x0330b1 (0xa5df bytes) 000175 1715333302.006151 mapread 0x0097db thru 0x0196b6 (0xfedc bytes) 000177 1715333302.015741 mapwrite 0x037f79 thru 0x03fe8d (0x7f15 bytes) 000178 1715333302.062841 write 0x0228d9 thru 0x026ca3 (0x43cb bytes) 000179 1715333302.066202 write 0x02e611 thru 0x038870 (0xa260 bytes) 000180 1715333302.069094 mapwrite 0x020682 thru 0x02313c (0x2abb bytes) 000181 1715333302.084532 read 0x01bde7 thru 0x01c952 (0xb6c bytes) 000182 1715333302.087861 write 0x00e1ce thru 0x015482 (0x72b5 bytes) 000183 1715333302.091410 read 0x0037e1 thru 0x011fe7 (0xe807 bytes) 000185 1715333302.093634 MIRROR_EXTEND to 8 mirrors 000187 1715333302.260620 mapread 0x01190d thru 0x01cc41 (0xb335 bytes) 000188 1715333302.270525 mapwrite 0x01b7b1 thru 0x028fa8 (0xd7f8 bytes) 000190 1715333302.326187 read 0x00f371 thru 0x013cb5 (0x4945 bytes) 000191 1715333302.329248 mapwrite 0x00764b thru 0x0115a0 (0x9f56 bytes) 000192 1715333302.351597 write 0x03edd1 thru 0x03ffff (0x122f bytes) 000193 1715333302.354895 trunc from 0x040000 to 0x00a508 (0x35af7 bytes) 000196 1715333302.368074 MIRROR_EXTEND to 9 mirrors 000198 1715333302.501150 MIRROR_RESYNC 9 mirrors 000199 1715333302.784794 mapwrite 0x0009f3 thru 0x00f0a7 (0xe6b5 bytes) 000200 1715333302.846270 read 0x00200f thru 0x0080dc (0x60ce bytes) 000201 1715333302.846959 mapwrite 0x0298dc thru 0x036425 (0xcb4a bytes) truncating to largest ever: 0x3f8fe 000203 1715333302.880631 trunc from 0x036426 to 0x03f8fd (0x94d8 bytes) 000204 1715333302.885050 write 0x0325e3 thru 0x03e21d (0xbc3b bytes) 000208 1715333302.888388 trunc from 0x03f8fe to 0x0095bf (0x3633e bytes) 000211 1715333302.894163 mapread 0x009175 thru 0x0095bf (0x44b bytes) 000212 1715333302.894591 trunc from 0x0095c0 to 0x032d4c (0x2978d bytes) 000214 1715333302.898918 write 0x024d56 thru 0x0303ea (0xb695 bytes) 000215 1715333302.903695 mapwrite 0x034492 thru 0x03aab8 (0x6627 bytes) 000216 1715333302.928844 mapread 0x006d87 thru 0x0074c2 (0x73c bytes) 000217 1715333302.929373 mapread 0x02e51c thru 0x0304e4 (0x1fc9 bytes) 000219 1715333302.931837 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000220 1715333302.983262 write 0x03d9b8 thru 0x03fe5d (0x24a6 bytes) 000221 1715333302.983865 write 0x03cd40 thru 0x03ffff (0x32c0 bytes) 000223 1715333302.985731 mapwrite 0x03d552 thru 0x03e7bf (0x126e bytes) 000224 1715333302.998058 read 0x0370a8 thru 0x03b6a5 (0x45fe bytes) 000227 1715333303.002362 MIRROR_EXTEND to 9 mirrors 000228 1715333303.153206 read 0x0285ca thru 0x030893 (0x82ca bytes) 000233 1715333303.167297 read 0x0390d2 thru 0x03ffff (0x6f2e bytes) 000235 1715333303.175015 write 0x0270ab thru 0x032ee6 (0xbe3c bytes) 000236 1715333303.195954 mapread 0x0391b6 thru 0x0393fb (0x246 bytes) 000237 1715333303.200219 MIRROR_EXTEND to 10 mirrors 000242 1715333303.352641 trunc from 0x040000 to 0x00a93e (0x356c1 bytes) 000243 1715333303.378241 write 0x005ae5 thru 0x0095f3 (0x3b0f bytes) 000245 1715333303.381527 mapwrite 0x005838 thru 0x00cc94 (0x745d bytes) 000248 1715333303.406913 trunc from 0x00cc95 to 0x009b3f (0x3155 bytes) 000250 1715333303.411585 read 0x006cb8 thru 0x009b3f (0x2e88 bytes) 000256 1715333303.413535 trunc from 0x009b40 to 0x00de54 (0x4315 bytes) 000258 1715333303.418781 write 0x02b677 thru 0x02c141 (0xacb bytes) 000260 1715333303.419861 read 0x02284e thru 0x02c141 (0x98f4 bytes) 000262 1715333303.425498 write 0x01acb6 thru 0x028b51 (0xde9c bytes) 000264 1715333303.426734 mapwrite 0x018a12 thru 0x0190b7 (0x6a6 bytes) 000265 1715333303.450109 trunc from 0x02c142 to 0x021af3 (0xa64e bytes) 000267 1715333303.456873 write 0x022a87 thru 0x02cd7b (0xa2f5 bytes) 000269 1715333303.458051 trunc from 0x02cd7c to 0x00b0b6 (0x21cc5 bytes) 000270 1715333303.463188 write 0x03f597 thru 0x03ffff (0xa69 bytes) 000274 1715333303.464887 read 0x03f251 thru 0x03ffff (0xdaf bytes) 000275 1715333303.465313 mapwrite 0x001a24 thru 0x006900 (0x4edd bytes) 000278 1715333303.488834 MIRROR_RESYNC 10 mirrors 000279 1715333303.827754 mapwrite 0x009f0a thru 0x00c448 (0x253f bytes) 000281 1715333303.892520 write 0x02500e thru 0x02759c (0x258f bytes) 000284 1715333303.899291 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000287 1715333303.961106 trunc from 0x040000 to 0x00594c (0x3a6b3 bytes) 000289 1715333303.974459 write 0x000675 thru 0x00568a (0x5016 bytes) 000290 1715333303.978812 trunc from 0x00594d to 0x0207e7 (0x1ae9b bytes) 000291 1715333303.985056 write 0x014b8c thru 0x01aa3f (0x5eb4 bytes) 000294 1715333303.992007 write 0x03d0b3 thru 0x03ffff (0x2f4d bytes) 000295 1715333303.993062 write 0x01b8c5 thru 0x02b85e (0xff9a bytes) 000298 1715333304.001538 trunc from 0x040000 to 0x031fe7 (0xe018 bytes) 000300 1715333304.010593 mapwrite 0x0109fa thru 0x01ece0 (0xe2e7 bytes) 000304 1715333304.042935 mapwrite 0x03e509 thru 0x03ffff (0x1af7 bytes) 000305 1715333304.074057 read 0x035897 thru 0x036765 (0xecf bytes) 000307 1715333304.076484 read 0x00f32d thru 0x0143dd (0x50b1 bytes) 000308 1715333304.079912 MIRROR_EXTEND to 10 mirrors 000309 1715333304.248019 read 0x003371 thru 0x006ac8 (0x3758 bytes) 000310 1715333304.252466 trunc from 0x040000 to 0x028411 (0x17bee bytes) 000311 1715333304.290176 mapread 0x01d152 thru 0x028411 (0xb2c0 bytes) 000312 1715333304.300309 read 0x01103f thru 0x012b3b (0x1afd bytes) 000313 1715333304.305964 MIRROR_EXTEND to 11 mirrors 000314 1715333304.465432 trunc from 0x028412 to 0x003e08 (0x24609 bytes) 000317 1715333304.503859 write 0x021679 thru 0x02c80e (0xb196 bytes) 000318 1715333304.504904 trunc from 0x02c80f to 0x02211b (0xa6f3 bytes) 000320 1715333304.512916 mapwrite 0x00971d thru 0x00f2b1 (0x5b95 bytes) 000325 1715333304.547919 read 0x010a01 thru 0x01a737 (0x9d37 bytes) 000326 1715333304.553546 MIRROR_EXTEND to 12 mirrors 000328 1715333304.719545 write 0x0291a8 thru 0x032182 (0x8fdb bytes) 000330 1715333304.744944 read 0x01c684 thru 0x025b71 (0x94ee bytes) 000331 1715333304.749738 mapwrite 0x0228a3 thru 0x02d6cf (0xae2d bytes) 000332 1715333304.793079 mapwrite 0x0384b2 thru 0x03ffff (0x7b4e bytes) 000333 1715333304.843505 trunc from 0x040000 to 0x02b40b (0x14bf4 bytes) 000336 1715333304.852167 write 0x01e1be thru 0x0202db (0x211e bytes) 000337 1715333304.852749 mapwrite 0x01b29c thru 0x02b290 (0xfff5 bytes) 000339 1715333304.920616 read 0x016ab0 thru 0x0178b4 (0xe05 bytes) 000340 1715333304.923254 MIRROR_RESYNC 12 mirrors 000343 1715333305.464174 write 0x032193 thru 0x039186 (0x6ff4 bytes) 000346 1715333305.489135 write 0x0312de thru 0x03bbb4 (0xa8d7 bytes) 000348 1715333305.491911 mapread 0x00ccfe thru 0x01b98f (0xec92 bytes) 000350 1715333305.502003 read 0x00c0a0 thru 0x014c94 (0x8bf5 bytes) 000351 1715333305.502497 mapwrite 0x0031e4 thru 0x0076c8 (0x44e5 bytes) 000352 1715333305.540941 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000353 1715333305.613233 read 0x019057 thru 0x01b7be (0x2768 bytes) 000355 1715333305.616020 write 0x03edb0 thru 0x03ffbc (0x120d bytes) 000358 1715333305.618986 MIRROR_EXTEND to 12 mirrors 000359 1715333305.798354 mapwrite 0x016622 thru 0x01c67a (0x6059 bytes) 000360 1715333305.885190 read 0x026ec2 thru 0x028895 (0x19d4 bytes) 000361 1715333305.887193 mapread 0x00ff54 thru 0x01597f (0x5a2c bytes) 000362 1715333305.890166 write 0x021e5b thru 0x027c1d (0x5dc3 bytes) 000363 1715333305.897006 MIRROR_RESYNC 12 mirrors 000364 1715333306.394493 write 0x03d026 thru 0x03f2a2 (0x227d bytes) 000366 1715333306.423293 mapwrite 0x016de6 thru 0x020581 (0x979c bytes) 000371 1715333306.458997 write 0x0081ba thru 0x016c39 (0xea80 bytes) 000372 1715333306.465937 MIRROR_EXTEND to 13 mirrors 000373 1715333306.660163 mapwrite 0x00318f thru 0x005601 (0x2473 bytes) 000374 1715333306.742454 mapread 0x01f386 thru 0x025b1d (0x6798 bytes) 000375 1715333306.746736 mapwrite 0x02de45 thru 0x02e3b7 (0x573 bytes) 000376 1715333306.779530 MIRROR_SPLIT mirror 2 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000377 1715333306.852475 read 0x03f526 thru 0x03ffbc (0xa97 bytes) 000378 1715333306.854896 read 0x020aba thru 0x03045d (0xf9a4 bytes) 000379 1715333306.863227 MIRROR_EXTEND to 13 mirrors 000382 1715333307.029162 mapread 0x0345a3 thru 0x03ffbc (0xba1a bytes) 000384 1715333307.040787 trunc from 0x03ffbd to 0x00a4e9 (0x35ad3 bytes) 000385 1715333307.078416 write 0x024ed2 thru 0x02d5d6 (0x8705 bytes) 000386 1715333307.079425 write 0x031821 thru 0x034588 (0x2d68 bytes) 000390 1715333307.081221 write 0x02ed7c thru 0x0354ab (0x6730 bytes) 000393 1715333307.086581 MIRROR_RESYNC 13 mirrors 000396 1715333307.611781 read 0x011398 thru 0x0208ea (0xf553 bytes) 000400 1715333307.615795 mapwrite 0x039f25 thru 0x03ffff (0x60db bytes) 000401 1715333307.668630 mapwrite 0x0336d0 thru 0x03625d (0x2b8e bytes) 000402 1715333307.689303 trunc from 0x040000 to 0x030cd7 (0xf328 bytes) 000403 1715333307.701869 mapread 0x004e67 thru 0x00eba5 (0x9d3f bytes) 000405 1715333307.710380 mapread 0x01afed thru 0x026d9e (0xbdb2 bytes) 000412 1715333307.720867 trunc from 0x030cd8 to 0x023672 (0xd665 bytes) 000413 1715333307.727085 trunc from 0x023673 to 0x02b06e (0x79fc bytes) 000414 1715333307.732909 trunc from 0x02b06f to 0x028f85 (0x20e9 bytes) 000415 1715333307.738431 write 0x018cfe thru 0x026c90 (0xdf93 bytes) 000416 1715333307.741441 read 0x016690 thru 0x022715 (0xc086 bytes) 000417 1715333307.741881 write 0x016e21 thru 0x0243d1 (0xd5b1 bytes) 000418 1715333307.742791 trunc from 0x028f86 to 0x007fc3 (0x20fc2 bytes) 000420 1715333307.749895 MIRROR_EXTEND to 14 mirrors 000421 1715333307.928481 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000422 1715333308.009825 write 0x038916 thru 0x03ffff (0x76ea bytes) 000424 1715333308.041651 trunc from 0x040000 to 0x019899 (0x26766 bytes) 000426 1715333308.051926 MIRROR_SPLIT mirror 8 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000427 1715333308.131324 trunc from 0x01989a to 0x008030 (0x11869 bytes) 000429 1715333308.138621 trunc from 0x008031 to 0x021978 (0x19948 bytes) 000430 1715333308.149610 read 0x01be72 thru 0x01ff61 (0x40f0 bytes) 000433 1715333308.157616 write 0x027d78 thru 0x02b947 (0x3bd0 bytes) 000434 1715333308.158530 write 0x0042e3 thru 0x007826 (0x3544 bytes) 000435 1715333308.163720 trunc from 0x02b948 to 0x01d39c (0xe5ab bytes) 000437 1715333308.170929 mapwrite 0x0127dd thru 0x021b5c (0xf380 bytes) 000438 1715333308.222016 trunc from 0x021b5d to 0x01f779 (0x23e3 bytes) 000439 1715333308.228781 write 0x01f2b3 thru 0x02233b (0x3089 bytes) 000440 1715333308.229532 write 0x0317d9 thru 0x03bb97 (0xa3bf bytes) 000441 1715333308.230172 trunc from 0x03bb98 to 0x001a5e (0x3a139 bytes) 000442 1715333308.237332 trunc from 0x001a5f to 0x02cc58 (0x2b1fa bytes) 000443 1715333308.242697 write 0x02938e thru 0x02d3a5 (0x4018 bytes) 000444 1715333308.247615 write 0x03275a thru 0x03fc4f (0xd4f6 bytes) 000448 1715333308.249778 write 0x03402f thru 0x03dce8 (0x9cba bytes) 000449 1715333308.250627 mapread 0x003d6f thru 0x00bdc2 (0x8054 bytes) 000452 1715333308.258156 mapread 0x019427 thru 0x025628 (0xc202 bytes) 000453 1715333308.266740 mapread 0x028c8d thru 0x03843b (0xf7af bytes) 000454 1715333308.269974 mapwrite 0x00cd82 thru 0x018f5f (0xc1de bytes) 000457 1715333308.307490 trunc from 0x03fc50 to 0x036570 (0x96df bytes) 000458 1715333308.314495 read 0x03353a thru 0x036570 (0x3037 bytes) 000461 1715333308.315582 write 0x02863e thru 0x02b4fd (0x2ec0 bytes) 000462 1715333308.318165 MIRROR_EXTEND to 13 mirrors 000465 1715333308.560404 trunc from 0x036571 to 0x028784 (0xddec bytes) 000466 1715333308.613079 read 0x00dcad thru 0x017e27 (0xa17b bytes) 000468 1715333308.621594 read 0x00e4d4 thru 0x015957 (0x7484 bytes) 000469 1715333308.622099 trunc from 0x028785 to 0x03119c (0x8a18 bytes) 000470 1715333308.632329 write 0x032edb thru 0x03a868 (0x798e bytes) 000471 1715333308.633370 write 0x011aa5 thru 0x016791 (0x4ced bytes) 000473 1715333308.635980 MIRROR_EXTEND to 14 mirrors 000475 1715333308.875733 write 0x02f8fc thru 0x0367e9 (0x6eee bytes) 000477 1715333308.914759 read 0x009b58 thru 0x014456 (0xa8ff bytes) 000479 1715333308.926972 read 0x012f1e thru 0x01e807 (0xb8ea bytes) 000480 1715333308.935349 mapread 0x023e63 thru 0x027357 (0x34f5 bytes) 000481 1715333308.936026 mapread 0x01eda1 thru 0x025c13 (0x6e73 bytes) 000484 1715333308.942194 MIRROR_RESYNC 14 mirrors 000485 1715333309.645276 trunc from 0x03a869 to 0x010526 (0x2a342 bytes) 000488 1715333309.685275 trunc from 0x010527 to 0x03aa34 (0x2a50e bytes) 000490 1715333309.695953 mapwrite 0x018c01 thru 0x01bba6 (0x2fa6 bytes) 000492 1715333309.749310 MIRROR_SPLIT mirror 5 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000493 1715333309.822237 read 0x01b276 thru 0x01b48e (0x219 bytes) 000494 1715333309.824820 write 0x02773f thru 0x02b0e0 (0x39a2 bytes) 000496 1715333309.830648 MIRROR_RESYNC 13 mirrors 000497 1715333310.425655 trunc from 0x03aa35 to 0x02d28b (0xd7a9 bytes) 000498 1715333310.459479 write 0x033992 thru 0x03ffff (0xc66e bytes) 000499 1715333310.460449 write 0x0301d9 thru 0x0376d3 (0x74fb bytes) 000501 1715333310.463761 trunc from 0x040000 to 0x02d48d (0x12b72 bytes) 000503 1715333310.475320 mapread 0x026859 thru 0x02d48d (0x6c35 bytes) 000504 1715333310.484223 MIRROR_EXTEND to 14 mirrors 000505 1715333310.710485 write 0x023230 thru 0x02e773 (0xb544 bytes) 000506 1715333310.749480 mapread 0x00f197 thru 0x01a031 (0xae9b bytes) 000507 1715333310.766583 mapwrite 0x0258c3 thru 0x02f879 (0x9fb7 bytes) 000508 1715333310.827450 MIRROR_SPLIT mirror 13 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000511 1715333310.912424 trunc from 0x02f87a to 0x0191a6 (0x166d3 bytes) 000512 1715333310.918441 mapwrite 0x03fe2c thru 0x03ffff (0x1d4 bytes) 000514 1715333310.957118 trunc from 0x040000 to 0x01deb1 (0x2214e bytes) 000516 1715333310.966121 MIRROR_RESYNC 13 mirrors 000518 1715333311.604258 trunc from 0x01deb2 to 0x003c28 (0x1a289 bytes) 000520 1715333311.635248 mapread 0x002c05 thru 0x003c28 (0x1024 bytes) 000521 1715333311.642111 mapwrite 0x03949d thru 0x03ffff (0x6b63 bytes) 000523 1715333311.686769 trunc from 0x040000 to 0x03a03a (0x5fc5 bytes) 000525 1715333311.700351 write 0x013d1a thru 0x01ed0a (0xaff1 bytes) 000526 1715333311.709582 read 0x010fdc thru 0x018c1d (0x7c42 bytes) 000527 1715333311.712502 MIRROR_RESYNC 13 mirrors 000531 1715333312.328897 trunc from 0x03a03b to 0x00009a (0x39fa0 bytes) 000533 1715333312.352365 mapread 0x000043 thru 0x00009a (0x58 bytes) 000535 1715333312.355415 trunc from 0x00009b to 0x0131ab (0x13111 bytes) 000538 1715333312.363201 trunc from 0x0131ac to 0x014562 (0x13b7 bytes) 000540 1715333312.368715 mapread 0x012025 thru 0x0136fd (0x16d9 bytes) 000542 1715333312.373786 write 0x012e7d thru 0x01f89a (0xca1e bytes) 000543 1715333312.376895 MIRROR_SPLIT mirror 6 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000545 1715333312.447934 MIRROR_EXTEND to 13 mirrors 000546 1715333312.636020 write 0x00f868 thru 0x018f7c (0x9715 bytes) 000547 1715333312.661058 trunc from 0x01f89b to 0x03e803 (0x1ef69 bytes) 000550 1715333312.668005 mapwrite 0x02c7ba thru 0x034030 (0x7877 bytes) 000551 1715333312.700356 MIRROR_RESYNC 13 mirrors 000552 1715333313.254462 mapread 0x01a236 thru 0x028d18 (0xeae3 bytes) 000556 1715333313.269456 trunc from 0x03e804 to 0x01c6df (0x22124 bytes) 000559 1715333313.306584 MIRROR_SPLIT mirror 6 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000560 1715333313.399272 read 0x00fa75 thru 0x01bde2 (0xc36e bytes) 000561 1715333313.404882 write 0x00a1d6 thru 0x00d92e (0x3759 bytes) 000562 1715333313.411213 read 0x01acfb thru 0x01c6df (0x19e5 bytes) 000564 1715333313.417791 trunc from 0x01c6e0 to 0x0318e6 (0x15207 bytes) 000566 1715333313.429379 trunc from 0x0318e7 to 0x037c09 (0x6323 bytes) 000568 1715333313.435992 mapwrite 0x030859 thru 0x0344d8 (0x3c80 bytes) 000569 1715333313.476176 write 0x03227e thru 0x03ce67 (0xabea bytes) 000572 1715333313.477730 read 0x028a09 thru 0x02c833 (0x3e2b bytes) 000573 1715333313.478131 write 0x00694b thru 0x007a17 (0x10cd bytes) 000575 1715333313.479106 write 0x00b52f thru 0x00e185 (0x2c57 bytes) 000577 1715333313.480060 write 0x0143b0 thru 0x01e3e6 (0xa037 bytes) 000578 1715333313.480883 mapwrite 0x03a316 thru 0x03ffff (0x5cea bytes) 000581 1715333313.533050 read 0x030722 thru 0x039d0e (0x95ed bytes) 000583 1715333313.533893 trunc from 0x040000 to 0x024cc8 (0x1b337 bytes) 000584 1715333313.541519 mapwrite 0x005ff4 thru 0x010f07 (0xaf14 bytes) 000588 1715333313.582453 write 0x0165f2 thru 0x02453c (0xdf4b bytes) 000589 1715333313.583743 trunc from 0x024cc9 to 0x02238b (0x293d bytes) 000592 1715333313.593753 mapread 0x00a0d5 thru 0x00f0de (0x500a bytes) 000594 1715333313.597480 MIRROR_SPLIT mirror 4 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000596 1715333313.678781 read 0x015d78 thru 0x01b493 (0x571c bytes) 000597 1715333313.684457 read 0x009202 thru 0x009ed9 (0xcd8 bytes) 000599 1715333313.690602 write 0x0085cf thru 0x013e98 (0xb8ca bytes) 000602 1715333313.692303 write 0x03231f thru 0x03ffff (0xdce1 bytes) 000603 1715333313.693792 write 0x036f1f thru 0x03ffff (0x90e1 bytes) 000604 1715333313.697070 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000609 1715333313.789508 mapwrite 0x0103e7 thru 0x011504 (0x111e bytes) 000610 1715333313.825574 trunc from 0x040000 to 0x00661d (0x399e2 bytes) 000611 1715333313.833376 write 0x01efa4 thru 0x027fea (0x9047 bytes) 000612 1715333313.836222 MIRROR_RESYNC 10 mirrors 000615 1715333314.371311 write 0x002c26 thru 0x00a345 (0x7720 bytes) 000616 1715333314.401400 trunc from 0x027feb to 0x018586 (0xfa64 bytes) 000619 1715333314.414735 MIRROR_EXTEND to 11 mirrors 000621 1715333314.626404 mapwrite 0x0317a5 thru 0x03e82c (0xd088 bytes) 000623 1715333314.721519 MIRROR_EXTEND to 12 mirrors 000624 1715333314.891281 read 0x009c85 thru 0x00e5f6 (0x4972 bytes) 000626 1715333314.897315 read 0x0039f6 thru 0x00f0b7 (0xb6c2 bytes) 000632 1715333314.913245 mapwrite 0x039b0f thru 0x03ffff (0x64f1 bytes) 000633 1715333314.991029 write 0x00ac22 thru 0x01159b (0x697a bytes) 000634 1715333314.997263 MIRROR_EXTEND to 13 mirrors 000635 1715333315.198341 write 0x00fee7 thru 0x011da3 (0x1ebd bytes) 000637 1715333315.228600 mapread 0x003f24 thru 0x00843a (0x4517 bytes) 000640 1715333315.233583 trunc from 0x040000 to 0x02318b (0x1ce74 bytes) 000642 1715333315.242295 MIRROR_EXTEND to 14 mirrors 000644 1715333315.449957 MIRROR_RESYNC 14 mirrors 000647 1715333315.998275 write 0x0394f5 thru 0x03ffff (0x6b0b bytes) 000648 1715333316.020326 mapwrite 0x03809c thru 0x03ffff (0x7f64 bytes) 000650 1715333316.048338 mapwrite 0x031a39 thru 0x033a49 (0x2011 bytes) 000651 1715333316.072601 read 0x032904 thru 0x03acb8 (0x83b5 bytes) 000652 1715333316.073156 mapread 0x00c83f thru 0x01b4a1 (0xec63 bytes) 000653 1715333316.083397 MIRROR_SPLIT mirror 4 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000655 1715333316.154938 read 0x00b4de thru 0x01b2e5 (0xfe08 bytes) 000657 1715333316.161210 MIRROR_EXTEND to 14 mirrors 000659 1715333316.318094 read 0x027c0b thru 0x02878d (0xb83 bytes) 000660 1715333316.322726 MIRROR_EXTEND to 15 mirrors 000661 1715333316.509079 write 0x022a69 thru 0x02c684 (0x9c1c bytes) 000663 1715333316.531661 mapwrite 0x004670 thru 0x010d2e (0xc6bf bytes) 000664 1715333316.563176 MIRROR_EXTEND to 16 mirrors 000665 1715333316.719056 MIRROR_SPLIT mirror 7 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000667 1715333316.811951 trunc from 0x040000 to 0x026f0b (0x190f4 bytes) 000668 1715333316.848428 MIRROR_RESYNC 15 mirrors 000670 1715333317.446179 write 0x00bd66 thru 0x012d59 (0x6ff4 bytes) 000671 1715333317.477180 trunc from 0x026f0c to 0x000294 (0x26c77 bytes) 000673 1715333317.487500 MIRROR_SPLIT mirror 2 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000674 1715333317.561714 mapread 0x00018d thru 0x000294 (0x108 bytes) 000675 1715333317.566054 MIRROR_RESYNC 14 mirrors 000677 1715333318.001444 read 0x00026c thru 0x000294 (0x29 bytes) 000678 1715333318.001733 read 0x0000b8 thru 0x000294 (0x1dd bytes) 000679 1715333318.001959 trunc from 0x000295 to 0x005e63 (0x5bcf bytes) 000680 1715333318.025891 write 0x037623 thru 0x03ffff (0x89dd bytes) 000683 1715333318.027427 mapwrite 0x02af5b thru 0x037dcf (0xce75 bytes) 000686 1715333318.054700 mapwrite 0x0252df thru 0x02d06e (0x7d90 bytes) 000689 1715333318.077608 write 0x031b4e thru 0x03ef76 (0xd429 bytes) 000696 1715333318.079920 mapread 0x0077b2 thru 0x01022b (0x8a7a bytes) 000697 1715333318.080510 mapwrite 0x01087c thru 0x013daa (0x352f bytes) 000699 1715333318.102507 MIRROR_SPLIT mirror 13 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000700 1715333318.164702 MIRROR_EXTEND to 14 mirrors 000701 1715333318.317767 MIRROR_EXTEND to 15 mirrors 000702 1715333318.488559 read 0x013ea8 thru 0x023c64 (0xfdbd bytes) 000703 1715333318.491725 mapread 0x00aeb5 thru 0x00e09f (0x31eb bytes) 000704 1715333318.498389 read 0x01ee8a thru 0x02cb4d (0xdcc4 bytes) 000705 1715333318.502464 trunc from 0x040000 to 0x02ead0 (0x1152f bytes) 000710 1715333318.531912 read 0x00c12c thru 0x01641f (0xa2f4 bytes) 000714 1715333318.541003 MIRROR_RESYNC 15 mirrors 000719 1715333319.071637 MIRROR_RESYNC 15 mirrors 000720 1715333319.168370 read 0x019358 thru 0x01bbde (0x2887 bytes) 000722 1715333319.171614 trunc from 0x02ead1 to 0x00190a (0x2d1c6 bytes) 000723 1715333319.199194 write 0x004f6f thru 0x00ac97 (0x5d29 bytes) 000725 1715333319.201967 MIRROR_EXTEND to 16 mirrors 000726 1715333319.367928 mapread 0x0050d2 thru 0x00ac97 (0x5bc6 bytes) 000730 1715333319.377761 MIRROR_SPLIT mirror 2 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000731 1715333319.461742 write 0x025d16 thru 0x02b648 (0x5933 bytes) 000733 1715333319.485509 trunc from 0x02b649 to 0x00f13e (0x1c50a bytes) 000734 1715333319.495999 mapread 0x0092a2 thru 0x00f13e (0x5e9d bytes) 000736 1715333319.508556 MIRROR_RESYNC 15 mirrors 000737 1715333320.012734 MIRROR_SPLIT mirror 1 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000739 1715333320.133798 read 0x005a4e thru 0x009b7e (0x4131 bytes) 000740 1715333320.138860 mapwrite 0x036133 thru 0x03ffff (0x9ecd bytes) 000741 1715333320.193764 read 0x028c91 thru 0x03177a (0x8aea bytes) 000744 1715333320.198967 read 0x023a2a thru 0x02d763 (0x9d3a bytes) 000745 1715333320.201019 MIRROR_RESYNC 14 mirrors 000747 1715333320.628587 MIRROR_SPLIT mirror 10 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000748 1715333320.748671 write 0x00f1ae thru 0x00fc2b (0xa7e bytes) 000752 1715333320.771437 mapwrite 0x00292b thru 0x0101ac (0xd882 bytes) 000753 1715333320.800752 mapread 0x03f5a5 thru 0x03ffff (0xa5b bytes) 000754 1715333320.804005 trunc from 0x040000 to 0x027a6f (0x18590 bytes) 000755 1715333320.814730 mapwrite 0x00d804 thru 0x01712a (0x9927 bytes) 000756 1715333320.839480 read 0x0157e4 thru 0x01dc91 (0x84ae bytes) 000757 1715333320.842339 trunc from 0x027a70 to 0x0142f5 (0x1377a bytes) 000760 1715333320.849230 mapread 0x0049b6 thru 0x0139bd (0xf008 bytes) 000762 1715333320.850244 write 0x02847f thru 0x03221c (0x9d9e bytes) 000764 1715333320.851358 read 0x0098b4 thru 0x0160d4 (0xc821 bytes) 000765 1715333320.856023 write 0x020c07 thru 0x025b38 (0x4f32 bytes) 000768 1715333320.857300 read 0x0029fd thru 0x002f2a (0x52e bytes) 000770 1715333320.857993 write 0x00c74e thru 0x00ed9c (0x264f bytes) 000772 1715333320.858832 read 0x00676e thru 0x013053 (0xc8e6 bytes) 000774 1715333320.859436 mapwrite 0x0142b1 thru 0x01d737 (0x9487 bytes) 000775 1715333320.889758 MIRROR_EXTEND to 14 mirrors 000776 1715333321.031489 trunc from 0x03221d to 0x033fe7 (0x1dcb bytes) 000777 1715333321.062604 read 0x02a03d thru 0x033fe7 (0x9fab bytes) 000778 1715333321.065386 read 0x033991 thru 0x033fe7 (0x657 bytes) 000779 1715333321.065777 trunc from 0x033fe8 to 0x03a7cc (0x67e5 bytes) 000780 1715333321.071437 read 0x001161 thru 0x009435 (0x82d5 bytes) 000782 1715333321.077025 MIRROR_RESYNC 14 mirrors 000787 1715333321.743287 mapwrite 0x007efb thru 0x00c222 (0x4328 bytes) 000789 1715333321.820333 write 0x008a26 thru 0x010f69 (0x8544 bytes) 000790 1715333321.821750 mapread 0x02e153 thru 0x033158 (0x5006 bytes) 000791 1715333321.831161 MIRROR_RESYNC 14 mirrors 000794 1715333322.383723 trunc from 0x03a7cd to 0x016031 (0x2479b bytes) 000795 1715333322.405381 mapwrite 0x00e50b thru 0x01c28c (0xdd82 bytes) 000796 1715333322.444897 write 0x01f614 thru 0x022321 (0x2d0e bytes) 000798 1715333322.445871 write 0x029c95 thru 0x02f433 (0x579f bytes) 000799 1715333322.446416 trunc from 0x02f434 to 0x03726c (0x7e39 bytes) 000800 1715333322.450966 read 0x002e8d thru 0x0059ed (0x2b61 bytes) 000802 1715333322.453259 write 0x0063f1 thru 0x011ef0 (0xbb00 bytes) 000804 1715333322.456243 write 0x01604b thru 0x01c2fe (0x62b4 bytes) 000807 1715333322.457623 write 0x009762 thru 0x013f0f (0xa7ae bytes) 000810 1715333322.460392 MIRROR_RESYNC 14 mirrors 000812 1715333322.974041 read 0x00bcd0 thru 0x00c23d (0x56e bytes) 000815 1715333322.976552 mapwrite 0x005014 thru 0x006095 (0x1082 bytes) 000816 1715333323.040005 mapread 0x0054ab thru 0x011d75 (0xc8cb bytes) 000820 1715333323.045313 mapread 0x0069ac thru 0x0151ae (0xe803 bytes) 000821 1715333323.046007 read 0x022c14 thru 0x02f93b (0xcd28 bytes) 000823 1715333323.053403 MIRROR_SPLIT mirror 13 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000824 1715333323.122454 MIRROR_SPLIT mirror 10 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000825 1715333323.189296 MIRROR_RESYNC 12 mirrors 000827 1715333323.623730 MIRROR_RESYNC 12 mirrors 000828 1715333323.720270 mapread 0x023069 thru 0x025c82 (0x2c1a bytes) 000834 1715333323.731179 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000835 1715333323.850436 read 0x0099bc thru 0x015cf6 (0xc33b bytes) 000836 1715333323.854115 write 0x03a92c thru 0x03ffff (0x56d4 bytes) 000837 1715333323.877089 mapread 0x00edc5 thru 0x012a08 (0x3c44 bytes) 000838 1715333323.887395 write 0x002107 thru 0x0021f8 (0xf2 bytes) 000839 1715333323.893136 MIRROR_RESYNC 11 mirrors 000840 1715333324.239650 MIRROR_RESYNC 11 mirrors 000841 1715333324.324473 trunc from 0x040000 to 0x01d084 (0x22f7b bytes) 000844 1715333324.345931 mapwrite 0x025a57 thru 0x026525 (0xacf bytes) 000846 1715333324.381236 mapread 0x0242ac thru 0x026525 (0x227a bytes) 000848 1715333324.386270 MIRROR_EXTEND to 12 mirrors 000849 1715333324.539300 trunc from 0x026526 to 0x00cd6d (0x197b8 bytes) 000850 1715333324.565513 write 0x034be5 thru 0x03a28c (0x56a8 bytes) 000851 1715333324.566481 read 0x025c37 thru 0x02e411 (0x87db bytes) 000852 1715333324.570478 mapread 0x02caf0 thru 0x033368 (0x6879 bytes) 000854 1715333324.577697 write 0x03b357 thru 0x03ccc5 (0x196f bytes) 000857 1715333324.579240 mapwrite 0x03a566 thru 0x03ffff (0x5a9a bytes) 000858 1715333324.617794 mapwrite 0x003384 thru 0x007dea (0x4a67 bytes) 000859 1715333324.648710 MIRROR_EXTEND to 13 mirrors 000860 1715333324.844833 MIRROR_SPLIT mirror 6 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000863 1715333324.972325 mapread 0x032b77 thru 0x035228 (0x26b2 bytes) 000867 1715333324.997574 mapwrite 0x03c06a thru 0x03ffff (0x3f96 bytes) 000869 1715333325.066849 trunc from 0x040000 to 0x00723b (0x38dc4 bytes) 000870 1715333325.080043 write 0x022d5b thru 0x02760e (0x48b4 bytes) 000871 1715333325.082698 MIRROR_EXTEND to 13 mirrors 000872 1715333325.295391 trunc from 0x02760f to 0x028ab2 (0x14a4 bytes) 000875 1715333325.330131 write 0x0391f1 thru 0x03ffff (0x6e0f bytes) 000879 1715333325.331899 mapwrite 0x019165 thru 0x027a1f (0xe8bb bytes) 000880 1715333325.376418 read 0x0203e1 thru 0x026cbd (0x68dd bytes) 000882 1715333325.378238 write 0x00d788 thru 0x0166d6 (0x8f4f bytes) 000884 1715333325.385048 trunc from 0x040000 to 0x000eae (0x3f151 bytes) 000890 1715333325.396499 mapread 0x000d7d thru 0x000eae (0x132 bytes) 000896 1715333325.400711 trunc from 0x000eaf to 0x02df43 (0x2d095 bytes) 000897 1715333325.407115 mapwrite 0x017276 thru 0x026731 (0xf4bc bytes) 000898 1715333325.458972 trunc from 0x02df44 to 0x020675 (0xd8ce bytes) 000900 1715333325.468576 mapread 0x01d3ae thru 0x020675 (0x32c8 bytes) 000901 1715333325.473738 MIRROR_EXTEND to 14 mirrors 000904 1715333325.677348 mapwrite 0x014332 thru 0x0225a2 (0xe271 bytes) 000905 1715333325.759741 trunc from 0x0225a3 to 0x03068c (0xe0ea bytes) 000906 1715333325.771148 trunc from 0x03068d to 0x030c62 (0x5d6 bytes) 000908 1715333325.780444 MIRROR_SPLIT mirror 2 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000909 1715333325.865456 write 0x02f644 thru 0x039028 (0x99e5 bytes) 000910 1715333325.869466 MIRROR_RESYNC 13 mirrors 000911 1715333326.500970 read 0x0028eb thru 0x004f04 (0x261a bytes) 000912 1715333326.503718 mapread 0x0044d6 thru 0x00484e (0x379 bytes) 000914 1715333326.506241 MIRROR_EXTEND to 14 mirrors 000915 1715333326.768659 trunc from 0x039029 to 0x02faeb (0x953d bytes) 000920 1715333326.803562 trunc from 0x02faec to 0x0124b6 (0x1d635 bytes) 000922 1715333326.812248 trunc from 0x0124b7 to 0x01da11 (0xb55b bytes) 000923 1715333326.819019 trunc from 0x01da12 to 0x034e96 (0x17485 bytes) 000924 1715333326.824410 write 0x02eb8c thru 0x034ef4 (0x6369 bytes) 000925 1715333326.829997 write 0x03df30 thru 0x03ffff (0x20d0 bytes) 000926 1715333326.830573 mapwrite 0x02a28a thru 0x039211 (0xef88 bytes) 000929 1715333326.884137 read 0x03fc90 thru 0x03ffff (0x370 bytes) 000930 1715333326.884613 read 0x01b1c6 thru 0x01f63f (0x447a bytes) 000931 1715333326.893695 read 0x02bb66 thru 0x02d686 (0x1b21 bytes) 000935 1715333326.895039 trunc from 0x040000 to 0x030143 (0xfebc bytes) 000936 1715333326.905370 mapwrite 0x022785 thru 0x022efa (0x776 bytes) 000941 1715333326.938343 read 0x00da51 thru 0x01afa7 (0xd557 bytes) 000943 1715333326.939344 read 0x004ced thru 0x01084f (0xbb63 bytes) 000946 1715333326.940420 read 0x007385 thru 0x00a9a1 (0x361d bytes) 000947 1715333326.940817 trunc from 0x030144 to 0x02fea0 (0x2a3 bytes) 000949 1715333326.950898 MIRROR_EXTEND to 15 mirrors 000950 1715333327.188886 write 0x0101f3 thru 0x016e7e (0x6c8c bytes) 000952 1715333327.225024 mapread 0x00cc89 thru 0x011e0e (0x5186 bytes) 000954 1715333327.231236 write 0x02fb03 thru 0x03129d (0x179b bytes) 000955 1715333327.238437 write 0x016eeb thru 0x024705 (0xd81b bytes) 000956 1715333327.243315 read 0x0076d7 thru 0x009c40 (0x256a bytes) 000958 1715333327.248620 MIRROR_EXTEND to 16 mirrors 000959 1715333327.526472 mapread 0x014dc3 thru 0x024944 (0xfb82 bytes) 000960 1715333327.542192 MIRROR_RESYNC 16 mirrors 000961 1715333328.431632 mapread 0x00ac4a thru 0x0148a3 (0x9c5a bytes) 000962 1715333328.444980 trunc from 0x03129e to 0x030ae1 (0x7bc bytes) 000963 1715333328.486965 write 0x010807 thru 0x013784 (0x2f7e bytes) 000964 1715333328.492610 mapwrite 0x0374c2 thru 0x03dd49 (0x6888 bytes) 000965 1715333328.543895 read 0x0228d3 thru 0x026f57 (0x4685 bytes) 000969 1715333328.549861 read 0x02662f thru 0x02af81 (0x4953 bytes) 000970 1715333328.560664 MIRROR_SPLIT mirror 2 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000971 1715333328.637390 trunc from 0x03dd4a to 0x00cc89 (0x310c0 bytes) 000973 1715333328.646337 MIRROR_EXTEND to 16 mirrors 000975 1715333328.852593 trunc from 0x00cc8a to 0x000ad7 (0xc1b2 bytes) 000980 1715333328.891637 trunc from 0x000ad8 to 0x017dbd (0x172e6 bytes) 000981 1715333328.898697 write 0x00b3c1 thru 0x011033 (0x5c73 bytes) 000983 1715333328.902908 write 0x0305e5 thru 0x03e9f2 (0xe40e bytes) 000990 1715333328.905829 mapwrite 0x01655d thru 0x0255f5 (0xf099 bytes) 000991 1715333328.945657 mapwrite 0x01284b thru 0x01c112 (0x98c8 bytes) 000992 1715333328.979451 mapread 0x00f384 thru 0x01245c (0x30d9 bytes) 000993 1715333328.980430 write 0x01f7a0 thru 0x02e6e0 (0xef41 bytes) 000994 1715333328.984620 mapread 0x009aba thru 0x013e4a (0xa391 bytes) 000997 1715333328.988462 write 0x029ee2 thru 0x02dc8a (0x3da9 bytes) 000999 1715333328.989614 write 0x00dd52 thru 0x014836 (0x6ae5 bytes) 001000 1715333328.992374 MIRROR_RESYNC 16 mirrors All operations completed A-OK! PASS 70a (33s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200a: stress test ===================== 05:28:52 (1715333332) Starting client: oleg253-client.virtnet: -o user_xattr,flock oleg253-server@tcp:/lustre /mnt/lustre2 Starting client: oleg253-client.virtnet: -o user_xattr,flock oleg253-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1210656 .. Extending file size to 1655040 .. done Extending file size to 6021760 .. Extending file size to 4115232 .. Extending file size to 6943520 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 5363808 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6099840 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8380000 .. Extending file size to 8016224 .. Extending file size to 8941536 .. Extending file size to 8285088 .. Extending file size to 8660448 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8675392 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8818112 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8889952 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9002272 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9054880 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9108672 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 3016 3017 3018 3020 3022 192.168.202.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg253-client.virtnet /mnt/lustre2 (opts:) 192.168.202.153@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg253-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 05:29:58 (1715333398) Starting client: oleg253-client.virtnet: -o user_xattr,flock oleg253-server@tcp:/lustre /mnt/lustre2 Starting client: oleg253-client.virtnet: -o user_xattr,flock oleg253-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3378944 .. Extending file size to 3917280 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7364224 .. Extending file size to 4951744 .. Extending file size to 7469760 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8435616 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8881312 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8677568 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8768832 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8772576 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9302016 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8915168 .. Waiting 6138 6139 6140 6142 6145 failed 192.168.202.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg253-client.virtnet /mnt/lustre2 (opts:) 192.168.202.153@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg253-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 05:31:03 (1715333463) Starting client: oleg253-client.virtnet: -o user_xattr,flock oleg253-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4864 2203648 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4864 2203648 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0359924 s, 58.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0708627 s, 29.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg253-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 05:31:09 (1715333469) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0271262 s, 77.3 MB/s PASS 202 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 203: mirror file preserve mirror ID === 05:31:12 (1715333472) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x803:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x803:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x804:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x806:0x0] } PASS 203 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 05:31:15 (1715333475) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0266526 s, 78.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x805:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x807:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x808:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 05:31:19 (1715333479) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0274239 s, 76.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x806:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x807:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x809:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x806:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x807:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x809:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x80a:0x0] } PASS 204b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 05:31:22 (1715333482) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0245567 s, 85.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x808:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x80b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x808:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x80b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x80c:0x0] } PASS 204c (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 05:31:46 (1715333506) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg253-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0197547 s, 53.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x809:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x80e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x80d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x809:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x80e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x80d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x80a:0x0] } PASS 204d (43s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 05:32:31 (1715333551) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg253-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x80b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x80c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0229888 s, 91.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x80b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x80c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x80f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x80b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x80c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x80f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (39s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 05:33:11 (1715333591) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg253-server mds-ost sync done. Creating new pool oleg253-server: Pool lustre.test_204f created Adding targets to pool oleg253-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg253-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0237346 s, 88.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x80d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x810:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x80d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x810:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg253-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg253-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg253-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (49s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 205: lfs mirror extend to set prefer and nocompr flags ========================================================== 05:34:01 (1715333641) PASS 205 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 05:34:05 (1715333645) oleg253-server: Pool lustre.test_206 created Adding targets to pool oleg253-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg253-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg253-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg253-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg253-server: Pool lustre.test_206 destroyed PASS 206 (11s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 05:34:18 (1715333658) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0160275 s, 65.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00908403 s, 115 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x54562034 Mirror 3: 0x54562034 PASS 207 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 05:34:21 (1715333661) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 05:34:24 (1715333664) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 05:34:26 (1715333666) Starting client: oleg253-client.virtnet: -o user_xattr,flock oleg253-server@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0137855 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xb70000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x9b0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x9b0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x9b0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xaf0000 Read this (4 bytes): BBBB 192.168.202.153@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg253-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 05:34:40 (1715333680) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00293436 s, 2.8 MB/s the very first read read with warm cache PASS 209b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210a: handle broken mirrored lovea ==== 05:34:44 (1715333684) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0085474 s, 123 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x49d lmm_fid: [0x200000404:0x49d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x49d lmm_fid: [0x200000404:0x49d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x816:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835101 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-05-10 05:34:44.000000000 -0400 Modify: 2024-05-10 05:34:44.000000000 -0400 Change: 2024-05-10 05:34:44.000000000 -0400 Birth: - PASS 210a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 05:34:47 (1715333687) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) debug_raw_pointers=0 debug_raw_pointers=0 == sanity-flr test complete, duration 885 sec ============ 05:34:48 (1715333688) === sanity-flr: start cleanup 05:34:48 (1715333688) === === sanity-flr: finish cleanup 05:34:49 (1715333689) ===