-----============= acceptance-small: sanity-flr ============----- Fri May 10 05:20:40 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 6 201 44c === sanity-flr: start setup 05:20:46 (1715332846) === oleg301-client.virtnet: executing check_config_client /mnt/lustre oleg301-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg301-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b5920000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b5920000.idle_timeout=debug disable quota as required oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 05:20:57 (1715332857) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f7500] debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0a: lfs mirror create with -N option == 05:20:59 (1715332859) PASS 0a (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 05:21:06 (1715332866) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 05:21:10 (1715332870) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0d: lfs mirror extend with -N option == 05:21:13 (1715332873) PASS 0d (8s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 05:21:23 (1715332883) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 05:21:27 (1715332887) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0g: lfs mirror create flags support === 05:21:31 (1715332891) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b5920000.stats=clear osc.lustre-OST0001-osc-ffff8800b5920000.stats=clear PASS 0g (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 05:21:36 (1715332896) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 05:21:42 (1715332902) PASS 0j (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 05:21:46 (1715332906) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 1: create components with setstripe options ========================================================== 05:21:50 (1715332910) lfs mirror mirror: cannot get UNLOCK lease, ext 4: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 2: create components from existing files ========================================================== 05:21:55 (1715332915) PASS 2 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 3: create components from files located on different MDTs ========================================================== 05:22:00 (1715332920) PASS 3 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 05:22:05 (1715332925) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 05:22:09 (1715332929) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 7: Create mirror with DoM component === 05:22:14 (1715332934) PASS 7 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 05:22:18 (1715332938) 15+0 records in 15+0 records out 15728640 bytes (16 MB) copied, 0.503493 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.041278 s, 25.4 MB/s PASS 21 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 05:22:24 (1715332944) 11+0 records in 11+0 records out 11534336 bytes (12 MB) copied, 0.108184 s, 107 MB/s PASS 22 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 05:22:29 (1715332949) Stopping /mnt/lustre-ost1 (opts:) on oleg301-server oleg301-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5920000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5920000.ost_server_uuid in DISCONN state after 2 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg301-server oleg301-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5920000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5920000.ost_server_uuid in DISCONN state after 3 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg301-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (32s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 05:23:04 (1715332984) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.157315 s, 53.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg301-server oleg301-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5920000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5920000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (13s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33a: read can choose available mirror to read ========================================================== 05:23:19 (1715332999) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg301-server oleg301-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5920000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5920000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg301-server oleg301-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5920000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5920000.ost_server_uuid in DISCONN state after 3 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg301-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 05:23:44 (1715333024) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.947735 s, 55.3 MB/s -rw-r--r-- 1 root root 52428800 May 10 05:23 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.329s user 0m0.000s sys 0m0.120s Stopping /mnt/lustre-ost1 (opts:) on oleg301-server oleg301-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5920000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5920000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.265s user 0m0.000s sys 0m0.115s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg301-server oleg301-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5920000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5920000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.403s user 0m0.000s sys 0m0.133s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg301-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg301-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b5920000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5920000.ost_server_uuid in FULL state after 0 sec PASS 33b (26s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 05:24:13 (1715333053) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 05:24:16 (1715333056) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 05:24:20 (1715333060) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 35: allow to write to mirrored files == 05:24:23 (1715333063) PASS 35 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36a: write to mirrored files ========== 05:24:29 (1715333069) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.204869 s, 61.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 05:24:36 (1715333076) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0157609 s, 66.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0133779 s, 78.4 MB/s PASS 36b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 05:24:41 (1715333081) PASS 36c (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 05:24:47 (1715333087) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00145812 s, 702 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000705213 s, 1.5 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00199138 s, 514 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00126106 s, 812 kB/s flv=10 olv=10 PASS 36d (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 37: mirror I/O API verification ======= 05:24:53 (1715333093) create test files with size 15746242 .. 1fc5faac9378ac122918112554d60ef1 - c2ec0cfe4d76b9b80e91dcb8f162b290 - 782c9140f1a22b4c5371b602711309de - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg301-client.virtnet /mnt/lustre (opts:) Starting client: oleg301-client.virtnet: -o user_xattr,flock oleg301-server@tcp:/lustre /mnt/lustre PASS 37 (7s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 38: resync ============================ 05:25:02 (1715333102) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 05:25:08 (1715333108) PASS 39 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 05:25:12 (1715333112) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0198485 s, 52.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x35:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x36:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x38:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109024 s, 96.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x39:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3a:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x37:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 41: lfs mirror resync check =========== 05:25:16 (1715333116) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.030792 s, 68.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0986032 s, 42.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0565751 s, 37.1 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 42: lfs mirror verify ================= 05:25:21 (1715333121) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43a: mirror pick on write ============= 05:25:24 (1715333124) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 05:25:27 (1715333127) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0129363 s, 81.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44a: lfs mirror split check =========== 05:25:32 (1715333132) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.112166 s, 28.0 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 44a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44b: mirror split does not create stale file ========================================================== 05:25:38 (1715333138) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x40:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x44:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 05:25:42 (1715333142) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.10026 s, 105 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg301-client.virtnet /mnt/lustre (opts:) Starting client: oleg301-client.virtnet: -o user_xattr,flock oleg301-server@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 05:25:49 (1715333149) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800a2466800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0929123 s, 113 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg301-client.virtnet /mnt/lustre (opts:) Starting client: oleg301-client.virtnet: -o user_xattr,flock oleg301-server@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff88013698b000.xattr_cache=1 PASS 44e (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 05:25:55 (1715333155) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (11s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 46: Verify setstripe --copy option ==== 05:26:08 (1715333168) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (6s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 47: Verify mirror obj alloc =========== 05:26:16 (1715333176) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 48: Verify snapshot mirror ============ 05:26:19 (1715333179) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.088051 s, 35.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0549858 s, 57.2 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 9a7f8308572777b2c2eee6a4571ebdfb - mirror 1 checksum: 523d45d9f800b4d31be4255c633088e0 - mirror 2 checksum: 9a7f8308572777b2c2eee6a4571ebdfb - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri May 10 05:26:20 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x49:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri May 10 05:26:20 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4a:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 9a7f8308572777b2c2eee6a4571ebdfb - mirror 1 checksum: 523d45d9f800b4d31be4255c633088e0 - mirror 2 checksum: 523d45d9f800b4d31be4255c633088e0 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri May 10 05:26:20 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x49:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri May 10 05:26:20 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4a:0x0] } PASS 48 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 49a: FIEMAP upon FLR file ============= 05:26:25 (1715333185) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0167024 s, 62.8 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50A: mirror split update layout generation ========================================================== 05:26:29 (1715333189) Failing mds1 on oleg301-server Stopping /mnt/lustre-mds1 (opts:) on oleg301-server 05:26:37 (1715333197) shut down Failover mds1 to oleg301-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-MDT0000 05:26:50 (1715333210) targets are mounted 05:26:50 (1715333210) facet_failover done oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0117323 s, 89.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4c:0x0] } PASS 50A (29s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 05:27:00 (1715333220) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0202136 s, 1.0 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00619081 s, 3.3 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50b: mirror rsync handles sparseness == 05:27:05 (1715333225) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0140461 s, 1.5 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00446866 s, 4.6 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 05:27:10 (1715333230) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00324878 s, 5.0 MB/s PASS 50c (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50d: mirror rsync keep holes ========== 05:27:14 (1715333234) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0179214 s, 1.1 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 05:27:19 (1715333239) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00310261 s, 1.3 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0187526 s, 218 kB/s PASS 60a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 05:27:24 (1715333244) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 60b (7s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 05:27:33 (1715333253) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 05:27:55 (1715333275) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 05:28:14 (1715333294) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 05:28:34 (1715333314) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70: mirror create and split race ====== 05:28:39 (1715333319) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 30614 mirror split pid 30615 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4716: 30614 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4716: 30615 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (63s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70a: flr mode fsx test ================ 05:29:44 (1715333384) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 3384 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read truncating to largest ever: 0x121ee 000005 1715333385.036590 trunc from 00000000 to 0x0121ed (0x121ee bytes) fallocating to largest ever: 0x1e090 000008 1715333385.045891 fallocate 0x00ecd7 thru 0x01e08f (0xf3b9 bytes) 000009 1715333385.050468 mapwrite 0x02a626 thru 0x02da0c (0x33e7 bytes) 000010 1715333385.076767 mapread 0x00052b thru 0x00ac94 (0xa76a bytes) truncating to largest ever: 0x2bd38 000011 1715333385.084807 trunc from 0x02da0d to 0x02bd37 (0x1cd5 bytes) 000013 1715333385.091715 mapwrite 0x03958e thru 0x03f8dd (0x6350 bytes) truncating to largest ever: 0x3987f 000014 1715333385.122165 trunc from 0x03f8de to 0x03987e (0x605f bytes) 000017 1715333385.133565 trunc from 0x03987f to 0x0353c8 (0x44b6 bytes) 000018 1715333385.140847 mapread 0x021038 thru 0x02f08d (0xe056 bytes) 000019 1715333385.149094 punch from 0x029aa9 to 0x0353c8 (0xb920 bytes) 000020 1715333385.152806 mapwrite 0x02405c thru 0x02715d (0x3102 bytes) fallocating to largest ever: 0x3b5dc 000021 1715333385.165620 fallocate 0x0347b5 thru 0x03b5db (0x6e27 bytes) 000022 1715333385.168607 punch from 0x022c0a to 0x02544d (0x2844 bytes) 000023 1715333385.172981 write 0x006528 thru 0x00a74a (0x4223 bytes) 000024 1715333385.173848 mapwrite 0x033af5 thru 0x03628f (0x279b bytes) 000025 1715333385.191786 read 0x01e58f thru 0x02245b (0x3ecd bytes) 000028 1715333385.195413 trunc from 0x03b5dc to 0x005974 (0x35c67 bytes) 000030 1715333385.204983 mapwrite 0x0339e0 thru 0x03a1c7 (0x67e8 bytes) 000031 1715333385.235842 trunc from 0x03a1c8 to 0x01a374 (0x1fe53 bytes) 000032 1715333385.246220 trunc from 0x01a375 to 0x0032e8 (0x1708c bytes) 000033 1715333385.256810 read 0x003067 thru 0x0032e8 (0x282 bytes) 000034 1715333385.257241 fallocate 0x01c42b thru 0x02abc8 (0xe79e bytes) 000036 1715333385.260415 write 0x03db29 thru 0x03ea35 (0xf0d bytes) 000037 1715333385.261049 trunc from 0x03ea36 to 0x010d1d (0x2dd18 bytes) 000039 1715333385.267539 write 0x0147c9 thru 0x01ef7a (0xa7b2 bytes) 000041 1715333385.268899 mapwrite 0x034fab thru 0x03ffff (0xb055 bytes) 000044 1715333385.296993 mapread 0x02f091 thru 0x038117 (0x9087 bytes) 000045 1715333385.297624 fallocate 0x013b00 thru 0x0172f3 (0x37f4 bytes) 000049 1715333385.301679 fallocate 0x021a61 thru 0x02a225 (0x87c5 bytes) 000050 1715333385.305171 read 0x0328bb thru 0x03ffff (0xd745 bytes) 000053 1715333385.307441 mapwrite 0x034e30 thru 0x03af7a (0x614b bytes) 000056 1715333385.316574 trunc from 0x040000 to 0x014388 (0x2bc77 bytes) 000059 1715333385.327110 fallocate 0x01c392 thru 0x02b45b (0xf0ca bytes) 000060 1715333385.329870 mapwrite 0x03a78e thru 0x03ffff (0x5872 bytes) 000061 1715333385.356550 mapread 0x034307 thru 0x0356cc (0x13c6 bytes) 000062 1715333385.357135 read 0x01c5e3 thru 0x022b33 (0x6551 bytes) 000065 1715333385.358186 mapread 0x01f9a5 thru 0x02c153 (0xc7af bytes) 000066 1715333385.358823 read 0x034a74 thru 0x03ffff (0xb58c bytes) 000067 1715333385.359242 mapread 0x01c3c2 thru 0x0276cc (0xb30b bytes) 000068 1715333385.362595 MIRROR_EXTEND to 2 mirrors 000069 1715333385.577565 write 0x0380a0 thru 0x03b0df (0x3040 bytes) 000070 1715333385.598020 punch from 0x002442 to 0x00ce82 (0xaa41 bytes) 000071 1715333385.602912 MIRROR_EXTEND to 3 mirrors 000072 1715333385.861209 write 0x0352eb thru 0x03ffff (0xad15 bytes) 000073 1715333385.884573 write 0x02d840 thru 0x038006 (0xa7c7 bytes) 000074 1715333385.887694 mapwrite 0x01a7a6 thru 0x0254ad (0xad08 bytes) 000075 1715333385.908809 read 0x0148b0 thru 0x01a041 (0x5792 bytes) truncating to largest ever: 0x3f6a3 000076 1715333385.909404 trunc from 0x040000 to 0x03f6a2 (0x95d bytes) 000077 1715333385.933201 punch from 0x030eff to 0x03558f (0x4691 bytes) 000078 1715333385.937339 trunc from 0x03f6a3 to 0x036e8a (0x8818 bytes) 000079 1715333385.945649 mapread 0x027ffb thru 0x02a9f6 (0x29fc bytes) 000081 1715333385.946877 mapread 0x00ed1f thru 0x014084 (0x5366 bytes) 000084 1715333385.953643 MIRROR_RESYNC 3 mirrors 000085 1715333386.356234 mapread 0x01adbf thru 0x024f0d (0xa14f bytes) 000086 1715333386.367325 mapwrite 0x038383 thru 0x03ffff (0x7c7d bytes) 000087 1715333386.396785 mapread 0x007cf1 thru 0x00e98b (0x6c9b bytes) 000088 1715333386.400762 write 0x01abfb thru 0x024265 (0x966b bytes) 000089 1715333386.410231 mapwrite 0x032946 thru 0x0385e5 (0x5ca0 bytes) 000090 1715333386.426296 write 0x03c8f5 thru 0x03ffff (0x370b bytes) 000091 1715333386.427023 fallocate 0x00621f thru 0x01436d (0xe14f bytes) 000092 1715333386.431660 write 0x0162d5 thru 0x01ed6d (0x8a99 bytes) 000093 1715333386.432465 trunc from 0x040000 to 0x0134b3 (0x2cb4c bytes) 000094 1715333386.442192 punch from 0x0056af to 0x008fb0 (0x3902 bytes) 000096 1715333386.448884 MIRROR_RESYNC 3 mirrors 000098 1715333386.778438 MIRROR_EXTEND to 4 mirrors 000099 1715333387.018845 mapread 0x00a9ad thru 0x00f2d8 (0x492c bytes) 000101 1715333387.032556 fallocate 0x015fca thru 0x0168ec (0x923 bytes) 000102 1715333387.053343 mapread 0x010e1e thru 0x0134b3 (0x2696 bytes) 000103 1715333387.060883 trunc from 0x0134b4 to 0x016da9 (0x38f6 bytes) truncating to largest ever: 0x3ff06 000104 1715333387.078208 trunc from 0x016daa to 0x03ff05 (0x2915c bytes) 000105 1715333387.088382 punch from 0x032507 to 0x03babc (0x95b6 bytes) 000106 1715333387.091945 write 0x005b05 thru 0x013e4f (0xe34b bytes) 000107 1715333387.096675 punch from 0x00a1aa to 0x0199ef (0xf846 bytes) 000108 1715333387.103263 punch from 0x01a495 to 0x025a9e (0xb60a bytes) 000109 1715333387.105644 fallocate 0x00366c thru 0x00507d (0x1a12 bytes) 000110 1715333387.108271 mapread 0x00fdc2 thru 0x0151cf (0x540e bytes) 000111 1715333387.118199 MIRROR_EXTEND to 5 mirrors 000113 1715333387.389394 mapwrite 0x01bb5e thru 0x0218e5 (0x5d88 bytes) 000114 1715333387.444213 mapwrite 0x0171ce thru 0x01fd23 (0x8b56 bytes) 000115 1715333387.466839 mapread 0x018001 thru 0x0180c2 (0xc2 bytes) 000116 1715333387.468120 trunc from 0x03ff06 to 0x01e555 (0x219b0 bytes) 000117 1715333387.480521 punch from 0x005f88 to 0x014cac (0xed25 bytes) 000118 1715333387.486761 MIRROR_EXTEND to 6 mirrors 000119 1715333387.759009 mapread 0x00503b thru 0x00df6d (0x8f33 bytes) 000123 1715333387.795618 mapread 0x012a72 thru 0x01d5a8 (0xab37 bytes) 000124 1715333387.812170 trunc from 0x01e556 to 0x010353 (0xe202 bytes) 000126 1715333387.833333 fallocate 0x01e73b thru 0x0249a9 (0x626f bytes) 000127 1715333387.835879 write 0x01db66 thru 0x01f3cb (0x1866 bytes) 000128 1715333387.836814 write 0x0353f7 thru 0x03ffff (0xac09 bytes) fallocating to largest ever: 0x40000 000129 1715333387.838017 fallocate 0x03d675 thru 0x03ffff (0x298b bytes) 000130 1715333387.840595 trunc from 0x040000 to 0x0208a7 (0x1f758 bytes) 000131 1715333387.851469 MIRROR_EXTEND to 7 mirrors 000133 1715333388.118226 punch from 0x011605 to 0x013731 (0x212d bytes) 000134 1715333388.141111 trunc from 0x0208a8 to 0x03f8b2 (0x1f00b bytes) 000135 1715333388.155316 read 0x02ce8c thru 0x0374d4 (0xa649 bytes) 000136 1715333388.161222 trunc from 0x03f8b3 to 0x03e9e8 (0xeca bytes) 000138 1715333388.173459 fallocate 0x000e6b thru 0x00f53a (0xe6d0 bytes) 000139 1715333388.177073 write 0x00360c thru 0x00b53f (0x7f34 bytes) 000140 1715333388.184224 write 0x03135b thru 0x03678f (0x5435 bytes) 000141 1715333388.184957 read 0x0028cb thru 0x00703f (0x4775 bytes) 000143 1715333388.195465 MIRROR_EXTEND to 8 mirrors 000144 1715333388.451432 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000145 1715333388.548084 write 0x01b6a7 thru 0x01bc79 (0x5d3 bytes) 000147 1715333388.565602 write 0x0059c0 thru 0x0112f2 (0xb933 bytes) 000148 1715333388.572016 MIRROR_EXTEND to 8 mirrors 000149 1715333388.819759 punch from 0x039d9c to 0x03e9e8 (0x4c4d bytes) 000150 1715333388.839367 fallocate 0x028b11 thru 0x0339f7 (0xaee7 bytes) 000151 1715333388.841627 fallocate 0x0036bb thru 0x00be94 (0x87da bytes) 000153 1715333388.845765 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000154 1715333388.913876 punch from 0x0247c3 to 0x02a068 (0x58a6 bytes) 000156 1715333388.920649 read 0x0066e1 thru 0x0138c0 (0xd1e0 bytes) 000157 1715333388.927635 MIRROR_RESYNC 7 mirrors 000158 1715333389.704027 trunc from 0x03e9e9 to 0x01e17a (0x2086e bytes) 000159 1715333389.725085 write 0x00dc1b thru 0x00e91c (0xd02 bytes) 000160 1715333389.730518 punch from 0x001905 to 0x006236 (0x4932 bytes) 000161 1715333389.734781 punch from 0x0045f9 to 0x0050d9 (0xae1 bytes) 000162 1715333389.737684 mapread 0x0122e6 thru 0x01b649 (0x9364 bytes) 000163 1715333389.752797 write 0x01e6cf thru 0x026041 (0x7973 bytes) 000164 1715333389.753705 fallocate 0x01c75c thru 0x01f675 (0x2f1a bytes) 000165 1715333389.758516 mapwrite 0x021e27 thru 0x023367 (0x1541 bytes) 000166 1715333389.776362 mapwrite 0x011c69 thru 0x012c5a (0xff2 bytes) 000167 1715333389.786656 mapread 0x025a15 thru 0x026041 (0x62d bytes) 000168 1715333389.787412 fallocate 0x001ae7 thru 0x0031b8 (0x16d2 bytes) 000170 1715333389.791607 mapread 0x021d86 thru 0x026041 (0x42bc bytes) 000171 1715333389.795000 MIRROR_RESYNC 7 mirrors 000172 1715333390.373974 punch from 0x00c4f7 to 0x00cb03 (0x60d bytes) 000173 1715333390.393694 read 0x00a754 thru 0x0120e2 (0x798f bytes) 000174 1715333390.401208 punch from 0x009fe1 to 0x010d6d (0x6d8d bytes) 000175 1715333390.406164 punch from 0x00502d to 0x005369 (0x33d bytes) 000176 1715333390.412741 write 0x00ae82 thru 0x00e31b (0x349a bytes) 000178 1715333390.419325 trunc from 0x026042 to 0x03f0c1 (0x19080 bytes) 000179 1715333390.432526 punch from 0x01848e to 0x01e825 (0x6398 bytes) 000180 1715333390.434678 trunc from 0x03f0c2 to 0x006af4 (0x385cd bytes) 000181 1715333390.442510 mapread 0x00143a thru 0x0060cf (0x4c96 bytes) 000182 1715333390.449645 trunc from 0x006af5 to 0x02e5d8 (0x27ae4 bytes) 000183 1715333390.461544 write 0x0025ed thru 0x0043ef (0x1e03 bytes) 000184 1715333390.462317 mapwrite 0x00e80b thru 0x00ea61 (0x257 bytes) 000187 1715333390.476586 punch from 0x0250ad to 0x02e5d8 (0x952c bytes) 000188 1715333390.480805 fallocate 0x008750 thru 0x011bdb (0x948c bytes) 000189 1715333390.483206 trunc from 0x02e5d9 to 0x0306b6 (0x20de bytes) 000191 1715333390.491331 mapread 0x00e695 thru 0x01bd67 (0xd6d3 bytes) 000192 1715333390.496261 mapread 0x01bf9e thru 0x027b39 (0xbb9c bytes) 000193 1715333390.501485 mapread 0x025eea thru 0x0306b6 (0xa7cd bytes) 000194 1715333390.504185 trunc from 0x0306b7 to 0x03a440 (0x9d8a bytes) 000195 1715333390.513942 trunc from 0x03a441 to 0x02a759 (0xfce7 bytes) 000196 1715333390.521138 mapread 0x022dfd thru 0x02a759 (0x795d bytes) 000198 1715333390.522250 punch from 0x018242 to 0x01c098 (0x3e57 bytes) 000199 1715333390.524988 read 0x02456f thru 0x02a759 (0x61eb bytes) 000200 1715333390.525422 read 0x02a5cc thru 0x02a759 (0x18e bytes) 000201 1715333390.525802 trunc from 0x02a75a to 0x036d6f (0xc616 bytes) 000202 1715333390.533762 trunc from 0x036d70 to 0x003142 (0x33c2d bytes) 000203 1715333390.543313 read 0x000f1a thru 0x003142 (0x2229 bytes) 000206 1715333390.548069 fallocate 0x007eac thru 0x016e15 (0xef6a bytes) 000207 1715333390.551440 mapread 0x00a4b7 thru 0x01209d (0x7be7 bytes) 000208 1715333390.557329 read 0x008016 thru 0x00a198 (0x2183 bytes) 000209 1715333390.558133 trunc from 0x016e16 to 0x00c8b9 (0xa55c bytes) 000211 1715333390.567980 punch from 0x004d4f to 0x00c8b9 (0x7b6b bytes) 000215 1715333390.573152 trunc from 0x00c8ba to 0x008767 (0x4152 bytes) 000216 1715333390.587236 MIRROR_EXTEND to 8 mirrors 000217 1715333390.826471 fallocate 0x024d89 thru 0x02f1ea (0xa462 bytes) 000218 1715333390.847401 mapwrite 0x0118a8 thru 0x018c40 (0x7399 bytes) 000219 1715333390.872765 mapread 0x015732 thru 0x0200f6 (0xa9c5 bytes) 000221 1715333390.882279 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000222 1715333390.996603 write 0x032501 thru 0x03d20f (0xad0f bytes) 000223 1715333391.000121 mapread 0x027686 thru 0x02f01a (0x7995 bytes) 000224 1715333391.014659 MIRROR_RESYNC 7 mirrors 000225 1715333391.628608 fallocate 0x014e9f thru 0x021d40 (0xcea2 bytes) 000226 1715333391.646881 mapwrite 0x0252a2 thru 0x02dc0d (0x896c bytes) 000227 1715333391.669557 write 0x00e6bd thru 0x01402c (0x5970 bytes) 000228 1715333391.674368 read 0x0378a8 thru 0x03d20f (0x5968 bytes) 000229 1715333391.678319 trunc from 0x03d210 to 0x01c7b7 (0x20a58 bytes) 000230 1715333391.686330 trunc from 0x01c7b8 to 0x032fa9 (0x167f2 bytes) 000231 1715333391.692398 mapwrite 0x03fb51 thru 0x03ffff (0x4af bytes) 000232 1715333391.713033 punch from 0x0235e4 to 0x02a0cd (0x6aea bytes) 000233 1715333391.715558 write 0x035ea3 thru 0x0392a8 (0x3406 bytes) 000234 1715333391.720431 punch from 0x038b05 to 0x03ffff (0x74fb bytes) 000235 1715333391.726571 punch from 0x02dcf9 to 0x02fb12 (0x1e1a bytes) 000237 1715333391.730052 trunc from 0x040000 to 0x028136 (0x17ec9 bytes) 000238 1715333391.737950 punch from 0x01010c to 0x018c96 (0x8b8b bytes) 000239 1715333391.742590 MIRROR_RESYNC 7 mirrors 000240 1715333392.328539 fallocate 0x0065ed thru 0x0115ca (0xafde bytes) 000242 1715333392.350267 write 0x03faca thru 0x03ffff (0x536 bytes) 000243 1715333392.350990 punch from 0x03fc72 to 0x03ffff (0x38e bytes) 000244 1715333392.356439 fallocate 0x016190 thru 0x0256b4 (0xf525 bytes) 000246 1715333392.360219 read 0x03c151 thru 0x03ce42 (0xcf2 bytes) 000247 1715333392.363720 mapwrite 0x006f53 thru 0x00a781 (0x382f bytes) 000248 1715333392.387999 write 0x01a10a thru 0x01d6dd (0x35d4 bytes) 000249 1715333392.394998 punch from 0x00eebd to 0x01557b (0x66bf bytes) 000250 1715333392.400582 mapread 0x035dde thru 0x03ffff (0xa222 bytes) 000252 1715333392.412299 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000254 1715333392.509755 read 0x03d719 thru 0x03ffff (0x28e7 bytes) 000255 1715333392.512853 punch from 0x01817e to 0x0271d2 (0xf055 bytes) 000256 1715333392.517716 MIRROR_RESYNC 6 mirrors 000257 1715333393.119560 punch from 0x01bac6 to 0x01ead8 (0x3013 bytes) 000258 1715333393.137313 fallocate 0x020177 thru 0x0215ac (0x1436 bytes) 000259 1715333393.141698 MIRROR_EXTEND to 7 mirrors 000260 1715333393.380751 mapwrite 0x027238 thru 0x03505a (0xde23 bytes) 000261 1715333393.447901 mapwrite 0x0181c0 thru 0x01c4e8 (0x4329 bytes) 000262 1715333393.466654 read 0x030ac2 thru 0x03c258 (0xb797 bytes) 000263 1715333393.472431 write 0x0165cf thru 0x01e2b1 (0x7ce3 bytes) 000264 1715333393.478029 MIRROR_EXTEND to 8 mirrors 000265 1715333393.756306 trunc from 0x040000 to 0x0368fe (0x9701 bytes) 000266 1715333393.784491 read 0x01879d thru 0x01c34f (0x3bb3 bytes) 000267 1715333393.788322 write 0x02bb85 thru 0x033812 (0x7c8e bytes) 000268 1715333393.792944 mapwrite 0x015f19 thru 0x01a764 (0x484c bytes) 000269 1715333393.810594 write 0x0120eb thru 0x01721e (0x5134 bytes) 000270 1715333393.811349 mapread 0x00fb87 thru 0x012b07 (0x2f81 bytes) 000271 1715333393.812297 fallocate 0x001c6e thru 0x00ccd8 (0xb06b bytes) 000272 1715333393.818412 MIRROR_RESYNC 8 mirrors 000274 1715333394.832959 read 0x01b951 thru 0x01c3c8 (0xa78 bytes) 000276 1715333394.839146 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000278 1715333394.969838 mapwrite 0x011286 thru 0x0126e9 (0x1464 bytes) 000279 1715333395.019455 fallocate 0x008a61 thru 0x010d7b (0x831b bytes) 000280 1715333395.022178 write 0x014c77 thru 0x0175e9 (0x2973 bytes) 000281 1715333395.026961 mapread 0x02c29f thru 0x02d5ef (0x1351 bytes) 000282 1715333395.033670 write 0x038070 thru 0x03ffff (0x7f90 bytes) 000283 1715333395.034647 fallocate 0x039a07 thru 0x03ffff (0x65f9 bytes) 000284 1715333395.038509 read 0x01eb0b thru 0x02c93c (0xde32 bytes) 000285 1715333395.046794 write 0x0398d8 thru 0x03ffff (0x6728 bytes) 000286 1715333395.047597 mapwrite 0x01455a thru 0x023368 (0xee0f bytes) 000289 1715333395.062402 write 0x02fc3b thru 0x037fdf (0x83a5 bytes) 000290 1715333395.074270 MIRROR_RESYNC 7 mirrors 000291 1715333395.662127 mapread 0x004178 thru 0x00e89f (0xa728 bytes) 000294 1715333395.670538 write 0x01f98b thru 0x02e113 (0xe789 bytes) 000295 1715333395.697451 punch from 0x022edd to 0x02dfdc (0xb100 bytes) 000296 1715333395.708590 read 0x0186b5 thru 0x021c12 (0x955e bytes) 000297 1715333395.709179 read 0x01dd33 thru 0x02bfac (0xe27a bytes) 000300 1715333395.718943 fallocate 0x01444c thru 0x0159e0 (0x1595 bytes) 000302 1715333395.726788 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000303 1715333395.813083 mapwrite 0x03ac9e thru 0x03ffff (0x5362 bytes) 000304 1715333395.838560 write 0x01051a thru 0x01ca67 (0xc54e bytes) 000305 1715333395.852668 mapwrite 0x000b21 thru 0x00254c (0x1a2c bytes) 000306 1715333395.870100 trunc from 0x040000 to 0x01b3b2 (0x24c4d bytes) 000307 1715333395.891136 trunc from 0x01b3b3 to 0x0038fe (0x17ab4 bytes) 000309 1715333395.904181 fallocate 0x01c57a thru 0x025dbc (0x9843 bytes) 000310 1715333395.908043 punch from 0x01f723 to 0x025dbc (0x669a bytes) 000311 1715333395.912080 read 0x009a1d thru 0x019282 (0xf866 bytes) 000312 1715333395.920048 MIRROR_RESYNC 6 mirrors 000314 1715333396.417406 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000315 1715333396.555145 mapread 0x0071af thru 0x008645 (0x1497 bytes) 000316 1715333396.561874 punch from 0x00e8b7 to 0x019b07 (0xb251 bytes) 000317 1715333396.579393 mapread 0x01ec3d thru 0x025dbc (0x7180 bytes) 000318 1715333396.592636 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000319 1715333396.716372 write 0x0188d6 thru 0x01c2fd (0x3a28 bytes) 000321 1715333396.730161 fallocate 0x017c92 thru 0x018fe8 (0x1357 bytes) 000322 1715333396.735752 punch from 0x01eb01 to 0x025dbc (0x72bc bytes) 000323 1715333396.741129 MIRROR_EXTEND to 5 mirrors 000324 1715333397.025382 mapwrite 0x0265fa thru 0x0307bb (0xa1c2 bytes) 000325 1715333397.070249 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000326 1715333397.161210 fallocate 0x03d03f thru 0x03ffff (0x2fc1 bytes) 000328 1715333397.165171 fallocate 0x004ab2 thru 0x00c532 (0x7a81 bytes) 000329 1715333397.167905 mapread 0x019da8 thru 0x023256 (0x94af bytes) 000330 1715333397.185309 fallocate 0x02c486 thru 0x036de1 (0xa95c bytes) 000331 1715333397.188444 mapread 0x005aa1 thru 0x00dedf (0x843f bytes) 000332 1715333397.196102 trunc from 0x040000 to 0x0384b0 (0x7b4f bytes) 000333 1715333397.207449 mapread 0x00c428 thru 0x017e2b (0xba04 bytes) 000334 1715333397.212067 MIRROR_EXTEND to 5 mirrors 000336 1715333397.498913 mapwrite 0x0087bd thru 0x00c6d2 (0x3f16 bytes) 000337 1715333397.546563 mapread 0x029891 thru 0x03533c (0xbaac bytes) 000338 1715333397.558210 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000339 1715333397.662960 MIRROR_EXTEND to 5 mirrors 000340 1715333397.960593 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000344 1715333398.047033 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000347 1715333398.137870 trunc from 0x0384b1 to 0x015d28 (0x22788 bytes) 000348 1715333398.161608 punch from 0x012e2a to 0x015d28 (0x2eff bytes) 000349 1715333398.164219 punch from 0x00398f to 0x0137c6 (0xfe38 bytes) 000351 1715333398.167785 write 0x03a159 thru 0x03f6ae (0x5556 bytes) 000352 1715333398.169259 write 0x03f3af thru 0x03ffff (0xc51 bytes) 000353 1715333398.172592 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000354 1715333398.281316 fallocate 0x02183c thru 0x02f9ac (0xe171 bytes) 000355 1715333398.284857 mapread 0x018b2f thru 0x025235 (0xc707 bytes) 000356 1715333398.302270 trunc from 0x040000 to 0x01a13e (0x25ec1 bytes) 000357 1715333398.314728 write 0x03844c thru 0x03ae6e (0x2a23 bytes) 000358 1715333398.315668 write 0x008833 thru 0x00e882 (0x6050 bytes) 000359 1715333398.321591 mapwrite 0x00480f thru 0x00b464 (0x6c56 bytes) 000360 1715333398.341195 write 0x01f8c8 thru 0x0294ee (0x9c27 bytes) 000361 1715333398.347887 mapread 0x022f9d thru 0x02946a (0x64ce bytes) 000362 1715333398.348611 trunc from 0x03ae6f to 0x01ad23 (0x2014b bytes) 000363 1715333398.362884 read 0x014249 thru 0x01ad23 (0x6adb bytes) 000364 1715333398.363418 trunc from 0x01ad24 to 0x02378e (0x8a6b bytes) 000366 1715333398.374917 fallocate 0x021112 thru 0x024b41 (0x3a30 bytes) 000367 1715333398.380793 trunc from 0x02378f to 0x022e29 (0x965 bytes) 000368 1715333398.390103 trunc from 0x022e2a to 0x02f1aa (0xc381 bytes) 000369 1715333398.395483 punch from 0x013192 to 0x019cbe (0x6b2d bytes) 000372 1715333398.397727 trunc from 0x02f1ab to 0x02dc97 (0x1513 bytes) 000374 1715333398.403681 fallocate 0x03097a thru 0x03119d (0x824 bytes) 000375 1715333398.406254 read 0x01602b thru 0x01a1ff (0x41d5 bytes) 000376 1715333398.412537 fallocate 0x008bc1 thru 0x01210d (0x954d bytes) 000377 1715333398.418347 read 0x0090cf thru 0x0127de (0x9710 bytes) 000378 1715333398.418773 mapread 0x00c95b thru 0x011136 (0x47dc bytes) 000381 1715333398.420116 fallocate 0x01c5f5 thru 0x02a0c7 (0xdad3 bytes) 000382 1715333398.423979 mapwrite 0x008c49 thru 0x0120dd (0x9495 bytes) 000384 1715333398.440215 fallocate 0x038627 thru 0x03ffff (0x79d9 bytes) 000385 1715333398.445100 MIRROR_RESYNC 2 mirrors 000387 1715333398.676010 read 0x027367 thru 0x02b9d4 (0x466e bytes) 000388 1715333398.682577 mapwrite 0x03c44d thru 0x03ffff (0x3bb3 bytes) 000389 1715333398.748728 punch from 0x001d76 to 0x005615 (0x38a0 bytes) 000391 1715333398.752515 fallocate 0x0144da thru 0x01fba7 (0xb6ce bytes) 000396 1715333398.760247 MIRROR_RESYNC 2 mirrors 000397 1715333399.004566 read 0x00a19c thru 0x00dc9b (0x3b00 bytes) 000399 1715333399.008392 read 0x023071 thru 0x02d017 (0x9fa7 bytes) 000400 1715333399.019220 fallocate 0x0312d9 thru 0x03c7ca (0xb4f2 bytes) 000401 1715333399.042080 fallocate 0x037a08 thru 0x03ffff (0x85f8 bytes) 000403 1715333399.047815 trunc from 0x040000 to 0x0019b3 (0x3e64c bytes) 000405 1715333399.076578 MIRROR_RESYNC 2 mirrors 000407 1715333399.251363 punch from 0x0018a7 to 0x0019b3 (0x10d bytes) 000408 1715333399.271677 MIRROR_RESYNC 2 mirrors 000409 1715333399.442525 punch from 0x000a07 to 0x0019b3 (0xfad bytes) 000410 1715333399.470616 MIRROR_EXTEND to 3 mirrors 000411 1715333399.679525 mapread 0x001275 thru 0x0019b3 (0x73f bytes) 000412 1715333399.681420 mapread 0x00084f thru 0x0019b3 (0x1165 bytes) 000414 1715333399.685025 read 0x00183a thru 0x0019b3 (0x17a bytes) 000415 1715333399.685298 trunc from 0x0019b4 to 0x031c77 (0x302c4 bytes) 000416 1715333399.713493 punch from 0x02df27 to 0x031c77 (0x3d51 bytes) 000417 1715333399.718760 MIRROR_RESYNC 3 mirrors 000418 1715333399.992677 mapwrite 0x01d1f8 thru 0x026f34 (0x9d3d bytes) 000419 1715333400.049316 trunc from 0x031c78 to 0x015a51 (0x1c226 bytes) 000421 1715333400.062979 write 0x00c5bb thru 0x00dcc2 (0x1708 bytes) 000422 1715333400.071076 MIRROR_EXTEND to 4 mirrors 000425 1715333400.337620 punch from 0x00bd22 to 0x015a51 (0x9d30 bytes) 000426 1715333400.358311 punch from 0x00111e to 0x005a07 (0x48ea bytes) 000427 1715333400.361420 trunc from 0x015a52 to 0x02357e (0xdb2d bytes) 000428 1715333400.376732 read 0x01b06b thru 0x022a97 (0x7a2d bytes) 000429 1715333400.381591 mapwrite 0x01bf73 thru 0x022073 (0x6101 bytes) 000430 1715333400.394087 mapread 0x0030cc thru 0x00aa4d (0x7982 bytes) 000431 1715333400.403368 punch from 0x019af0 to 0x01c8d3 (0x2de4 bytes) 000432 1715333400.407065 mapwrite 0x010578 thru 0x01d657 (0xd0e0 bytes) 000434 1715333400.428321 fallocate 0x004353 thru 0x00ade5 (0x6a93 bytes) 000435 1715333400.431485 fallocate 0x00f7fb thru 0x018e8e (0x9694 bytes) 000436 1715333400.436787 MIRROR_EXTEND to 5 mirrors 000437 1715333400.688350 MIRROR_RESYNC 5 mirrors 000438 1715333401.004171 MIRROR_EXTEND to 6 mirrors 000439 1715333401.302824 mapread 0x007fda thru 0x011188 (0x91af bytes) 000440 1715333401.320394 punch from 0x010159 to 0x0128e2 (0x278a bytes) 000441 1715333401.346832 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000443 1715333401.429486 punch from 0x004c02 to 0x004e33 (0x232 bytes) 000444 1715333401.433674 write 0x000800 thru 0x00aa4e (0xa24f bytes) 000447 1715333401.445339 write 0x01120f thru 0x0173fd (0x61ef bytes) 000448 1715333401.456148 write 0x033638 thru 0x035c79 (0x2642 bytes) 000451 1715333401.458231 write 0x00792c thru 0x00a8b8 (0x2f8d bytes) 000454 1715333401.461480 punch from 0x00f614 to 0x013bcb (0x45b8 bytes) 000455 1715333401.471120 mapwrite 0x029560 thru 0x0385ff (0xf0a0 bytes) 000456 1715333401.513644 trunc from 0x038600 to 0x00c4ee (0x2c111 bytes) 000457 1715333401.526462 mapread 0x00479b thru 0x00a728 (0x5f8e bytes) 000458 1715333401.527192 read 0x0041ba thru 0x00c4ee (0x8335 bytes) 000459 1715333401.531975 trunc from 0x00c4ef to 0x02b135 (0x1ec47 bytes) 000460 1715333401.542880 trunc from 0x02b136 to 0x005fd6 (0x2515f bytes) 000461 1715333401.558989 MIRROR_RESYNC 5 mirrors 000462 1715333401.809858 punch from 0x00278e to 0x005fd6 (0x3849 bytes) 000463 1715333401.829086 MIRROR_EXTEND to 6 mirrors 000464 1715333402.078100 read 0x000660 thru 0x0059db (0x537c bytes) 000467 1715333402.095095 read 0x001a06 thru 0x005fd6 (0x45d1 bytes) 000469 1715333402.103501 read 0x005689 thru 0x005fd6 (0x94e bytes) 000470 1715333402.107554 trunc from 0x005fd7 to 0x01c98b (0x169b5 bytes) 000471 1715333402.141863 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000472 1715333402.228876 trunc from 0x01c98c to 0x00e989 (0xe002 bytes) 000473 1715333402.244064 mapwrite 0x0007b9 thru 0x002d61 (0x25a9 bytes) 000474 1715333402.272016 write 0x01e046 thru 0x02a44c (0xc407 bytes) 000475 1715333402.273613 trunc from 0x02a44d to 0x012773 (0x17cd9 bytes) 000476 1715333402.281573 trunc from 0x012774 to 0x032a37 (0x202c4 bytes) 000477 1715333402.289638 fallocate 0x03dfa2 thru 0x03ffff (0x205e bytes) 000478 1715333402.292292 write 0x01b039 thru 0x02a137 (0xf0ff bytes) 000479 1715333402.301209 fallocate 0x00c9d2 thru 0x017f47 (0xb576 bytes) 000480 1715333402.304526 mapwrite 0x00057b thru 0x00e259 (0xdcdf bytes) 000481 1715333402.327693 mapwrite 0x02ab80 thru 0x037125 (0xc5a6 bytes) 000482 1715333402.345805 read 0x03a212 thru 0x03aead (0xc9c bytes) 000483 1715333402.348869 MIRROR_EXTEND to 6 mirrors 000484 1715333402.663568 write 0x02705f thru 0x02e29a (0x723c bytes) 000485 1715333402.696062 mapread 0x02a442 thru 0x038cb0 (0xe86f bytes) 000487 1715333402.708783 write 0x02f0ae thru 0x02fbd9 (0xb2c bytes) 000488 1715333402.710601 mapread 0x037b14 thru 0x03ffff (0x84ec bytes) 000489 1715333402.711464 trunc from 0x040000 to 0x009a8f (0x36570 bytes) 000490 1715333402.725057 trunc from 0x009a90 to 0x023142 (0x196b3 bytes) 000491 1715333402.734245 read 0x005fc2 thru 0x0144e9 (0xe528 bytes) 000492 1715333402.742181 read 0x020402 thru 0x023142 (0x2d41 bytes) 000493 1715333402.748742 fallocate 0x01d308 thru 0x02644f (0x9148 bytes) 000494 1715333402.753211 fallocate 0x027abd thru 0x036339 (0xe87d bytes) 000495 1715333402.758054 mapwrite 0x033539 thru 0x03ffff (0xcac7 bytes) 000496 1715333402.796085 punch from 0x00a509 to 0x01579b (0xb293 bytes) 000497 1715333402.800368 punch from 0x0150e3 to 0x024e6b (0xfd89 bytes) 000498 1715333402.803792 mapwrite 0x00f9c1 thru 0x01cdea (0xd42a bytes) 000499 1715333402.863526 mapwrite 0x016638 thru 0x01acb2 (0x467b bytes) 000500 1715333402.879834 read 0x008aa9 thru 0x015427 (0xc97f bytes) 000501 1715333402.884833 mapwrite 0x038cbb thru 0x03ffff (0x7345 bytes) 000502 1715333402.900453 punch from 0x007156 to 0x012fee (0xbe99 bytes) 000503 1715333402.905298 write 0x025265 thru 0x02f4b4 (0xa250 bytes) 000504 1715333402.906284 mapwrite 0x02b593 thru 0x039085 (0xdaf3 bytes) 000505 1715333402.921044 mapwrite 0x01d891 thru 0x02930e (0xba7e bytes) 000506 1715333402.934569 mapwrite 0x0271a6 thru 0x02bec3 (0x4d1e bytes) 000507 1715333402.943871 read 0x0398f0 thru 0x03ffff (0x6710 bytes) 000508 1715333402.944350 punch from 0x028358 to 0x0365e7 (0xe290 bytes) 000509 1715333402.949159 mapwrite 0x0041eb thru 0x00f35d (0xb173 bytes) 000510 1715333402.971204 mapwrite 0x015184 thru 0x0207eb (0xb668 bytes) 000511 1715333402.985182 mapwrite 0x012353 thru 0x01af2c (0x8bda bytes) 000512 1715333402.997622 read 0x002e11 thru 0x00791a (0x4b0a bytes) 000513 1715333403.000724 MIRROR_EXTEND to 7 mirrors 000514 1715333403.314673 fallocate 0x03c00e thru 0x03ffff (0x3ff2 bytes) 000515 1715333403.332949 MIRROR_EXTEND to 8 mirrors 000516 1715333403.653417 trunc from 0x040000 to 0x00fdc2 (0x3023d bytes) 000517 1715333403.677640 fallocate 0x01af9a thru 0x026e69 (0xbed0 bytes) 000518 1715333403.682099 read 0x011c77 thru 0x016b79 (0x4f03 bytes) 000519 1715333403.689018 fallocate 0x03ce09 thru 0x03d527 (0x71f bytes) 000520 1715333403.693375 punch from 0x0284b7 to 0x02ea29 (0x6573 bytes) 000521 1715333403.696511 fallocate 0x039015 thru 0x03d52a (0x4516 bytes) 000522 1715333403.699684 fallocate 0x02f13b thru 0x03c0bc (0xcf82 bytes) 000523 1715333403.702691 mapwrite 0x0388a4 thru 0x03a247 (0x19a4 bytes) 000524 1715333403.730543 punch from 0x00b8e4 to 0x0116d9 (0x5df6 bytes) 000525 1715333403.733512 trunc from 0x03d52b to 0x005acd (0x37a5d bytes) 000528 1715333403.748577 fallocate 0x034cf1 thru 0x035a06 (0xd16 bytes) 000530 1715333403.752621 punch from 0x003abe to 0x005acd (0x2010 bytes) 000531 1715333403.757337 mapread 0x0039db thru 0x005acd (0x20f3 bytes) 000532 1715333403.763646 fallocate 0x0361b7 thru 0x03cdac (0x6bf6 bytes) 000533 1715333403.766910 write 0x0332a1 thru 0x039b0d (0x686d bytes) 000534 1715333403.768777 mapwrite 0x03f4bf thru 0x03ffff (0xb41 bytes) 000536 1715333403.792062 fallocate 0x008774 thru 0x01704c (0xe8d9 bytes) 000537 1715333403.795342 mapread 0x021a05 thru 0x021a19 (0x15 bytes) 000539 1715333403.802324 mapread 0x00f0ef thru 0x01585b (0x676d bytes) 000540 1715333403.807547 read 0x021e59 thru 0x029746 (0x78ee bytes) 000541 1715333403.812354 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000542 1715333403.922171 mapread 0x02e39a thru 0x03c599 (0xe200 bytes) 000543 1715333403.945965 MIRROR_EXTEND to 8 mirrors 000544 1715333404.257903 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000545 1715333404.374946 mapread 0x004600 thru 0x00e7fb (0xa1fc bytes) 000546 1715333404.391419 mapwrite 0x023792 thru 0x02af94 (0x7803 bytes) 000547 1715333404.450101 punch from 0x01c427 to 0x026620 (0xa1fa bytes) 000548 1715333404.453146 mapwrite 0x004126 thru 0x0125cb (0xe4a6 bytes) 000549 1715333404.476433 mapread 0x03beec thru 0x03ffff (0x4114 bytes) 000550 1715333404.481448 trunc from 0x040000 to 0x00c27f (0x33d80 bytes) 000551 1715333404.493091 mapwrite 0x012e8d thru 0x02105f (0xe1d3 bytes) 000552 1715333404.532067 fallocate 0x01fbaa thru 0x02f925 (0xfd7c bytes) 000553 1715333404.535040 punch from 0x00a038 to 0x014056 (0xa01f bytes) 000554 1715333404.538148 trunc from 0x021060 to 0x025cef (0x4c90 bytes) 000555 1715333404.547084 mapwrite 0x000ed7 thru 0x010b86 (0xfcb0 bytes) 000556 1715333404.572011 mapread 0x00eb6c thru 0x016123 (0x75b8 bytes) 000558 1715333404.573350 mapread 0x020f28 thru 0x025cef (0x4dc8 bytes) 000559 1715333404.576694 mapread 0x00d77c thru 0x01d564 (0xfde9 bytes) 000560 1715333404.577802 fallocate 0x006601 thru 0x0103d9 (0x9dd9 bytes) 000561 1715333404.581289 fallocate 0x0260e9 thru 0x026ada (0x9f2 bytes) 000563 1715333404.585211 write 0x034a57 thru 0x03d51e (0x8ac8 bytes) 000564 1715333404.589301 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000565 1715333404.695174 fallocate 0x024b3f thru 0x02714f (0x2611 bytes) 000566 1715333404.698782 write 0x0035ac thru 0x009979 (0x63ce bytes) 000567 1715333404.706973 mapread 0x005714 thru 0x013d45 (0xe632 bytes) 000568 1715333404.713829 punch from 0x00fcce to 0x013f23 (0x4256 bytes) 000569 1715333404.716788 mapread 0x011eed thru 0x021a63 (0xfb77 bytes) 000570 1715333404.727479 fallocate 0x015d6d thru 0x01a3ec (0x4680 bytes) 000571 1715333404.730398 read 0x01de48 thru 0x0207f1 (0x29aa bytes) 000572 1715333404.731451 write 0x009fb1 thru 0x00fd83 (0x5dd3 bytes) 000573 1715333404.732265 read 0x03ba20 thru 0x03cee9 (0x14ca bytes) 000574 1715333404.740889 mapwrite 0x00e0d5 thru 0x017ae8 (0x9a14 bytes) 000575 1715333404.759283 punch from 0x01178e to 0x01d9ca (0xc23d bytes) 000576 1715333404.763296 mapread 0x01a235 thru 0x025a51 (0xb81d bytes) 000577 1715333404.771818 punch from 0x026b31 to 0x02d259 (0x6729 bytes) 000578 1715333404.775628 trunc from 0x03d51f to 0x003975 (0x39ba9 bytes) 000579 1715333404.786665 write 0x007d00 thru 0x00dde3 (0x60e4 bytes) 000580 1715333404.787467 mapread 0x003325 thru 0x0063a0 (0x307c bytes) 000581 1715333404.798615 mapwrite 0x018136 thru 0x022551 (0xa41c bytes) 000583 1715333404.852313 mapread 0x004b01 thru 0x011877 (0xcd77 bytes) 000584 1715333404.864859 read 0x011810 thru 0x014f27 (0x3718 bytes) 000585 1715333404.869096 read 0x010df8 thru 0x01b510 (0xa719 bytes) 000586 1715333404.869546 trunc from 0x022552 to 0x0034ae (0x1f0a3 bytes) 000587 1715333404.879629 fallocate 0x024834 thru 0x033b30 (0xf2fd bytes) 000588 1715333404.884041 mapwrite 0x00ecf0 thru 0x0190d4 (0xa3e5 bytes) 000589 1715333404.903388 fallocate 0x037aa3 thru 0x03af9c (0x34fa bytes) 000590 1715333404.908236 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000591 1715333404.992480 MIRROR_RESYNC 5 mirrors 000592 1715333405.390079 write 0x02fedb thru 0x03c713 (0xc839 bytes) 000595 1715333405.407458 mapwrite 0x023631 thru 0x0333d4 (0xfda4 bytes) 000596 1715333405.434000 fallocate 0x003075 thru 0x005c80 (0x2c0c bytes) 000598 1715333405.437084 mapread 0x01100b thru 0x017726 (0x671c bytes) 000599 1715333405.443940 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000600 1715333405.526097 read 0x0144d9 thru 0x01e938 (0xa460 bytes) 000603 1715333405.532632 fallocate 0x02e887 thru 0x0333b0 (0x4b2a bytes) 000604 1715333405.536039 write 0x01c130 thru 0x02a5f9 (0xe4ca bytes) 000605 1715333405.541144 punch from 0x03aaf3 to 0x03c713 (0x1c21 bytes) 000606 1715333405.543907 write 0x019f17 thru 0x01c5f7 (0x26e1 bytes) 000607 1715333405.544734 mapread 0x033df7 thru 0x034bbd (0xdc7 bytes) 000608 1715333405.551109 trunc from 0x03c714 to 0x0246f4 (0x1801f bytes) 000609 1715333405.558067 mapread 0x014160 thru 0x016512 (0x23b3 bytes) 000610 1715333405.558859 punch from 0x019304 to 0x0235a2 (0xa29f bytes) 000611 1715333405.566312 write 0x01b42c thru 0x022b2e (0x7703 bytes) 000612 1715333405.570808 mapread 0x00ac3f thru 0x0150c7 (0xa489 bytes) 000613 1715333405.579654 mapread 0x017d6d thru 0x01c0ce (0x4362 bytes) 000614 1715333405.580249 trunc from 0x0246f5 to 0x0270ed (0x29f9 bytes) 000616 1715333405.587171 mapwrite 0x00878b thru 0x0091d8 (0xa4e bytes) 000617 1715333405.599460 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000618 1715333405.679196 mapread 0x002ca6 thru 0x00e4a4 (0xb7ff bytes) 000619 1715333405.689757 write 0x0247ab thru 0x0273fa (0x2c50 bytes) 000620 1715333405.695727 read 0x01df8f thru 0x0273fa (0x946c bytes) 000623 1715333405.696783 punch from 0x001147 to 0x007f74 (0x6e2e bytes) 000624 1715333405.702375 MIRROR_EXTEND to 4 mirrors 000625 1715333405.939974 mapread 0x013bcc thru 0x01e409 (0xa83e bytes) 000626 1715333405.963410 trunc from 0x0273fb to 0x01c40e (0xafec bytes) 000627 1715333405.995440 mapwrite 0x03c4ba thru 0x03ef4a (0x2a91 bytes) 000628 1715333406.026406 trunc from 0x03ef4b to 0x033c8e (0xb2bc bytes) 000630 1715333406.034525 MIRROR_RESYNC 4 mirrors 000631 1715333406.439723 trunc from 0x033c8f to 0x007b2d (0x2c161 bytes) 000632 1715333406.461516 mapwrite 0x03870d thru 0x03a548 (0x1e3c bytes) 000633 1715333406.492356 trunc from 0x03a549 to 0x004894 (0x35cb4 bytes) 000634 1715333406.498300 mapwrite 0x03ccfa thru 0x03ffff (0x3306 bytes) 000635 1715333406.517125 write 0x00a2a8 thru 0x00ad0e (0xa67 bytes) 000636 1715333406.522485 mapread 0x0232e8 thru 0x03040f (0xd128 bytes) 000637 1715333406.534002 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000638 1715333406.636708 MIRROR_RESYNC 3 mirrors 000640 1715333406.910325 MIRROR_RESYNC 3 mirrors 000641 1715333406.969234 mapwrite 0x01bdb4 thru 0x01d021 (0x126e bytes) 000642 1715333407.013330 fallocate 0x01cf7b thru 0x029da5 (0xce2b bytes) 000645 1715333407.017635 punch from 0x019cd4 to 0x028eb7 (0xf1e4 bytes) 000647 1715333407.021069 write 0x033e76 thru 0x03d15e (0x92e9 bytes) 000648 1715333407.032173 fallocate 0x033abc thru 0x039e6f (0x63b4 bytes) 000651 1715333407.040854 MIRROR_RESYNC 3 mirrors 000653 1715333407.420381 punch from 0x00f4c0 to 0x014a02 (0x5543 bytes) 000654 1715333407.441332 fallocate 0x00d90d thru 0x0161e8 (0x88dc bytes) 000656 1715333407.445126 punch from 0x00d72a to 0x011696 (0x3f6d bytes) 000657 1715333407.450616 write 0x01abb1 thru 0x0258d4 (0xad24 bytes) 000658 1715333407.458861 write 0x03b1a6 thru 0x03ffff (0x4e5a bytes) 000659 1715333407.464844 fallocate 0x0092ea thru 0x015859 (0xc570 bytes) 000660 1715333407.467462 read 0x01ec85 thru 0x023722 (0x4a9e bytes) 000661 1715333407.467861 mapread 0x016a6f thru 0x0259c2 (0xef54 bytes) 000663 1715333407.475189 mapwrite 0x02d532 thru 0x031daa (0x4879 bytes) 000664 1715333407.500490 trunc from 0x040000 to 0x0096d2 (0x3692d bytes) 000665 1715333407.512441 write 0x03c84c thru 0x03ffff (0x37b4 bytes) 000666 1715333407.513324 punch from 0x004b81 to 0x005984 (0xe04 bytes) 000667 1715333407.515529 trunc from 0x040000 to 0x03d5e0 (0x2a1f bytes) 000668 1715333407.523031 mapwrite 0x02946d thru 0x02c930 (0x34c4 bytes) 000669 1715333407.540886 mapread 0x01052b thru 0x01fdb2 (0xf888 bytes) 000670 1715333407.555426 MIRROR_RESYNC 3 mirrors 000676 1715333407.905591 write 0x00233d thru 0x00892e (0x65f2 bytes) 000678 1715333407.929198 trunc from 0x03d5e1 to 0x0059d2 (0x37c0e bytes) 000679 1715333407.951503 trunc from 0x0059d3 to 0x03e1ce (0x387fc bytes) 000680 1715333407.965590 mapread 0x018714 thru 0x01f719 (0x7006 bytes) 000681 1715333407.972644 punch from 0x02f47f to 0x03d00c (0xdb8e bytes) 000683 1715333407.977244 fallocate 0x03c172 thru 0x03ffff (0x3e8e bytes) 000684 1715333407.981361 fallocate 0x03381a thru 0x03dc16 (0xa3fd bytes) 000685 1715333407.984916 write 0x0256a4 thru 0x02d4f5 (0x7e52 bytes) 000686 1715333407.988699 write 0x035ce5 thru 0x03a1b7 (0x44d3 bytes) 000687 1715333407.992624 fallocate 0x03e72f thru 0x03ffff (0x18d1 bytes) 000688 1715333407.994439 fallocate 0x0279fb thru 0x02be82 (0x4488 bytes) 000689 1715333407.996486 fallocate 0x034498 thru 0x03f8c7 (0xb430 bytes) 000690 1715333407.998604 fallocate 0x020fa0 thru 0x02b9e5 (0xaa46 bytes) 000691 1715333408.000455 read 0x0351b6 thru 0x03923e (0x4089 bytes) 000693 1715333408.001358 fallocate 0x03fd6a thru 0x03ffff (0x296 bytes) 000696 1715333408.003982 punch from 0x02d327 to 0x033957 (0x6631 bytes) 000697 1715333408.009851 mapread 0x0192be thru 0x01fcf9 (0x6a3c bytes) 000698 1715333408.010620 write 0x0181dd thru 0x025d19 (0xdb3d bytes) 000699 1715333408.011655 read 0x033f89 thru 0x036677 (0x26ef bytes) 000700 1715333408.018693 MIRROR_RESYNC 3 mirrors 000701 1715333408.285045 mapread 0x001f78 thru 0x008b3e (0x6bc7 bytes) 000702 1715333408.294651 write 0x00a197 thru 0x019a32 (0xf89c bytes) 000703 1715333408.316188 mapwrite 0x034e42 thru 0x039fb6 (0x5175 bytes) 000704 1715333408.331738 punch from 0x0307f2 to 0x03b992 (0xb1a1 bytes) 000705 1715333408.334233 trunc from 0x040000 to 0x01b242 (0x24dbd bytes) 000706 1715333408.352875 read 0x004089 thru 0x012c9c (0xec14 bytes) 000707 1715333408.354308 mapwrite 0x029502 thru 0x039212 (0xfd11 bytes) 000708 1715333408.380483 fallocate 0x00f028 thru 0x00fd7f (0xd58 bytes) 000712 1715333408.384260 punch from 0x027b78 to 0x0338d5 (0xbd5e bytes) 000713 1715333408.387400 mapread 0x0066c9 thru 0x007d90 (0x16c8 bytes) 000719 1715333408.390047 read 0x0069d3 thru 0x007dbc (0x13ea bytes) 000720 1715333408.390805 punch from 0x00dbd6 to 0x0142aa (0x66d5 bytes) 000721 1715333408.395376 punch from 0x02e995 to 0x0342cb (0x5937 bytes) 000722 1715333408.398070 write 0x034b88 thru 0x03ffff (0xb478 bytes) 000724 1715333408.406689 MIRROR_RESYNC 3 mirrors 000725 1715333408.691122 mapwrite 0x01b92a thru 0x024c64 (0x933b bytes) 000726 1715333408.749138 read 0x033ed8 thru 0x03ffff (0xc128 bytes) 000727 1715333408.753661 trunc from 0x040000 to 0x025b7b (0x1a484 bytes) 000728 1715333408.772985 fallocate 0x0387eb thru 0x03ffff (0x7815 bytes) 000729 1715333408.775274 write 0x03a4b5 thru 0x03ffff (0x5b4b bytes) 000730 1715333408.776116 mapread 0x02ee04 thru 0x035b37 (0x6d34 bytes) 000731 1715333408.782981 MIRROR_RESYNC 3 mirrors 000732 1715333409.071061 read 0x035862 thru 0x03ffff (0xa79e bytes) 000733 1715333409.078566 punch from 0x00cd00 to 0x00d992 (0xc93 bytes) 000738 1715333409.102001 trunc from 0x040000 to 0x020d47 (0x1f2b8 bytes) 000739 1715333409.122098 MIRROR_EXTEND to 4 mirrors 000740 1715333409.339966 fallocate 0x033526 thru 0x03f4c7 (0xbfa2 bytes) 000741 1715333409.354583 mapread 0x002d61 thru 0x00ad64 (0x8004 bytes) 000742 1715333409.363203 fallocate 0x028e07 thru 0x0354a0 (0xc69a bytes) 000743 1715333409.365554 read 0x0030f0 thru 0x00d669 (0xa57a bytes) 000745 1715333409.366727 write 0x01b510 thru 0x01bb80 (0x671 bytes) 000746 1715333409.373090 read 0x01eba6 thru 0x028681 (0x9adc bytes) 000749 1715333409.378282 trunc from 0x03f4c8 to 0x01cee3 (0x225e4 bytes) 000751 1715333409.388441 write 0x007e3b thru 0x00f80d (0x79d3 bytes) 000752 1715333409.389391 fallocate 0x025bbb thru 0x0330bb (0xd501 bytes) 000753 1715333409.392552 write 0x027810 thru 0x02ed1c (0x750d bytes) 000754 1715333409.396236 MIRROR_EXTEND to 5 mirrors 000755 1715333409.599630 read 0x02a648 thru 0x02ed1c (0x46d5 bytes) 000756 1715333409.602439 write 0x021833 thru 0x03052b (0xecf9 bytes) 000759 1715333409.616819 mapwrite 0x0348cc thru 0x03862f (0x3d64 bytes) 000760 1715333409.643765 fallocate 0x00bc3f thru 0x00f783 (0x3b45 bytes) 000761 1715333409.646191 write 0x02935e thru 0x038a36 (0xf6d9 bytes) 000762 1715333409.647333 punch from 0x0081b0 to 0x008c43 (0xa94 bytes) 000763 1715333409.649874 mapwrite 0x0205e3 thru 0x024b16 (0x4534 bytes) 000765 1715333409.662790 fallocate 0x035acb thru 0x03ffff (0xa535 bytes) 000766 1715333409.667196 mapwrite 0x0120b8 thru 0x0215e7 (0xf530 bytes) 000767 1715333409.686414 write 0x02c47d thru 0x031d5d (0x58e1 bytes) 000768 1715333409.687283 trunc from 0x038a37 to 0x00079b (0x3829b bytes) 000772 1715333409.695231 fallocate 0x03a1f0 thru 0x03ffff (0x5e10 bytes) 000773 1715333409.697571 fallocate 0x012bd6 thru 0x020122 (0xd54d bytes) 000774 1715333409.701115 trunc from 0x020123 to 0x007f88 (0x1819a bytes) 000776 1715333409.708280 read 0x001a12 thru 0x007f88 (0x6577 bytes) 000777 1715333409.715667 mapwrite 0x0227e7 thru 0x030ab0 (0xe2ca bytes) 000779 1715333409.747555 punch from 0x0170c4 to 0x024598 (0xd4d5 bytes) 000780 1715333409.750436 read 0x02e97b thru 0x030ab0 (0x2136 bytes) 000781 1715333409.751141 read 0x00e0a3 thru 0x019b0e (0xba6c bytes) 000782 1715333409.755826 fallocate 0x02e390 thru 0x03ba1e (0xd68f bytes) 000783 1715333409.758488 trunc from 0x03ba1f to 0x0089c7 (0x33057 bytes) 000784 1715333409.765170 fallocate 0x03d0af thru 0x03e8ea (0x183c bytes) 000785 1715333409.770001 trunc from 0x03e8eb to 0x0124f7 (0x2c3f3 bytes) 000786 1715333409.780736 punch from 0x00495c to 0x007e03 (0x34a8 bytes) 000787 1715333409.783304 fallocate 0x0065da thru 0x007166 (0xb8d bytes) 000788 1715333409.785457 fallocate 0x024188 thru 0x026f48 (0x2dc1 bytes) 000789 1715333409.789542 MIRROR_RESYNC 5 mirrors 000790 1715333410.068161 trunc from 0x0124f8 to 0x03a00d (0x27b16 bytes) 000791 1715333410.092173 read 0x029cdc thru 0x0313f7 (0x771c bytes) 000792 1715333410.099539 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000793 1715333410.177052 read 0x004fa0 thru 0x006bbf (0x1c20 bytes) 000794 1715333410.180513 read 0x0134ad thru 0x0233e9 (0xff3d bytes) 000796 1715333410.187981 write 0x002b04 thru 0x011b9c (0xf099 bytes) 000797 1715333410.192358 trunc from 0x03a00e to 0x02b896 (0xe777 bytes) 000798 1715333410.201070 punch from 0x016c1d to 0x022ce9 (0xc0cd bytes) 000800 1715333410.206357 trunc from 0x02b897 to 0x024230 (0x7666 bytes) 000801 1715333410.218081 mapread 0x001014 thru 0x00cd36 (0xbd23 bytes) 000802 1715333410.219088 trunc from 0x024231 to 0x0019e4 (0x2284c bytes) 000803 1715333410.227155 mapwrite 0x01c336 thru 0x0245f1 (0x82bc bytes) 000804 1715333410.272771 trunc from 0x0245f2 to 0x023aa0 (0xb51 bytes) 000805 1715333410.282499 MIRROR_RESYNC 4 mirrors 000806 1715333410.500912 write 0x03413f thru 0x037a51 (0x3913 bytes) 000807 1715333410.517285 mapread 0x02670f thru 0x03204b (0xb93d bytes) 000808 1715333410.531670 mapread 0x02f775 thru 0x035218 (0x5aa4 bytes) 000809 1715333410.532260 punch from 0x0170d1 to 0x025fd6 (0xef06 bytes) 000810 1715333410.534912 read 0x01a195 thru 0x01fdb4 (0x5c20 bytes) 000811 1715333410.538181 write 0x018b93 thru 0x01e950 (0x5dbe bytes) 000812 1715333410.542270 write 0x035bc9 thru 0x03ffff (0xa437 bytes) 000813 1715333410.543796 write 0x00d84d thru 0x00fee6 (0x269a bytes) 000814 1715333410.552708 MIRROR_EXTEND to 5 mirrors 000815 1715333410.785328 mapwrite 0x02f75f thru 0x03d523 (0xddc5 bytes) 000816 1715333410.844429 mapread 0x01b392 thru 0x020a15 (0x5684 bytes) 000817 1715333410.852552 MIRROR_EXTEND to 6 mirrors 000818 1715333411.074280 read 0x00b2fd thru 0x00c58c (0x1290 bytes) 000819 1715333411.079795 punch from 0x03eef5 to 0x03ffff (0x110b bytes) 000820 1715333411.099544 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000821 1715333411.185565 trunc from 0x040000 to 0x03c181 (0x3e7e bytes) 000822 1715333411.203462 punch from 0x023454 to 0x0303d4 (0xcf81 bytes) 000825 1715333411.206417 write 0x01ce55 thru 0x0261bc (0x9368 bytes) 000826 1715333411.212373 fallocate 0x00c417 thru 0x01afa2 (0xeb8c bytes) 000827 1715333411.215403 trunc from 0x03c182 to 0x036f75 (0x520c bytes) 000829 1715333411.222823 read 0x00e49a thru 0x016d6e (0x88d5 bytes) 000830 1715333411.229672 MIRROR_EXTEND to 6 mirrors 000831 1715333411.433515 trunc from 0x036f76 to 0x03b0be (0x4149 bytes) 000832 1715333411.462232 mapread 0x00c52c thru 0x01a3c8 (0xde9d bytes) 000833 1715333411.474756 trunc from 0x03b0bf to 0x03152b (0x9b93 bytes) 000834 1715333411.482146 punch from 0x031064 to 0x03152b (0x4c8 bytes) 000836 1715333411.485176 trunc from 0x03152c to 0x00eda9 (0x22782 bytes) 000839 1715333411.494155 trunc from 0x00edaa to 0x017268 (0x84bf bytes) 000840 1715333411.502403 read 0x0007a0 thru 0x00f38f (0xebf0 bytes) 000841 1715333411.508660 punch from 0x00c0c1 to 0x0147a0 (0x86e0 bytes) 000843 1715333411.511704 read 0x009357 thru 0x00bddb (0x2a85 bytes) 000844 1715333411.512204 read 0x0036a6 thru 0x0135ef (0xff4a bytes) 000845 1715333411.517029 read 0x00f93d thru 0x010f92 (0x1656 bytes) 000846 1715333411.517622 mapread 0x013c6a thru 0x0141d3 (0x56a bytes) 000847 1715333411.518208 trunc from 0x017269 to 0x0197a2 (0x253a bytes) 000848 1715333411.524875 write 0x030c15 thru 0x03c5a0 (0xb98c bytes) 000849 1715333411.526005 mapread 0x035c48 thru 0x03c5a0 (0x6959 bytes) 000850 1715333411.528669 MIRROR_RESYNC 6 mirrors 000851 1715333411.981814 punch from 0x020bef to 0x02c8c7 (0xbcd9 bytes) 000852 1715333412.007266 punch from 0x0187c5 to 0x01a26a (0x1aa6 bytes) 000853 1715333412.010458 punch from 0x02a535 to 0x03885d (0xe329 bytes) 000855 1715333412.017654 MIRROR_RESYNC 6 mirrors 000857 1715333412.511693 write 0x016131 thru 0x016543 (0x413 bytes) 000858 1715333412.533751 fallocate 0x033f8e thru 0x03ffff (0xc072 bytes) 000859 1715333412.536337 fallocate 0x03bd4b thru 0x03ffff (0x42b5 bytes) 000860 1715333412.540446 punch from 0x01bce9 to 0x01e180 (0x2498 bytes) 000863 1715333412.545182 write 0x006609 thru 0x013759 (0xd151 bytes) 000864 1715333412.552781 mapwrite 0x02c1c9 thru 0x0389f6 (0xc82e bytes) 000865 1715333412.577567 punch from 0x03430f to 0x03ffff (0xbcf1 bytes) 000866 1715333412.580638 punch from 0x00f986 to 0x014e1c (0x5497 bytes) 000867 1715333412.583473 punch from 0x0142a5 to 0x016cd9 (0x2a35 bytes) 000868 1715333412.586446 read 0x0340d1 thru 0x03a660 (0x6590 bytes) 000869 1715333412.597474 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000870 1715333412.691979 mapread 0x02402f thru 0x02e2a2 (0xa274 bytes) 000873 1715333412.706252 mapwrite 0x02f039 thru 0x03d95e (0xe926 bytes) 000874 1715333412.723598 fallocate 0x0059c7 thru 0x00921c (0x3856 bytes) 000875 1715333412.727138 trunc from 0x040000 to 0x003501 (0x3cafe bytes) 000876 1715333412.746005 MIRROR_RESYNC 5 mirrors 000877 1715333424.018203 mapwrite 0x00216d thru 0x00e2c2 (0xc156 bytes) 000878 1715333424.065979 punch from 0x0079a6 to 0x00e2c2 (0x691d bytes) 000880 1715333424.073494 write 0x0292c8 thru 0x02b05a (0x1d93 bytes) 000881 1715333424.081165 MIRROR_RESYNC 5 mirrors 000882 1715333424.489281 punch from 0x023ab3 to 0x02b05a (0x75a8 bytes) 000883 1715333424.506696 fallocate 0x036106 thru 0x037979 (0x1874 bytes) 000884 1715333424.509583 read 0x016be8 thru 0x024d2c (0xe145 bytes) 000885 1715333424.513916 read 0x020a88 thru 0x025c2a (0x51a3 bytes) 000886 1715333424.518123 mapwrite 0x01051c thru 0x015c1a (0x56ff bytes) 000887 1715333424.527444 mapread 0x01300f thru 0x022d0a (0xfcfc bytes) 000888 1715333424.528074 read 0x021602 thru 0x0236de (0x20dd bytes) 000889 1715333424.528457 mapread 0x02aee5 thru 0x02b05a (0x176 bytes) 000890 1715333424.531139 MIRROR_RESYNC 5 mirrors 000892 1715333424.993156 fallocate 0x012aad thru 0x015589 (0x2add bytes) 000893 1715333425.008246 write 0x0188e5 thru 0x01a393 (0x1aaf bytes) 000894 1715333425.014689 fallocate 0x023b67 thru 0x02fa2a (0xbec4 bytes) 000895 1715333425.017545 mapwrite 0x015546 thru 0x01b8a3 (0x635e bytes) 000896 1715333425.029657 mapwrite 0x032452 thru 0x03607d (0x3c2c bytes) 000897 1715333425.061990 write 0x000e2d thru 0x003319 (0x24ed bytes) 000898 1715333425.070239 MIRROR_EXTEND to 6 mirrors 000899 1715333425.299048 mapread 0x006468 thru 0x015b63 (0xf6fc bytes) 000900 1715333425.314715 mapread 0x0176cf thru 0x01e6f5 (0x7027 bytes) 000901 1715333425.324352 trunc from 0x03607e to 0x02d8ee (0x878f bytes) 000902 1715333425.343445 mapwrite 0x02d933 thru 0x039fec (0xc6ba bytes) 000903 1715333425.374023 fallocate 0x03ace3 thru 0x03ffff (0x531d bytes) 000904 1715333425.376519 trunc from 0x040000 to 0x02f6b5 (0x1094a bytes) 000905 1715333425.383185 mapread 0x00a001 thru 0x018783 (0xe783 bytes) 000906 1715333425.389440 trunc from 0x02f6b6 to 0x02045b (0xf25a bytes) 000907 1715333425.394833 mapwrite 0x01ed23 thru 0x02c537 (0xd815 bytes) 000908 1715333425.414130 write 0x012b8e thru 0x016281 (0x36f4 bytes) 000909 1715333425.414806 write 0x011966 thru 0x017dd8 (0x6473 bytes) 000911 1715333425.415960 fallocate 0x0093a8 thru 0x01716b (0xddc4 bytes) 000912 1715333425.418247 write 0x01f4d2 thru 0x02b3da (0xbf09 bytes) 000913 1715333425.421355 MIRROR_EXTEND to 7 mirrors 000914 1715333425.644743 fallocate 0x03d7b7 thru 0x03ffff (0x2849 bytes) 000915 1715333425.662806 mapwrite 0x02eefd thru 0x03b888 (0xc98c bytes) 000916 1715333425.688117 write 0x027cda thru 0x02d104 (0x542b bytes) 000917 1715333425.694675 punch from 0x032ae4 to 0x03c6eb (0x9c08 bytes) 000918 1715333425.698438 trunc from 0x040000 to 0x009d62 (0x3629d bytes) 000919 1715333425.708487 punch from 0x006395 to 0x009d62 (0x39ce bytes) 000920 1715333425.713376 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000921 1715333425.792105 punch from 0x002574 to 0x005003 (0x2a90 bytes) 000922 1715333425.795553 mapread 0x006e47 thru 0x009d62 (0x2f1c bytes) 000923 1715333425.804612 trunc from 0x009d63 to 0x012f03 (0x91a1 bytes) 000924 1715333425.813948 trunc from 0x012f04 to 0x03f963 (0x2ca60 bytes) 000925 1715333425.821784 mapwrite 0x0073aa thru 0x01658e (0xf1e5 bytes) 000926 1715333425.843619 mapread 0x018ea9 thru 0x01e5e8 (0x5740 bytes) 000929 1715333425.849949 mapread 0x011019 thru 0x011dc2 (0xdaa bytes) 000930 1715333425.850554 mapread 0x02168c thru 0x02f2d1 (0xdc46 bytes) 000932 1715333425.858013 trunc from 0x03f964 to 0x015cbc (0x29ca7 bytes) 000934 1715333425.865486 mapwrite 0x01b308 thru 0x01f1d5 (0x3ece bytes) 000937 1715333425.889717 read 0x012f22 thru 0x018a48 (0x5b27 bytes) 000938 1715333425.890975 punch from 0x00807d to 0x0144df (0xc463 bytes) 000939 1715333425.895154 read 0x0087e1 thru 0x017fb1 (0xf7d1 bytes) 000940 1715333425.900228 write 0x027b36 thru 0x02e79e (0x6c69 bytes) 000941 1715333425.901139 mapwrite 0x00635e thru 0x00c60f (0x62b2 bytes) 000943 1715333425.911870 fallocate 0x0218a1 thru 0x02e7f6 (0xcf56 bytes) 000944 1715333425.914522 mapread 0x01bb09 thru 0x01e18e (0x2686 bytes) 000945 1715333425.915082 fallocate 0x0328ac thru 0x035c7b (0x33d0 bytes) 000946 1715333425.921231 MIRROR_EXTEND to 7 mirrors 000947 1715333426.168395 mapwrite 0x02ea41 thru 0x039e51 (0xb411 bytes) 000948 1715333426.231429 fallocate 0x02f183 thru 0x03d0b4 (0xdf32 bytes) 000949 1715333426.233605 write 0x013e39 thru 0x01f02c (0xb1f4 bytes) 000950 1715333426.240340 mapread 0x028916 thru 0x036c6b (0xe356 bytes) 000951 1715333426.245197 read 0x021570 thru 0x029346 (0x7dd7 bytes) 000952 1715333426.245821 punch from 0x03752b to 0x037d59 (0x82f bytes) 000953 1715333426.248277 trunc from 0x039e52 to 0x0251b4 (0x14c9d bytes) 000954 1715333426.258141 fallocate 0x020465 thru 0x02756d (0x7109 bytes) 000955 1715333426.260127 punch from 0x01548c to 0x01d10a (0x7c7f bytes) 000956 1715333426.266735 read 0x017c27 thru 0x019e6c (0x2246 bytes) 000958 1715333426.275549 MIRROR_RESYNC 7 mirrors 000959 1715333426.871972 write 0x009434 thru 0x00d4dc (0x40a9 bytes) 000960 1715333426.903529 write 0x01535d thru 0x01f4ab (0xa14f bytes) 000961 1715333426.911623 mapread 0x000daa thru 0x00f715 (0xe96c bytes) 000962 1715333426.930485 MIRROR_RESYNC 7 mirrors 000964 1715333427.411792 punch from 0x004a16 to 0x013d7b (0xf366 bytes) 000965 1715333427.428039 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000966 1715333427.503973 mapwrite 0x018fb6 thru 0x027986 (0xe9d1 bytes) 000967 1715333427.533584 read 0x01c9ff thru 0x027986 (0xaf88 bytes) 000968 1715333427.534118 mapwrite 0x016354 thru 0x025616 (0xf2c3 bytes) 000969 1715333427.543625 mapwrite 0x016986 thru 0x01e717 (0x7d92 bytes) 000971 1715333427.551905 write 0x02d1d6 thru 0x02d8d4 (0x6ff bytes) 000972 1715333427.552617 read 0x0141a4 thru 0x023bf4 (0xfa51 bytes) 000973 1715333427.553417 punch from 0x023dad to 0x026d27 (0x2f7b bytes) 000974 1715333427.556243 fallocate 0x01953f thru 0x01cc57 (0x3719 bytes) 000975 1715333427.559017 write 0x0229be thru 0x0312e1 (0xe924 bytes) 000979 1715333427.561579 punch from 0x00e7f5 to 0x01914f (0xa95b bytes) 000980 1715333427.564191 mapread 0x02d596 thru 0x0312e1 (0x3d4c bytes) 000981 1715333427.564740 fallocate 0x03ecfe thru 0x03ffff (0x1302 bytes) 000982 1715333427.567893 fallocate 0x035c7c thru 0x039589 (0x390e bytes) 000983 1715333427.570267 fallocate 0x00c441 thru 0x016aef (0xa6af bytes) 000985 1715333427.573291 mapwrite 0x00fb05 thru 0x014ba5 (0x50a1 bytes) 000986 1715333427.586051 trunc from 0x040000 to 0x01bb97 (0x24468 bytes) 000987 1715333427.591993 read 0x00b455 thru 0x00cb89 (0x1735 bytes) 000988 1715333427.592450 punch from 0x018786 to 0x01acb3 (0x252e bytes) 000989 1715333427.594648 punch from 0x01a8f2 to 0x01bb97 (0x12a6 bytes) 000990 1715333427.597277 fallocate 0x012e82 thru 0x01e703 (0xb882 bytes) 000991 1715333427.599816 fallocate 0x023736 thru 0x0326e2 (0xefad bytes) 000993 1715333427.602909 write 0x0361a7 thru 0x03a1f6 (0x4050 bytes) 000994 1715333427.603724 read 0x0023b4 thru 0x006595 (0x41e2 bytes) 000997 1715333427.604668 mapread 0x00765a thru 0x00dc5b (0x6602 bytes) 000998 1715333427.605601 mapread 0x019609 thru 0x0245dc (0xafd4 bytes) 000999 1715333427.610329 trunc from 0x03a1f7 to 0x01a410 (0x1fde6 bytes) 001000 1715333427.615863 fallocate 0x01c215 thru 0x02acdd (0xeac9 bytes) All operations completed A-OK! PASS 70a (45s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200a: stress test ===================== 05:30:30 (1715333430) Starting client: oleg301-client.virtnet: -o user_xattr,flock oleg301-server@tcp:/lustre /mnt/lustre2 Starting client: oleg301-client.virtnet: -o user_xattr,flock oleg301-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3578528 .. Extending file size to 3703552 .. failed Extending file size to 7953920 .. Extending file size to 9068416 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8654144 .. Extending file size to 8903200 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 5881 5882 5883 5885 5887 192.168.203.101@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg301-client.virtnet /mnt/lustre2 (opts:) 192.168.203.101@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg301-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (66s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 05:31:37 (1715333497) Starting client: oleg301-client.virtnet: -o user_xattr,flock oleg301-server@tcp:/lustre /mnt/lustre2 Starting client: oleg301-client.virtnet: -o user_xattr,flock oleg301-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4528352 .. Extending file size to 3927360 .. done Extending file size to 6179488 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8055040 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 4324096 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 5793152 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7233696 .. Extending file size to 8893568 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8774016 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8959456 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8848224 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9056992 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9045632 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9200800 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9203808 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 8608 8609 8610 8612 8614 done 192.168.203.101@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg301-client.virtnet /mnt/lustre2 (opts:) 192.168.203.101@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg301-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (65s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 05:32:44 (1715333564) Starting client: oleg301-client.virtnet: -o user_xattr,flock oleg301-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2716 1284972 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 22736 3584284 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 14832 3592188 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37568 7176472 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2716 1284972 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 22736 3584284 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 14832 3592188 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37568 7176472 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0405327 s, 51.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.060567 s, 34.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.101@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg301-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 05:32:51 (1715333571) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0323159 s, 64.9 MB/s PASS 202 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 203: mirror file preserve mirror ID === 05:32:56 (1715333576) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x81:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x81:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x82:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x9e:0x0] } PASS 203 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 05:33:00 (1715333580) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0415119 s, 50.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x77e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x77f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x780:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 05:33:05 (1715333585) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0352516 s, 59.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x83:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x84:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x9f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x83:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x84:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x9f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa0:0x0] } PASS 204b (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 05:33:10 (1715333590) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0253146 s, 82.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x77f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x781:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x77f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x781:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x782:0x0] } PASS 204c (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 05:33:36 (1715333616) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg301-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0216688 s, 48.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x780:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x784:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x783:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x780:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x784:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x783:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x781:0x0] } PASS 204d (35s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 05:34:13 (1715333653) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg301-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x785:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x786:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0294732 s, 71.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x785:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x786:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x782:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x785:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x786:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x782:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (31s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 05:34:46 (1715333686) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg301-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool oleg301-server: Pool lustre.test_204f created Adding targets to pool oleg301-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg301-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0501063 s, 41.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x85:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x85:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg301-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg301-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg301-server: Pool lustre.test_204f destroyed PASS 204f (51s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 205: lfs mirror extend to set prefer and nocompr flags ========================================================== 05:35:39 (1715333739) PASS 205 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 05:35:44 (1715333744) oleg301-server: Pool lustre.test_206 created Adding targets to pool oleg301-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg301-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg301-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg301-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg301-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 05:36:03 (1715333763) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.028093 s, 37.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0152335 s, 68.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x1f4cec1 Mirror 3: 0x1f4cec1 PASS 207 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 05:36:08 (1715333768) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 05:36:12 (1715333772) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 05:36:15 (1715333775) Starting client: oleg301-client.virtnet: -o user_xattr,flock oleg301-server@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0195082 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2060000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xce0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xce0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xce0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x740000 Read this (4 bytes): BBBB 192.168.203.101@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg301-client.virtnet /mnt/lustre2 (opts:) PASS 209a (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 05:36:35 (1715333795) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.0052547 s, 1.6 MB/s the very first read read with warm cache PASS 209b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210a: handle broken mirrored lovea ==== 05:36:39 (1715333799) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0142056 s, 73.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x53 lmm_fid: [0x200000405:0x53:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x53 lmm_fid: [0x200000405:0x53:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611219 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-05-10 05:36:39.000000000 -0400 Modify: 2024-05-10 05:36:39.000000000 -0400 Change: 2024-05-10 05:36:40.000000000 -0400 Birth: - PASS 210a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 05:36:43 (1715333803) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0120151 s, 87.3 MB/s oleg301-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg301-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg301-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg301-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg301-server: O/280000401/d12/ac: File not found by ext2_lookup PASS 210b (14s) debug_raw_pointers=0 debug_raw_pointers=0 == sanity-flr test complete, duration 977 sec ============ 05:36:58 (1715333818) === sanity-flr: start cleanup 05:36:58 (1715333818) === === sanity-flr: finish cleanup 05:37:00 (1715333820) ===