-----============= acceptance-small: sanity-flr ============----- Wed Jul 24 23:55:54 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 6 201 44c === sanity-flr: start setup 23:55:59 (1721879759) === oleg443-client.virtnet: executing check_config_client /mnt/lustre oleg443-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg443-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012a62c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012a62c800.idle_timeout=debug disable quota as required oleg443-server: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 23:56:06 (1721879766) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f7568] == sanity-flr test 0a: lfs mirror create with -N option == 23:56:06 (1721879766) PASS 0a (4s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 23:56:10 (1721879770) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 23:56:11 (1721879771) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 23:56:12 (1721879772) PASS 0d (5s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 23:56:17 (1721879777) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (0s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 23:56:18 (1721879778) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (0s) == sanity-flr test 0g: lfs mirror create flags support === 23:56:18 (1721879778) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88012a62c800.stats=clear osc.lustre-OST0001-osc-ffff88012a62c800.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 23:56:20 (1721879780) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 23:56:23 (1721879783) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 23:56:25 (1721879785) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 23:56:26 (1721879786) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 23:56:28 (1721879788) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 23:56:30 (1721879790) PASS 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 23:56:32 (1721879792) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (1s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 23:56:34 (1721879794) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (1s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 23:56:36 (1721879796) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 23:56:38 (1721879798) 17+0 records in 17+0 records out 17825792 bytes (18 MB) copied, 0.524651 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0370045 s, 28.3 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 23:56:40 (1721879800) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB) copied, 0.0600797 s, 122 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 23:56:42 (1721879802) Stopping /mnt/lustre-ost1 (opts:) on oleg443-server oleg443-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a62c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a62c800.ost_server_uuid in DISCONN state after 1 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg443-server: oleg443-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg443-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg443-server oleg443-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a62c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a62c800.ost_server_uuid in DISCONN state after 4 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg443-server: oleg443-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg443-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (26s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 23:57:08 (1721879828) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB) copied, 0.116015 s, 63.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg443-server oleg443-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a62c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a62c800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg443-server: oleg443-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg443-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (8s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 23:57:16 (1721879836) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg443-server oleg443-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a62c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a62c800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg443-server: oleg443-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg443-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg443-server oleg443-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a62c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a62c800.ost_server_uuid in DISCONN state after 4 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg443-server: oleg443-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg443-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (21s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 23:57:37 (1721879857) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.878551 s, 59.7 MB/s -rw-r--r-- 1 root root 52428800 Jul 24 23:57 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.237s user 0m0.000s sys 0m0.118s Stopping /mnt/lustre-ost1 (opts:) on oleg443-server oleg443-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a62c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a62c800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.244s user 0m0.000s sys 0m0.142s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg443-server: oleg443-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg443-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg443-server oleg443-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a62c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a62c800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.287s user 0m0.000s sys 0m0.132s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg443-server: oleg443-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg443-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg443-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012a62c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a62c800.ost_server_uuid in FULL state after 0 sec PASS 33b (23s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 23:58:00 (1721879880) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 23:58:01 (1721879881) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 23:58:02 (1721879882) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 23:58:03 (1721879883) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 23:58:06 (1721879886) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.182157 s, 69.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 23:58:09 (1721879889) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0118649 s, 88.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0165404 s, 63.4 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 23:58:12 (1721879892) PASS 36c (2s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 23:58:14 (1721879894) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000564253 s, 1.8 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000560438 s, 1.8 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000811508 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000722442 s, 1.4 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 23:58:17 (1721879897) create test files with size 15738776 .. a6b8e6e3f6ad2ac09594e1dc40011a6e - c8f0b50972c5772de6c4ad9b6bb5f0b5 - 11cc45de61621c4074b30050be9a002a - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg443-client.virtnet /mnt/lustre (opts:) Starting client: oleg443-client.virtnet: -o user_xattr,flock oleg443-server@tcp:/lustre /mnt/lustre PASS 37 (5s) == sanity-flr test 38: resync ============================ 23:58:22 (1721879902) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 23:58:25 (1721879905) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 23:58:27 (1721879907) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0212323 s, 49.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x37:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x38:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102501 s, 102 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 23:58:30 (1721879910) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0239414 s, 87.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0944455 s, 44.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0504846 s, 41.5 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 23:58:32 (1721879912) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 23:58:33 (1721879913) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 23:58:34 (1721879914) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0103763 s, 101 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 23:58:37 (1721879917) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0948313 s, 33.2 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x10:0x0]: Invalid argument (22) PASS 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 23:58:40 (1721879920) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x42:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x46:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x15:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 23:58:42 (1721879922) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0773069 s, 136 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg443-client.virtnet /mnt/lustre (opts:) Starting client: oleg443-client.virtnet: -o user_xattr,flock oleg443-server@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (3s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 23:58:45 (1721879925) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff880136891000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0774775 s, 135 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg443-client.virtnet /mnt/lustre (opts:) Starting client: oleg443-client.virtnet: -o user_xattr,flock oleg443-server@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800a957b800.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 23:58:49 (1721879929) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (8s) == sanity-flr test 46: Verify setstripe --copy option ==== 23:58:57 (1721879937) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (8s) == sanity-flr test 47: Verify mirror obj alloc =========== 23:59:05 (1721879945) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 23:59:06 (1721879946) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0783241 s, 40.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0607446 s, 51.8 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 59bd40e27a6f5f9030efdd780eada101 - mirror 1 checksum: bf21e60ea4c3ff72f99979dbf85b9b4a - mirror 2 checksum: 59bd40e27a6f5f9030efdd780eada101 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Jul 24 23:59:07 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4c:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Jul 24 23:59:07 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 59bd40e27a6f5f9030efdd780eada101 - mirror 1 checksum: bf21e60ea4c3ff72f99979dbf85b9b4a - mirror 2 checksum: bf21e60ea4c3ff72f99979dbf85b9b4a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Jul 24 23:59:07 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4c:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Jul 24 23:59:07 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } PASS 48 (3s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 23:59:09 (1721879949) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0165365 s, 63.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 23:59:11 (1721879951) Failing mds1 on oleg443-server Stopping /mnt/lustre-mds1 (opts:) on oleg443-server 23:59:19 (1721879959) shut down Failover mds1 to oleg443-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg443-server: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-MDT0000 23:59:32 (1721879972) targets are mounted 23:59:32 (1721879972) facet_failover done oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00989129 s, 106 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } PASS 50A (28s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 23:59:40 (1721879980) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00862792 s, 2.4 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00531052 s, 3.9 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 23:59:42 (1721879982) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0167803 s, 1.2 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00584502 s, 3.5 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 23:59:44 (1721879984) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00248688 s, 6.6 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 23:59:47 (1721879987) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0100088 s, 2.0 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 23:59:49 (1721879989) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0022401 s, 1.8 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0116878 s, 350 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 23:59:51 (1721879991) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) PASS 60b (4s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 23:59:55 (1721879995) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 00:00:13 (1721880013) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 00:00:30 (1721880030) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 00:00:47 (1721880047) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 00:00:49 (1721880049) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 25283 mirror split pid 25284 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4759: 25283 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4759: 25284 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (61s) == sanity-flr test 70a: flr mode fsx test ================ 00:01:50 (1721880110) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 111 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported 000001 1721880111.344818 write 0x002298 thru 0x009182 (0x6eeb bytes) 000002 1721880111.348140 write 0x0329cd thru 0x03fb1e (0xd152 bytes) 000004 1721880111.349442 write 0x02b377 thru 0x036f18 (0xbba2 bytes) 000005 1721880111.351975 write 0x03cba3 thru 0x03f927 (0x2d85 bytes) 000006 1721880111.352567 mapwrite 0x02ccf7 thru 0x030c47 (0x3f51 bytes) 000008 1721880111.363835 write 0x037411 thru 0x03a4d9 (0x30c9 bytes) 000009 1721880111.364492 mapread 0x013042 thru 0x0222d0 (0xf28f bytes) 000013 1721880111.374492 mapread 0x0375a6 thru 0x03fb1e (0x8579 bytes) truncating to largest ever: 0xe092 000014 1721880111.375108 trunc from 0x03fb1f to 0x00e091 (0x31a8d bytes) 000015 1721880111.382831 read 0x00b3e5 thru 0x00e091 (0x2cad bytes) 000016 1721880111.385543 read 0x00966e thru 0x00e091 (0x4a24 bytes) 000017 1721880111.388869 mapwrite 0x01015d thru 0x010d58 (0xbfc bytes) 000018 1721880111.405734 MIRROR_EXTEND to 2 mirrors 000019 1721880111.553685 mapread 0x0010cf thru 0x008d82 (0x7cb4 bytes) 000020 1721880111.564439 mapread 0x00fe4e thru 0x010d58 (0xf0b bytes) truncating to largest ever: 0x3e852 000021 1721880111.571867 trunc from 0x010d59 to 0x03e851 (0x2daf9 bytes) 000023 1721880111.595568 read 0x00f297 thru 0x01e279 (0xefe3 bytes) 000024 1721880111.599320 write 0x00d70c thru 0x010962 (0x3257 bytes) 000026 1721880111.604867 MIRROR_EXTEND to 3 mirrors 000027 1721880111.753611 trunc from 0x03e852 to 0x00d8a1 (0x30fb0 bytes) 000028 1721880111.775543 mapread 0x001b21 thru 0x00363a (0x1b1a bytes) 000029 1721880111.783137 trunc from 0x00d8a2 to 0x00a165 (0x373c bytes) 000030 1721880111.792376 punch from 0x00015e to 0x00975c (0x95ff bytes) 000031 1721880111.794677 fallocate 0x00ed62 thru 0x019838 (0xaad7 bytes) 000033 1721880111.797412 mapwrite 0x02e497 thru 0x03a54c (0xc0b6 bytes) 000035 1721880111.825006 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000036 1721880111.895568 mapread 0x005116 thru 0x012ff6 (0xdee1 bytes) 000037 1721880111.907444 mapwrite 0x0211ff thru 0x027ff2 (0x6df4 bytes) 000038 1721880111.928042 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000039 1721880112.009075 mapread 0x012cd1 thru 0x01b11f (0x844f bytes) 000040 1721880112.021927 write 0x02698a thru 0x03535c (0xe9d3 bytes) 000041 1721880112.031421 mapwrite 0x02ed07 thru 0x030e9c (0x2196 bytes) 000042 1721880112.040864 punch from 0x028ed5 to 0x033798 (0xa8c4 bytes) 000043 1721880112.043003 write 0x01ce8f thru 0x02c99d (0xfb0f bytes) 000044 1721880112.046209 write 0x035f23 thru 0x03ffff (0xa0dd bytes) 000045 1721880112.047016 mapwrite 0x03043b thru 0x03c525 (0xc0eb bytes) 000047 1721880112.058352 read 0x01dca4 thru 0x02d2ec (0xf649 bytes) 000048 1721880112.058723 trunc from 0x040000 to 0x02016b (0x1fe94 bytes) 000049 1721880112.065476 mapread 0x017d6a thru 0x02016b (0x8402 bytes) 000050 1721880112.066186 read 0x0168b5 thru 0x01fc06 (0x9352 bytes) 000051 1721880112.066478 fallocate 0x02abd8 thru 0x02bf43 (0x136c bytes) 000052 1721880112.070276 MIRROR_EXTEND to 2 mirrors 000053 1721880112.245675 fallocate 0x01adf1 thru 0x01cfcb (0x21db bytes) 000054 1721880112.261594 MIRROR_EXTEND to 3 mirrors 000056 1721880112.437927 write 0x03ba18 thru 0x03ffff (0x45e8 bytes) 000057 1721880112.450297 write 0x021984 thru 0x0239a6 (0x2023 bytes) 000059 1721880112.454218 write 0x031176 thru 0x03ffff (0xee8a bytes) 000060 1721880112.457879 write 0x0071fd thru 0x016ea6 (0xfcaa bytes) 000061 1721880112.462961 fallocate 0x02b3e5 thru 0x032097 (0x6cb3 bytes) 000062 1721880112.465113 punch from 0x012b32 to 0x0151c4 (0x2693 bytes) 000063 1721880112.471509 punch from 0x009141 to 0x00ed83 (0x5c43 bytes) 000064 1721880112.475029 trunc from 0x040000 to 0x01aa01 (0x255fe bytes) 000066 1721880112.489675 read 0x006491 thru 0x013892 (0xd402 bytes) 000067 1721880112.493610 write 0x01b633 thru 0x023cde (0x86ac bytes) 000068 1721880112.494676 fallocate 0x00db8f thru 0x015839 (0x7cab bytes) 000069 1721880112.497944 mapread 0x010485 thru 0x0175b5 (0x7131 bytes) 000070 1721880112.503559 trunc from 0x023cdf to 0x02e48e (0xa7b0 bytes) 000071 1721880112.510363 trunc from 0x02e48f to 0x03cbd2 (0xe744 bytes) 000072 1721880112.516717 punch from 0x00b914 to 0x00e634 (0x2d21 bytes) 000073 1721880112.518914 mapread 0x03b39b thru 0x03cbd2 (0x1838 bytes) 000074 1721880112.522784 trunc from 0x03cbd3 to 0x03b554 (0x167e bytes) 000075 1721880112.531085 trunc from 0x03b555 to 0x02fabf (0xba95 bytes) 000076 1721880112.537945 fallocate 0x0072e7 thru 0x0163be (0xf0d8 bytes) 000077 1721880112.540732 read 0x013d79 thru 0x02269a (0xe922 bytes) 000078 1721880112.541153 read 0x015c92 thru 0x01efe1 (0x9350 bytes) 000079 1721880112.543697 MIRROR_RESYNC 3 mirrors 000081 1721880112.774793 mapread 0x02e140 thru 0x02fabf (0x1980 bytes) 000082 1721880112.780046 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000083 1721880112.878428 mapread 0x0078f0 thru 0x01676d (0xee7e bytes) 000084 1721880112.889495 mapwrite 0x02aa98 thru 0x02eb56 (0x40bf bytes) 000085 1721880112.945604 punch from 0x019863 to 0x019d76 (0x514 bytes) 000086 1721880112.947907 mapwrite 0x0319a7 thru 0x03e955 (0xcfaf bytes) 000088 1721880112.985215 mapwrite 0x002395 thru 0x0027d6 (0x442 bytes) 000089 1721880112.996301 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000090 1721880113.036494 punch from 0x019f12 to 0x0297ba (0xf8a9 bytes) 000091 1721880113.039006 read 0x002c8d thru 0x00358d (0x901 bytes) 000092 1721880113.042136 trunc from 0x03e956 to 0x008686 (0x362cf bytes) 000093 1721880113.051783 punch from 0x003998 to 0x008686 (0x4cef bytes) 000095 1721880113.057245 MIRROR_EXTEND to 2 mirrors 000096 1721880113.203889 trunc from 0x008687 to 0x02a317 (0x21c91 bytes) 000098 1721880113.227310 trunc from 0x02a318 to 0x03d220 (0x12f09 bytes) 000099 1721880113.234314 read 0x01642a thru 0x01d72b (0x7302 bytes) 000100 1721880113.237741 write 0x036cd8 thru 0x03ffff (0x9328 bytes) 000101 1721880113.240465 trunc from 0x040000 to 0x036947 (0x96b8 bytes) 000102 1721880113.245110 trunc from 0x036948 to 0x014eb0 (0x21a97 bytes) 000103 1721880113.249772 write 0x0318e5 thru 0x037978 (0x6094 bytes) 000104 1721880113.250522 write 0x029b63 thru 0x034033 (0xa4d1 bytes) 000105 1721880113.253055 fallocate 0x020120 thru 0x02911c (0x8ffd bytes) 000106 1721880113.255140 fallocate 0x012a6b thru 0x0153e2 (0x2978 bytes) 000107 1721880113.256706 fallocate 0x006702 thru 0x00a877 (0x4176 bytes) 000108 1721880113.258085 trunc from 0x037979 to 0x005707 (0x32271 bytes) 000109 1721880113.262101 mapwrite 0x03d53d thru 0x03f3e6 (0x1eaa bytes) 000110 1721880113.274196 fallocate 0x002662 thru 0x0093e6 (0x6d85 bytes) 000111 1721880113.276160 punch from 0x00e330 to 0x0183de (0xa0af bytes) 000112 1721880113.277816 fallocate 0x0005fb thru 0x001efb (0x1901 bytes) 000113 1721880113.279520 punch from 0x010155 to 0x0198e2 (0x978e bytes) 000114 1721880113.280915 read 0x03ba13 thru 0x03e52f (0x2b1d bytes) 000115 1721880113.281232 mapwrite 0x023eb2 thru 0x0284da (0x4629 bytes) 000116 1721880113.293269 read 0x013e27 thru 0x021ae3 (0xdcbd bytes) 000117 1721880113.293718 read 0x006ae1 thru 0x00f5f9 (0x8b19 bytes) 000118 1721880113.294124 punch from 0x00cd78 to 0x00dea2 (0x112b bytes) 000120 1721880113.296334 mapwrite 0x02568d thru 0x02fe58 (0xa7cc bytes) 000121 1721880113.306694 MIRROR_EXTEND to 3 mirrors 000122 1721880113.448554 write 0x03b710 thru 0x03ffff (0x48f0 bytes) 000124 1721880113.464605 trunc from 0x040000 to 0x031795 (0xe86a bytes) 000126 1721880113.478406 read 0x005c22 thru 0x00c463 (0x6842 bytes) 000127 1721880113.483045 MIRROR_EXTEND to 4 mirrors 000128 1721880113.613376 read 0x006cb6 thru 0x00f6ed (0x8a38 bytes) 000129 1721880113.624320 mapread 0x01f359 thru 0x0275a7 (0x824f bytes) 000130 1721880113.639245 mapwrite 0x001f89 thru 0x00e3d9 (0xc451 bytes) 000131 1721880113.697184 trunc from 0x031796 to 0x0331b9 (0x1a24 bytes) 000133 1721880113.717901 MIRROR_RESYNC 4 mirrors 000137 1721880113.946770 fallocate 0x00ad7f thru 0x00f393 (0x4615 bytes) 000138 1721880113.957340 write 0x0201b2 thru 0x027a96 (0x78e5 bytes) 000139 1721880113.963737 mapread 0x00e0f3 thru 0x019063 (0xaf71 bytes) 000140 1721880113.969082 punch from 0x01f10d to 0x02c001 (0xcef5 bytes) 000141 1721880113.976040 MIRROR_EXTEND to 5 mirrors 000143 1721880114.120233 trunc from 0x0331ba to 0x02148c (0x11d2d bytes) 000144 1721880114.134334 punch from 0x00f361 to 0x01c887 (0xd527 bytes) 000145 1721880114.136201 mapread 0x00a814 thru 0x00fa47 (0x5234 bytes) 000146 1721880114.141447 fallocate 0x022915 thru 0x031c6f (0xf35b bytes) 000147 1721880114.143064 write 0x01a34d thru 0x0272ec (0xcfa0 bytes) 000149 1721880114.146476 mapread 0x024c42 thru 0x026f7b (0x233a bytes) 000150 1721880114.148717 MIRROR_RESYNC 5 mirrors 000153 1721880114.316697 read 0x006413 thru 0x00d833 (0x7421 bytes) 000154 1721880114.319251 fallocate 0x025487 thru 0x02b761 (0x62db bytes) 000155 1721880114.328232 trunc from 0x02b762 to 0x00c60a (0x1f157 bytes) fallocating to largest ever: 0x40000 000156 1721880114.338439 fallocate 0x03fc76 thru 0x03ffff (0x38a bytes) 000159 1721880114.340875 read 0x001b2d thru 0x00c5fd (0xaad1 bytes) 000160 1721880114.345242 read 0x004d82 thru 0x00b65c (0x68db bytes) 000163 1721880114.346292 mapwrite 0x03acc8 thru 0x03d569 (0x28a2 bytes) 000166 1721880114.354058 mapwrite 0x0221f6 thru 0x02e638 (0xc443 bytes) 000167 1721880114.361316 mapwrite 0x02e11c thru 0x03b2e5 (0xd1ca bytes) 000169 1721880114.370347 read 0x02fce1 thru 0x0313bd (0x16dd bytes) 000170 1721880114.372511 MIRROR_RESYNC 5 mirrors 000172 1721880114.563335 fallocate 0x007d79 thru 0x00ac0f (0x2e97 bytes) 000173 1721880114.574516 fallocate 0x0298f6 thru 0x03663a (0xcd45 bytes) 000175 1721880114.576464 mapwrite 0x0361d3 thru 0x03c928 (0x6756 bytes) 000177 1721880114.586676 punch from 0x037c7b to 0x03fb23 (0x7ea9 bytes) 000178 1721880114.588380 mapread 0x00ad95 thru 0x0174fd (0xc769 bytes) 000179 1721880114.593082 mapread 0x012638 thru 0x01c907 (0xa2d0 bytes) 000180 1721880114.593558 write 0x004d93 thru 0x00c998 (0x7c06 bytes) 000181 1721880114.596729 write 0x0114bc thru 0x01fd89 (0xe8ce bytes) 000182 1721880114.597488 punch from 0x01c419 to 0x02c2f9 (0xfee1 bytes) 000183 1721880114.601566 read 0x0208c6 thru 0x0285eb (0x7d26 bytes) 000184 1721880114.604081 read 0x030839 thru 0x036ef1 (0x66b9 bytes) 000185 1721880114.606680 fallocate 0x0078ab thru 0x014068 (0xc7be bytes) 000186 1721880114.609039 fallocate 0x02ab6f thru 0x02eab5 (0x3f47 bytes) 000187 1721880114.610932 mapwrite 0x03ec2c thru 0x03ffff (0x13d4 bytes) 000188 1721880114.615208 read 0x016a84 thru 0x025e6d (0xf3ea bytes) 000191 1721880114.616355 mapwrite 0x02af5e thru 0x03714b (0xc1ee bytes) 000192 1721880114.626387 write 0x02cb70 thru 0x0332e3 (0x6774 bytes) 000193 1721880114.627152 mapread 0x0227b7 thru 0x030e18 (0xe662 bytes) 000194 1721880114.627666 write 0x022b13 thru 0x0253d5 (0x28c3 bytes) 000195 1721880114.628242 punch from 0x00c563 to 0x00df14 (0x19b2 bytes) 000198 1721880114.630811 trunc from 0x040000 to 0x03f8e0 (0x71f bytes) 000199 1721880114.641941 mapwrite 0x022a23 thru 0x02e794 (0xbd72 bytes) 000200 1721880114.650451 write 0x01526a thru 0x0155f8 (0x38f bytes) 000203 1721880114.653477 MIRROR_RESYNC 5 mirrors 000204 1721880114.895750 mapread 0x01de7a thru 0x026210 (0x8397 bytes) 000207 1721880114.904421 MIRROR_EXTEND to 6 mirrors 000208 1721880115.057935 write 0x025748 thru 0x02f782 (0xa03b bytes) 000209 1721880115.074188 fallocate 0x02956d thru 0x02daee (0x4582 bytes) 000210 1721880115.076339 mapread 0x030ee0 thru 0x032a89 (0x1baa bytes) 000212 1721880115.080323 trunc from 0x03f8e1 to 0x001a23 (0x3debd bytes) 000213 1721880115.094527 mapwrite 0x03bff6 thru 0x03ffff (0x400a bytes) 000214 1721880115.109371 punch from 0x026852 to 0x028258 (0x1a07 bytes) 000215 1721880115.111187 trunc from 0x040000 to 0x008dca (0x37235 bytes) 000216 1721880115.116933 read 0x0066be thru 0x008dca (0x270d bytes) 000217 1721880115.119073 mapread 0x0006f8 thru 0x0049da (0x42e3 bytes) 000218 1721880115.123349 write 0x02a284 thru 0x030d13 (0x6a90 bytes) 000219 1721880115.124083 trunc from 0x030d14 to 0x01529e (0x1ba75 bytes) 000220 1721880115.129842 fallocate 0x035686 thru 0x03ebc0 (0x953b bytes) 000221 1721880115.132596 read 0x00bd06 thru 0x00dd60 (0x205b bytes) 000222 1721880115.137190 mapread 0x010c01 thru 0x013105 (0x2505 bytes) 000223 1721880115.137824 mapwrite 0x02151b thru 0x029696 (0x817c bytes) 000224 1721880115.145471 mapread 0x03bd10 thru 0x03ebc0 (0x2eb1 bytes) 000225 1721880115.146069 mapread 0x016a2f thru 0x018d0f (0x22e1 bytes) 000226 1721880115.148111 MIRROR_RESYNC 6 mirrors 000228 1721880115.371607 fallocate 0x00ebdc thru 0x014c64 (0x6089 bytes) 000229 1721880115.385030 MIRROR_RESYNC 6 mirrors 000230 1721880115.616909 mapread 0x019a6f thru 0x02959b (0xfb2d bytes) 000232 1721880115.626994 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000233 1721880115.695497 trunc from 0x03ebc1 to 0x020c1a (0x1dfa6 bytes) 000234 1721880115.711868 mapread 0x01b37e thru 0x01f468 (0x40eb bytes) 000235 1721880115.717281 read 0x003acc thru 0x00b540 (0x7a75 bytes) 000237 1721880115.720253 mapwrite 0x026ae1 thru 0x03545e (0xe97e bytes) 000238 1721880115.737206 trunc from 0x03545f to 0x01dd30 (0x1772e bytes) 000239 1721880115.741181 punch from 0x003e14 to 0x00773d (0x392a bytes) 000240 1721880115.742875 mapwrite 0x030df1 thru 0x03fcb0 (0xeec0 bytes) 000242 1721880115.758907 write 0x03cc1d thru 0x03f7da (0x2bbe bytes) 000243 1721880115.759421 punch from 0x02cd07 to 0x03179b (0x4a95 bytes) 000245 1721880115.761474 punch from 0x01aab3 to 0x022e7f (0x83cd bytes) 000246 1721880115.763285 punch from 0x0185bc to 0x01bf77 (0x39bc bytes) 000247 1721880115.764882 mapread 0x0244d6 thru 0x03060a (0xc135 bytes) 000248 1721880115.769678 mapwrite 0x01c16a thru 0x022212 (0x60a9 bytes) 000249 1721880115.778417 mapread 0x011e81 thru 0x01c296 (0xa416 bytes) 000250 1721880115.781807 mapwrite 0x0285a1 thru 0x0382ae (0xfd0e bytes) 000251 1721880115.791068 fallocate 0x0122f0 thru 0x01c65b (0xa36c bytes) 000252 1721880115.794647 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000253 1721880115.843331 fallocate 0x028084 thru 0x03611e (0xe09b bytes) 000254 1721880115.845265 fallocate 0x0189db thru 0x021d63 (0x9389 bytes) 000256 1721880115.847265 trunc from 0x03fcb1 to 0x0210c9 (0x1ebe7 bytes) 000259 1721880115.853085 trunc from 0x0210ca to 0x030693 (0xf5ca bytes) 000260 1721880115.857240 mapwrite 0x006b7f thru 0x00d207 (0x6689 bytes) 000261 1721880115.868449 mapwrite 0x00ce34 thru 0x0110eb (0x42b8 bytes) 000262 1721880115.876358 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000263 1721880115.933167 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000264 1721880115.984425 punch from 0x0116be to 0x01fa01 (0xe344 bytes) 000265 1721880115.986713 mapwrite 0x02608e thru 0x03217a (0xc0ed bytes) 000266 1721880116.012158 punch from 0x006de0 to 0x016201 (0xf422 bytes) 000267 1721880116.014003 mapwrite 0x02848e thru 0x03329c (0xae0f bytes) 000269 1721880116.029069 mapread 0x00b8ef thru 0x019f22 (0xe634 bytes) 000270 1721880116.036009 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000271 1721880116.076945 fallocate 0x02bd86 thru 0x03aae2 (0xed5d bytes) 000272 1721880116.079085 mapwrite 0x00abad thru 0x00e8dd (0x3d31 bytes) 000273 1721880116.092547 write 0x008978 thru 0x013873 (0xaefc bytes) 000275 1721880116.095714 mapread 0x00a991 thru 0x017b76 (0xd1e6 bytes) 000276 1721880116.100475 punch from 0x01bf30 to 0x02217d (0x624e bytes) 000277 1721880116.102580 write 0x035263 thru 0x0356b6 (0x454 bytes) 000278 1721880116.103162 read 0x0198fc thru 0x022f99 (0x969e bytes) 000279 1721880116.106204 fallocate 0x02f2eb thru 0x033e9c (0x4bb2 bytes) 000280 1721880116.108400 read 0x00084a thru 0x00b481 (0xac38 bytes) 000282 1721880116.112279 trunc from 0x0356b7 to 0x01bc2a (0x19a8c bytes) 000283 1721880116.117221 punch from 0x018bfe to 0x01bc2a (0x302d bytes) 000285 1721880116.120230 mapread 0x00719c thru 0x00de2f (0x6c94 bytes) 000287 1721880116.121085 mapread 0x01a495 thru 0x01bc2a (0x1796 bytes) 000290 1721880116.127421 MIRROR_EXTEND to 2 mirrors 000291 1721880116.271710 mapread 0x01b071 thru 0x01bc2a (0xbba bytes) 000294 1721880116.279994 trunc from 0x01bc2b to 0x003eb9 (0x17d71 bytes) 000295 1721880116.296352 write 0x01b8f8 thru 0x027470 (0xbb79 bytes) 000297 1721880116.297484 punch from 0x005f32 to 0x013ea0 (0xdf6f bytes) 000299 1721880116.299452 fallocate 0x03be71 thru 0x03e29f (0x242f bytes) 000300 1721880116.301425 trunc from 0x027471 to 0x033a2b (0xc5bb bytes) 000301 1721880116.307307 punch from 0x027e4a to 0x0337cd (0xb984 bytes) 000302 1721880116.313640 write 0x001981 thru 0x00f214 (0xd894 bytes) 000303 1721880116.318217 fallocate 0x031c93 thru 0x03ffff (0xe36d bytes) 000304 1721880116.322700 MIRROR_RESYNC 2 mirrors 000305 1721880116.465862 MIRROR_RESYNC 2 mirrors 000306 1721880116.514267 read 0x0111be thru 0x01c031 (0xae74 bytes) 000307 1721880116.517342 punch from 0x01f828 to 0x02f075 (0xf84e bytes) 000308 1721880116.529200 mapwrite 0x00ea48 thru 0x00f34b (0x904 bytes) 000311 1721880116.538778 read 0x02102a thru 0x0239aa (0x2981 bytes) 000312 1721880116.542315 mapwrite 0x0320db thru 0x03ffff (0xdf25 bytes) 000313 1721880116.565490 read 0x002ff5 thru 0x005ab0 (0x2abc bytes) 000314 1721880116.566226 write 0x00400d thru 0x00c158 (0x814c bytes) 000315 1721880116.567002 read 0x00ef72 thru 0x01cc65 (0xdcf4 bytes) 000316 1721880116.567404 write 0x012bb3 thru 0x01afa7 (0x83f5 bytes) 000317 1721880116.570031 MIRROR_EXTEND to 3 mirrors 000318 1721880116.744357 trunc from 0x040000 to 0x007b0e (0x384f1 bytes) 000320 1721880116.762609 mapwrite 0x02ab4b thru 0x034dcf (0xa285 bytes) 000322 1721880116.785834 read 0x001c06 thru 0x00dae6 (0xbee1 bytes) 000323 1721880116.789131 fallocate 0x03741a thru 0x038c6c (0x1853 bytes) 000324 1721880116.792484 MIRROR_RESYNC 3 mirrors 000326 1721880116.947850 mapwrite 0x01aeef thru 0x026b02 (0xbc14 bytes) 000327 1721880116.996710 mapwrite 0x01fbca thru 0x02129f (0x16d6 bytes) 000328 1721880117.002729 write 0x0368d0 thru 0x03fe79 (0x95aa bytes) 000329 1721880117.009564 trunc from 0x03fe7a to 0x030360 (0xfb19 bytes) 000330 1721880117.029030 write 0x024f13 thru 0x02ed98 (0x9e86 bytes) 000331 1721880117.031596 mapread 0x008731 thru 0x017dcc (0xf69c bytes) 000332 1721880117.040778 write 0x00882f thru 0x00d9fe (0x51d0 bytes) 000333 1721880117.041724 trunc from 0x030361 to 0x026bfe (0x9762 bytes) 000335 1721880117.050076 mapwrite 0x02f7a6 thru 0x035d96 (0x65f1 bytes) 000336 1721880117.073338 write 0x02b0e9 thru 0x0308bd (0x57d5 bytes) 000337 1721880117.074135 fallocate 0x0024a9 thru 0x00645d (0x3fb5 bytes) 000338 1721880117.076263 write 0x00b590 thru 0x00e49b (0x2f0c bytes) 000339 1721880117.076955 punch from 0x02f3c8 to 0x035d96 (0x69cf bytes) 000341 1721880117.083194 fallocate 0x026d6f thru 0x0284ff (0x1791 bytes) 000342 1721880117.085030 mapwrite 0x002cc4 thru 0x009b1e (0x6e5b bytes) 000344 1721880117.094492 trunc from 0x035d97 to 0x0282ce (0xdac8 bytes) 000345 1721880117.100347 mapread 0x017e4e thru 0x01edd4 (0x6f87 bytes) 000346 1721880117.101504 punch from 0x000f9f to 0x008a74 (0x7ad6 bytes) 000347 1721880117.103574 read 0x00da4c thru 0x0169e6 (0x8f9b bytes) 000348 1721880117.104006 fallocate 0x00cad1 thru 0x0190e0 (0xc610 bytes) 000349 1721880117.106349 trunc from 0x0282cf to 0x011470 (0x16e5e bytes) 000351 1721880117.112789 mapread 0x0057ef thru 0x00a30c (0x4b1e bytes) 000352 1721880117.115914 read 0x009375 thru 0x00eaf5 (0x5781 bytes) 000354 1721880117.116598 write 0x030c8e thru 0x03cd49 (0xc0bc bytes) 000355 1721880117.117755 mapwrite 0x00b850 thru 0x00cb11 (0x12c2 bytes) 000356 1721880117.126374 trunc from 0x03cd4a to 0x0315f5 (0xb754 bytes) 000357 1721880117.133290 punch from 0x00161b to 0x00e664 (0xd04a bytes) 000359 1721880117.137995 MIRROR_EXTEND to 4 mirrors 000362 1721880117.324554 mapwrite 0x017771 thru 0x01df44 (0x67d4 bytes) 000364 1721880117.367982 read 0x01d34b thru 0x02d200 (0xfeb6 bytes) 000365 1721880117.373998 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000366 1721880117.435295 fallocate 0x021f91 thru 0x02c7fa (0xa86a bytes) 000367 1721880117.437916 punch from 0x01a07c to 0x01d635 (0x35ba bytes) 000368 1721880117.442200 MIRROR_EXTEND to 4 mirrors 000369 1721880117.600562 MIRROR_EXTEND to 5 mirrors 000370 1721880117.779097 mapread 0x002394 thru 0x009215 (0x6e82 bytes) 000371 1721880117.784309 write 0x03b461 thru 0x03ffff (0x4b9f bytes) 000372 1721880117.798743 write 0x0294e2 thru 0x032913 (0x9432 bytes) 000373 1721880117.805247 MIRROR_RESYNC 5 mirrors 000374 1721880118.225578 mapwrite 0x03827b thru 0x03ffff (0x7d85 bytes) 000375 1721880118.255454 trunc from 0x040000 to 0x017427 (0x28bd8 bytes) 000376 1721880118.261855 punch from 0x00864b to 0x0151df (0xcb95 bytes) 000377 1721880118.263157 read 0x016919 thru 0x017427 (0xb0f bytes) 000378 1721880118.264743 fallocate 0x0371f0 thru 0x03ffff (0x8e10 bytes) 000379 1721880118.266257 mapwrite 0x0323b8 thru 0x035918 (0x3561 bytes) 000380 1721880118.273275 trunc from 0x040000 to 0x033d80 (0xc27f bytes) 000382 1721880118.277524 mapread 0x01f69e thru 0x02c81d (0xd180 bytes) 000383 1721880118.282738 read 0x017def thru 0x020403 (0x8615 bytes) 000385 1721880118.285952 read 0x01bd55 thru 0x025a2a (0x9cd6 bytes) 000386 1721880118.286303 read 0x008a47 thru 0x011b78 (0x9132 bytes) 000387 1721880118.286661 trunc from 0x033d81 to 0x00190f (0x32471 bytes) 000388 1721880118.291096 mapread 0x00150e thru 0x00190f (0x402 bytes) 000389 1721880118.291607 write 0x009ded thru 0x00dc4d (0x3e61 bytes) 000390 1721880118.292267 fallocate 0x013308 thru 0x01d4a4 (0xa19d bytes) 000391 1721880118.293989 punch from 0x005dda to 0x009789 (0x39b0 bytes) 000392 1721880118.297801 mapwrite 0x015c7d thru 0x021720 (0xbaa4 bytes) 000394 1721880118.313202 punch from 0x01804a to 0x018b36 (0xaed bytes) 000395 1721880118.315022 read 0x00e5e8 thru 0x00f9fd (0x1416 bytes) 000396 1721880118.315423 read 0x00a468 thru 0x012e0b (0x89a4 bytes) 000397 1721880118.315848 mapwrite 0x023943 thru 0x02459a (0xc58 bytes) 000398 1721880118.329171 MIRROR_EXTEND to 6 mirrors 000399 1721880118.465554 MIRROR_EXTEND to 7 mirrors 000401 1721880118.624730 write 0x02b5a1 thru 0x03583b (0xa29b bytes) 000402 1721880118.639323 trunc from 0x03583c to 0x01ec88 (0x16bb3 bytes) 000403 1721880118.653480 mapread 0x0196ef thru 0x01ec88 (0x559a bytes) 000404 1721880118.660685 punch from 0x005d6e to 0x00f234 (0x94c7 bytes) 000405 1721880118.663033 mapread 0x0129ca thru 0x01cffc (0xa633 bytes) 000406 1721880118.667389 write 0x02c087 thru 0x02f3e6 (0x3360 bytes) 000408 1721880118.668693 mapread 0x003021 thru 0x006842 (0x3822 bytes) 000409 1721880118.673572 write 0x018ded thru 0x01c8da (0x3aee bytes) 000411 1721880118.674486 write 0x03c8a3 thru 0x03ffff (0x375d bytes) 000412 1721880118.675262 punch from 0x032f8f to 0x03da26 (0xaa98 bytes) 000413 1721880118.684827 MIRROR_EXTEND to 8 mirrors 000415 1721880118.838041 mapwrite 0x032b8a thru 0x03568b (0x2b02 bytes) 000416 1721880118.874315 read 0x01974c thru 0x024fb9 (0xb86e bytes) 000417 1721880118.877305 fallocate 0x007c22 thru 0x014329 (0xc708 bytes) 000418 1721880118.878821 mapread 0x0029bf thru 0x00961f (0x6c61 bytes) 000419 1721880118.883229 MIRROR_EXTEND to 9 mirrors 000420 1721880119.017590 punch from 0x016211 to 0x01b3c0 (0x51b0 bytes) 000422 1721880119.028766 mapread 0x03508f thru 0x03ffff (0xaf71 bytes) 000423 1721880119.035045 mapread 0x002037 thru 0x011591 (0xf55b bytes) 000424 1721880119.041475 MIRROR_RESYNC 9 mirrors 000425 1721880119.535798 write 0x00611c thru 0x00a9a6 (0x488b bytes) 000426 1721880119.549507 MIRROR_EXTEND to 10 mirrors 000427 1721880119.700766 mapread 0x0050c8 thru 0x00c69a (0x75d3 bytes) 000428 1721880119.709508 punch from 0x0363cb to 0x03ef76 (0x8bac bytes) 000430 1721880119.721011 mapread 0x0258eb thru 0x034b1f (0xf235 bytes) 000431 1721880119.726159 fallocate 0x015175 thru 0x01a64d (0x54d9 bytes) 000432 1721880119.727912 punch from 0x012f29 to 0x01ab31 (0x7c09 bytes) 000433 1721880119.730805 fallocate 0x003beb thru 0x00ded7 (0xa2ed bytes) 000434 1721880119.732824 write 0x0201cf thru 0x027271 (0x70a3 bytes) 000435 1721880119.739070 MIRROR_EXTEND to 11 mirrors 000436 1721880119.905848 write 0x01fd4f thru 0x029418 (0x96ca bytes) 000437 1721880119.921408 write 0x01f580 thru 0x02350b (0x3f8c bytes) 000438 1721880119.922174 mapread 0x02fda9 thru 0x03cca4 (0xcefc bytes) 000439 1721880119.928235 write 0x014f03 thru 0x01e644 (0x9742 bytes) 000441 1721880119.933221 fallocate 0x03b7a1 thru 0x03ffff (0x485f bytes) 000442 1721880119.936754 MIRROR_RESYNC 11 mirrors 000443 1721880120.763025 punch from 0x031f88 to 0x036ba7 (0x4c20 bytes) 000444 1721880120.774214 mapwrite 0x0070f1 thru 0x0101b8 (0x90c8 bytes) 000445 1721880120.788618 mapwrite 0x012bc0 thru 0x0213c2 (0xe803 bytes) 000446 1721880120.803413 write 0x03bc6b thru 0x03ffff (0x4395 bytes) 000447 1721880120.808219 read 0x02ecaa thru 0x032d94 (0x40eb bytes) 000448 1721880120.812939 MIRROR_RESYNC 11 mirrors 000449 1721880121.363167 punch from 0x039f19 to 0x03ffff (0x60e7 bytes) 000450 1721880121.377510 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000452 1721880121.438083 mapread 0x003526 thru 0x004204 (0xcdf bytes) 000454 1721880121.442083 punch from 0x021c8f to 0x0314c7 (0xf839 bytes) 000455 1721880121.445202 mapread 0x004c93 thru 0x013c9c (0xf00a bytes) 000457 1721880121.453109 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000458 1721880121.513852 mapwrite 0x00f601 thru 0x01f0ae (0xfaae bytes) 000459 1721880121.535338 read 0x0224f9 thru 0x02481e (0x2326 bytes) 000460 1721880121.537813 fallocate 0x0313e4 thru 0x033a78 (0x2695 bytes) 000461 1721880121.540032 read 0x03cc4f thru 0x03f4d0 (0x2882 bytes) 000462 1721880121.543912 mapwrite 0x0394f9 thru 0x03ffff (0x6b07 bytes) 000463 1721880121.550346 read 0x03cfa7 thru 0x03ffff (0x3059 bytes) 000464 1721880121.550807 fallocate 0x039dc5 thru 0x03ffff (0x623b bytes) 000465 1721880121.552699 write 0x03a9dc thru 0x03b6ba (0xcdf bytes) 000467 1721880121.555019 fallocate 0x000028 thru 0x00e676 (0xe64f bytes) 000468 1721880121.556777 mapwrite 0x039d4b thru 0x03ffff (0x62b5 bytes) 000469 1721880121.564498 MIRROR_EXTEND to 10 mirrors 000470 1721880121.704275 mapread 0x03c91d thru 0x03ffff (0x36e3 bytes) 000471 1721880121.714660 fallocate 0x004ed0 thru 0x00660a (0x173b bytes) 000472 1721880121.725813 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000473 1721880121.778341 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000474 1721880121.834337 mapread 0x025fe4 thru 0x028898 (0x28b5 bytes) 000475 1721880121.840786 fallocate 0x02dc62 thru 0x039312 (0xb6b1 bytes) 000477 1721880121.843391 punch from 0x0398bc to 0x03ffff (0x6744 bytes) 000478 1721880121.845236 read 0x021180 thru 0x0248fe (0x377f bytes) 000479 1721880121.845696 read 0x02e3ab thru 0x037e9d (0x9af3 bytes) 000480 1721880121.849480 write 0x021f8a thru 0x030fc0 (0xf037 bytes) 000481 1721880121.851860 mapwrite 0x009cc8 thru 0x016ce7 (0xd020 bytes) 000482 1721880121.860530 mapwrite 0x01427c thru 0x01d64f (0x93d4 bytes) 000483 1721880121.868020 mapwrite 0x01514c thru 0x01e6b0 (0x9565 bytes) 000484 1721880121.875934 mapwrite 0x02a6a8 thru 0x02aa29 (0x382 bytes) 000485 1721880121.881430 punch from 0x0399bd to 0x03eccf (0x5313 bytes) 000486 1721880121.883819 write 0x02ceca thru 0x031c72 (0x4da9 bytes) 000488 1721880121.884878 punch from 0x0224a1 to 0x02726d (0x4dcd bytes) 000489 1721880121.887881 mapread 0x03aa3f thru 0x03f23d (0x47ff bytes) 000490 1721880121.898343 trunc from 0x040000 to 0x00ec6e (0x31391 bytes) 000491 1721880121.908729 trunc from 0x00ec6f to 0x03d5fa (0x2e98c bytes) 000493 1721880121.913586 write 0x0193df thru 0x01a739 (0x135b bytes) 000494 1721880121.917040 punch from 0x01627b to 0x01abf6 (0x497c bytes) 000495 1721880121.921270 write 0x008f9e thru 0x00965f (0x6c2 bytes) 000496 1721880121.921931 punch from 0x03bf63 to 0x03d5fa (0x1698 bytes) 000501 1721880121.927359 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000502 1721880121.991944 mapread 0x02ffb1 thru 0x0363de (0x642e bytes) 000503 1721880121.998720 mapread 0x02d13d thru 0x030aec (0x39b0 bytes) 000504 1721880121.999249 mapread 0x036277 thru 0x037343 (0x10cd bytes) 000505 1721880121.999748 mapwrite 0x02952b thru 0x038470 (0xef46 bytes) 000507 1721880122.011175 mapread 0x00e7d0 thru 0x0171e6 (0x8a17 bytes) 000508 1721880122.017372 write 0x012fbf thru 0x02069f (0xd6e1 bytes) 000509 1721880122.021129 mapread 0x01820b thru 0x020882 (0x8678 bytes) 000510 1721880122.021813 mapwrite 0x011c0b thru 0x01fd93 (0xe189 bytes) 000511 1721880122.028918 trunc from 0x03d5fb to 0x028c07 (0x149f3 bytes) 000512 1721880122.036152 MIRROR_EXTEND to 8 mirrors 000514 1721880122.216391 MIRROR_EXTEND to 9 mirrors 000516 1721880122.387142 write 0x00528b thru 0x005f6a (0xce0 bytes) 000517 1721880122.409816 mapwrite 0x000bc4 thru 0x00cdfb (0xc238 bytes) 000518 1721880122.432250 write 0x026e5c thru 0x032d7f (0xbf24 bytes) 000519 1721880122.438779 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000520 1721880122.529665 MIRROR_RESYNC 8 mirrors 000522 1721880122.988223 MIRROR_EXTEND to 9 mirrors 000523 1721880123.149377 write 0x033f8d thru 0x036fc1 (0x3035 bytes) 000524 1721880123.163764 trunc from 0x036fc2 to 0x008c45 (0x2e37c bytes) 000525 1721880123.170397 write 0x03840f thru 0x03a454 (0x2046 bytes) 000526 1721880123.171177 mapread 0x02e8a3 thru 0x0374cf (0x8c2d bytes) 000527 1721880123.175815 fallocate 0x00a5f1 thru 0x011e44 (0x7854 bytes) 000528 1721880123.177286 mapread 0x0305a3 thru 0x0330ae (0x2b0c bytes) 000529 1721880123.177799 punch from 0x02c594 to 0x0342dd (0x7d4a bytes) 000530 1721880123.179651 write 0x022a78 thru 0x02caf9 (0xa082 bytes) 000531 1721880123.185011 write 0x007367 thru 0x012417 (0xb0b1 bytes) 000532 1721880123.190308 fallocate 0x015afc thru 0x023b9b (0xe0a0 bytes) 000533 1721880123.192563 mapwrite 0x004557 thru 0x013eab (0xf955 bytes) 000534 1721880123.210932 MIRROR_RESYNC 9 mirrors 000535 1721880123.614321 mapread 0x02a106 thru 0x02a680 (0x57b bytes) 000536 1721880123.617209 punch from 0x01b8b3 to 0x021ce4 (0x6432 bytes) 000537 1721880123.631312 read 0x0285e8 thru 0x0328eb (0xa304 bytes) 000538 1721880123.636554 MIRROR_RESYNC 9 mirrors 000539 1721880124.115101 trunc from 0x03a455 to 0x018f28 (0x2152c bytes) 000541 1721880124.128766 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000542 1721880124.180220 mapwrite 0x038842 thru 0x03dce3 (0x54a2 bytes) 000543 1721880124.196660 mapwrite 0x01f695 thru 0x024258 (0x4bc4 bytes) 000544 1721880124.203839 trunc from 0x03dce4 to 0x0293d2 (0x14911 bytes) 000546 1721880124.208324 mapwrite 0x038c25 thru 0x03ffff (0x73db bytes) 000547 1721880124.222206 MIRROR_RESYNC 8 mirrors 000548 1721880124.584437 punch from 0x036413 to 0x0380b4 (0x1ca2 bytes) 000549 1721880124.596653 read 0x02fe09 thru 0x0308e5 (0xadd bytes) 000550 1721880124.599069 punch from 0x03bdee to 0x03ffff (0x4212 bytes) 000552 1721880124.601052 fallocate 0x01d549 thru 0x027eab (0xa963 bytes) 000553 1721880124.602981 read 0x0108da thru 0x011183 (0x8aa bytes) 000556 1721880124.607324 write 0x01b41a thru 0x01ffbf (0x4ba6 bytes) 000557 1721880124.609318 write 0x01d42d thru 0x01d9b5 (0x589 bytes) 000558 1721880124.609700 mapread 0x006312 thru 0x00a79a (0x4489 bytes) 000559 1721880124.610178 mapwrite 0x030aa8 thru 0x033927 (0x2e80 bytes) 000561 1721880124.617261 trunc from 0x040000 to 0x0114b8 (0x2eb47 bytes) 000564 1721880124.627570 mapwrite 0x03d75c thru 0x03ffff (0x28a4 bytes) 000566 1721880124.640662 MIRROR_EXTEND to 9 mirrors 000568 1721880124.757388 punch from 0x019442 to 0x0248d4 (0xb493 bytes) 000569 1721880124.770393 fallocate 0x02c55a thru 0x02d1e1 (0xc88 bytes) 000571 1721880124.772473 mapread 0x01a7eb thru 0x0281c8 (0xd9de bytes) 000573 1721880124.782275 MIRROR_RESYNC 9 mirrors 000574 1721880124.990303 MIRROR_RESYNC 9 mirrors 000575 1721880125.063221 trunc from 0x040000 to 0x026881 (0x1977e bytes) 000576 1721880125.079582 fallocate 0x0100f5 thru 0x01db0a (0xda16 bytes) 000577 1721880125.081468 trunc from 0x026882 to 0x00c978 (0x19f09 bytes) 000578 1721880125.088476 MIRROR_RESYNC 9 mirrors 000579 1721880125.283408 fallocate 0x01f7fb thru 0x02311b (0x3921 bytes) 000580 1721880125.294346 punch from 0x007aaa to 0x011fa0 (0xa4f7 bytes) 000581 1721880125.295999 write 0x000d19 thru 0x00b8f2 (0xabda bytes) 000582 1721880125.299472 mapread 0x01ff34 thru 0x02311b (0x31e8 bytes) 000583 1721880125.305328 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000584 1721880125.366801 read 0x015c29 thru 0x0166ef (0xac7 bytes) 000585 1721880125.369233 mapread 0x019e2a thru 0x020d2e (0x6f05 bytes) 000588 1721880125.377365 fallocate 0x004762 thru 0x011a9a (0xd339 bytes) 000591 1721880125.379753 fallocate 0x00f33b thru 0x0154e8 (0x61ae bytes) 000592 1721880125.381419 read 0x012f37 thru 0x0225df (0xf6a9 bytes) 000593 1721880125.385119 punch from 0x00418f to 0x007b0f (0x3981 bytes) 000594 1721880125.387026 trunc from 0x02311c to 0x031398 (0xe27d bytes) 000595 1721880125.392124 punch from 0x00d88b to 0x013ee3 (0x6659 bytes) 000596 1721880125.393691 fallocate 0x02552b thru 0x02eb2c (0x9602 bytes) 000598 1721880125.396284 read 0x022aaf thru 0x02e268 (0xb7ba bytes) 000599 1721880125.399574 punch from 0x00cf47 to 0x010a96 (0x3b50 bytes) 000600 1721880125.401147 fallocate 0x03e074 thru 0x03ffff (0x1f8c bytes) 000601 1721880125.403783 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000603 1721880125.457771 read 0x028e88 thru 0x03639e (0xd517 bytes) 000604 1721880125.460872 mapwrite 0x00933a thru 0x017f2c (0xebf3 bytes) 000606 1721880125.476057 fallocate 0x0086a5 thru 0x014a37 (0xc393 bytes) 000608 1721880125.478109 trunc from 0x040000 to 0x02062e (0x1f9d1 bytes) 000609 1721880125.483823 mapwrite 0x030960 thru 0x03cec3 (0xc564 bytes) 000610 1721880125.499480 mapread 0x03c093 thru 0x03cec3 (0xe31 bytes) 000612 1721880125.500382 read 0x03a180 thru 0x03cec3 (0x2d44 bytes) 000613 1721880125.500734 read 0x03b70f thru 0x03cec3 (0x17b5 bytes) 000615 1721880125.501354 fallocate 0x03861a thru 0x03ffff (0x79e6 bytes) 000616 1721880125.503117 trunc from 0x03cec4 to 0x028935 (0x1458e bytes) 000617 1721880125.506838 mapread 0x01bcf4 thru 0x026a4a (0xad57 bytes) 000619 1721880125.509766 write 0x00f4d3 thru 0x011e72 (0x29a0 bytes) 000620 1721880125.510380 trunc from 0x028936 to 0x03e911 (0x15fdc bytes) 000621 1721880125.516468 MIRROR_RESYNC 7 mirrors 000622 1721880125.830616 mapread 0x010ed5 thru 0x0120b0 (0x11dc bytes) 000623 1721880125.835496 mapwrite 0x0264ba thru 0x02bd7f (0x58c6 bytes) 000624 1721880125.871431 fallocate 0x03ef3d thru 0x03ffff (0x10c3 bytes) 000625 1721880125.874669 MIRROR_RESYNC 7 mirrors 000626 1721880126.241061 trunc from 0x03e912 to 0x037693 (0x727e bytes) 000627 1721880126.250537 mapwrite 0x003973 thru 0x0130ab (0xf739 bytes) 000628 1721880126.266384 punch from 0x00e1a1 to 0x017d71 (0x9bd1 bytes) 000629 1721880126.268319 trunc from 0x037694 to 0x00d5c9 (0x2a0ca bytes) 000630 1721880126.273808 mapwrite 0x024007 thru 0x026d1b (0x2d15 bytes) 000631 1721880126.284083 fallocate 0x030a06 thru 0x0388ab (0x7ea6 bytes) 000632 1721880126.286775 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000633 1721880126.333377 mapwrite 0x032c17 thru 0x03ffff (0xd3e9 bytes) 000634 1721880126.349179 trunc from 0x040000 to 0x028701 (0x178fe bytes) 000635 1721880126.352729 punch from 0x01ffcc to 0x028701 (0x8736 bytes) 000636 1721880126.354094 punch from 0x0118ea to 0x01ce5f (0xb576 bytes) 000637 1721880126.355325 trunc from 0x028702 to 0x0075dc (0x21125 bytes) 000638 1721880126.360015 mapread 0x000454 thru 0x0075dc (0x7189 bytes) 000639 1721880126.364793 trunc from 0x0075dd to 0x0003e5 (0x71f7 bytes) 000640 1721880126.370752 write 0x032cbb thru 0x03c65d (0x99a3 bytes) 000642 1721880126.372112 write 0x00b879 thru 0x015d21 (0xa4a9 bytes) 000643 1721880126.377356 trunc from 0x03c65e to 0x02cc9a (0xf9c3 bytes) 000644 1721880126.382147 trunc from 0x02cc9b to 0x02518f (0x7b0b bytes) 000645 1721880126.386305 read 0x022607 thru 0x02518f (0x2b89 bytes) 000646 1721880126.388641 fallocate 0x00d060 thru 0x0140d3 (0x7074 bytes) 000647 1721880126.390496 mapread 0x01c37d thru 0x021a93 (0x5717 bytes) 000648 1721880126.393382 read 0x01a675 thru 0x01b862 (0x11ee bytes) 000649 1721880126.395578 MIRROR_RESYNC 6 mirrors 000650 1721880126.639558 fallocate 0x030088 thru 0x039943 (0x98bc bytes) 000651 1721880126.651295 mapread 0x0211f2 thru 0x02518f (0x3f9e bytes) 000653 1721880126.657218 mapwrite 0x01b765 thru 0x027cd6 (0xc572 bytes) 000654 1721880126.685480 mapread 0x00e893 thru 0x0142f0 (0x5a5e bytes) 000655 1721880126.688914 read 0x02006c thru 0x0230a4 (0x3039 bytes) 000656 1721880126.689271 trunc from 0x027cd7 to 0x025175 (0x2b61 bytes) 000657 1721880126.694518 read 0x00af41 thru 0x0194d1 (0xe591 bytes) 000658 1721880126.697653 fallocate 0x028fd3 thru 0x030281 (0x72af bytes) 000660 1721880126.701970 MIRROR_EXTEND to 7 mirrors 000661 1721880126.842985 write 0x036517 thru 0x03c9f8 (0x64e2 bytes) 000662 1721880126.855030 punch from 0x0342c3 to 0x03be45 (0x7b83 bytes) 000663 1721880126.861085 MIRROR_RESYNC 7 mirrors 000664 1721880127.249911 fallocate 0x0148b4 thru 0x01db0a (0x9257 bytes) 000665 1721880127.262293 mapread 0x010cba thru 0x018fa5 (0x82ec bytes) 000666 1721880127.269343 write 0x010c88 thru 0x01d00b (0xc384 bytes) 000667 1721880127.270226 mapwrite 0x018c07 thru 0x01b8cf (0x2cc9 bytes) 000669 1721880127.279543 mapread 0x03798a thru 0x03944b (0x1ac2 bytes) 000670 1721880127.283365 read 0x02a0ac thru 0x039bc1 (0xfb16 bytes) 000671 1721880127.287062 mapread 0x0061d5 thru 0x00c5f5 (0x6421 bytes) 000672 1721880127.291897 write 0x02a906 thru 0x02ca6d (0x2168 bytes) 000673 1721880127.292493 trunc from 0x03c9f9 to 0x03abd4 (0x1e24 bytes) 000674 1721880127.303056 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000675 1721880127.389726 write 0x03db83 thru 0x03ffff (0x247d bytes) 000677 1721880127.392231 read 0x00da3c thru 0x00f7f2 (0x1db7 bytes) 000678 1721880127.394552 mapwrite 0x0324b7 thru 0x03ffff (0xdb49 bytes) 000679 1721880127.408236 trunc from 0x040000 to 0x00dd1f (0x322e0 bytes) 000680 1721880127.414372 mapwrite 0x019b5c thru 0x0299d4 (0xfe79 bytes) 000681 1721880127.436369 fallocate 0x017955 thru 0x01c2b6 (0x4962 bytes) 000682 1721880127.438811 mapwrite 0x01c2e9 thru 0x01e122 (0x1e3a bytes) 000683 1721880127.446178 read 0x01bfca thru 0x0224b8 (0x64ef bytes) 000684 1721880127.446552 read 0x00327a thru 0x00c222 (0x8fa9 bytes) 000685 1721880127.450337 punch from 0x00882d to 0x01573a (0xcf0e bytes) 000686 1721880127.453179 punch from 0x009c48 to 0x01117e (0x7537 bytes) 000687 1721880127.455491 read 0x002365 thru 0x0113d0 (0xf06c bytes) 000688 1721880127.461810 MIRROR_RESYNC 6 mirrors 000691 1721880127.723903 mapread 0x02357b thru 0x026d3b (0x37c1 bytes) 000692 1721880127.732339 read 0x021a36 thru 0x028340 (0x690b bytes) 000693 1721880127.732666 mapread 0x01ed0a thru 0x0299d4 (0xaccb bytes) 000694 1721880127.733087 trunc from 0x0299d5 to 0x01a4e5 (0xf4ef bytes) 000695 1721880127.754163 write 0x028901 thru 0x037ee5 (0xf5e5 bytes) 000696 1721880127.755450 read 0x028500 thru 0x031d21 (0x9822 bytes) 000697 1721880127.755891 mapread 0x013e81 thru 0x01c192 (0x8312 bytes) 000698 1721880127.760757 mapread 0x023c8f thru 0x024cf5 (0x1067 bytes) 000699 1721880127.761455 fallocate 0x01307b thru 0x01f7ec (0xc772 bytes) 000700 1721880127.764674 read 0x000149 thru 0x006437 (0x62ef bytes) 000702 1721880127.765485 fallocate 0x021eb3 thru 0x024e69 (0x2fb7 bytes) 000705 1721880127.769408 write 0x012047 thru 0x01e00a (0xbfc4 bytes) 000706 1721880127.770375 mapread 0x02f787 thru 0x03096d (0x11e7 bytes) 000708 1721880127.771298 read 0x01e1c4 thru 0x02b86d (0xd6aa bytes) 000709 1721880127.771785 fallocate 0x00ae7d thru 0x01139a (0x651e bytes) 000710 1721880127.775141 mapwrite 0x0069dd thru 0x015633 (0xec57 bytes) 000711 1721880127.790519 trunc from 0x037ee6 to 0x01f481 (0x18a64 bytes) 000712 1721880127.797706 write 0x036d0d thru 0x03ffff (0x92f3 bytes) 000713 1721880127.799032 read 0x03d16a thru 0x03ffff (0x2e96 bytes) 000714 1721880127.799424 read 0x0380de thru 0x03ffff (0x7f22 bytes) 000715 1721880127.799861 read 0x03caf4 thru 0x03ffff (0x350c bytes) 000716 1721880127.801953 MIRROR_EXTEND to 7 mirrors 000717 1721880128.000063 trunc from 0x040000 to 0x0119ef (0x2e610 bytes) 000719 1721880128.030119 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000720 1721880128.095766 fallocate 0x01a06d thru 0x022ba2 (0x8b36 bytes) 000721 1721880128.098554 read 0x022a7b thru 0x022ba2 (0x128 bytes) 000722 1721880128.100943 punch from 0x010842 to 0x019854 (0x9013 bytes) 000723 1721880128.103868 trunc from 0x022ba3 to 0x01b50e (0x7694 bytes) 000724 1721880128.112930 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000725 1721880128.173991 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000726 1721880128.238486 read 0x003f1b thru 0x0075f8 (0x36de bytes) 000727 1721880128.241460 punch from 0x005834 to 0x0058dc (0xa9 bytes) 000728 1721880128.243509 read 0x012713 thru 0x015132 (0x2a20 bytes) 000730 1721880128.247610 punch from 0x018d22 to 0x01b50e (0x27ed bytes) 000731 1721880128.250199 mapread 0x001b3f thru 0x010873 (0xed35 bytes) 000732 1721880128.250811 read 0x009ed8 thru 0x019292 (0xf3bb bytes) 000733 1721880128.252926 read 0x003c87 thru 0x0055ea (0x1964 bytes) 000734 1721880128.253290 mapread 0x0020b7 thru 0x00a5d3 (0x851d bytes) 000735 1721880128.253796 read 0x009a6c thru 0x014803 (0xad98 bytes) 000737 1721880128.254298 write 0x0080d0 thru 0x008c7c (0xbad bytes) 000738 1721880128.256461 trunc from 0x01b50f to 0x004335 (0x171d9 bytes) 000739 1721880128.261894 write 0x03dfda thru 0x03ffff (0x2026 bytes) 000740 1721880128.262560 write 0x034eca thru 0x03ffff (0xb136 bytes) 000741 1721880128.266943 write 0x027d83 thru 0x02d3ed (0x566b bytes) 000742 1721880128.267752 write 0x038ba9 thru 0x03ffff (0x7457 bytes) 000745 1721880128.269269 write 0x00ee1f thru 0x01b462 (0xc644 bytes) 000746 1721880128.270225 mapread 0x015f49 thru 0x0247e4 (0xe89c bytes) 000747 1721880128.270819 write 0x0071ae thru 0x00e2e0 (0x7133 bytes) 000748 1721880128.271418 fallocate 0x019de5 thru 0x021e9c (0x80b8 bytes) 000749 1721880128.273363 trunc from 0x040000 to 0x0355c2 (0xaa3d bytes) 000750 1721880128.277300 write 0x00a2f5 thru 0x00ddb2 (0x3abe bytes) 000751 1721880128.277938 fallocate 0x0083f7 thru 0x011e81 (0x9a8b bytes) 000752 1721880128.279527 fallocate 0x019ed9 thru 0x02299e (0x8ac6 bytes) 000753 1721880128.281163 fallocate 0x031c1b thru 0x034006 (0x23ec bytes) 000754 1721880128.283974 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000756 1721880128.333351 fallocate 0x02c9c8 thru 0x038053 (0xb68c bytes) 000757 1721880128.335126 mapwrite 0x028b3b thru 0x02d045 (0x450b bytes) 000758 1721880128.345529 read 0x030cde thru 0x034515 (0x3838 bytes) 000759 1721880128.347468 mapwrite 0x03bf8c thru 0x03d1cc (0x1241 bytes) 000760 1721880128.360048 trunc from 0x03d1cd to 0x016c42 (0x2658a bytes) 000761 1721880128.364541 mapread 0x00c6b4 thru 0x00f746 (0x3093 bytes) 000762 1721880128.368271 trunc from 0x016c43 to 0x03e22f (0x275ed bytes) 000763 1721880128.373025 mapwrite 0x0108be thru 0x020794 (0xfed7 bytes) 000764 1721880128.388251 mapwrite 0x0212cb thru 0x030dc7 (0xfafd bytes) 000765 1721880128.398842 punch from 0x0014aa to 0x008520 (0x7077 bytes) 000766 1721880128.402982 MIRROR_EXTEND to 4 mirrors 000767 1721880128.560834 MIRROR_RESYNC 4 mirrors 000768 1721880128.756083 punch from 0x03d059 to 0x03e22f (0x11d7 bytes) 000770 1721880128.771809 punch from 0x00779d to 0x0125a7 (0xae0b bytes) 000771 1721880128.774946 read 0x0221c7 thru 0x02f8a7 (0xd6e1 bytes) 000772 1721880128.780644 punch from 0x03427b to 0x03e22f (0x9fb5 bytes) 000773 1721880128.782937 punch from 0x016e6e to 0x023d0f (0xcea2 bytes) 000775 1721880128.785820 write 0x036a10 thru 0x03ffff (0x95f0 bytes) 000776 1721880128.791997 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000777 1721880128.859803 write 0x00a6ae thru 0x01a167 (0xfaba bytes) 000778 1721880128.865204 punch from 0x03547d to 0x03cc5f (0x77e3 bytes) 000779 1721880128.868519 MIRROR_EXTEND to 4 mirrors 000780 1721880129.038475 read 0x03eeed thru 0x03ffff (0x1113 bytes) 000781 1721880129.043839 trunc from 0x040000 to 0x02770f (0x188f0 bytes) 000782 1721880129.064191 fallocate 0x00501e thru 0x009469 (0x444c bytes) 000783 1721880129.066467 fallocate 0x028b35 thru 0x034f5a (0xc426 bytes) 000784 1721880129.068457 mapread 0x020615 thru 0x02a12d (0x9b19 bytes) 000786 1721880129.079943 MIRROR_EXTEND to 5 mirrors 000787 1721880129.192256 write 0x02d616 thru 0x0317c4 (0x41af bytes) 000789 1721880129.205335 mapread 0x019228 thru 0x01dacd (0x48a6 bytes) 000791 1721880129.208576 fallocate 0x02030c thru 0x026426 (0x611b bytes) 000792 1721880129.210288 read 0x00101c thru 0x0069c0 (0x59a5 bytes) 000793 1721880129.212702 mapwrite 0x033566 thru 0x038662 (0x50fd bytes) 000794 1721880129.231249 fallocate 0x00e979 thru 0x01311b (0x47a3 bytes) 000795 1721880129.232974 trunc from 0x038663 to 0x0170f1 (0x21571 bytes) 000796 1721880129.237146 fallocate 0x0123bc thru 0x013798 (0x13dd bytes) 000797 1721880129.238924 trunc from 0x0170f2 to 0x0087fd (0xe8f4 bytes) 000798 1721880129.243597 punch from 0x0000f4 to 0x0087fd (0x870a bytes) 000799 1721880129.245845 punch from 0x007b0a to 0x007e53 (0x34a bytes) 000800 1721880129.248856 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000801 1721880129.303788 mapread 0x000cc1 thru 0x0087fd (0x7b3d bytes) 000802 1721880129.310531 read 0x00346e thru 0x0087fd (0x5390 bytes) 000803 1721880129.310920 trunc from 0x0087fe to 0x01c591 (0x13d94 bytes) 000805 1721880129.320593 read 0x01bdcf thru 0x01c591 (0x7c3 bytes) 000806 1721880129.323994 fallocate 0x006904 thru 0x00f32c (0x8a29 bytes) 000808 1721880129.326524 mapwrite 0x00d0ac thru 0x00fd43 (0x2c98 bytes) 000809 1721880129.334036 mapread 0x0131a2 thru 0x01c591 (0x93f0 bytes) 000810 1721880129.334697 punch from 0x000464 to 0x00709d (0x6c3a bytes) 000811 1721880129.336958 trunc from 0x01c592 to 0x000d58 (0x1b839 bytes) 000812 1721880129.341765 trunc from 0x000d59 to 0x01b44b (0x1a6f3 bytes) 000814 1721880129.345825 fallocate 0x03fdc8 thru 0x03ffff (0x238 bytes) 000815 1721880129.347986 mapwrite 0x03fbbd thru 0x03ffff (0x443 bytes) 000816 1721880129.360117 read 0x02abde thru 0x03aab8 (0xfedb bytes) 000818 1721880129.361124 fallocate 0x03b2ec thru 0x03eb4c (0x3861 bytes) 000819 1721880129.363897 mapread 0x02bd6b thru 0x02ffa6 (0x423c bytes) 000820 1721880129.364453 read 0x011c1b thru 0x01ead7 (0xcebd bytes) 000821 1721880129.368283 fallocate 0x016707 thru 0x01c15d (0x5a57 bytes) 000823 1721880129.370475 trunc from 0x040000 to 0x00e1f1 (0x31e0e bytes) 000824 1721880129.377722 MIRROR_EXTEND to 5 mirrors 000825 1721880129.502703 read 0x008952 thru 0x00e1f1 (0x58a0 bytes) 000826 1721880129.505073 read 0x0082dd thru 0x00e1f1 (0x5f15 bytes) 000828 1721880129.505533 fallocate 0x0263e1 thru 0x030853 (0xa473 bytes) 000829 1721880129.516429 fallocate 0x01b22d thru 0x021d16 (0x6aea bytes) 000830 1721880129.517985 punch from 0x00b6e8 to 0x00e1f1 (0x2b0a bytes) 000831 1721880129.519698 mapwrite 0x00f31f thru 0x0162a5 (0x6f87 bytes) 000835 1721880129.544177 punch from 0x00cc83 to 0x0162a5 (0x9623 bytes) 000836 1721880129.547979 MIRROR_EXTEND to 6 mirrors 000837 1721880129.662818 write 0x036754 thru 0x0387cc (0x2079 bytes) 000838 1721880129.674229 MIRROR_RESYNC 6 mirrors 000840 1721880129.918198 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000841 1721880129.982903 read 0x00cf4f thru 0x0194d0 (0xc582 bytes) 000842 1721880129.987534 MIRROR_RESYNC 5 mirrors 000843 1721880130.039955 mapread 0x021988 thru 0x02aa8f (0x9108 bytes) 000846 1721880130.045923 mapread 0x006ed9 thru 0x013556 (0xc67e bytes) 000848 1721880130.049308 write 0x03d659 thru 0x03ffff (0x29a7 bytes) 000849 1721880130.060123 fallocate 0x027f71 thru 0x032bfd (0xac8d bytes) 000852 1721880130.063007 mapwrite 0x038f57 thru 0x03ffff (0x70a9 bytes) 000853 1721880130.077108 punch from 0x007075 to 0x01093a (0x98c6 bytes) 000855 1721880130.080026 punch from 0x03df4a to 0x03ffff (0x20b6 bytes) 000856 1721880130.082316 mapread 0x037803 thru 0x03ffff (0x87fd bytes) 000857 1721880130.084974 fallocate 0x010cb2 thru 0x01313b (0x248a bytes) 000858 1721880130.087032 read 0x00870e thru 0x0135df (0xaed2 bytes) 000859 1721880130.090180 write 0x03dd5c thru 0x03ffff (0x22a4 bytes) 000860 1721880130.090888 punch from 0x02bb84 to 0x02e827 (0x2ca4 bytes) 000863 1721880130.092949 punch from 0x00ad20 to 0x01186b (0x6b4c bytes) 000865 1721880130.094846 trunc from 0x040000 to 0x017f6a (0x28095 bytes) 000866 1721880130.108786 trunc from 0x017f6b to 0x017e8c (0xde bytes) 000867 1721880130.113576 mapread 0x00984f thru 0x00c26d (0x2a1f bytes) 000868 1721880130.117592 mapread 0x00413d thru 0x010dbb (0xcc7f bytes) 000870 1721880130.121098 read 0x003b14 thru 0x00432c (0x819 bytes) 000871 1721880130.121516 mapwrite 0x021a9d thru 0x029db3 (0x8317 bytes) 000872 1721880130.138205 mapwrite 0x011904 thru 0x01b600 (0x9cfd bytes) 000873 1721880130.147057 punch from 0x01117a to 0x017a9b (0x6922 bytes) 000874 1721880130.148999 mapwrite 0x039aa6 thru 0x03ffff (0x655a bytes) 000875 1721880130.164852 mapwrite 0x010da5 thru 0x017195 (0x63f1 bytes) 000876 1721880130.180063 mapwrite 0x02fa63 thru 0x03c16b (0xc709 bytes) 000877 1721880130.187741 fallocate 0x021c0e thru 0x02a8e3 (0x8cd6 bytes) 000878 1721880130.190096 read 0x02d88c thru 0x03168a (0x3dff bytes) 000879 1721880130.190361 mapread 0x01bfe8 thru 0x0245a9 (0x85c2 bytes) 000880 1721880130.191015 trunc from 0x040000 to 0x01ad7d (0x25282 bytes) 000881 1721880130.196887 trunc from 0x01ad7e to 0x01eb39 (0x3dbc bytes) 000883 1721880130.202422 trunc from 0x01eb3a to 0x0160c3 (0x8a76 bytes) 000884 1721880130.207404 read 0x014677 thru 0x0160c3 (0x1a4d bytes) 000885 1721880130.207832 fallocate 0x03a495 thru 0x03ffff (0x5b6b bytes) 000886 1721880130.210569 read 0x026624 thru 0x029ab7 (0x3494 bytes) 000888 1721880130.214805 punch from 0x01b0ce to 0x023718 (0x864b bytes) 000890 1721880130.216947 read 0x01f148 thru 0x02de25 (0xecde bytes) 000891 1721880130.221174 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000892 1721880130.272100 mapread 0x0315d0 thru 0x035f4f (0x4980 bytes) 000893 1721880130.277488 trunc from 0x040000 to 0x016475 (0x29b8a bytes) 000894 1721880130.283796 write 0x03ab45 thru 0x03d8ca (0x2d86 bytes) 000895 1721880130.284709 mapread 0x020b04 thru 0x026b3c (0x6039 bytes) 000896 1721880130.288362 write 0x03b9cb thru 0x03ffff (0x4635 bytes) 000897 1721880130.289076 trunc from 0x040000 to 0x033e10 (0xc1ef bytes) 000898 1721880130.294153 read 0x01a681 thru 0x02848f (0xde0f bytes) 000899 1721880130.297874 trunc from 0x033e11 to 0x02b716 (0x86fa bytes) 000901 1721880130.302987 trunc from 0x02b717 to 0x033caf (0x8599 bytes) 000902 1721880130.307173 mapwrite 0x00a72d thru 0x00b45d (0xd31 bytes) 000904 1721880130.315931 write 0x00d597 thru 0x00e0e8 (0xb52 bytes) 000905 1721880130.316437 trunc from 0x033cb0 to 0x030cba (0x2ff5 bytes) 000906 1721880130.320595 fallocate 0x034f40 thru 0x03633c (0x13fd bytes) 000907 1721880130.323581 MIRROR_EXTEND to 5 mirrors 000908 1721880130.464332 mapwrite 0x002b09 thru 0x00f67c (0xcb74 bytes) 000909 1721880130.498832 read 0x00c9ba thru 0x00f2ff (0x2946 bytes) 000910 1721880130.499241 fallocate 0x0102f1 thru 0x013288 (0x2f98 bytes) 000911 1721880130.501255 mapwrite 0x00891f thru 0x00c95c (0x403e bytes) 000912 1721880130.507615 read 0x020057 thru 0x028703 (0x86ad bytes) 000913 1721880130.513463 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000915 1721880130.564169 mapread 0x00036b thru 0x009caf (0x9945 bytes) 000916 1721880130.572534 read 0x03512e thru 0x03633c (0x120f bytes) 000917 1721880130.574798 read 0x00b4df thru 0x01b1c8 (0xfcea bytes) 000919 1721880130.578708 trunc from 0x03633d to 0x0029de (0x3395e bytes) 000920 1721880130.590295 read 0x001878 thru 0x0029de (0x1167 bytes) 000922 1721880130.590890 mapwrite 0x01d689 thru 0x0257bf (0x8137 bytes) 000923 1721880130.606464 punch from 0x008630 to 0x0120ac (0x9a7d bytes) 000924 1721880130.608715 trunc from 0x0257c0 to 0x008130 (0x1d68f bytes) 000925 1721880130.614348 write 0x034581 thru 0x03ffff (0xba7f bytes) 000926 1721880130.615460 read 0x01e423 thru 0x024ae9 (0x66c7 bytes) 000927 1721880130.618688 trunc from 0x040000 to 0x009ba1 (0x3645e bytes) 000928 1721880130.624313 read 0x0011be thru 0x005926 (0x4769 bytes) 000930 1721880130.625111 mapwrite 0x029471 thru 0x037752 (0xe2e2 bytes) 000931 1721880130.645345 MIRROR_EXTEND to 5 mirrors 000933 1721880130.761579 write 0x02fc5c thru 0x03e869 (0xec0e bytes) 000934 1721880130.775830 MIRROR_EXTEND to 6 mirrors 000936 1721880130.883550 read 0x0231da thru 0x0312cd (0xe0f4 bytes) 000938 1721880130.891083 fallocate 0x01819d thru 0x01b6f0 (0x3554 bytes) 000940 1721880130.901720 punch from 0x001e51 to 0x00e559 (0xc709 bytes) 000941 1721880130.903309 punch from 0x02f406 to 0x03af73 (0xbb6e bytes) 000942 1721880130.905204 mapread 0x00f488 thru 0x015501 (0x607a bytes) 000943 1721880130.910701 punch from 0x033b80 to 0x03e7b1 (0xac32 bytes) 000944 1721880130.912090 write 0x01a408 thru 0x029c80 (0xf879 bytes) 000945 1721880130.915488 punch from 0x02fe70 to 0x034d17 (0x4ea8 bytes) 000946 1721880130.917123 read 0x010337 thru 0x01bdc5 (0xba8f bytes) 000947 1721880130.917479 mapread 0x007c11 thru 0x0176e8 (0xfad8 bytes) 000948 1721880130.920708 punch from 0x0080ac to 0x0083b5 (0x30a bytes) 000950 1721880130.923297 MIRROR_EXTEND to 7 mirrors 000952 1721880131.048531 fallocate 0x00fee5 thru 0x01fc63 (0xfd7f bytes) 000953 1721880131.059601 punch from 0x013977 to 0x0193e3 (0x5a6d bytes) 000954 1721880131.061290 write 0x005e4c thru 0x01498d (0xeb42 bytes) 000955 1721880131.067377 MIRROR_EXTEND to 8 mirrors 000957 1721880131.210443 punch from 0x01c72e to 0x02a20d (0xdae0 bytes) 000958 1721880131.220300 read 0x039e9d thru 0x03e869 (0x49cd bytes) 000959 1721880131.222476 read 0x00e46e thru 0x016bce (0x8761 bytes) 000962 1721880131.228273 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000963 1721880131.280211 fallocate 0x026c8b thru 0x0314ee (0xa864 bytes) 000964 1721880131.281975 mapwrite 0x036461 thru 0x03ffff (0x9b9f bytes) 000965 1721880131.298326 mapread 0x03f1a7 thru 0x03ffff (0xe59 bytes) 000966 1721880131.298814 mapread 0x02727c thru 0x02d754 (0x64d9 bytes) 000967 1721880131.303419 MIRROR_EXTEND to 8 mirrors 000968 1721880131.473399 mapread 0x010c22 thru 0x011334 (0x713 bytes) 000969 1721880131.483329 read 0x019243 thru 0x01d7d4 (0x4592 bytes) 000970 1721880131.489173 write 0x0148e5 thru 0x0171e9 (0x2905 bytes) 000971 1721880131.505659 read 0x0326a3 thru 0x0372c1 (0x4c1f bytes) 000972 1721880131.509443 mapwrite 0x021105 thru 0x02a2b8 (0x91b4 bytes) 000975 1721880131.519457 punch from 0x020b31 to 0x02e784 (0xdc54 bytes) 000976 1721880131.521850 fallocate 0x001755 thru 0x00624d (0x4af9 bytes) 000977 1721880131.523524 write 0x00529d thru 0x009bf7 (0x495b bytes) 000978 1721880131.524267 read 0x00cbe2 thru 0x00dc0e (0x102d bytes) 000979 1721880131.524565 fallocate 0x02db09 thru 0x032f9c (0x5494 bytes) 000980 1721880131.526847 punch from 0x01eda0 to 0x027bea (0x8e4b bytes) 000981 1721880131.528873 fallocate 0x031683 thru 0x036d25 (0x56a3 bytes) 000982 1721880131.531019 write 0x021de1 thru 0x0259e4 (0x3c04 bytes) 000983 1721880131.534582 fallocate 0x02200c thru 0x025377 (0x336c bytes) 000984 1721880131.536792 write 0x009983 thru 0x00f23c (0x58ba bytes) 000985 1721880131.537547 punch from 0x0250f1 to 0x02c8bd (0x77cd bytes) 000986 1721880131.541672 mapread 0x037308 thru 0x038ffd (0x1cf6 bytes) 000987 1721880131.542221 mapread 0x016341 thru 0x01a3d9 (0x4099 bytes) 000988 1721880131.542668 mapwrite 0x03d621 thru 0x03ffff (0x29df bytes) 000989 1721880131.547483 mapread 0x035c0f thru 0x03cbb2 (0x6fa4 bytes) 000990 1721880131.548119 mapread 0x0110ea thru 0x01ef77 (0xde8e bytes) 000991 1721880131.548736 punch from 0x01ecc6 to 0x022831 (0x3b6c bytes) 000992 1721880131.551210 fallocate 0x0023d7 thru 0x0090fa (0x6d24 bytes) 000993 1721880131.552830 write 0x0099da thru 0x010e48 (0x746f bytes) 000994 1721880131.553517 trunc from 0x040000 to 0x027b79 (0x18486 bytes) 000995 1721880131.564419 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8bf:0x0]: Invalid argument (22) 000996 1721880131.614843 fallocate 0x020af3 thru 0x02d9ab (0xceb9 bytes) 000997 1721880131.616432 mapwrite 0x037e34 thru 0x03bd5a (0x3f27 bytes) 000998 1721880131.632240 mapread 0x012204 thru 0x01b456 (0x9253 bytes) 001000 1721880131.637216 punch from 0x0075ab to 0x016e5f (0xf8b5 bytes) All operations completed A-OK! PASS 70a (22s) == sanity-flr test 200a: stress test ===================== 00:02:12 (1721880132) Starting client: oleg443-client.virtnet: -o user_xattr,flock oleg443-server@tcp:/lustre /mnt/lustre2 Starting client: oleg443-client.virtnet: -o user_xattr,flock oleg443-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4455616 .. Extending file size to 1535968 .. failed Extending file size to 7361952 .. Extending file size to 8326880 .. Extending file size to 8504832 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8476736 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8686368 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8990112 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9362112 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8907616 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9261728 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 3657 3658 3659 3662 3663 192.168.204.143@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg443-client.virtnet /mnt/lustre2 (opts:) 192.168.204.143@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg443-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 00:03:16 (1721880196) Starting client: oleg443-client.virtnet: -o user_xattr,flock oleg443-server@tcp:/lustre /mnt/lustre2 Starting client: oleg443-client.virtnet: -o user_xattr,flock oleg443-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4775872 .. Extending file size to 1696416 .. failed Extending file size to 6659776 .. Extending file size to 6087712 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7519168 .. Extending file size to 6824928 .. Extending file size to 7455296 .. Extending file size to 7726848 .. Extending file size to 8013408 .. Extending file size to 7815648 .. Extending file size to 8620704 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8032192 .. Extending file size to 8054976 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8711808 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8373152 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8819008 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8934112 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8937376 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9272736 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8989536 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 6499 6500 6502 6503 6506 192.168.204.143@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg443-client.virtnet /mnt/lustre2 (opts:) 192.168.204.143@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg443-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 00:04:20 (1721880260) Starting client: oleg443-client.virtnet: -o user_xattr,flock oleg443-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2516 1285172 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3032 1284656 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 15824 3591196 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 21864 3585156 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37688 7176352 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2516 1285172 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 3032 1284656 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 15824 3591196 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 21864 3585156 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37688 7176352 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0370694 s, 56.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0509293 s, 41.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.143@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg443-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 00:04:25 (1721880265) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0313861 s, 66.8 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 00:04:27 (1721880267) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x38:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x9f:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x9f:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa0:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x81:0x0] } PASS 203 (1s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 00:04:28 (1721880268) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0250637 s, 83.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xb3d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xb3d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xb3e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 00:04:30 (1721880270) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0243979 s, 86.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xb3e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xb3f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xb3f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xb3e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xb3f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xb3f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xb40:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 00:04:32 (1721880272) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0272941 s, 76.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xb40:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xb41:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xb40:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xb41:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xb42:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 00:04:55 (1721880295) Waiting for MDT destroys to complete oleg443-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg443-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg443-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0151464 s, 69.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xb41:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xb44:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xb43:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xb41:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xb44:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xb43:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xb42:0x0] } PASS 204d (32s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 00:05:27 (1721880327) Waiting for MDT destroys to complete oleg443-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg443-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg443-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0270934 s, 77.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x82:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x82:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (31s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 00:05:58 (1721880358) Waiting for MDT destroys to complete oleg443-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg443-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg443-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool oleg443-server: Pool lustre.test_204f created Adding targets to pool oleg443-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg443-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0223431 s, 93.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x83:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x83:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg443-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg443-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg443-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (47s) == sanity-flr test 205: lfs mirror extend to set prefer and nocompr flags ========================================================== 00:06:45 (1721880405) PASS 205 (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 00:06:47 (1721880407) oleg443-server: Pool lustre.test_206 created Adding targets to pool oleg443-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg443-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg443-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg443-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg443-server: Pool lustre.test_206 destroyed PASS 206 (12s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 00:06:59 (1721880419) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.019375 s, 54.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00996255 s, 105 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xace8cc0a Mirror 3: 0xace8cc0a PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 00:07:01 (1721880421) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 00:07:02 (1721880422) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 00:07:03 (1721880423) Starting client: oleg443-client.virtnet: -o user_xattr,flock oleg443-server@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0100702 s, 0.4 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1c70000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1160000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1160000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1160000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2450000 Read this (4 bytes): BBBB 192.168.204.143@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg443-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 00:07:16 (1721880436) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00211544 s, 3.9 MB/s the very first read read with warm cache PASS 209b (1s) == sanity-flr test 210a: handle broken mirrored lovea ==== 00:07:17 (1721880437) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00993667 s, 106 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x54 lmm_fid: [0x200000405:0x54:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x54 lmm_fid: [0x200000405:0x54:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611220 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-25 00:07:17.000000000 -0400 Modify: 2024-07-25 00:07:17.000000000 -0400 Change: 2024-07-25 00:07:18.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 00:07:19 (1721880439) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00932056 s, 113 MB/s oleg443-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg443-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg443-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg443-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 oleg443-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg443-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg443-server: O/280000401/d14/ae: File not found by ext2_lookup PASS 210b (10s) == sanity-flr test complete, duration 694 sec ============ 00:07:29 (1721880449) === sanity-flr: start cleanup 00:07:29 (1721880449) === === sanity-flr: finish cleanup 00:07:30 (1721880450) ===