-----============= acceptance-small: sanity-flr ============----- Sun Feb 2 02:01:51 EST 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg158-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 02:01:56 (1738479716) === oleg158-client.virtnet: executing check_config_client /mnt/lustre oleg158-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg158-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012e181000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012e181000.idle_timeout=debug disable quota as required oleg158-server: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 02:02:04 (1738479724) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7608] == sanity-flr test 0a: lfs mirror create with -N option == 02:02:04 (1738479724) PASS 0a (4s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 02:02:08 (1738479728) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 02:02:09 (1738479729) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (0s) == sanity-flr test 0d: lfs mirror extend with -N option == 02:02:10 (1738479730) PASS 0d (4s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 02:02:14 (1738479734) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (0s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 02:02:14 (1738479734) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 02:02:15 (1738479735) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88012e181000.stats=clear osc.lustre-OST0001-osc-ffff88012e181000.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 02:02:17 (1738479737) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 02:02:19 (1738479739) PASS 0j (1s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 02:02:20 (1738479740) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 02:02:21 (1738479741) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 02:02:23 (1738479743) PASS 2 (1s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 02:02:24 (1738479744) PASS 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 02:02:26 (1738479746) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td lfs extend: cannot get the last init comp error: lfs extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 02:02:28 (1738479748) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (1s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 02:02:30 (1738479750) PASS 7 (1s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 02:02:32 (1738479752) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB) copied, 0.115296 s, 54.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.020827 s, 50.3 MB/s PASS 21 (1s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 02:02:33 (1738479753) 20+0 records in 20+0 records out 20971520 bytes (21 MB) copied, 0.148897 s, 141 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 02:02:35 (1738479755) Stopping /mnt/lustre-ost1 (opts:) on oleg158-server oleg158-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012e181000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012e181000.ost_server_uuid in DISCONN state after 2 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg158-server: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg158-server: oleg158-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg158-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg158-server oleg158-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012e181000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012e181000.ost_server_uuid in DISCONN state after 2 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg158-server: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg158-server: oleg158-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg158-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (25s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 02:03:00 (1738479780) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0546089 s, 76.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg158-server oleg158-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012e181000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012e181000.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg158-server: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg158-server: oleg158-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg158-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (10s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 02:03:10 (1738479790) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg158-server oleg158-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012e181000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012e181000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg158-server: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg158-server: oleg158-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg158-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg158-server oleg158-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012e181000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012e181000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg158-server: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg158-server: oleg158-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg158-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (16s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 02:03:26 (1738479806) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.78592 s, 66.7 MB/s -rw-r--r-- 1 root root 52428800 Feb 2 02:03 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.208s user 0m0.002s sys 0m0.114s Stopping /mnt/lustre-ost1 (opts:) on oleg158-server oleg158-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012e181000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012e181000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.205s user 0m0.001s sys 0m0.135s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg158-server: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg158-server: oleg158-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg158-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg158-server oleg158-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012e181000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012e181000.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost1)... real 0m0.243s user 0m0.000s sys 0m0.127s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg158-server: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg158-server: oleg158-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg158-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg158-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012e181000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012e181000.ost_server_uuid in FULL state after 0 sec PASS 33b (20s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 02:03:46 (1738479826) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 02:03:47 (1738479827) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 02:03:48 (1738479828) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 02:03:49 (1738479829) PASS 35 (1s) == sanity-flr test 36a: write to mirrored files ========== 02:03:50 (1738479830) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.112823 s, 112 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (2s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 02:03:53 (1738479833) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00836238 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00946925 s, 111 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 02:03:54 (1738479834) PASS 36c (2s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 02:03:56 (1738479836) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000525624 s, 1.9 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000607772 s, 1.7 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000433013 s, 2.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000559582 s, 1.8 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 02:03:58 (1738479838) create test files with size 15748157 .. fdeec6c8ceedc1a8ce82e2c61dbfdad1 - d863a05ebbb3794ec93ec2f14bdbba13 - 4b7b3898081a5f26c319233fe28cf282 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg158-client.virtnet /mnt/lustre (opts:) Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre PASS 37 (5s) == sanity-flr test 38: resync ============================ 02:04:03 (1738479843) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (2s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 02:04:05 (1738479845) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 02:04:07 (1738479847) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0199169 s, 52.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x37:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x38:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x37:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00783043 s, 134 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x38:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x39:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (1s) == sanity-flr test 41: lfs mirror resync check =========== 02:04:08 (1738479848) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0190124 s, 110 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0749406 s, 56.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0305385 s, 68.7 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) == sanity-flr test 42: lfs mirror verify ================= 02:04:11 (1738479851) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (0s) == sanity-flr test 43a: mirror pick on write ============= 02:04:11 (1738479851) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 02:04:12 (1738479852) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00774773 s, 135 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 02:04:14 (1738479854) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0653008 s, 48.2 MB/s PASS 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 02:04:15 (1738479855) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x44:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x44:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x44:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 02:04:17 (1738479857) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.075044 s, 140 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg158-client.virtnet /mnt/lustre (opts:) Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (24s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 02:04:41 (1738479881) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800aa0f7800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0905174 s, 116 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg158-client.virtnet /mnt/lustre (opts:) Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800a292d000.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 02:04:44 (1738479884) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (6s) == sanity-flr test 46: Verify setstripe --copy option ==== 02:04:50 (1738479890) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (7s) == sanity-flr test 47: Verify mirror obj alloc =========== 02:04:57 (1738479897) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (0s) == sanity-flr test 48: Verify snapshot mirror ============ 02:04:57 (1738479897) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0578168 s, 54.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0484133 s, 65.0 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: fd5ebbccb41277e21f4d0156117735bb - mirror 1 checksum: f6ffaaf2a08ddb574668de4fc59fff67 - mirror 2 checksum: fd5ebbccb41277e21f4d0156117735bb - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Feb 2 02:04:58 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Feb 2 02:04:58 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: fd5ebbccb41277e21f4d0156117735bb - mirror 1 checksum: f6ffaaf2a08ddb574668de4fc59fff67 - mirror 2 checksum: f6ffaaf2a08ddb574668de4fc59fff67 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Feb 2 02:04:58 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Feb 2 02:04:58 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } PASS 48 (2s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 02:04:59 (1738479899) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00739906 s, 142 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 02:05:01 (1738479901) Failing mds1 on oleg158-server Stopping /mnt/lustre-mds1 (opts:) on oleg158-server 02:05:08 (1738479908) shut down facet: mds1 facet_host: oleg158-server facet_failover_host: oleg158-server Failover mds1 to oleg158-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg158-server: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-MDT0000 02:05:20 (1738479920) targets are mounted 02:05:20 (1738479920) facet_failover done oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111112 s, 94.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } PASS 50A (27s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 02:05:28 (1738479928) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00731759 s, 2.8 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00348626 s, 5.9 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 02:05:30 (1738479930) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00818102 s, 2.5 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00268466 s, 7.6 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 02:05:31 (1738479931) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.0018955 s, 8.6 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 02:05:33 (1738479933) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00840486 s, 2.4 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 02:05:35 (1738479935) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00135484 s, 3.0 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00605513 s, 676 kB/s PASS 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 02:05:36 (1738479936) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) PASS 60b (4s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 02:05:40 (1738479940) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (17s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 02:05:57 (1738479957) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 02:06:14 (1738479974) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (16s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 02:06:30 (1738479990) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 02:06:32 (1738479992) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 26233 mirror split pid 26234 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4859: 26233 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4859: 26234 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (61s) == sanity-flr test 70a: flr mode fsx test ================ 02:07:33 (1738480053) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 54 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported 000002 1738480054.449372 write 0x0370a4 thru 0x03ffff (0x8f5c bytes) truncating to largest ever: 0x30927 000003 1738480054.451294 trunc from 0x040000 to 0x030926 (0xf6d9 bytes) 000004 1738480054.454588 write 0x034423 thru 0x03ffff (0xbbdd bytes) 000005 1738480054.455684 mapread 0x0077b7 thru 0x00fe1f (0x8669 bytes) 000006 1738480054.460581 punch from 0x021dde to 0x02811c (0x633f bytes) truncating to largest ever: 0x31125 000007 1738480054.461860 trunc from 0x040000 to 0x031124 (0xeedb bytes) 000008 1738480054.465400 read 0x002fea thru 0x00565f (0x2676 bytes) truncating to largest ever: 0x33e5e 000009 1738480054.466888 trunc from 0x031125 to 0x033e5d (0x2d39 bytes) 000011 1738480054.471276 MIRROR_EXTEND to 2 mirrors 000012 1738480054.571096 trunc from 0x033e5e to 0x015dac (0x1e0b1 bytes) 000013 1738480054.584777 mapwrite 0x000801 thru 0x00c96f (0xc16f bytes) 000014 1738480054.597834 write 0x020930 thru 0x02f705 (0xedd6 bytes) 000015 1738480054.598835 mapwrite 0x016ae6 thru 0x01e464 (0x797f bytes) 000016 1738480054.606846 punch from 0x026def to 0x02f705 (0x8917 bytes) 000017 1738480054.608589 trunc from 0x02f706 to 0x004eb9 (0x2a84c bytes) 000019 1738480054.614098 mapread 0x0002c6 thru 0x004eb9 (0x4bf4 bytes) 000020 1738480054.614790 punch from 0x002452 to 0x004eb9 (0x2a68 bytes) 000021 1738480054.616352 fallocate 0x012bbb thru 0x020c6b (0xe0b1 bytes) 000022 1738480054.619210 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000023 1738480054.650916 punch from 0x000c84 to 0x004eb9 (0x4236 bytes) fallocating to largest ever: 0x40000 000024 1738480054.652671 fallocate 0x03e003 thru 0x03ffff (0x1ffd bytes) 000026 1738480054.654492 fallocate 0x0318e3 thru 0x03be4e (0xa56c bytes) 000027 1738480054.656015 fallocate 0x035814 thru 0x03a8c9 (0x50b6 bytes) 000029 1738480054.657450 mapread 0x022a9b thru 0x02eeb2 (0xc418 bytes) 000030 1738480054.664459 write 0x016a25 thru 0x0257d4 (0xedb0 bytes) 000031 1738480054.669063 mapwrite 0x03d17e thru 0x03ffff (0x2e82 bytes) 000035 1738480054.679848 MIRROR_EXTEND to 2 mirrors 000036 1738480054.784318 punch from 0x035127 to 0x03ffff (0xaed9 bytes) 000037 1738480054.792585 fallocate 0x01052b thru 0x0161cd (0x5ca3 bytes) 000038 1738480054.794208 punch from 0x01af90 to 0x021967 (0x69d8 bytes) 000039 1738480054.795683 read 0x01e027 thru 0x029875 (0xb84f bytes) 000040 1738480054.798390 write 0x03e737 thru 0x03ffff (0x18c9 bytes) 000041 1738480054.801276 read 0x03ac5a thru 0x03ffff (0x53a6 bytes) 000042 1738480054.804427 punch from 0x00724c to 0x008c39 (0x19ee bytes) 000043 1738480054.805835 fallocate 0x01174a thru 0x018935 (0x71ec bytes) 000044 1738480054.807237 punch from 0x01e031 to 0x0201c6 (0x2196 bytes) 000045 1738480054.808639 read 0x01bcff thru 0x01cdd8 (0x10da bytes) 000046 1738480054.809136 mapwrite 0x0053e5 thru 0x011777 (0xc393 bytes) 000048 1738480054.817417 fallocate 0x005370 thru 0x009bbc (0x484d bytes) 000049 1738480054.819772 MIRROR_EXTEND to 3 mirrors 000050 1738480054.938359 write 0x01efe1 thru 0x01f63f (0x65f bytes) 000052 1738480054.949524 write 0x028e72 thru 0x02c332 (0x34c1 bytes) 000053 1738480054.951913 mapread 0x02e3c6 thru 0x0342f2 (0x5f2d bytes) 000054 1738480054.954315 read 0x0109a1 thru 0x013795 (0x2df5 bytes) 000056 1738480054.956314 write 0x0321ff thru 0x03ffff (0xde01 bytes) 000057 1738480054.958680 MIRROR_RESYNC 3 mirrors 000058 1738480055.155432 mapread 0x01960f thru 0x019659 (0x4b bytes) 000059 1738480055.156913 mapread 0x004062 thru 0x00ebde (0xab7d bytes) 000060 1738480055.161827 mapread 0x02ff2f thru 0x031724 (0x17f6 bytes) 000062 1738480055.164242 read 0x02600e thru 0x02864f (0x2642 bytes) 000063 1738480055.166434 MIRROR_EXTEND to 4 mirrors 000064 1738480055.288052 mapwrite 0x01d7c4 thru 0x024fb0 (0x77ed bytes) 000065 1738480055.318514 write 0x019149 thru 0x01d5d0 (0x4488 bytes) 000066 1738480055.321285 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000067 1738480055.371391 trunc from 0x040000 to 0x027120 (0x18edf bytes) 000068 1738480055.375591 punch from 0x009bd8 to 0x00be11 (0x223a bytes) 000070 1738480055.377276 punch from 0x019f86 to 0x021df9 (0x7e74 bytes) 000071 1738480055.378527 trunc from 0x027121 to 0x0304bb (0x939b bytes) 000073 1738480055.382027 trunc from 0x0304bc to 0x02f212 (0x12a9 bytes) 000076 1738480055.387064 MIRROR_RESYNC 3 mirrors 000077 1738480055.548853 punch from 0x02ad41 to 0x02f212 (0x44d2 bytes) 000078 1738480055.556681 mapwrite 0x031cea thru 0x03ab6c (0x8e83 bytes) 000079 1738480055.571558 mapread 0x026b1b thru 0x033e71 (0xd357 bytes) 000080 1738480055.573834 read 0x017c09 thru 0x0188a4 (0xc9c bytes) 000081 1738480055.575508 mapwrite 0x03061c thru 0x03312b (0x2b10 bytes) 000082 1738480055.579786 trunc from 0x03ab6d to 0x03b7d5 (0xc69 bytes) 000083 1738480055.583532 read 0x017a76 thru 0x027871 (0xfdfc bytes) 000084 1738480055.586488 punch from 0x006bc2 to 0x00b367 (0x47a6 bytes) 000085 1738480055.588024 mapwrite 0x0157e8 thru 0x017763 (0x1f7c bytes) 000086 1738480055.592417 mapread 0x017896 thru 0x023d86 (0xc4f1 bytes) 000087 1738480055.592849 trunc from 0x03b7d6 to 0x02706d (0x14768 bytes) 000088 1738480055.596751 read 0x002b01 thru 0x00aeef (0x83ef bytes) 000090 1738480055.599150 trunc from 0x02706e to 0x010b4d (0x16520 bytes) 000091 1738480055.602649 fallocate 0x01e6f4 thru 0x025d88 (0x7695 bytes) 000092 1738480055.605253 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000093 1738480055.651715 punch from 0x0028ab to 0x005a03 (0x3159 bytes) 000094 1738480055.654682 MIRROR_EXTEND to 3 mirrors 000095 1738480055.749867 mapwrite 0x0381ec thru 0x03ffff (0x7e14 bytes) 000096 1738480055.771413 mapwrite 0x013396 thru 0x0140dd (0xd48 bytes) 000098 1738480055.777402 mapwrite 0x00b4d4 thru 0x017838 (0xc365 bytes) 000099 1738480055.785716 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000100 1738480055.828703 trunc from 0x040000 to 0x009442 (0x36bbd bytes) 000104 1738480055.835336 MIRROR_RESYNC 2 mirrors 000105 1738480055.918675 fallocate 0x019a9f thru 0x022a0d (0x8f6f bytes) 000106 1738480055.925914 mapread 0x01224a thru 0x01836f (0x6126 bytes) 000107 1738480055.929995 punch from 0x00111c to 0x00f9f8 (0xe8dd bytes) 000109 1738480055.931504 mapread 0x00dfe2 thru 0x00f109 (0x1128 bytes) 000110 1738480055.933467 trunc from 0x022a0e to 0x036583 (0x13b76 bytes) 000112 1738480055.941369 mapread 0x018d75 thru 0x0263ea (0xd676 bytes) 000113 1738480055.943710 mapread 0x0065f7 thru 0x00c454 (0x5e5e bytes) 000114 1738480055.945882 punch from 0x033ca8 to 0x036583 (0x28dc bytes) 000116 1738480055.947693 fallocate 0x026c1e thru 0x0323ba (0xb79d bytes) 000117 1738480055.949250 write 0x02023f thru 0x02f9e3 (0xf7a5 bytes) 000118 1738480055.951286 read 0x01d4fd thru 0x027c55 (0xa759 bytes) 000120 1738480055.951767 fallocate 0x00637e thru 0x00b01a (0x4c9d bytes) 000122 1738480055.953415 write 0x0082d1 thru 0x0166f5 (0xe425 bytes) 000123 1738480055.954149 punch from 0x00e79f to 0x014b8a (0x63ec bytes) 000124 1738480055.957727 mapwrite 0x00fa29 thru 0x01acf5 (0xb2cd bytes) 000125 1738480055.966839 mapread 0x0270ac thru 0x0273fb (0x350 bytes) 000126 1738480055.967467 punch from 0x016a20 to 0x01b705 (0x4ce6 bytes) 000128 1738480055.969637 mapread 0x00f1cf thru 0x01c59c (0xd3ce bytes) 000129 1738480055.972413 write 0x00b387 thru 0x00ba7d (0x6f7 bytes) 000130 1738480055.972913 mapwrite 0x00a547 thru 0x016a72 (0xc52c bytes) 000131 1738480055.978817 fallocate 0x01c7e6 thru 0x028f44 (0xc75f bytes) 000132 1738480055.980365 read 0x005c9d thru 0x01417b (0xe4df bytes) 000134 1738480055.981115 write 0x0200c1 thru 0x027230 (0x7170 bytes) 000135 1738480055.981864 read 0x0215dd thru 0x02632e (0x4d52 bytes) 000136 1738480055.982237 trunc from 0x036584 to 0x033065 (0x351e bytes) 000137 1738480055.986346 mapwrite 0x012e7e thru 0x022700 (0xf883 bytes) 000138 1738480055.993849 fallocate 0x0291a5 thru 0x02b31b (0x2177 bytes) 000139 1738480055.995404 punch from 0x000774 to 0x00a7da (0xa067 bytes) 000141 1738480055.997369 read 0x00d948 thru 0x015913 (0x7fcc bytes) 000143 1738480055.998228 fallocate 0x022aae thru 0x02e261 (0xb7b4 bytes) 000144 1738480056.002233 write 0x01e26c thru 0x020440 (0x21d5 bytes) 000145 1738480056.002720 read 0x022a84 thru 0x022ead (0x42a bytes) 000146 1738480056.002975 write 0x01d5ea thru 0x029923 (0xc33a bytes) 000147 1738480056.003615 punch from 0x011cad to 0x01c9ec (0xad40 bytes) 000148 1738480056.005396 punch from 0x011054 to 0x01937a (0x8327 bytes) 000150 1738480056.007003 mapread 0x006c29 thru 0x00f12c (0x8504 bytes) 000152 1738480056.009949 trunc from 0x033066 to 0x0357c3 (0x275e bytes) 000153 1738480056.013337 mapread 0x016bde thru 0x016d43 (0x166 bytes) 000154 1738480056.014890 MIRROR_RESYNC 2 mirrors 000156 1738480056.115496 punch from 0x0284aa to 0x02a48b (0x1fe2 bytes) 000157 1738480056.122965 mapread 0x010393 thru 0x011490 (0x10fe bytes) 000158 1738480056.126532 MIRROR_RESYNC 2 mirrors 000159 1738480056.230244 mapwrite 0x037d62 thru 0x03d6d0 (0x596f bytes) 000161 1738480056.250508 write 0x03b5db thru 0x03f162 (0x3b88 bytes) 000162 1738480056.251093 fallocate 0x0130b7 thru 0x01f574 (0xc4be bytes) 000163 1738480056.252392 mapread 0x031bc8 thru 0x039e3c (0x8275 bytes) 000164 1738480056.254796 trunc from 0x03f163 to 0x00285c (0x3c906 bytes) 000165 1738480056.259633 trunc from 0x00285d to 0x023713 (0x20eb7 bytes) 000167 1738480056.263389 mapwrite 0x03f01f thru 0x03ffff (0xfe1 bytes) 000168 1738480056.272358 trunc from 0x040000 to 0x020045 (0x1ffba bytes) 000169 1738480056.275853 mapwrite 0x023bba thru 0x029822 (0x5c69 bytes) 000170 1738480056.286181 write 0x02d32f thru 0x032032 (0x4d04 bytes) 000171 1738480056.286860 read 0x0240de thru 0x030c18 (0xcb3b bytes) 000172 1738480056.288539 write 0x02b33d thru 0x02be20 (0xae4 bytes) 000173 1738480056.289202 write 0x0093ac thru 0x0124a5 (0x90fa bytes) 000174 1738480056.292875 write 0x03155b thru 0x0333a4 (0x1e4a bytes) 000175 1738480056.293590 write 0x00e31c thru 0x01c269 (0xdf4e bytes) 000176 1738480056.294766 mapread 0x00a79f thru 0x0160ce (0xb930 bytes) 000177 1738480056.295480 fallocate 0x0381ad thru 0x038389 (0x1dd bytes) 000178 1738480056.297248 write 0x022c50 thru 0x0262f1 (0x36a2 bytes) 000179 1738480056.297960 trunc from 0x0333a5 to 0x00d4c0 (0x25ee4 bytes) 000180 1738480056.301593 read 0x001e96 thru 0x00d4c0 (0xb62b bytes) 000181 1738480056.303037 mapwrite 0x00fc75 thru 0x01992a (0x9cb6 bytes) 000182 1738480056.315757 punch from 0x00f35c to 0x012643 (0x32e8 bytes) 000183 1738480056.317553 trunc from 0x01992b to 0x03ccb5 (0x2338b bytes) 000184 1738480056.321578 mapwrite 0x0121ff thru 0x014f3d (0x2d3f bytes) 000185 1738480056.328794 punch from 0x012dbe to 0x02219e (0xf3e1 bytes) 000187 1738480056.332156 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000188 1738480056.362594 fallocate 0x025e71 thru 0x029ec3 (0x4053 bytes) 000191 1738480056.364834 punch from 0x01302a to 0x013328 (0x2ff bytes) 000192 1738480056.366193 read 0x00deb3 thru 0x013265 (0x53b3 bytes) 000193 1738480056.368438 read 0x01fb5e thru 0x02ee7b (0xf31e bytes) 000196 1738480056.372530 mapwrite 0x0220db thru 0x0277cb (0x56f1 bytes) 000197 1738480056.377598 mapread 0x02a238 thru 0x0317a5 (0x756e bytes) 000198 1738480056.378034 read 0x00d232 thru 0x0197c0 (0xc58f bytes) 000200 1738480056.378610 write 0x015ecf thru 0x019864 (0x3996 bytes) 000202 1738480056.380804 write 0x01177b thru 0x01d058 (0xb8de bytes) 000203 1738480056.381903 mapread 0x006104 thru 0x0091bc (0x30b9 bytes) 000205 1738480056.382906 write 0x017836 thru 0x01a5ed (0x2db8 bytes) 000206 1738480056.383601 trunc from 0x03ccb6 to 0x02bb84 (0x11131 bytes) 000207 1738480056.387348 write 0x00c5d9 thru 0x013c77 (0x769f bytes) 000209 1738480056.388610 mapread 0x00cbdc thru 0x013868 (0x6c8d bytes) 000210 1738480056.389279 trunc from 0x02bb85 to 0x03584d (0x9cc9 bytes) 000211 1738480056.392925 fallocate 0x03d54d thru 0x03ffff (0x2ab3 bytes) 000212 1738480056.394469 punch from 0x028ac9 to 0x0364fe (0xda36 bytes) 000213 1738480056.395979 write 0x034e1d thru 0x03fda0 (0xaf84 bytes) 000214 1738480056.398726 mapread 0x01afe2 thru 0x01c162 (0x1181 bytes) 000215 1738480056.399149 read 0x01df2b thru 0x024cab (0x6d81 bytes) 000216 1738480056.399411 fallocate 0x02a4c1 thru 0x0386a8 (0xe1e8 bytes) 000217 1738480056.400644 read 0x03118a thru 0x031742 (0x5b9 bytes) 000218 1738480056.400886 mapread 0x0009b3 thru 0x00c409 (0xba57 bytes) 000219 1738480056.401319 mapwrite 0x034534 thru 0x03a8c8 (0x6395 bytes) 000220 1738480056.406579 trunc from 0x040000 to 0x02ce37 (0x131c8 bytes) 000221 1738480056.410032 write 0x01d4a0 thru 0x028354 (0xaeb5 bytes) 000222 1738480056.410807 read 0x001708 thru 0x00aad7 (0x93d0 bytes) 000226 1738480056.411718 fallocate 0x035baa thru 0x03b549 (0x59a0 bytes) 000227 1738480056.413028 punch from 0x0008c6 to 0x00e97e (0xe0b9 bytes) 000228 1738480056.414359 mapwrite 0x01584d thru 0x01a34b (0x4aff bytes) 000229 1738480056.419335 mapwrite 0x0305db thru 0x038738 (0x815e bytes) 000231 1738480056.426560 trunc from 0x03b54a to 0x013c90 (0x278b9 bytes) 000232 1738480056.430440 mapwrite 0x01b9f0 thru 0x02469d (0x8cae bytes) 000235 1738480056.449846 read 0x00aced thru 0x012e77 (0x818b bytes) 000236 1738480056.450190 mapwrite 0x033433 thru 0x03ffff (0xcbcd bytes) 000240 1738480056.462600 punch from 0x0123d4 to 0x014bfe (0x282b bytes) 000241 1738480056.465298 MIRROR_EXTEND to 2 mirrors 000242 1738480056.565427 fallocate 0x01bf26 thru 0x01fffc (0x40d7 bytes) 000243 1738480056.572542 trunc from 0x040000 to 0x026d55 (0x192aa bytes) 000244 1738480056.580902 punch from 0x020456 to 0x026d55 (0x6900 bytes) 000246 1738480056.582704 write 0x002a55 thru 0x00320e (0x7ba bytes) 000247 1738480056.585160 read 0x00d444 thru 0x015c73 (0x8830 bytes) 000251 1738480056.588324 write 0x02c4b7 thru 0x0341d9 (0x7d23 bytes) 000252 1738480056.589262 mapread 0x0234a6 thru 0x024aeb (0x1646 bytes) 000253 1738480056.592999 MIRROR_RESYNC 2 mirrors 000254 1738480056.705562 write 0x03245b thru 0x037ccf (0x5875 bytes) 000255 1738480056.713615 punch from 0x022da5 to 0x0274d1 (0x472d bytes) 000256 1738480056.714857 punch from 0x02f1d1 to 0x034835 (0x5665 bytes) 000258 1738480056.718322 trunc from 0x037cd0 to 0x01d0bd (0x1ac12 bytes) 000259 1738480056.726182 trunc from 0x01d0be to 0x00d3e8 (0xfcd5 bytes) 000260 1738480056.729519 trunc from 0x00d3e9 to 0x0091bc (0x422c bytes) 000261 1738480056.733166 trunc from 0x0091bd to 0x02ece1 (0x25b25 bytes) 000262 1738480056.736730 punch from 0x015f6c to 0x01a7dc (0x4871 bytes) 000263 1738480056.737924 mapwrite 0x037580 thru 0x03b3ec (0x3e6d bytes) 000264 1738480056.749684 read 0x004601 thru 0x00b0d1 (0x6ad1 bytes) 000265 1738480056.751834 mapread 0x026e97 thru 0x02f7c4 (0x892e bytes) 000266 1738480056.755598 read 0x0319d5 thru 0x0342cc (0x28f8 bytes) 000267 1738480056.755874 fallocate 0x0049d7 thru 0x006f6c (0x2596 bytes) 000268 1738480056.758345 MIRROR_RESYNC 2 mirrors 000269 1738480056.840017 read 0x014d1c thru 0x0219f2 (0xccd7 bytes) 000270 1738480056.842268 write 0x02fa4b thru 0x037ab0 (0x8066 bytes) 000271 1738480056.850915 mapread 0x00d44e thru 0x00fdc4 (0x2977 bytes) 000272 1738480056.853448 mapread 0x01de56 thru 0x02656c (0x8717 bytes) 000273 1738480056.857295 write 0x032656 thru 0x037412 (0x4dbd bytes) 000274 1738480056.858122 trunc from 0x03b3ed to 0x031183 (0xa269 bytes) 000275 1738480056.866541 mapread 0x01fba5 thru 0x025dcc (0x6228 bytes) 000276 1738480056.866992 mapwrite 0x01a21f thru 0x020488 (0x626a bytes) 000277 1738480056.872574 punch from 0x02b9d0 to 0x031183 (0x57b4 bytes) 000278 1738480056.874156 mapwrite 0x007ee4 thru 0x0096c3 (0x17e0 bytes) 000279 1738480056.879473 read 0x02b463 thru 0x02fd8c (0x492a bytes) 000280 1738480056.881552 mapwrite 0x00215e thru 0x0101ce (0xe071 bytes) 000281 1738480056.888844 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000282 1738480056.918689 punch from 0x018810 to 0x01f242 (0x6a33 bytes) 000283 1738480056.920232 trunc from 0x031184 to 0x033ec9 (0x2d46 bytes) 000286 1738480056.925642 punch from 0x02b576 to 0x033ec9 (0x8954 bytes) 000287 1738480056.927003 trunc from 0x033eca to 0x00a0d0 (0x29df9 bytes) 000288 1738480056.930602 write 0x032b8e thru 0x034c17 (0x208a bytes) 000290 1738480056.931812 punch from 0x004588 to 0x00c400 (0x7e79 bytes) 000292 1738480056.933519 mapread 0x034b98 thru 0x034c17 (0x80 bytes) 000293 1738480056.934162 mapwrite 0x01e739 thru 0x01f6ab (0xf73 bytes) 000295 1738480056.941652 read 0x00a949 thru 0x017112 (0xc7ca bytes) 000296 1738480056.943935 fallocate 0x036f63 thru 0x03a6c9 (0x3767 bytes) 000297 1738480056.945633 fallocate 0x00ea8f thru 0x0198de (0xae50 bytes) 000298 1738480056.946960 read 0x021f7f thru 0x02ada4 (0x8e26 bytes) 000299 1738480056.949820 mapread 0x022579 thru 0x02ad51 (0x87d9 bytes) 000300 1738480056.950462 trunc from 0x03a6ca to 0x01764d (0x2307c bytes) 000302 1738480056.954884 write 0x033f5d thru 0x03ed95 (0xae39 bytes) 000304 1738480056.956145 mapread 0x03eb7f thru 0x03ed95 (0x217 bytes) 000306 1738480056.956958 read 0x02dcd4 thru 0x03638e (0x86bb bytes) 000307 1738480056.959417 fallocate 0x014b7c thru 0x01b228 (0x66ad bytes) 000308 1738480056.960939 write 0x02b7f8 thru 0x0372a6 (0xbaaf bytes) 000309 1738480056.961607 trunc from 0x03ed96 to 0x02b9d9 (0x133bc bytes) 000310 1738480056.965307 mapread 0x01af61 thru 0x0241da (0x927a bytes) 000311 1738480056.965995 punch from 0x0137e8 to 0x01442c (0xc45 bytes) 000312 1738480056.967434 mapread 0x027075 thru 0x02b9d9 (0x4965 bytes) 000313 1738480056.967952 trunc from 0x02b9da to 0x0136ad (0x1832c bytes) 000315 1738480056.971983 fallocate 0x0040e3 thru 0x00e549 (0xa467 bytes) 000318 1738480056.974391 fallocate 0x00b0f3 thru 0x016da4 (0xbcb2 bytes) 000319 1738480056.975798 fallocate 0x03edb3 thru 0x03ffff (0x124d bytes) 000320 1738480056.977341 write 0x00a57d thru 0x00a685 (0x109 bytes) 000321 1738480056.977956 mapread 0x024be0 thru 0x024c74 (0x95 bytes) 000323 1738480056.980431 punch from 0x00d622 to 0x01c288 (0xec67 bytes) 000324 1738480056.981898 mapread 0x029a30 thru 0x035d65 (0xc336 bytes) 000327 1738480056.996662 fallocate 0x0222d3 thru 0x028f4c (0x6c7a bytes) 000330 1738480056.998758 fallocate 0x0283e9 thru 0x0297cf (0x13e7 bytes) 000331 1738480057.000354 write 0x027dca thru 0x02da44 (0x5c7b bytes) 000332 1738480057.002215 fallocate 0x0157b9 thru 0x01db49 (0x8391 bytes) 000333 1738480057.003670 read 0x02f371 thru 0x03e9c6 (0xf656 bytes) 000334 1738480057.006334 fallocate 0x017802 thru 0x0236b6 (0xbeb5 bytes) 000335 1738480057.007863 trunc from 0x040000 to 0x0157a1 (0x2a85e bytes) 000336 1738480057.012056 mapread 0x00bb1a thru 0x0157a1 (0x9c88 bytes) 000337 1738480057.012526 read 0x00bb0d thru 0x014d9e (0x9292 bytes) 000338 1738480057.012779 fallocate 0x01cf95 thru 0x01d9ed (0xa59 bytes) 000339 1738480057.014374 trunc from 0x0157a2 to 0x00894a (0xce57 bytes) 000340 1738480057.018378 write 0x00163a thru 0x00b7d7 (0xa19e bytes) 000343 1738480057.020115 write 0x0054cd thru 0x0087f6 (0x332a bytes) 000345 1738480057.020692 write 0x006eae thru 0x00f360 (0x84b3 bytes) 000347 1738480057.022612 MIRROR_EXTEND to 2 mirrors 000348 1738480057.094569 MIRROR_RESYNC 2 mirrors 000349 1738480057.126774 read 0x00642b thru 0x00f071 (0x8c47 bytes) 000350 1738480057.130818 MIRROR_RESYNC 2 mirrors 000351 1738480057.161182 punch from 0x0010d2 to 0x00f360 (0xe28f bytes) 000353 1738480057.168917 mapread 0x00c4f0 thru 0x00f360 (0x2e71 bytes) 000354 1738480057.173201 mapwrite 0x02b82e thru 0x039b62 (0xe335 bytes) 000355 1738480057.192340 mapwrite 0x02008d thru 0x02f28c (0xf200 bytes) 000356 1738480057.200138 mapread 0x01e09b thru 0x027830 (0x9796 bytes) 000361 1738480057.201494 punch from 0x010875 to 0x0118a3 (0x102f bytes) 000362 1738480057.202679 mapread 0x022dcb thru 0x031617 (0xe84d bytes) 000363 1738480057.203160 mapread 0x005d20 thru 0x01497b (0xec5c bytes) 000364 1738480057.207346 fallocate 0x010e8e thru 0x0192bc (0x842f bytes) 000367 1738480057.209343 trunc from 0x039b63 to 0x01767e (0x224e4 bytes) 000369 1738480057.213708 read 0x012555 thru 0x01767e (0x512a bytes) 000370 1738480057.214221 mapread 0x001517 thru 0x01145f (0xff49 bytes) 000371 1738480057.214832 fallocate 0x03dbeb thru 0x03ffff (0x2415 bytes) 000372 1738480057.216400 write 0x00059a thru 0x0066c9 (0x6130 bytes) 000374 1738480057.218205 mapwrite 0x028ba0 thru 0x0314f9 (0x895a bytes) 000375 1738480057.227796 fallocate 0x00403a thru 0x006cc3 (0x2c8a bytes) 000376 1738480057.229232 fallocate 0x00d606 thru 0x011801 (0x41fc bytes) 000379 1738480057.230996 fallocate 0x00da91 thru 0x019b18 (0xc088 bytes) 000380 1738480057.232366 mapread 0x018f0f thru 0x01d8d7 (0x49c9 bytes) 000381 1738480057.234736 write 0x03346c thru 0x0338c8 (0x45d bytes) 000382 1738480057.235300 fallocate 0x01b223 thru 0x027973 (0xc751 bytes) 000383 1738480057.236697 write 0x016fc7 thru 0x01a5d8 (0x3612 bytes) 000384 1738480057.237320 trunc from 0x040000 to 0x01879b (0x27864 bytes) 000385 1738480057.241135 mapread 0x000661 thru 0x0040a1 (0x3a41 bytes) 000387 1738480057.242024 punch from 0x015167 to 0x01879b (0x3635 bytes) 000388 1738480057.244701 mapwrite 0x00bd4e thru 0x01249f (0x6752 bytes) 000389 1738480057.250101 mapwrite 0x01df8c thru 0x02b28a (0xd2ff bytes) 000390 1738480057.263181 fallocate 0x01b508 thru 0x027edf (0xc9d8 bytes) 000391 1738480057.264654 trunc from 0x02b28b to 0x034c4f (0x99c5 bytes) 000393 1738480057.268594 fallocate 0x00fc33 thru 0x010b28 (0xef6 bytes) 000394 1738480057.270104 mapread 0x0349e2 thru 0x034c4f (0x26e bytes) 000395 1738480057.272755 mapwrite 0x028e15 thru 0x031b59 (0x8d45 bytes) 000396 1738480057.280396 fallocate 0x0137c4 thru 0x01c7d1 (0x900e bytes) 000397 1738480057.281765 trunc from 0x034c50 to 0x037880 (0x2c31 bytes) 000398 1738480057.285027 read 0x0098a6 thru 0x01352e (0x9c89 bytes) 000401 1738480057.285999 trunc from 0x037881 to 0x035ac7 (0x1db9 bytes) 000402 1738480057.289528 trunc from 0x035ac8 to 0x00d5d1 (0x284f6 bytes) 000403 1738480057.293370 trunc from 0x00d5d2 to 0x03613a (0x28b69 bytes) 000404 1738480057.296935 mapread 0x01dd2b thru 0x025953 (0x7c29 bytes) 000405 1738480057.300007 mapread 0x005215 thru 0x006e75 (0x1c61 bytes) 000406 1738480057.300466 trunc from 0x03613b to 0x03f17a (0x9040 bytes) 000407 1738480057.304136 trunc from 0x03f17b to 0x02002e (0x1f14c bytes) 000408 1738480057.308414 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000409 1738480057.337805 MIRROR_EXTEND to 2 mirrors 000410 1738480057.430143 trunc from 0x02002f to 0x0244f0 (0x44c2 bytes) 000411 1738480057.442583 mapwrite 0x028b33 thru 0x02ca80 (0x3f4e bytes) 000412 1738480057.456246 mapread 0x0149f3 thru 0x024517 (0xfb25 bytes) 000413 1738480057.460593 trunc from 0x02ca81 to 0x039fa7 (0xd527 bytes) 000414 1738480057.464159 read 0x00fb00 thru 0x01f3a4 (0xf8a5 bytes) 000415 1738480057.466980 MIRROR_RESYNC 2 mirrors 000416 1738480057.563970 fallocate 0x01eff1 thru 0x02a8de (0xb8ee bytes) 000417 1738480057.573610 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000418 1738480057.602790 mapread 0x0331b6 thru 0x034534 (0x137f bytes) 000419 1738480057.605782 write 0x018da9 thru 0x01b2e1 (0x2539 bytes) 000420 1738480057.609684 read 0x02c290 thru 0x03851d (0xc28e bytes) 000421 1738480057.611961 read 0x017c0f thru 0x023732 (0xbb24 bytes) 000422 1738480057.614278 mapread 0x012454 thru 0x015bcd (0x377a bytes) 000424 1738480057.615133 read 0x00adb1 thru 0x018bb9 (0xde09 bytes) 000425 1738480057.615411 read 0x00e45f thru 0x01dbcb (0xf76d bytes) 000426 1738480057.615667 mapwrite 0x012c1f thru 0x01d889 (0xac6b bytes) 000427 1738480057.622117 mapread 0x038cb7 thru 0x039fa7 (0x12f1 bytes) 000428 1738480057.622527 fallocate 0x0062b6 thru 0x00641d (0x168 bytes) 000429 1738480057.623923 write 0x0369af thru 0x036e21 (0x473 bytes) 000430 1738480057.624454 mapwrite 0x035945 thru 0x035a94 (0x150 bytes) 000432 1738480057.628175 mapwrite 0x035dd2 thru 0x03ffff (0xa22e bytes) 000434 1738480057.643873 trunc from 0x040000 to 0x006e62 (0x3919d bytes) 000435 1738480057.648012 punch from 0x006ad9 to 0x006e62 (0x38a bytes) 000436 1738480057.649593 fallocate 0x02433e thru 0x02fa0d (0xb6d0 bytes) 000438 1738480057.651702 mapread 0x0017bc thru 0x006e62 (0x56a7 bytes) 000441 1738480057.654447 fallocate 0x03050d thru 0x0310f8 (0xbec bytes) 000442 1738480057.656275 write 0x001935 thru 0x01037c (0xea48 bytes) 000443 1738480057.658631 fallocate 0x033864 thru 0x03ffff (0xc79c bytes) 000444 1738480057.660186 mapwrite 0x03d228 thru 0x03ffff (0x2dd8 bytes) 000445 1738480057.673171 read 0x0236ad thru 0x031c2f (0xe583 bytes) 000446 1738480057.676571 fallocate 0x026264 thru 0x02d950 (0x76ed bytes) 000447 1738480057.678207 fallocate 0x017b56 thru 0x01f033 (0x74de bytes) 000448 1738480057.679646 write 0x02ea6e thru 0x03b5c3 (0xcb56 bytes) 000449 1738480057.680724 write 0x015278 thru 0x018444 (0x31cd bytes) 000452 1738480057.682398 mapwrite 0x039cab thru 0x03ffff (0x6355 bytes) 000454 1738480057.688015 mapread 0x02559a thru 0x034d09 (0xf770 bytes) 000455 1738480057.688467 punch from 0x0303d9 to 0x037bbb (0x77e3 bytes) 000456 1738480057.689808 read 0x03f07f thru 0x03f571 (0x4f3 bytes) 000457 1738480057.690172 trunc from 0x040000 to 0x0029fe (0x3d601 bytes) 000458 1738480057.694222 read 0x0018db thru 0x0029fe (0x1124 bytes) 000460 1738480057.694931 write 0x007b18 thru 0x01475b (0xcc44 bytes) 000461 1738480057.697189 MIRROR_EXTEND to 2 mirrors 000464 1738480057.785562 read 0x0058c6 thru 0x01475b (0xee96 bytes) 000466 1738480057.787844 trunc from 0x01475c to 0x00bf53 (0x8808 bytes) 000468 1738480057.801865 mapread 0x001180 thru 0x00bf53 (0xadd4 bytes) 000469 1738480057.807036 mapwrite 0x0118c0 thru 0x019e95 (0x85d6 bytes) 000470 1738480057.819357 write 0x02161c thru 0x02f656 (0xe03b bytes) 000471 1738480057.820373 trunc from 0x02f657 to 0x021417 (0xe23f bytes) 000472 1738480057.823740 trunc from 0x021418 to 0x01c97e (0x4a99 bytes) 000473 1738480057.827023 mapread 0x016b3f thru 0x01c97e (0x5e40 bytes) 000474 1738480057.828846 punch from 0x01950c to 0x01c97e (0x3473 bytes) 000475 1738480057.830273 mapread 0x015a54 thru 0x01c97e (0x6f2b bytes) 000476 1738480057.831916 write 0x010463 thru 0x019ec6 (0x9a64 bytes) 000477 1738480057.832548 trunc from 0x01c97f to 0x00029c (0x1c6e2 bytes) 000478 1738480057.836193 punch from 0x000158 to 0x00029c (0x145 bytes) 000479 1738480057.837453 mapread 0x0001a9 thru 0x00029c (0xf4 bytes) 000481 1738480057.839182 punch from 0x00000d to 0x00029c (0x290 bytes) 000482 1738480057.841485 MIRROR_RESYNC 2 mirrors 000483 1738480057.901140 fallocate 0x028031 thru 0x02ab98 (0x2b68 bytes) 000484 1738480057.908656 mapwrite 0x018ea6 thru 0x027a58 (0xebb3 bytes) 000488 1738480057.922296 fallocate 0x022205 thru 0x02368e (0x148a bytes) 000489 1738480057.923890 fallocate 0x02de75 thru 0x036c2a (0x8db6 bytes) 000490 1738480057.925533 fallocate 0x022781 thru 0x02e6ae (0xbf2e bytes) 000491 1738480057.927178 fallocate 0x0008f8 thru 0x007b49 (0x7252 bytes) 000492 1738480057.930147 MIRROR_RESYNC 2 mirrors 000493 1738480058.004976 write 0x005533 thru 0x00fa07 (0xa4d5 bytes) 000495 1738480058.014020 trunc from 0x02ab99 to 0x025f42 (0x4c56 bytes) 000496 1738480058.021819 mapread 0x018bad thru 0x018f4c (0x3a0 bytes) 000497 1738480058.024449 trunc from 0x025f43 to 0x02dd80 (0x7e3e bytes) 000498 1738480058.028140 read 0x00190d thru 0x00d75d (0xbe51 bytes) 000502 1738480058.030373 mapread 0x02d474 thru 0x02dd80 (0x90d bytes) 000503 1738480058.032589 trunc from 0x02dd81 to 0x02cb51 (0x122f bytes) 000504 1738480058.036342 write 0x016a38 thru 0x019738 (0x2d01 bytes) 000505 1738480058.038124 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000506 1738480058.072254 mapread 0x01ca76 thru 0x02c221 (0xf7ac bytes) 000508 1738480058.079498 punch from 0x0149ca to 0x01f921 (0xaf58 bytes) 000509 1738480058.081249 read 0x021960 thru 0x029269 (0x790a bytes) 000510 1738480058.081691 fallocate 0x016474 thru 0x01d3db (0x6f68 bytes) 000511 1738480058.083289 fallocate 0x0220ec thru 0x023130 (0x1045 bytes) 000512 1738480058.084671 mapwrite 0x016d37 thru 0x016f8d (0x257 bytes) 000513 1738480058.091584 fallocate 0x022092 thru 0x022424 (0x393 bytes) 000514 1738480058.093263 read 0x008b01 thru 0x00dea7 (0x53a7 bytes) 000517 1738480058.096433 mapwrite 0x022d1c thru 0x023ff5 (0x12da bytes) 000518 1738480058.099701 write 0x010ae2 thru 0x0142a9 (0x37c8 bytes) 000519 1738480058.101354 mapwrite 0x031839 thru 0x03e64b (0xce13 bytes) 000521 1738480058.115513 read 0x028696 thru 0x02fcef (0x765a bytes) 000523 1738480058.116008 trunc from 0x03e64c to 0x0089b3 (0x35c98 bytes) 000524 1738480058.121031 mapwrite 0x0132b5 thru 0x015f8f (0x2cdb bytes) 000525 1738480058.131387 trunc from 0x015f90 to 0x03e4bd (0x2852e bytes) 000526 1738480058.134949 write 0x00ac39 thru 0x00dc84 (0x304c bytes) 000527 1738480058.135519 punch from 0x0171fa to 0x021430 (0xa237 bytes) 000528 1738480058.136814 read 0x020e40 thru 0x029150 (0x8311 bytes) 000530 1738480058.139692 punch from 0x0170b5 to 0x01894f (0x189b bytes) 000531 1738480058.140829 mapread 0x03779e thru 0x03cc8a (0x54ed bytes) 000533 1738480058.141472 mapread 0x02f636 thru 0x03406d (0x4a38 bytes) 000534 1738480058.141822 punch from 0x03cbfc to 0x03e4bd (0x18c2 bytes) 000535 1738480058.143137 write 0x03aad0 thru 0x03ffff (0x5530 bytes) 000536 1738480058.143690 trunc from 0x040000 to 0x01ed01 (0x212fe bytes) 000537 1738480058.147466 punch from 0x019b6a to 0x01a119 (0x5b0 bytes) 000538 1738480058.148790 mapwrite 0x00e7e2 thru 0x015064 (0x6883 bytes) 000539 1738480058.154664 fallocate 0x01a485 thru 0x024af9 (0xa675 bytes) 000540 1738480058.156248 trunc from 0x01ed02 to 0x018b28 (0x61d9 bytes) 000541 1738480058.159830 fallocate 0x0253d8 thru 0x02d682 (0x82ab bytes) 000542 1738480058.161400 mapread 0x00d114 thru 0x014b32 (0x7a1f bytes) 000543 1738480058.162091 punch from 0x0176dc to 0x017796 (0xbb bytes) 000544 1738480058.163382 mapread 0x003edc thru 0x012e4e (0xef73 bytes) 000545 1738480058.163966 mapread 0x00dc96 thru 0x0107a4 (0x2b0f bytes) 000548 1738480058.165235 mapread 0x00694b thru 0x00e96f (0x8025 bytes) 000549 1738480058.165657 write 0x005389 thru 0x006598 (0x1210 bytes) 000551 1738480058.166455 write 0x0085c4 thru 0x00f9f9 (0x7436 bytes) 000552 1738480058.167145 mapread 0x014aac thru 0x018b28 (0x407d bytes) 000557 1738480058.170834 write 0x01bf99 thru 0x01e73a (0x27a2 bytes) 000559 1738480058.171665 mapread 0x00cd2e thru 0x0114a7 (0x477a bytes) 000560 1738480058.172115 mapwrite 0x03d1f6 thru 0x03e375 (0x1180 bytes) 000564 1738480058.184228 trunc from 0x03e376 to 0x0313fc (0xcf79 bytes) 000565 1738480058.187466 write 0x02d7a6 thru 0x02f9e7 (0x2242 bytes) 000567 1738480058.190376 mapwrite 0x004236 thru 0x00f743 (0xb50e bytes) 000571 1738480058.197845 fallocate 0x01f11d thru 0x02bd8c (0xcc70 bytes) 000573 1738480058.199660 read 0x02feb5 thru 0x0313fc (0x1548 bytes) 000574 1738480058.202130 mapwrite 0x00dd32 thru 0x013409 (0x56d8 bytes) 000576 1738480058.207377 read 0x023808 thru 0x02e033 (0xa82c bytes) 000578 1738480058.208269 punch from 0x028ec3 to 0x0313fc (0x853a bytes) 000579 1738480058.209662 trunc from 0x0313fd to 0x02d8a8 (0x3b54 bytes) 000580 1738480058.213076 punch from 0x0203dd to 0x029acc (0x96f0 bytes) 000582 1738480058.214616 fallocate 0x0318c7 thru 0x031ded (0x527 bytes) 000583 1738480058.216119 write 0x02a93a thru 0x0353bf (0xaa86 bytes) 000584 1738480058.219356 write 0x013493 thru 0x017223 (0x3d91 bytes) 000585 1738480058.220124 punch from 0x034f75 to 0x0353bf (0x44b bytes) 000586 1738480058.225415 MIRROR_EXTEND to 2 mirrors 000588 1738480058.325270 punch from 0x01e054 to 0x02474b (0x66f8 bytes) 000589 1738480058.332318 fallocate 0x00f323 thru 0x019db4 (0xaa92 bytes) 000590 1738480058.333622 punch from 0x02f579 to 0x0353bf (0x5e47 bytes) 000591 1738480058.336035 MIRROR_EXTEND to 3 mirrors 000592 1738480058.438707 mapread 0x01c0bc thru 0x0206bb (0x4600 bytes) 000593 1738480058.447424 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000594 1738480058.501384 read 0x02dcd1 thru 0x0345e2 (0x6912 bytes) 000595 1738480058.503884 write 0x011348 thru 0x01c65b (0xb314 bytes) 000596 1738480058.515237 fallocate 0x01e4d7 thru 0x024a79 (0x65a3 bytes) 000597 1738480058.516922 write 0x0238cf thru 0x0289f4 (0x5126 bytes) 000598 1738480058.519477 mapwrite 0x01dacb thru 0x024b7f (0x70b5 bytes) 000599 1738480058.525507 fallocate 0x01a517 thru 0x01df4a (0x3a34 bytes) 000601 1738480058.528691 MIRROR_RESYNC 2 mirrors 000602 1738480058.620074 read 0x00e1c3 thru 0x015720 (0x755e bytes) 000603 1738480058.622272 write 0x014263 thru 0x014bad (0x94b bytes) 000604 1738480058.630346 mapread 0x017fb7 thru 0x01d2c1 (0x530b bytes) 000605 1738480058.633924 write 0x02b4e9 thru 0x036e00 (0xb918 bytes) 000606 1738480058.636689 trunc from 0x036e01 to 0x0003a5 (0x36a5b bytes) 000607 1738480058.645713 write 0x01c1f9 thru 0x025496 (0x929e bytes) 000608 1738480058.646661 write 0x030b0a thru 0x037d89 (0x7280 bytes) 000611 1738480058.648276 fallocate 0x03628b thru 0x03ffff (0x9d75 bytes) 000612 1738480058.649901 write 0x029642 thru 0x0322dd (0x8c9c bytes) 000613 1738480058.652457 fallocate 0x02cfa9 thru 0x03b5f9 (0xe651 bytes) 000614 1738480058.653744 punch from 0x010558 to 0x017a16 (0x74bf bytes) 000615 1738480058.654961 punch from 0x00177f to 0x00a031 (0x88b3 bytes) 000616 1738480058.656361 mapread 0x01b657 thru 0x0283d6 (0xcd80 bytes) 000617 1738480058.658981 fallocate 0x018409 thru 0x01d4d9 (0x50d1 bytes) 000618 1738480058.660553 punch from 0x00768d to 0x012d69 (0xb6dd bytes) 000619 1738480058.663226 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000620 1738480058.696553 mapread 0x021f04 thru 0x029b91 (0x7c8e bytes) 000624 1738480058.702206 mapwrite 0x0222f4 thru 0x032201 (0xff0e bytes) 000625 1738480058.711816 mapread 0x0007c1 thru 0x0049af (0x41ef bytes) 000626 1738480058.715436 trunc from 0x040000 to 0x026b42 (0x194bd bytes) 000627 1738480058.720231 trunc from 0x026b43 to 0x00aff5 (0x1bb4d bytes) 000628 1738480058.723493 mapwrite 0x035eec thru 0x03c693 (0x67a8 bytes) 000630 1738480058.734791 fallocate 0x006ab8 thru 0x00af44 (0x448d bytes) 000631 1738480058.736384 fallocate 0x00234a thru 0x005bde (0x3895 bytes) 000632 1738480058.738185 fallocate 0x0376c2 thru 0x03abf4 (0x3533 bytes) 000633 1738480058.739738 mapwrite 0x00735f thru 0x00a6cc (0x336e bytes) 000634 1738480058.744089 fallocate 0x01263c thru 0x01a148 (0x7b0d bytes) 000635 1738480058.745662 trunc from 0x03c694 to 0x002fcd (0x396c6 bytes) 000636 1738480058.749533 trunc from 0x002fce to 0x036279 (0x332ac bytes) 000637 1738480058.752931 read 0x026305 thru 0x033c74 (0xd970 bytes) 000640 1738480058.755782 mapread 0x01a92e thru 0x01f816 (0x4ee9 bytes) 000641 1738480058.758349 punch from 0x023cc8 to 0x033072 (0xf3ab bytes) 000643 1738480058.759962 trunc from 0x03627a to 0x003e2a (0x3244f bytes) 000644 1738480058.764784 MIRROR_EXTEND to 2 mirrors 000646 1738480058.853568 trunc from 0x003e2b to 0x026b7b (0x22d51 bytes) 000647 1738480058.865474 write 0x03acff thru 0x03ffff (0x5301 bytes) 000648 1738480058.866246 fallocate 0x017d62 thru 0x021a77 (0x9d16 bytes) 000650 1738480058.869085 MIRROR_EXTEND to 3 mirrors 000651 1738480058.971934 mapread 0x028959 thru 0x034f22 (0xc5ca bytes) 000652 1738480058.981488 fallocate 0x00a850 thru 0x011566 (0x6d17 bytes) 000653 1738480058.990015 trunc from 0x040000 to 0x01c6a5 (0x2395a bytes) 000654 1738480058.999023 read 0x009836 thru 0x00c8fe (0x30c9 bytes) 000655 1738480059.001921 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000656 1738480059.052233 trunc from 0x01c6a6 to 0x01f8c6 (0x3221 bytes) 000657 1738480059.056719 write 0x03f817 thru 0x03ffff (0x7e9 bytes) 000658 1738480059.057759 punch from 0x0083ae to 0x0104b2 (0x8105 bytes) 000659 1738480059.059287 mapread 0x01b779 thru 0x01e687 (0x2f0f bytes) 000662 1738480059.065083 MIRROR_RESYNC 2 mirrors 000663 1738480059.176097 MIRROR_RESYNC 2 mirrors 000664 1738480059.208737 fallocate 0x00c4c2 thru 0x019091 (0xcbd0 bytes) 000665 1738480059.216374 trunc from 0x040000 to 0x00cf95 (0x3306a bytes) 000667 1738480059.224431 trunc from 0x00cf96 to 0x032f0c (0x25f77 bytes) 000669 1738480059.227787 write 0x0149df thru 0x02195c (0xcf7e bytes) 000671 1738480059.231230 mapread 0x026f20 thru 0x02d90e (0x69ef bytes) 000672 1738480059.233965 punch from 0x011d47 to 0x0176d7 (0x5991 bytes) 000673 1738480059.237507 fallocate 0x02b654 thru 0x037393 (0xbd40 bytes) 000675 1738480059.239016 fallocate 0x03ca63 thru 0x03ffff (0x359d bytes) 000679 1738480059.241075 write 0x00c559 thru 0x01319d (0x6c45 bytes) 000680 1738480059.243608 punch from 0x02c5d1 to 0x03048d (0x3ebd bytes) 000682 1738480059.246784 MIRROR_RESYNC 2 mirrors 000683 1738480059.380195 write 0x0194eb thru 0x023234 (0x9d4a bytes) 000684 1738480059.389820 write 0x003a24 thru 0x005e6c (0x2449 bytes) 000686 1738480059.392582 punch from 0x018fc1 to 0x02504a (0xc08a bytes) 000687 1738480059.395725 write 0x0170a2 thru 0x01bca0 (0x4bff bytes) 000688 1738480059.397821 mapread 0x01630b thru 0x01d9b2 (0x76a8 bytes) 000689 1738480059.398563 punch from 0x0151cd to 0x01feb5 (0xace9 bytes) 000690 1738480059.401451 mapwrite 0x00a9f2 thru 0x0127ab (0x7dba bytes) 000691 1738480059.407576 trunc from 0x032f0d to 0x018708 (0x1a804 bytes) 000692 1738480059.415487 mapread 0x008d85 thru 0x00f2c8 (0x6544 bytes) 000693 1738480059.415939 mapread 0x004f5b thru 0x00dcae (0x8d54 bytes) 000698 1738480059.417230 mapread 0x00ca5e thru 0x00e86d (0x1e10 bytes) 000699 1738480059.417582 mapwrite 0x021291 thru 0x02dc5c (0xc9cc bytes) 000701 1738480059.429704 mapwrite 0x035175 thru 0x038fb4 (0x3e40 bytes) 000702 1738480059.439416 fallocate 0x01b019 thru 0x0228dd (0x78c5 bytes) 000703 1738480059.440964 mapread 0x01be3b thru 0x025e06 (0x9fcc bytes) 000704 1738480059.441501 write 0x033206 thru 0x034504 (0x12ff bytes) 000707 1738480059.442329 write 0x01b537 thru 0x01c928 (0x13f2 bytes) 000708 1738480059.442698 read 0x000c57 thru 0x00922f (0x85d9 bytes) 000709 1738480059.444149 write 0x010d55 thru 0x01ee64 (0xe110 bytes) 000710 1738480059.444860 write 0x016d2e thru 0x02508c (0xe35f bytes) 000711 1738480059.445614 mapread 0x01290c thru 0x01bdb2 (0x94a7 bytes) 000712 1738480059.446105 mapwrite 0x007532 thru 0x00e6b8 (0x7187 bytes) 000714 1738480059.452330 MIRROR_EXTEND to 3 mirrors 000715 1738480059.548164 fallocate 0x02081c thru 0x026ba6 (0x638b bytes) 000716 1738480059.556162 MIRROR_RESYNC 3 mirrors 000718 1738480059.686185 MIRROR_RESYNC 3 mirrors 000719 1738480059.720471 fallocate 0x021d0a thru 0x023699 (0x1990 bytes) 000721 1738480059.729576 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000722 1738480059.771776 mapwrite 0x037c60 thru 0x03ffff (0x83a0 bytes) 000723 1738480059.788302 write 0x00a370 thru 0x011b26 (0x77b7 bytes) 000726 1738480059.790892 trunc from 0x040000 to 0x011e8e (0x2e171 bytes) 000729 1738480059.795758 MIRROR_RESYNC 2 mirrors 000730 1738480059.877457 punch from 0x003e32 to 0x00e8eb (0xaaba bytes) 000731 1738480059.884655 write 0x029556 thru 0x03593c (0xc3e7 bytes) 000732 1738480059.885533 punch from 0x02ab6f to 0x03593c (0xadce bytes) 000733 1738480059.888833 trunc from 0x03593d to 0x015527 (0x20415 bytes) 000734 1738480059.896625 mapwrite 0x00b928 thru 0x01a2b0 (0xe989 bytes) 000735 1738480059.912360 mapread 0x018dc7 thru 0x01a2b0 (0x14ea bytes) 000736 1738480059.912763 mapwrite 0x01cc6f thru 0x020622 (0x39b4 bytes) 000737 1738480059.923110 MIRROR_EXTEND to 3 mirrors 000738 1738480060.016156 mapwrite 0x02f7ef thru 0x03e71d (0xef2f bytes) 000740 1738480060.043247 trunc from 0x03e71e to 0x01189d (0x2ce80 bytes) 000741 1738480060.048089 mapwrite 0x005b2a thru 0x00fa88 (0x9f5f bytes) 000744 1738480060.056936 mapread 0x002871 thru 0x00ee98 (0xc628 bytes) 000745 1738480060.058771 MIRROR_RESYNC 3 mirrors 000746 1738480060.166392 trunc from 0x01189e to 0x00311e (0xe77f bytes) 000747 1738480060.178365 mapread 0x000f44 thru 0x00311e (0x21db bytes) 000748 1738480060.183230 mapread 0x0029f3 thru 0x00311e (0x72c bytes) 000749 1738480060.183788 write 0x01ee5f thru 0x023f36 (0x50d8 bytes) 000751 1738480060.184561 trunc from 0x023f37 to 0x014e96 (0xf0a0 bytes) 000755 1738480060.189467 punch from 0x00057c to 0x009b61 (0x95e6 bytes) 000756 1738480060.190881 trunc from 0x014e97 to 0x026922 (0x11a8c bytes) 000757 1738480060.194325 punch from 0x0085fe to 0x00c1d8 (0x3bdb bytes) 000758 1738480060.195805 fallocate 0x031b59 thru 0x03a6a5 (0x8b4d bytes) 000761 1738480060.197872 punch from 0x01aa09 to 0x01f904 (0x4efc bytes) 000762 1738480060.199121 mapwrite 0x00701e thru 0x012126 (0xb109 bytes) 000763 1738480060.209651 fallocate 0x014be4 thru 0x0200e4 (0xb501 bytes) 000764 1738480060.211102 mapread 0x0115b9 thru 0x01788f (0x62d7 bytes) 000765 1738480060.212697 MIRROR_EXTEND to 4 mirrors 000767 1738480060.308443 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000768 1738480060.349369 read 0x02db51 thru 0x032753 (0x4c03 bytes) 000769 1738480060.351484 punch from 0x023146 to 0x026b1f (0x39da bytes) 000770 1738480060.354555 mapwrite 0x0372a1 thru 0x03e444 (0x71a4 bytes) 000771 1738480060.371746 mapwrite 0x01cbe0 thru 0x024dba (0x81db bytes) 000772 1738480060.379652 MIRROR_RESYNC 3 mirrors 000775 1738480060.512257 mapwrite 0x01fdbf thru 0x02fd31 (0xff73 bytes) 000777 1738480060.543908 mapread 0x03380b thru 0x03c3db (0x8bd1 bytes) 000778 1738480060.547116 read 0x020f1c thru 0x02b2c1 (0xa3a6 bytes) 000779 1738480060.547415 trunc from 0x03e445 to 0x034ae5 (0x995f bytes) 000780 1738480060.554618 punch from 0x00987a to 0x00cb99 (0x3320 bytes) 000781 1738480060.556079 trunc from 0x034ae6 to 0x0028db (0x3220a bytes) 000784 1738480060.559967 mapread 0x000c10 thru 0x0028db (0x1ccc bytes) 000785 1738480060.562860 mapread 0x0016fb thru 0x0028db (0x11e1 bytes) 000786 1738480060.563351 mapread 0x002191 thru 0x0028db (0x74b bytes) 000787 1738480060.563889 fallocate 0x022407 thru 0x025c0f (0x3809 bytes) 000788 1738480060.565367 trunc from 0x0028dc to 0x012228 (0xf94d bytes) 000791 1738480060.568866 write 0x02da5c thru 0x033d7a (0x631f bytes) 000792 1738480060.569492 read 0x02b00e thru 0x033d7a (0x8d6d bytes) 000794 1738480060.572374 read 0x028388 thru 0x0337b1 (0xb42a bytes) 000795 1738480060.572665 write 0x031c78 thru 0x037380 (0x5709 bytes) 000796 1738480060.573530 read 0x014a87 thru 0x01ea8c (0xa006 bytes) 000797 1738480060.573924 mapwrite 0x03cc7e thru 0x03ffff (0x3382 bytes) 000798 1738480060.583853 fallocate 0x03f50e thru 0x03ffff (0xaf2 bytes) 000799 1738480060.585290 write 0x02b7f2 thru 0x02ccf4 (0x1503 bytes) 000800 1738480060.585904 mapread 0x026d05 thru 0x026e5e (0x15a bytes) 000801 1738480060.586305 fallocate 0x02cfb6 thru 0x03b319 (0xe364 bytes) 000803 1738480060.587803 mapwrite 0x01afd4 thru 0x029445 (0xe472 bytes) 000804 1738480060.594215 MIRROR_RESYNC 3 mirrors 000805 1738480060.702489 write 0x022809 thru 0x02420d (0x1a05 bytes) 000806 1738480060.710980 punch from 0x02008a to 0x02ee1f (0xed96 bytes) 000807 1738480060.713444 trunc from 0x040000 to 0x02edef (0x11210 bytes) 000809 1738480060.721497 punch from 0x01fa7a to 0x02cea3 (0xd42a bytes) 000810 1738480060.722866 trunc from 0x02edf0 to 0x0000af (0x2ed40 bytes) 000811 1738480060.726260 read 0x00004a thru 0x0000af (0x66 bytes) 000812 1738480060.727641 punch from 0x000032 to 0x0000af (0x7e bytes) 000813 1738480060.728748 mapwrite 0x03f1a6 thru 0x03ffff (0xe5a bytes) 000814 1738480060.737753 punch from 0x0086ed to 0x00f06e (0x6982 bytes) 000815 1738480060.738979 trunc from 0x040000 to 0x0288d0 (0x1772f bytes) 000816 1738480060.742321 fallocate 0x004553 thru 0x011fa9 (0xda57 bytes) 000817 1738480060.743671 punch from 0x01c557 to 0x0288d0 (0xc37a bytes) 000819 1738480060.745088 trunc from 0x0288d1 to 0x01f3be (0x9512 bytes) 000820 1738480060.748552 write 0x03bc07 thru 0x03ffff (0x43f9 bytes) 000821 1738480060.750604 MIRROR_EXTEND to 4 mirrors 000822 1738480060.842954 mapread 0x0069dd thru 0x01458e (0xdbb2 bytes) 000823 1738480060.852072 read 0x02ee29 thru 0x03a0e0 (0xb2b8 bytes) 000824 1738480060.855609 mapread 0x016a6d thru 0x021775 (0xad09 bytes) 000825 1738480060.860810 fallocate 0x0344dd thru 0x0363b7 (0x1edb bytes) 000826 1738480060.868514 mapwrite 0x011ffe thru 0x01e074 (0xc077 bytes) 000828 1738480060.878682 mapread 0x011ac2 thru 0x02002d (0xe56c bytes) 000829 1738480060.879238 punch from 0x013056 to 0x0181a5 (0x5150 bytes) 000830 1738480060.880503 mapread 0x03d0b6 thru 0x03ffff (0x2f4a bytes) 000831 1738480060.883721 MIRROR_EXTEND to 5 mirrors 000832 1738480060.978389 trunc from 0x040000 to 0x02a4bf (0x15b40 bytes) 000833 1738480060.992228 mapwrite 0x02cf13 thru 0x035ca9 (0x8d97 bytes) 000834 1738480061.006322 read 0x01204b thru 0x013e4e (0x1e04 bytes) 000835 1738480061.007942 write 0x0349aa thru 0x03ffff (0xb656 bytes) 000836 1738480061.009130 trunc from 0x040000 to 0x029788 (0x16877 bytes) 000837 1738480061.012870 mapread 0x02110e thru 0x02755a (0x644d bytes) 000838 1738480061.016179 MIRROR_RESYNC 5 mirrors 000839 1738480061.200096 read 0x022da0 thru 0x029788 (0x69e9 bytes) 000840 1738480061.202371 mapwrite 0x010d61 thru 0x01e34a (0xd5ea bytes) 000841 1738480061.231653 fallocate 0x011f1c thru 0x017c8f (0x5d74 bytes) 000842 1738480061.232910 trunc from 0x029789 to 0x018163 (0x11625 bytes) 000843 1738480061.241389 MIRROR_EXTEND to 6 mirrors 000844 1738480061.325077 mapread 0x000acb thru 0x010054 (0xf58a bytes) 000845 1738480061.336200 MIRROR_EXTEND to 7 mirrors 000846 1738480061.414963 trunc from 0x018164 to 0x0262d8 (0xe175 bytes) 000847 1738480061.429071 fallocate 0x03a1d9 thru 0x03ffff (0x5e27 bytes) 000848 1738480061.430516 write 0x020363 thru 0x02e45d (0xe0fb bytes) 000849 1738480061.432699 read 0x01ef4e thru 0x02827e (0x9331 bytes) 000850 1738480061.434255 mapwrite 0x002e84 thru 0x0118ab (0xea28 bytes) 000851 1738480061.445584 punch from 0x01ae89 to 0x020103 (0x527b bytes) 000852 1738480061.446952 trunc from 0x02e45e to 0x013792 (0x1accb bytes) 000853 1738480061.452611 MIRROR_RESYNC 7 mirrors 000854 1738480061.606412 mapwrite 0x0334a5 thru 0x03ac01 (0x775d bytes) 000855 1738480061.625987 mapwrite 0x02a170 thru 0x036656 (0xc4e7 bytes) 000856 1738480061.632465 mapread 0x02c3d0 thru 0x037b6f (0xb7a0 bytes) 000857 1738480061.633095 fallocate 0x00ea9b thru 0x0164d1 (0x7a37 bytes) 000858 1738480061.634404 mapread 0x01b245 thru 0x01e96c (0x3728 bytes) 000860 1738480061.636834 trunc from 0x03ac02 to 0x0007c5 (0x3a43c bytes) 000861 1738480061.641777 read 0x00000b thru 0x0007c5 (0x7bb bytes) 000862 1738480061.643052 punch from 0x00055a to 0x0007c5 (0x26c bytes) 000864 1738480061.644533 fallocate 0x02a336 thru 0x02de99 (0x3b64 bytes) 000865 1738480061.645964 punch from 0x00067f to 0x0007c5 (0x147 bytes) 000867 1738480061.647370 write 0x0371cf thru 0x038033 (0xe65 bytes) 000869 1738480061.648093 fallocate 0x011800 thru 0x01ebb7 (0xd3b8 bytes) 000870 1738480061.649446 punch from 0x00e6fa to 0x011ea5 (0x37ac bytes) 000871 1738480061.650517 read 0x025a0c thru 0x03185d (0xbe52 bytes) 000872 1738480061.653315 mapwrite 0x01aafc thru 0x01cc9b (0x21a0 bytes) 000873 1738480061.659315 MIRROR_EXTEND to 8 mirrors 000875 1738480061.751024 mapread 0x02a073 thru 0x02a3b5 (0x343 bytes) 000876 1738480061.755450 punch from 0x01b007 to 0x0201db (0x51d5 bytes) 000877 1738480061.763513 punch from 0x02e2d1 to 0x031968 (0x3698 bytes) 000878 1738480061.764694 mapwrite 0x008b46 thru 0x00f17d (0x6638 bytes) 000879 1738480061.772803 fallocate 0x034171 thru 0x03e705 (0xa595 bytes) 000880 1738480061.775415 MIRROR_RESYNC 8 mirrors 000881 1738480062.056844 read 0x029269 thru 0x038033 (0xedcb bytes) 000882 1738480062.059245 trunc from 0x038034 to 0x00dc2f (0x2a404 bytes) 000883 1738480062.071300 mapwrite 0x027dde thru 0x0313e9 (0x960c bytes) 000884 1738480062.084788 trunc from 0x0313ea to 0x007c1c (0x297cd bytes) 000885 1738480062.088368 trunc from 0x007c1d to 0x013e91 (0xc275 bytes) 000886 1738480062.091959 write 0x022594 thru 0x02929c (0x6d09 bytes) 000887 1738480062.092724 fallocate 0x03a5b5 thru 0x03ffff (0x5a4b bytes) 000888 1738480062.094142 write 0x00f00e thru 0x0191bd (0xa1b0 bytes) 000889 1738480062.097575 punch from 0x00172d to 0x00ed77 (0xd64b bytes) 000890 1738480062.099010 mapread 0x016efd thru 0x021e3f (0xaf43 bytes) 000891 1738480062.101433 punch from 0x00d8c7 to 0x01b428 (0xdb62 bytes) 000892 1738480062.104776 mapread 0x00a832 thru 0x017f85 (0xd754 bytes) 000893 1738480062.108728 punch from 0x0110d9 to 0x01416b (0x3093 bytes) 000894 1738480062.110121 mapread 0x00a039 thru 0x0188ea (0xe8b2 bytes) 000895 1738480062.111770 mapwrite 0x01124a thru 0x017e2f (0x6be6 bytes) 000896 1738480062.117229 mapwrite 0x0081f6 thru 0x012150 (0x9f5b bytes) 000897 1738480062.123594 MIRROR_EXTEND to 9 mirrors 000898 1738480062.217585 mapwrite 0x02baea thru 0x034eac (0x93c3 bytes) 000899 1738480062.241613 MIRROR_EXTEND to 10 mirrors 000901 1738480062.340401 fallocate 0x007727 thru 0x01010e (0x89e8 bytes) 000902 1738480062.348464 trunc from 0x034ead to 0x039135 (0x4289 bytes) 000904 1738480062.357598 punch from 0x0075bb to 0x01449e (0xcee4 bytes) 000905 1738480062.359024 punch from 0x02c83e to 0x02efb0 (0x2773 bytes) 000906 1738480062.360246 fallocate 0x01df3d thru 0x021229 (0x32ed bytes) 000907 1738480062.361720 mapread 0x00ba35 thru 0x016009 (0xa5d5 bytes) 000908 1738480062.367839 punch from 0x02590e to 0x0265ff (0xcf2 bytes) 000909 1738480062.369238 mapwrite 0x02a0a9 thru 0x039792 (0xf6ea bytes) 000910 1738480062.383392 write 0x032333 thru 0x03c480 (0xa14e bytes) 000912 1738480062.385718 MIRROR_EXTEND to 11 mirrors 000913 1738480062.489141 MIRROR_EXTEND to 12 mirrors 000914 1738480062.589278 fallocate 0x019937 thru 0x02750c (0xdbd6 bytes) 000915 1738480062.598414 fallocate 0x007fee thru 0x01213d (0xa150 bytes) 000916 1738480062.599662 mapwrite 0x028dcc thru 0x029aa5 (0xcda bytes) 000917 1738480062.605551 fallocate 0x000d9c thru 0x00e451 (0xd6b6 bytes) 000919 1738480062.607162 fallocate 0x035e72 thru 0x03ffff (0xa18e bytes) 000920 1738480062.608493 mapwrite 0x002553 thru 0x002860 (0x30e bytes) 000921 1738480062.614886 MIRROR_EXTEND to 13 mirrors 000922 1738480062.719256 write 0x01713f thru 0x01bb36 (0x49f8 bytes) 000923 1738480062.730534 read 0x008f8f thru 0x010cb6 (0x7d28 bytes) 000925 1738480062.733171 trunc from 0x040000 to 0x01846f (0x27b90 bytes) 000926 1738480062.741907 read 0x001a2a thru 0x011027 (0xf5fe bytes) 000927 1738480062.744289 read 0x008caf thru 0x0180c6 (0xf418 bytes) 000928 1738480062.744677 read 0x00cc07 thru 0x01846f (0xb869 bytes) 000929 1738480062.745053 read 0x00ec91 thru 0x0146ed (0x5a5d bytes) 000930 1738480062.745343 punch from 0x012879 to 0x01846f (0x5bf7 bytes) 000931 1738480062.748196 mapread 0x00cab1 thru 0x01846f (0xb9bf bytes) 000934 1738480062.752061 MIRROR_SPLIT mirror 3 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000935 1738480062.797161 mapread 0x001159 thru 0x0073a2 (0x624a bytes) 000936 1738480062.801623 write 0x000191 thru 0x00bb8c (0xb9fc bytes) 000937 1738480062.804361 write 0x01d872 thru 0x028ab9 (0xb248 bytes) 000938 1738480062.805316 punch from 0x02719b to 0x028ab9 (0x191f bytes) 000939 1738480062.810457 MIRROR_RESYNC 12 mirrors 000941 1738480063.166710 mapread 0x01f0d4 thru 0x01fba2 (0xacf bytes) 000942 1738480063.168283 write 0x016376 thru 0x01b548 (0x51d3 bytes) 000943 1738480063.178613 fallocate 0x03c284 thru 0x03ffff (0x3d7c bytes) 000946 1738480063.180761 mapread 0x032b9e thru 0x038e60 (0x62c3 bytes) 000947 1738480063.183137 mapwrite 0x01b177 thru 0x01d97c (0x2806 bytes) 000948 1738480063.189830 write 0x036844 thru 0x038af2 (0x22af bytes) 000949 1738480063.190386 punch from 0x029751 to 0x036a81 (0xd331 bytes) 000951 1738480063.193325 trunc from 0x040000 to 0x00caa6 (0x33559 bytes) 000952 1738480063.198141 read 0x0033a0 thru 0x00caa6 (0x9707 bytes) 000953 1738480063.200246 mapread 0x0005c8 thru 0x0071e2 (0x6c1b bytes) 000954 1738480063.203092 write 0x00cfa6 thru 0x01820b (0xb266 bytes) 000956 1738480063.204123 read 0x00eba8 thru 0x01820b (0x9664 bytes) 000957 1738480063.204523 trunc from 0x01820c to 0x0396e2 (0x214d7 bytes) 000958 1738480063.208146 read 0x00b53e thru 0x019abb (0xe57e bytes) 000959 1738480063.210645 read 0x00e072 thru 0x01d7f2 (0xf781 bytes) 000960 1738480063.210933 fallocate 0x019552 thru 0x01f4f9 (0x5fa8 bytes) 000962 1738480063.213476 MIRROR_EXTEND to 13 mirrors 000963 1738480063.309363 fallocate 0x03f545 thru 0x03ffff (0xabb bytes) 000965 1738480063.317628 read 0x01b7ca thru 0x029a0d (0xe244 bytes) 000966 1738480063.320424 fallocate 0x02f028 thru 0x033b23 (0x4afc bytes) 000968 1738480063.322268 write 0x019f58 thru 0x028293 (0xe33c bytes) 000969 1738480063.325445 write 0x03554f thru 0x03916f (0x3c21 bytes) 000970 1738480063.328324 mapwrite 0x0399ae thru 0x03b782 (0x1dd5 bytes) 000972 1738480063.335671 mapwrite 0x00a96c thru 0x012397 (0x7a2c bytes) 000973 1738480063.342351 write 0x0153da thru 0x02299d (0xd5c4 bytes) 000974 1738480063.346272 MIRROR_EXTEND to 14 mirrors 000975 1738480063.440759 write 0x00de7f thru 0x01b116 (0xd298 bytes) 000980 1738480063.452827 read 0x003ecf thru 0x00d90b (0x9a3d bytes) 000981 1738480063.455023 trunc from 0x040000 to 0x01c6e7 (0x23918 bytes) 000982 1738480063.464280 MIRROR_EXTEND to 15 mirrors 000984 1738480063.542145 write 0x01dd6b thru 0x025cba (0x7f50 bytes) 000985 1738480063.551228 write 0x023c9e thru 0x02e5a8 (0xa90b bytes) 000988 1738480063.553699 MIRROR_RESYNC 15 mirrors 000989 1738480063.777575 mapread 0x015ad1 thru 0x01f0d3 (0x9603 bytes) 000990 1738480063.783617 mapread 0x019778 thru 0x026a4d (0xd2d6 bytes) 000991 1738480063.785605 write 0x0328a8 thru 0x038469 (0x5bc2 bytes) 000992 1738480063.794431 trunc from 0x03846a to 0x00f2b8 (0x291b1 bytes) 000993 1738480063.802582 mapread 0x00c29f thru 0x00f2b8 (0x301a bytes) 000994 1738480063.805202 write 0x025cf4 thru 0x031632 (0xb93f bytes) 000996 1738480063.806414 punch from 0x003ba3 to 0x0064e8 (0x2946 bytes) 000997 1738480063.807674 mapread 0x019200 thru 0x023318 (0xa119 bytes) 000998 1738480063.811623 mapwrite 0x02c0df thru 0x036a3f (0xa961 bytes) 000999 1738480063.822741 punch from 0x02a570 to 0x035fd4 (0xba65 bytes) 001000 1738480063.824419 mapread 0x02fedd thru 0x036a3f (0x6b63 bytes) All operations completed A-OK! PASS 70a (11s) == sanity-flr test 71: check mirror extend parent fid ==== 02:07:44 (1738480064) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000404:0xb69:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (3s) == sanity-flr test 200a: stress test ===================== 02:07:47 (1738480067) Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre2 Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3332416 .. Extending file size to 5401120 .. failed Extending file size to 8637856 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 4429568 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8730720 .. Extending file size to 6303392 .. done Extending file size to 6905152 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8101440 .. Extending file size to 8630432 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 8732224 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9090944 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8892992 .. Extending file size to 9212128 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9121056 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 5122 5123 5124 5126 5129 192.168.201.158@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg158-client.virtnet /mnt/lustre2 (opts:) 192.168.201.158@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg158-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (63s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 02:08:50 (1738480130) Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre2 Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8182528 .. Extending file size to 1170176 .. done Extending file size to 8051712 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8987232 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8409632 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8936000 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9056640 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9215232 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9061504 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9264480 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Waiting 9597 9598 9600 9601 9604 192.168.201.158@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg158-client.virtnet /mnt/lustre2 (opts:) 192.168.201.158@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg158-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 02:09:53 (1738480193) Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2888 1284800 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3400 1284288 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 14828 3592192 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 22916 3584104 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37744 7176296 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2888 1284800 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 3400 1284288 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 14828 3592192 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 22916 3584104 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37744 7176296 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0461034 s, 45.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0440157 s, 47.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.158@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg158-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 02:09:57 (1738480197) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0240563 s, 87.2 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 02:09:59 (1738480199) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x66:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbf:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbf:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc0:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x9f:0x0] } PASS 203 (1s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 02:10:01 (1738480201) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0172126 s, 122 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1080:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1081:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1082:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (1s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 02:10:02 (1738480202) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0217495 s, 96.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa1:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 02:10:04 (1738480204) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0183628 s, 114 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa3:0x0] } PASS 204c (22s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 02:10:26 (1738480226) Waiting for MDT destroys to complete oleg158-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg158-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg158-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0148716 s, 70.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc5:0x0] } PASS 204d (34s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 02:11:00 (1738480260) Waiting for MDT destroys to complete oleg158-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 oleg158-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg158-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0198695 s, 106 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (36s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 02:11:36 (1738480296) Waiting for MDT destroys to complete oleg158-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg158-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg158-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg158-server: Pool lustre.test_204f created Adding targets to pool oleg158-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg158-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0196557 s, 107 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg158-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg158-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg158-server: Pool lustre.test_204f destroyed PASS 204f (45s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 02:12:21 (1738480341) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 02:12:23 (1738480343) PASS 205b (1s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 02:12:24 (1738480344) oleg158-server: Pool lustre.test_206 created Adding targets to pool oleg158-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg158-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg158-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg158-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg158-server: Pool lustre.test_206 destroyed PASS 206 (12s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 02:12:36 (1738480356) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0193036 s, 54.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0078686 s, 133 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xde03b064 Mirror 3: 0xde03b064 PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 02:12:38 (1738480358) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 02:12:39 (1738480359) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 02:12:40 (1738480360) Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0068701 s, 0.6 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1cc0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1920000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1920000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1920000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1900000 Read this (4 bytes): BBBB 192.168.201.158@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg158-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 02:12:53 (1738480373) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00241862 s, 3.4 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 02:12:55 (1738480375) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100459 s, 104 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x89 lmm_fid: [0x200000405:0x89:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x89 lmm_fid: [0x200000405:0x89:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb2:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611273 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-02-02 02:12:55.000000000 -0500 Modify: 2025-02-02 02:12:55.000000000 -0500 Change: 2025-02-02 02:12:55.000000000 -0500 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 02:12:57 (1738480377) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00924499 s, 113 MB/s oleg158-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg158-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg158-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg158-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 oleg158-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg158-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg158-server: O/280000401/d18/d2: File not found by ext2_lookup PASS 210b (11s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 02:13:08 (1738480388) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0193918 s, 2.1 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00408825 s, 1.0 MB/s size after second write -rw-r--r-- 1 root root 4096 Feb 2 02:13 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x8e:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Feb 2 02:13 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 679 sec ============ 02:13:10 (1738480390) === sanity-flr: start cleanup 02:13:10 (1738480390) === === sanity-flr: finish cleanup 02:13:12 (1738480392) ===