-----============= acceptance-small: sanity-flr ============----- Wed Oct 16 10:25:09 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 6 201 44c === sanity-flr: start setup 10:25:14 (1729088714) === oleg233-client.virtnet: executing check_config_client /mnt/lustre oleg233-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg233-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b63b1800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b63b1800.idle_timeout=debug disable quota as required oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 10:25:25 (1729088725) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f7605] == sanity-flr test 0a: lfs mirror create with -N option == 10:25:26 (1729088726) PASS 0a (5s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:25:31 (1729088731) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:25:32 (1729088732) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 10:25:33 (1729088733) PASS 0d (5s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:25:38 (1729088738) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:25:39 (1729088739) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 10:25:40 (1729088740) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b63b1800.stats=clear osc.lustre-OST0001-osc-ffff8800b63b1800.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:25:43 (1729088743) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:25:45 (1729088745) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:25:47 (1729088747) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:25:48 (1729088748) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 10:25:51 (1729088751) PASS 2 (1s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:25:53 (1729088753) PASS 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:25:55 (1729088755) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (1s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:25:57 (1729088757) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (1s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:25:59 (1729088759) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:26:01 (1729088761) 14+0 records in 14+0 records out 14680064 bytes (15 MB) copied, 0.419693 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0343986 s, 30.5 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:26:04 (1729088764) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0274079 s, 115 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:26:06 (1729088766) Stopping /mnt/lustre-ost1 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b63b1800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b63b1800.ost_server_uuid in DISCONN state after 0 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg233-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b63b1800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b63b1800.ost_server_uuid in DISCONN state after 3 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg233-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (26s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:26:31 (1729088791) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.133027 s, 63.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b63b1800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b63b1800.ost_server_uuid in DISCONN state after 3 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg233-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (14s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:26:45 (1729088805) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b63b1800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b63b1800.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg233-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b63b1800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b63b1800.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg233-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (26s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:27:11 (1729088831) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.986216 s, 53.2 MB/s -rw-r--r-- 1 root root 52428800 Oct 16 10:27 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.303s user 0m0.000s sys 0m0.120s Stopping /mnt/lustre-ost1 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b63b1800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b63b1800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.288s user 0m0.000s sys 0m0.121s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg233-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b63b1800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b63b1800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.283s user 0m0.000s sys 0m0.145s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg233-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg233-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b63b1800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b63b1800.ost_server_uuid in FULL state after 0 sec PASS 33b (27s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:27:38 (1729088858) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:27:39 (1729088859) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:27:40 (1729088860) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 10:27:41 (1729088861) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 10:27:43 (1729088863) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.167531 s, 75.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:27:47 (1729088867) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0099518 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109538 s, 95.7 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:27:49 (1729088869) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:27:52 (1729088872) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000898526 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00198893 s, 515 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000640994 s, 1.6 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000979748 s, 1.0 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 10:27:55 (1729088875) create test files with size 15749183 .. a46310229dbbb01c5a2c3dc55e03272a - 98e180fd4d3269a24310e76dca5fe90a - 6535f91be8554bebe85d50c2493d4f73 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg233-client.virtnet /mnt/lustre (opts:) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre PASS 37 (7s) == sanity-flr test 38: resync ============================ 10:28:01 (1729088881) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (4s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:28:05 (1729088885) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:28:07 (1729088887) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.023118 s, 45.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x41:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106374 s, 98.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x42:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 10:28:09 (1729088889) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0277124 s, 75.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0940868 s, 44.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0495338 s, 42.3 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 10:28:12 (1729088892) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 10:28:13 (1729088893) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:28:14 (1729088894) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122915 s, 85.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 10:28:16 (1729088896) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0803019 s, 39.2 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x10:0x0]: Invalid argument (22) PASS 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:28:19 (1729088899) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4b:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x49:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x15:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:28:21 (1729088901) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0831503 s, 126 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg233-client.virtnet /mnt/lustre (opts:) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (3s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:28:24 (1729088904) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff88012d48e800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.081976 s, 128 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg233-client.virtnet /mnt/lustre (opts:) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800b58c5000.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:28:28 (1729088908) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (11s) == sanity-flr test 46: Verify setstripe --copy option ==== 10:28:39 (1729088919) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (8s) == sanity-flr test 47: Verify mirror obj alloc =========== 10:28:47 (1729088927) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 10:28:48 (1729088928) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0766048 s, 41.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0561427 s, 56.0 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: f41f4f7048f4b11f1351c1e30c99a219 - mirror 1 checksum: ac0e80cf010626e5f6e6be92bef2d7d7 - mirror 2 checksum: f41f4f7048f4b11f1351c1e30c99a219 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Oct 16 10:28:50 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Oct 16 10:28:50 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: f41f4f7048f4b11f1351c1e30c99a219 - mirror 1 checksum: ac0e80cf010626e5f6e6be92bef2d7d7 - mirror 2 checksum: ac0e80cf010626e5f6e6be92bef2d7d7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Oct 16 10:28:50 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Oct 16 10:28:50 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } PASS 48 (3s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 10:28:51 (1729088931) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0158446 s, 66.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 529408.. 530431: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:28:53 (1729088933) Failing mds1 on oleg233-server Stopping /mnt/lustre-mds1 (opts:) on oleg233-server 10:29:01 (1729088941) shut down Failover mds1 to oleg233-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-MDT0000 10:29:14 (1729088954) targets are mounted 10:29:14 (1729088954) facet_failover done oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.010823 s, 96.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x57:0x0] } PASS 50A (29s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 10:29:22 (1729088962) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00874174 s, 2.3 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00536653 s, 3.8 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 10:29:24 (1729088964) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.011903 s, 1.7 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00367002 s, 5.6 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:29:26 (1729088966) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00247309 s, 6.6 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 10:29:28 (1729088968) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0116182 s, 1.8 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:29:30 (1729088970) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00596504 s, 687 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0125014 s, 328 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:29:33 (1729088973) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) PASS 60b (5s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:29:38 (1729088978) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 10:29:56 (1729088996) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x48:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 10:30:14 (1729089014) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 10:30:31 (1729089031) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 10:30:34 (1729089034) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 26118 mirror split pid 26119 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4827: 26118 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4827: 26119 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (61s) == sanity-flr test 70a: flr mode fsx test ================ 10:31:35 (1729089095) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 9096 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero length punch hole skipping zero size read 000003 1729089096.499287 write 0x034ed1 thru 0x03ffff (0xb12f bytes) 000005 1729089096.503752 MIRROR_EXTEND to 2 mirrors fallocating to largest ever: 0x40000 000007 1729089096.680419 fallocate 0x03e0bc thru 0x03ffff (0x1f44 bytes) 000008 1729089096.696757 punch from 0x014ea5 to 0x01c32e (0x748a bytes) 000009 1729089096.700717 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x776:0x0]: Invalid argument (22) 000010 1729089096.751491 fallocate 0x01f51f thru 0x023654 (0x4136 bytes) 000011 1729089096.753937 mapwrite 0x03f279 thru 0x03ffff (0xd87 bytes) 000012 1729089096.766535 fallocate 0x017fb1 thru 0x021478 (0x94c8 bytes) 000013 1729089096.770518 mapread 0x02b091 thru 0x03866d (0xd5dd bytes) 000014 1729089096.784117 write 0x016658 thru 0x01a062 (0x3a0b bytes) 000015 1729089096.790601 mapwrite 0x03642f thru 0x03f2a6 (0x8e78 bytes) 000017 1729089096.801221 punch from 0x015812 to 0x01740f (0x1bfe bytes) 000018 1729089096.804739 write 0x02ddf6 thru 0x03430d (0x6518 bytes) 000019 1729089096.805673 punch from 0x03cbd6 to 0x03ffff (0x342a bytes) 000020 1729089096.809610 mapwrite 0x03931a thru 0x03b1db (0x1ec2 bytes) 000021 1729089096.819985 punch from 0x021b05 to 0x02589e (0x3d9a bytes) 000022 1729089096.821937 mapread 0x012404 thru 0x014394 (0x1f91 bytes) 000023 1729089096.822474 punch from 0x01cc42 to 0x02629a (0x9659 bytes) 000024 1729089096.824735 punch from 0x0335fd to 0x035484 (0x1e88 bytes) 000025 1729089096.828458 write 0x01c766 thru 0x022142 (0x59dd bytes) 000026 1729089096.833874 read 0x0103a5 thru 0x01c825 (0xc481 bytes) 000027 1729089096.834326 trunc from 0x040000 to 0x00e673 (0x3198c bytes) 000028 1729089096.851343 punch from 0x005fa1 to 0x00e673 (0x86d3 bytes) 000030 1729089096.854343 punch from 0x004d62 to 0x00cb13 (0x7db2 bytes) 000031 1729089096.856737 read 0x009618 thru 0x009884 (0x26d bytes) 000032 1729089096.861340 punch from 0x0067ad to 0x00b83c (0x5090 bytes) 000036 1729089096.864898 write 0x0009e6 thru 0x00a4d5 (0x9af0 bytes) 000038 1729089096.870914 trunc from 0x00e674 to 0x016508 (0x7e95 bytes) 000039 1729089096.878397 mapwrite 0x027143 thru 0x02b3c7 (0x4285 bytes) 000040 1729089096.902501 trunc from 0x02b3c8 to 0x0176cd (0x13cfa bytes) 000042 1729089096.913935 mapwrite 0x033b99 thru 0x03ffff (0xc467 bytes) 000044 1729089096.943554 trunc from 0x040000 to 0x025e56 (0x1a1a9 bytes) 000045 1729089096.950821 trunc from 0x025e57 to 0x02e39c (0x8546 bytes) 000046 1729089096.957111 write 0x033e06 thru 0x03ed38 (0xaf33 bytes) 000047 1729089096.958355 write 0x034e3a thru 0x036b73 (0x1d3a bytes) 000048 1729089096.960866 MIRROR_EXTEND to 2 mirrors 000049 1729089097.158273 trunc from 0x03ed39 to 0x03d361 (0x19d7 bytes) 000050 1729089097.180014 punch from 0x02668f to 0x031045 (0xa9b7 bytes) 000051 1729089097.182448 mapread 0x028dd0 thru 0x02b40c (0x263d bytes) 000053 1729089097.193130 write 0x0249f6 thru 0x027cf7 (0x3302 bytes) 000055 1729089097.196991 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x776:0x0]: Invalid argument (22) 000056 1729089097.253108 read 0x00c72d thru 0x00c827 (0xfb bytes) 000058 1729089097.256964 punch from 0x000e67 to 0x007ba9 (0x6d43 bytes) 000059 1729089097.259393 write 0x000b50 thru 0x00872c (0x7bdd bytes) 000060 1729089097.266492 trunc from 0x03d362 to 0x00d1ed (0x30174 bytes) 000062 1729089097.273095 mapread 0x0076bd thru 0x00d1ed (0x5b31 bytes) 000064 1729089097.279729 write 0x03c790 thru 0x03ffff (0x3870 bytes) 000065 1729089097.280540 punch from 0x035862 to 0x03d953 (0x80f2 bytes) 000068 1729089097.289490 fallocate 0x037a44 thru 0x03ffff (0x85bc bytes) 000069 1729089097.293047 write 0x00dd86 thru 0x0147d1 (0x6a4c bytes) 000070 1729089097.295806 mapwrite 0x03c6e3 thru 0x03ffff (0x391d bytes) 000071 1729089097.307032 mapwrite 0x00edb5 thru 0x01aab2 (0xbcfe bytes) 000072 1729089097.321264 fallocate 0x01be6d thru 0x020f59 (0x50ed bytes) 000073 1729089097.324337 mapread 0x0063fd thru 0x0102fb (0x9eff bytes) 000074 1729089097.325180 fallocate 0x01f452 thru 0x02b7cf (0xc37e bytes) 000075 1729089097.327631 punch from 0x00fe4d to 0x01bee1 (0xc095 bytes) 000077 1729089097.330790 mapwrite 0x010865 thru 0x016a6f (0x620b bytes) 000078 1729089097.346134 trunc from 0x040000 to 0x03dc50 (0x23af bytes) 000080 1729089097.353959 mapwrite 0x010d34 thru 0x01a2d1 (0x959e bytes) 000081 1729089097.369405 write 0x00b757 thru 0x011032 (0x58dc bytes) 000082 1729089097.373241 MIRROR_EXTEND to 2 mirrors 000084 1729089097.580172 mapwrite 0x02eb98 thru 0x03b477 (0xc8e0 bytes) 000086 1729089097.636291 MIRROR_EXTEND to 3 mirrors 000088 1729089097.814186 write 0x03ab32 thru 0x03ffff (0x54ce bytes) 000089 1729089097.831099 fallocate 0x00a15c thru 0x014d44 (0xabe9 bytes) 000090 1729089097.835265 fallocate 0x03c93b thru 0x03e0ed (0x17b3 bytes) 000093 1729089097.838980 write 0x0252b5 thru 0x031e6a (0xcbb6 bytes) 000094 1729089097.843935 punch from 0x01a87d to 0x0223f0 (0x7b74 bytes) 000096 1729089097.847652 punch from 0x039e0f to 0x03ffff (0x61f1 bytes) 000098 1729089097.857333 MIRROR_RESYNC 3 mirrors 000099 1729089098.064938 punch from 0x0024d6 to 0x00353e (0x1069 bytes) 000101 1729089098.080677 trunc from 0x040000 to 0x03de19 (0x21e6 bytes) 000102 1729089098.097364 mapread 0x01b427 thru 0x02a886 (0xf460 bytes) 000103 1729089098.111181 read 0x0237ff thru 0x02a7a6 (0x6fa8 bytes) 000104 1729089098.111609 fallocate 0x036718 thru 0x03f81a (0x9103 bytes) 000105 1729089098.114278 trunc from 0x03de1a to 0x01cc26 (0x211f3 bytes) 000106 1729089098.121231 punch from 0x00729d to 0x015d88 (0xeaec bytes) 000107 1729089098.124783 punch from 0x01b008 to 0x01cc26 (0x1c1f bytes) 000108 1729089098.127360 fallocate 0x03ace3 thru 0x03c336 (0x1654 bytes) 000109 1729089098.133627 read 0x02a684 thru 0x035b49 (0xb4c6 bytes) 000111 1729089098.139468 punch from 0x013c1d to 0x016363 (0x2747 bytes) 000112 1729089098.141719 punch from 0x01e20a to 0x02136b (0x3162 bytes) 000113 1729089098.143905 trunc from 0x03c337 to 0x034114 (0x8222 bytes) 000114 1729089098.149637 write 0x016515 thru 0x01eecc (0x89b8 bytes) 000115 1729089098.155847 MIRROR_EXTEND to 4 mirrors 000117 1729089098.350819 punch from 0x01dca6 to 0x0296e9 (0xba44 bytes) 000118 1729089098.367318 fallocate 0x024021 thru 0x028b1c (0x4afc bytes) 000119 1729089098.369830 mapread 0x015506 thru 0x0181af (0x2caa bytes) 000121 1729089098.377094 punch from 0x0293c2 to 0x033765 (0xa3a4 bytes) 000122 1729089098.379643 fallocate 0x0228c7 thru 0x02ef03 (0xc63d bytes) 000123 1729089098.382337 read 0x026310 thru 0x02666f (0x360 bytes) 000124 1729089098.385511 trunc from 0x034115 to 0x029386 (0xad8e bytes) 000125 1729089098.401943 fallocate 0x02fd0a thru 0x03df44 (0xe23b bytes) 000127 1729089098.405667 mapwrite 0x02cb11 thru 0x03a8bd (0xddad bytes) 000130 1729089098.427082 fallocate 0x007c2f thru 0x0113b0 (0x9782 bytes) 000132 1729089098.433697 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x776:0x0]: Invalid argument (22) 000133 1729089098.506109 read 0x02f5e0 thru 0x03df44 (0xe965 bytes) 000134 1729089098.510699 mapwrite 0x0341a4 thru 0x035428 (0x1285 bytes) 000136 1729089098.517694 read 0x01f8a0 thru 0x0272d1 (0x7a32 bytes) 000137 1729089098.522869 fallocate 0x00474c thru 0x00b80d (0x70c2 bytes) 000138 1729089098.527922 mapwrite 0x0370a8 thru 0x03a06c (0x2fc5 bytes) 000139 1729089098.537000 punch from 0x01fd80 to 0x028559 (0x87da bytes) 000141 1729089098.540008 write 0x0262e8 thru 0x035f00 (0xfc19 bytes) 000142 1729089098.545305 mapwrite 0x023593 thru 0x028cb4 (0x5722 bytes) 000143 1729089098.560453 mapwrite 0x030378 thru 0x03995b (0x95e4 bytes) 000144 1729089098.570805 punch from 0x03cf70 to 0x03df44 (0xfd5 bytes) 000146 1729089098.574041 fallocate 0x0202a0 thru 0x02114c (0xead bytes) 000148 1729089098.577005 punch from 0x03cd4e to 0x03df44 (0x11f7 bytes) 000149 1729089098.580520 mapwrite 0x00af89 thru 0x017018 (0xc090 bytes) 000150 1729089098.590922 read 0x0061db thru 0x00ac7b (0x4aa1 bytes) 000152 1729089098.592269 trunc from 0x03df45 to 0x017438 (0x26b0c bytes) 000153 1729089098.600324 punch from 0x000a1d to 0x004a7f (0x4063 bytes) 000154 1729089098.608625 read 0x001d2a thru 0x0091d5 (0x74ac bytes) 000155 1729089098.612983 fallocate 0x0129d6 thru 0x018c17 (0x6242 bytes) 000156 1729089098.616168 fallocate 0x0045d5 thru 0x0117a2 (0xd1ce bytes) 000158 1729089098.618811 punch from 0x0050a3 to 0x01160c (0xc56a bytes) 000159 1729089098.622024 trunc from 0x018c18 to 0x028518 (0xf901 bytes) 000161 1729089098.630029 punch from 0x024eae to 0x028518 (0x366b bytes) 000165 1729089098.633826 mapwrite 0x00e762 thru 0x01a276 (0xbb15 bytes) 000166 1729089098.656718 trunc from 0x028519 to 0x03633e (0xde26 bytes) 000169 1729089098.665596 fallocate 0x0322c1 thru 0x03b88c (0x95cc bytes) 000171 1729089098.669734 trunc from 0x03b88d to 0x0054a3 (0x363e9 bytes) 000172 1729089098.678918 punch from 0x003f05 to 0x0054a3 (0x159f bytes) 000173 1729089098.681745 mapwrite 0x0144ac thru 0x020ba2 (0xc6f7 bytes) 000174 1729089098.731923 read 0x00bf89 thru 0x017e66 (0xbede bytes) 000175 1729089098.735937 mapread 0x01f494 thru 0x020ba2 (0x170f bytes) 000176 1729089098.736526 punch from 0x0188f7 to 0x01d2c6 (0x49d0 bytes) 000177 1729089098.739825 fallocate 0x02370b thru 0x0307c7 (0xd0bd bytes) 000179 1729089098.743636 trunc from 0x0307c8 to 0x020df4 (0xf9d3 bytes) 000181 1729089098.750462 mapread 0x008d1d thru 0x00c506 (0x37ea bytes) 000182 1729089098.750898 fallocate 0x033e65 thru 0x03ffff (0xc19b bytes) 000183 1729089098.753612 write 0x023eee thru 0x02f2c3 (0xb3d6 bytes) 000184 1729089098.760019 mapread 0x0235eb thru 0x0245c3 (0xfd9 bytes) 000185 1729089098.760910 read 0x008cc0 thru 0x011778 (0x8ab9 bytes) 000188 1729089098.762002 mapread 0x02ca35 thru 0x03a689 (0xdc55 bytes) 000191 1729089098.765656 MIRROR_EXTEND to 4 mirrors 000193 1729089098.991665 mapread 0x0171fe thru 0x026995 (0xf798 bytes) 000194 1729089099.012277 mapwrite 0x035f2b thru 0x039682 (0x3758 bytes) 000195 1729089099.063027 mapread 0x00ea60 thru 0x0182f4 (0x9895 bytes) 000196 1729089099.069934 trunc from 0x040000 to 0x02de6a (0x12195 bytes) 000197 1729089099.090154 read 0x01db35 thru 0x01dfa1 (0x46d bytes) 000199 1729089099.091150 trunc from 0x02de6b to 0x029e58 (0x4012 bytes) 000200 1729089099.097830 mapread 0x002d83 thru 0x00f279 (0xc4f7 bytes) 000201 1729089099.102616 write 0x029a8d thru 0x02e029 (0x459d bytes) 000202 1729089099.106265 mapwrite 0x02b41b thru 0x039253 (0xde39 bytes) 000203 1729089099.133459 read 0x0192aa thru 0x027747 (0xe49e bytes) 000204 1729089099.139229 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x776:0x0]: Invalid argument (22) 000205 1729089099.224772 MIRROR_EXTEND to 4 mirrors 000206 1729089099.450889 write 0x00d44e thru 0x013b59 (0x670c bytes) 000207 1729089099.475865 read 0x02a8ba thru 0x037fad (0xd6f4 bytes) 000208 1729089099.480953 trunc from 0x039254 to 0x0269f7 (0x1285c bytes) 000211 1729089099.501720 MIRROR_EXTEND to 5 mirrors 000213 1729089099.708715 mapwrite 0x015ad6 thru 0x0168f1 (0xe1c bytes) 000214 1729089099.752906 punch from 0x00bb74 to 0x00e4d8 (0x2965 bytes) 000215 1729089099.760096 MIRROR_RESYNC 5 mirrors 000216 1729089100.287661 read 0x025534 thru 0x025975 (0x442 bytes) 000217 1729089100.294057 MIRROR_RESYNC 5 mirrors 000218 1729089100.370659 fallocate 0x028dce thru 0x031c8c (0x8ebf bytes) 000219 1729089100.391274 trunc from 0x0269f8 to 0x02df60 (0x7569 bytes) 000220 1729089100.401262 mapwrite 0x0267f8 thru 0x036022 (0xf82b bytes) 000221 1729089100.430006 write 0x01c4bf thru 0x024099 (0x7bdb bytes) 000222 1729089100.436627 MIRROR_EXTEND to 6 mirrors 000223 1729089100.637291 mapread 0x01470a thru 0x02464a (0xff41 bytes) 000224 1729089100.651818 trunc from 0x036023 to 0x012f0d (0x23115 bytes) 000225 1729089100.681276 mapread 0x0044a0 thru 0x005c5a (0x17bb bytes) 000226 1729089100.688032 write 0x018dab thru 0x01e10b (0x5361 bytes) 000227 1729089100.688995 write 0x03fb58 thru 0x03ffff (0x4a8 bytes) 000228 1729089100.689662 fallocate 0x033fa0 thru 0x039414 (0x5475 bytes) 000230 1729089100.692773 trunc from 0x040000 to 0x02f2ac (0x10d53 bytes) 000231 1729089100.700931 read 0x01c017 thru 0x027de6 (0xbdd0 bytes) 000236 1729089100.706363 punch from 0x01fbd9 to 0x02772d (0x7b55 bytes) 000238 1729089100.709475 trunc from 0x02f2ad to 0x031a14 (0x2768 bytes) 000239 1729089100.716284 fallocate 0x01ad54 thru 0x01b39f (0x64c bytes) 000240 1729089100.720657 mapread 0x0281ae thru 0x02f9b0 (0x7803 bytes) 000241 1729089100.725227 trunc from 0x031a15 to 0x003167 (0x2e8ad bytes) 000242 1729089100.734393 fallocate 0x035923 thru 0x0375f6 (0x1cd4 bytes) 000243 1729089100.737047 fallocate 0x0189f3 thru 0x020025 (0x7633 bytes) 000244 1729089100.740045 write 0x025ab4 thru 0x031edc (0xc429 bytes) 000245 1729089100.741391 punch from 0x015a6d to 0x01d0e8 (0x767c bytes) 000246 1729089100.743956 mapread 0x010374 thru 0x011023 (0xcb0 bytes) 000247 1729089100.747876 read 0x00dfc3 thru 0x010bd4 (0x2c12 bytes) 000248 1729089100.748307 mapread 0x018bb4 thru 0x025abc (0xcf09 bytes) 000249 1729089100.752320 mapwrite 0x02c09c thru 0x036d30 (0xac95 bytes) 000251 1729089100.776968 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x776:0x0]: Invalid argument (22) 000252 1729089100.863905 read 0x00dbe4 thru 0x01315a (0x5577 bytes) 000253 1729089100.869124 fallocate 0x00bfd7 thru 0x019958 (0xd982 bytes) 000254 1729089100.872391 trunc from 0x036d31 to 0x035908 (0x1428 bytes) 000256 1729089100.882398 trunc from 0x035909 to 0x02f138 (0x67d0 bytes) 000257 1729089100.891309 read 0x0166e0 thru 0x01db35 (0x7456 bytes) 000259 1729089100.896656 write 0x01f294 thru 0x02e97d (0xf6ea bytes) 000260 1729089100.897749 write 0x00586e thru 0x008693 (0x2e26 bytes) 000261 1729089100.898319 mapread 0x00c4cc thru 0x01ba77 (0xf5ac bytes) 000262 1729089100.899167 write 0x02fbee thru 0x03a12f (0xa542 bytes) 000263 1729089100.900200 trunc from 0x03a130 to 0x007d0b (0x32424 bytes) 000266 1729089100.912816 mapread 0x003bfe thru 0x004ad7 (0xeda bytes) 000268 1729089100.914065 fallocate 0x01cdee thru 0x01f8fa (0x2b0d bytes) 000269 1729089100.918932 trunc from 0x01f8fb to 0x0116fa (0xe200 bytes) 000270 1729089100.928379 punch from 0x001b1a to 0x00b3b5 (0x989c bytes) 000271 1729089100.936522 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x776:0x0]: Invalid argument (22) 000272 1729089101.041446 MIRROR_EXTEND to 5 mirrors 000273 1729089101.228496 trunc from 0x0116fb to 0x0279ed (0x162f3 bytes) 000274 1729089101.247228 mapread 0x00c6ac thru 0x01701a (0xa96f bytes) 000275 1729089101.261490 mapread 0x010abd thru 0x0167dd (0x5d21 bytes) 000276 1729089101.262123 mapread 0x00f0ed thru 0x01a338 (0xb24c bytes) 000278 1729089101.263313 punch from 0x01eb22 to 0x0204e1 (0x19c0 bytes) 000279 1729089101.266177 mapread 0x00f897 thru 0x0134c3 (0x3c2d bytes) 000280 1729089101.268635 MIRROR_RESYNC 5 mirrors 000281 1729089101.504788 MIRROR_EXTEND to 6 mirrors 000283 1729089101.692189 read 0x026bcb thru 0x0279ed (0xe23 bytes) 000284 1729089101.695750 read 0x0056f4 thru 0x008a01 (0x330e bytes) 000285 1729089101.700355 mapread 0x00875a thru 0x009f23 (0x17ca bytes) 000286 1729089101.700894 write 0x014874 thru 0x015abe (0x124b bytes) 000288 1729089101.721654 read 0x020d7f thru 0x022d0a (0x1f8c bytes) 000289 1729089101.727634 write 0x03432c thru 0x036524 (0x21f9 bytes) 000290 1729089101.728463 write 0x005ab0 thru 0x00603b (0x58c bytes) 000291 1729089101.729083 write 0x02f76b thru 0x035af9 (0x638f bytes) 000292 1729089101.734460 trunc from 0x036525 to 0x0023f3 (0x34131 bytes) 000293 1729089101.754211 read 0x002359 thru 0x0023f3 (0x9b bytes) 000294 1729089101.757315 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x776:0x0]: Invalid argument (22) 000295 1729089101.831815 write 0x01a05e thru 0x01be32 (0x1dd5 bytes) 000296 1729089101.835806 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x776:0x0]: Invalid argument (22) 000297 1729089101.915595 trunc from 0x01be33 to 0x02d854 (0x11a22 bytes) 000298 1729089101.923016 read 0x01d1d3 thru 0x01fea8 (0x2cd6 bytes) 000300 1729089101.928016 fallocate 0x01f5cd thru 0x02dce5 (0xe719 bytes) 000302 1729089101.931099 fallocate 0x0020d3 thru 0x0091f9 (0x7127 bytes) 000303 1729089101.933384 fallocate 0x03c33d thru 0x03ffff (0x3cc3 bytes) 000304 1729089101.938018 MIRROR_EXTEND to 5 mirrors 000305 1729089102.145363 write 0x02481d thru 0x029c4e (0x5432 bytes) 000306 1729089102.167366 punch from 0x02c849 to 0x0382b8 (0xba70 bytes) 000307 1729089102.169935 fallocate 0x026563 thru 0x02c54b (0x5fe9 bytes) 000308 1729089102.174841 MIRROR_EXTEND to 6 mirrors 000309 1729089102.393165 mapwrite 0x021144 thru 0x025b82 (0x4a3f bytes) 000310 1729089102.445084 mapread 0x01d62b thru 0x0287e0 (0xb1b6 bytes) 000311 1729089102.445895 punch from 0x036a29 to 0x03c2c2 (0x589a bytes) 000313 1729089102.449649 mapread 0x00b770 thru 0x018694 (0xcf25 bytes) 000314 1729089102.458880 mapwrite 0x00f604 thru 0x01a76f (0xb16c bytes) 000316 1729089102.474994 fallocate 0x012a30 thru 0x01fbcd (0xd19e bytes) 000317 1729089102.478481 punch from 0x001ec7 to 0x00e0de (0xc218 bytes) 000318 1729089102.480994 mapwrite 0x004441 thru 0x00a8b4 (0x6474 bytes) 000319 1729089102.494381 fallocate 0x03a9cb thru 0x03e0cb (0x3701 bytes) 000321 1729089102.501043 mapread 0x01224f thru 0x021872 (0xf624 bytes) 000322 1729089102.502125 read 0x019ed4 thru 0x01d271 (0x339e bytes) 000325 1729089102.503284 write 0x0337ec thru 0x03f470 (0xbc85 bytes) 000326 1729089102.517013 trunc from 0x040000 to 0x02caf9 (0x13506 bytes) 000327 1729089102.527577 mapwrite 0x0015fd thru 0x0112c2 (0xfcc6 bytes) 000329 1729089102.545595 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x776:0x0]: Invalid argument (22) 000330 1729089102.622024 write 0x02a9bb thru 0x02ade5 (0x42b bytes) 000331 1729089102.626335 trunc from 0x02cafa to 0x02606b (0x6a8e bytes) 000332 1729089102.635857 MIRROR_EXTEND to 6 mirrors 000333 1729089102.812743 punch from 0x0163f3 to 0x016a3a (0x648 bytes) 000334 1729089102.828876 read 0x00c997 thru 0x018a32 (0xc09c bytes) 000335 1729089102.833498 trunc from 0x02606c to 0x0257bb (0x8b0 bytes) 000336 1729089102.844705 write 0x0186bd thru 0x02561d (0xcf61 bytes) 000337 1729089102.847511 mapwrite 0x00dd19 thru 0x010e4f (0x3137 bytes) 000339 1729089102.862816 punch from 0x01d9f5 to 0x020347 (0x2953 bytes) 000341 1729089102.865833 mapwrite 0x008a6b thru 0x01615f (0xd6f5 bytes) 000342 1729089102.881232 read 0x007ed4 thru 0x016a02 (0xeb2f bytes) 000343 1729089102.881711 fallocate 0x00eb99 thru 0x0107f5 (0x1c5d bytes) 000344 1729089102.884098 mapwrite 0x0032e6 thru 0x009f77 (0x6c92 bytes) 000345 1729089102.891940 mapread 0x01f7a5 thru 0x0257bb (0x6017 bytes) 000346 1729089102.899612 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x776:0x0]: Invalid argument (22) 000347 1729089102.971858 punch from 0x00a84c to 0x018699 (0xde4e bytes) 000349 1729089102.976707 MIRROR_EXTEND to 6 mirrors 000350 1729089103.172247 read 0x010682 thru 0x01de44 (0xd7c3 bytes) 000351 1729089103.176028 write 0x039355 thru 0x03ffff (0x6cab bytes) 000352 1729089103.192003 punch from 0x029efa to 0x031f1f (0x8026 bytes) 000353 1729089103.194189 write 0x03bf59 thru 0x03ffff (0x40a7 bytes) 000355 1729089103.195388 fallocate 0x01c5e7 thru 0x01fa00 (0x341a bytes) 000359 1729089103.200248 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x776:0x0]: Invalid argument (22) 000360 1729089103.287056 fallocate 0x0008a8 thru 0x00b373 (0xaacc bytes) 000361 1729089103.290937 read 0x0259e5 thru 0x035120 (0xf73c bytes) 000362 1729089103.295838 punch from 0x01e25f to 0x01f435 (0x11d7 bytes) 000363 1729089103.299021 mapwrite 0x01435c thru 0x015555 (0x11fa bytes) 000364 1729089103.311399 read 0x023320 thru 0x02d437 (0xa118 bytes) 000366 1729089103.317398 mapwrite 0x006e8e thru 0x013079 (0xc1ec bytes) 000367 1729089103.329769 write 0x000230 thru 0x00f831 (0xf602 bytes) 000368 1729089103.333137 mapwrite 0x038ce3 thru 0x03ffff (0x731d bytes) 000370 1729089103.347313 mapwrite 0x038497 thru 0x03fc5b (0x77c5 bytes) 000371 1729089103.356663 mapread 0x034d50 thru 0x036dc2 (0x2073 bytes) 000372 1729089103.357242 punch from 0x0202f6 to 0x02b70c (0xb417 bytes) 000373 1729089103.359952 mapread 0x0127a2 thru 0x01772b (0x4f8a bytes) 000374 1729089103.360458 fallocate 0x021df2 thru 0x026409 (0x4618 bytes) 000375 1729089103.362509 punch from 0x015195 to 0x01fda2 (0xac0e bytes) 000376 1729089103.365340 punch from 0x0059b5 to 0x00e0a7 (0x86f3 bytes) 000378 1729089103.368166 mapwrite 0x00b697 thru 0x0184b3 (0xce1d bytes) 000379 1729089103.391511 read 0x0245a9 thru 0x02ae3c (0x6894 bytes) 000380 1729089103.395390 fallocate 0x024c4e thru 0x034512 (0xf8c5 bytes) 000383 1729089103.398578 fallocate 0x037ca2 thru 0x03ffff (0x835e bytes) 000384 1729089103.400940 punch from 0x033908 to 0x03ffff (0xc6f8 bytes) 000386 1729089103.403339 write 0x01716f thru 0x026884 (0xf716 bytes) 000387 1729089103.404429 mapwrite 0x01049f thru 0x011c66 (0x17c8 bytes) 000388 1729089103.413160 trunc from 0x040000 to 0x00a79c (0x35863 bytes) 000389 1729089103.420913 punch from 0x0062cc to 0x00a79c (0x44d1 bytes) 000391 1729089103.423630 punch from 0x0051a2 to 0x00a79c (0x55fb bytes) 000392 1729089103.425790 fallocate 0x027ed4 thru 0x02978c (0x18b9 bytes) 000393 1729089103.428422 mapread 0x0217cd thru 0x02978c (0x7fc0 bytes) 000394 1729089103.445308 punch from 0x01a441 to 0x020e3e (0x69fe bytes) 000395 1729089103.447494 mapread 0x002b75 thru 0x00a86a (0x7cf6 bytes) 000397 1729089103.461043 fallocate 0x019ec7 thru 0x027de7 (0xdf21 bytes) 000398 1729089103.463690 read 0x00808e thru 0x01264d (0xa5c0 bytes) 000399 1729089103.467906 write 0x032b96 thru 0x03a0f5 (0x7560 bytes) 000400 1729089103.468915 fallocate 0x011df0 thru 0x01ddc7 (0xbfd8 bytes) 000401 1729089103.472356 trunc from 0x03a0f6 to 0x00edc9 (0x2b32c bytes) 000402 1729089103.478988 write 0x0067c5 thru 0x007c86 (0x14c2 bytes) 000403 1729089103.479616 fallocate 0x034ee4 thru 0x03c105 (0x7222 bytes) 000404 1729089103.482469 read 0x01030d thru 0x01b554 (0xb248 bytes) 000405 1729089103.489593 MIRROR_RESYNC 5 mirrors 000406 1729089103.726798 trunc from 0x03c106 to 0x013750 (0x289b5 bytes) 000407 1729089103.748382 mapwrite 0x024f12 thru 0x033228 (0xe317 bytes) 000408 1729089103.777995 fallocate 0x001099 thru 0x011013 (0xff7b bytes) 000409 1729089103.780830 trunc from 0x033229 to 0x00c867 (0x269c1 bytes) 000413 1729089103.787697 write 0x03f252 thru 0x03ffff (0xdae bytes) 000414 1729089103.788435 trunc from 0x040000 to 0x012dea (0x2d215 bytes) 000415 1729089103.795068 fallocate 0x00ae58 thru 0x01528b (0xa434 bytes) 000416 1729089103.796961 mapread 0x0108bc thru 0x01528b (0x49d0 bytes) 000417 1729089103.801075 punch from 0x0030f7 to 0x012a09 (0xf913 bytes) 000418 1729089103.803203 fallocate 0x034e74 thru 0x038733 (0x38c0 bytes) 000419 1729089103.805037 punch from 0x00f350 to 0x014714 (0x53c5 bytes) 000420 1729089103.807315 fallocate 0x0091a8 thru 0x0173c4 (0xe21d bytes) 000421 1729089103.810090 punch from 0x01519f to 0x0173c4 (0x2226 bytes) 000423 1729089103.813115 read 0x01198c thru 0x0173c4 (0x5a39 bytes) 000424 1729089103.816911 read 0x00e34c thru 0x0173c4 (0x9079 bytes) 000425 1729089103.821461 mapwrite 0x0265e7 thru 0x0297b7 (0x31d1 bytes) 000426 1729089103.844442 mapwrite 0x0393c0 thru 0x03ffff (0x6c40 bytes) 000427 1729089103.869407 mapread 0x00611d thru 0x00ab17 (0x49fb bytes) 000428 1729089103.870199 punch from 0x02de00 to 0x0315db (0x37dc bytes) 000429 1729089103.873277 mapwrite 0x02a40d thru 0x038342 (0xdf36 bytes) 000430 1729089103.899798 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x776:0x0]: Invalid argument (22) 000431 1729089103.984918 punch from 0x00b5c0 to 0x016803 (0xb244 bytes) 000434 1729089103.988328 trunc from 0x040000 to 0x028c87 (0x17378 bytes) 000435 1729089103.998001 punch from 0x026ce0 to 0x028c87 (0x1fa8 bytes) 000436 1729089104.003464 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x776:0x0]: Invalid argument (22) 000438 1729089104.076012 fallocate 0x005fb7 thru 0x01470f (0xe759 bytes) 000439 1729089104.080854 MIRROR_EXTEND to 4 mirrors 000440 1729089104.293528 MIRROR_RESYNC 4 mirrors 000441 1729089104.563157 fallocate 0x0057ed thru 0x0125af (0xcdc3 bytes) 000442 1729089104.579397 write 0x01ecf6 thru 0x01fcc1 (0xfcc bytes) 000443 1729089104.584393 mapread 0x01ad79 thru 0x0243fb (0x9683 bytes) 000444 1729089104.592465 mapwrite 0x00c117 thru 0x01055c (0x4446 bytes) 000445 1729089104.608331 mapread 0x01c663 thru 0x027884 (0xb222 bytes) 000446 1729089104.608971 mapread 0x016266 thru 0x01c920 (0x66bb bytes) 000447 1729089104.609892 read 0x0259f3 thru 0x028c87 (0x3295 bytes) 000448 1729089104.610901 write 0x00c198 thru 0x01b4f8 (0xf361 bytes) 000449 1729089104.612032 trunc from 0x028c88 to 0x0116ed (0x1759a bytes) 000450 1729089104.622520 MIRROR_EXTEND to 5 mirrors 000451 1729089104.800056 fallocate 0x017df3 thru 0x02404b (0xc259 bytes) 000453 1729089104.815016 write 0x011303 thru 0x0180ae (0x6dac bytes) 000455 1729089104.821733 fallocate 0x00836f thru 0x017a40 (0xf6d2 bytes) 000456 1729089104.824136 punch from 0x014cee to 0x01c117 (0x742a bytes) 000457 1729089104.831076 trunc from 0x02404c to 0x00f957 (0x146f4 bytes) 000458 1729089104.840311 mapwrite 0x023a02 thru 0x028576 (0x4b75 bytes) 000460 1729089104.861965 mapread 0x023cd5 thru 0x028576 (0x48a2 bytes) 000461 1729089104.862671 read 0x002f7d thru 0x003568 (0x5ec bytes) 000462 1729089104.865273 trunc from 0x028577 to 0x007d93 (0x207e3 bytes) 000463 1729089104.872109 read 0x001f27 thru 0x007d93 (0x5e6d bytes) 000464 1729089104.875747 mapwrite 0x016d4f thru 0x01a441 (0x36f3 bytes) 000465 1729089104.897181 MIRROR_RESYNC 5 mirrors 000466 1729089105.212972 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x776:0x0]: Invalid argument (22) 000467 1729089105.315282 mapread 0x00fe9b thru 0x0149b9 (0x4b1f bytes) 000468 1729089105.325619 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x776:0x0]: Invalid argument (22) 000469 1729089105.417445 read 0x016ce1 thru 0x01a441 (0x3761 bytes) 000470 1729089105.420731 mapread 0x007aa8 thru 0x00cc72 (0x51cb bytes) 000471 1729089105.432269 MIRROR_RESYNC 3 mirrors 000472 1729089105.496901 punch from 0x0005e8 to 0x00df22 (0xd93b bytes) 000473 1729089105.516160 write 0x006f92 thru 0x0122f4 (0xb363 bytes) 000474 1729089105.527137 read 0x007007 thru 0x00bb90 (0x4b8a bytes) 000475 1729089105.527631 read 0x00f0b3 thru 0x01a441 (0xb38f bytes) 000476 1729089105.532832 mapread 0x016ec8 thru 0x0190b1 (0x21ea bytes) 000477 1729089105.533577 trunc from 0x01a442 to 0x02f8e7 (0x154a6 bytes) 000478 1729089105.542146 punch from 0x000718 to 0x005668 (0x4f51 bytes) 000479 1729089105.548150 MIRROR_EXTEND to 4 mirrors 000480 1729089105.762191 trunc from 0x02f8e8 to 0x006edd (0x28a0a bytes) 000481 1729089105.790605 read 0x001da3 thru 0x005886 (0x3ae4 bytes) 000483 1729089105.795123 mapwrite 0x030682 thru 0x03883c (0x81bb bytes) 000484 1729089105.828234 MIRROR_RESYNC 4 mirrors 000485 1729089106.123120 trunc from 0x03883d to 0x0064a3 (0x32399 bytes) 000487 1729089106.140715 trunc from 0x0064a4 to 0x00eacc (0x8629 bytes) 000488 1729089106.149852 mapread 0x002cd9 thru 0x00b26f (0x8597 bytes) 000489 1729089106.160607 read 0x005421 thru 0x00b1d4 (0x5db4 bytes) 000492 1729089106.161721 mapread 0x00834e thru 0x00eacc (0x677f bytes) 000493 1729089106.164885 MIRROR_EXTEND to 5 mirrors 000494 1729089106.331465 read 0x001bd5 thru 0x00eacc (0xcef8 bytes) 000495 1729089106.335303 fallocate 0x00021b thru 0x004143 (0x3f29 bytes) 000496 1729089106.350914 mapread 0x004db4 thru 0x00b0f0 (0x633d bytes) 000497 1729089106.357734 write 0x02d978 thru 0x037868 (0x9ef1 bytes) 000498 1729089106.361413 mapwrite 0x003f7f thru 0x00d611 (0x9693 bytes) 000499 1729089106.373089 mapwrite 0x035dbc thru 0x037421 (0x1666 bytes) 000500 1729089106.382401 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x776:0x0]: Invalid argument (22) 000501 1729089106.454696 trunc from 0x037869 to 0x0329ff (0x4e69 bytes) 000503 1729089106.471301 trunc from 0x032a00 to 0x0302a4 (0x275b bytes) 000504 1729089106.481048 punch from 0x018fcf to 0x0218f2 (0x8924 bytes) 000505 1729089106.485894 mapread 0x020773 thru 0x023c30 (0x34be bytes) 000507 1729089106.496207 trunc from 0x0302a5 to 0x03bdc5 (0xbb21 bytes) 000508 1729089106.503237 punch from 0x035be2 to 0x03aa91 (0x4eb0 bytes) 000509 1729089106.505603 mapread 0x00eafb thru 0x011eab (0x33b1 bytes) 000511 1729089106.511976 trunc from 0x03bdc6 to 0x006881 (0x35544 bytes) 000513 1729089106.521374 write 0x00251e thru 0x00d189 (0xac6c bytes) 000514 1729089106.527950 trunc from 0x00d18a to 0x026f0a (0x19d81 bytes) 000515 1729089106.535051 punch from 0x024bda to 0x026f0a (0x2331 bytes) 000516 1729089106.537335 mapwrite 0x017b64 thru 0x026367 (0xe804 bytes) 000517 1729089106.558776 write 0x037ec4 thru 0x03c035 (0x4172 bytes) 000518 1729089106.559522 write 0x022ce2 thru 0x02fe02 (0xd121 bytes) 000519 1729089106.563403 write 0x021fee thru 0x030cb6 (0xecc9 bytes) 000520 1729089106.569016 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x776:0x0]: Invalid argument (22) 000521 1729089106.645277 read 0x03a39a thru 0x03c035 (0x1c9c bytes) 000522 1729089106.650272 MIRROR_RESYNC 3 mirrors 000523 1729089106.846359 punch from 0x032bdb to 0x036dbd (0x41e3 bytes) 000524 1729089106.856505 fallocate 0x01bd9e thru 0x029380 (0xd5e3 bytes) 000525 1729089106.858514 mapwrite 0x03b74b thru 0x03ffff (0x48b5 bytes) 000526 1729089106.887442 trunc from 0x040000 to 0x0316aa (0xe955 bytes) 000528 1729089106.892529 trunc from 0x0316ab to 0x021b62 (0xfb48 bytes) 000529 1729089106.899275 read 0x00c0df thru 0x0150d8 (0x8ffa bytes) 000530 1729089106.902717 punch from 0x020359 to 0x021ac8 (0x1770 bytes) 000531 1729089106.905026 write 0x033dec thru 0x037e9f (0x40b4 bytes) 000532 1729089106.905960 read 0x018551 thru 0x01ed2f (0x67df bytes) 000534 1729089106.910882 write 0x01c631 thru 0x024561 (0x7f31 bytes) 000535 1729089106.912758 mapwrite 0x01d86b thru 0x02bfe4 (0xe77a bytes) 000536 1729089106.924467 punch from 0x012d2a to 0x01a5bb (0x7892 bytes) 000537 1729089106.927718 trunc from 0x037ea0 to 0x007063 (0x30e3c bytes) 000538 1729089106.936287 punch from 0x005537 to 0x007063 (0x1b2d bytes) 000540 1729089106.939667 punch from 0x002708 to 0x007063 (0x495c bytes) 000541 1729089106.942157 punch from 0x002c99 to 0x007063 (0x43cb bytes) 000542 1729089106.944602 write 0x01de2e thru 0x0235fb (0x57ce bytes) 000544 1729089106.945972 trunc from 0x0235fc to 0x03a636 (0x1703b bytes) 000546 1729089106.953214 punch from 0x034a89 to 0x03a636 (0x5bae bytes) 000547 1729089106.956107 write 0x028a72 thru 0x02e0cf (0x565e bytes) 000548 1729089106.963448 trunc from 0x03a637 to 0x035c70 (0x49c6 bytes) 000549 1729089106.971295 punch from 0x00a8f5 to 0x0150b0 (0xa7bc bytes) 000550 1729089106.973498 mapwrite 0x012013 thru 0x01b4f8 (0x94e6 bytes) 000551 1729089107.003096 MIRROR_EXTEND to 4 mirrors 000552 1729089107.180883 fallocate 0x02f17e thru 0x03b65e (0xc4e1 bytes) 000553 1729089107.194006 write 0x030e0c thru 0x03a5a8 (0x979d bytes) 000554 1729089107.198983 mapwrite 0x00d624 thru 0x015c41 (0x861e bytes) 000556 1729089107.208996 punch from 0x0029ae to 0x00dcf7 (0xb34a bytes) 000557 1729089107.211240 fallocate 0x01f9cb thru 0x021d93 (0x23c9 bytes) 000558 1729089107.212863 punch from 0x010ac5 to 0x014402 (0x393e bytes) 000559 1729089107.214481 mapread 0x030b77 thru 0x033883 (0x2d0d bytes) 000561 1729089107.215232 trunc from 0x03b65f to 0x01ba39 (0x1fc25 bytes) 000562 1729089107.221276 read 0x005eba thru 0x00fdb4 (0x9efb bytes) 000563 1729089107.224005 mapwrite 0x0021a3 thru 0x00def5 (0xbd53 bytes) 000564 1729089107.235245 mapread 0x016d16 thru 0x01ba39 (0x4d24 bytes) 000565 1729089107.238742 mapwrite 0x00ded3 thru 0x0146bb (0x67e9 bytes) 000566 1729089107.247369 read 0x006fc3 thru 0x010419 (0x9457 bytes) 000567 1729089107.247830 read 0x01849c thru 0x01ba39 (0x359e bytes) 000569 1729089107.248449 trunc from 0x01ba3a to 0x02ac34 (0xf1fb bytes) 000570 1729089107.254552 write 0x012c4a thru 0x0171ad (0x4564 bytes) 000571 1729089107.255296 punch from 0x00aa77 to 0x010699 (0x5c23 bytes) 000574 1729089107.258536 read 0x005163 thru 0x009e9e (0x4d3c bytes) 000575 1729089107.258994 mapread 0x009ba2 thru 0x00f0c6 (0x5525 bytes) 000576 1729089107.263589 read 0x0165ef thru 0x024255 (0xdc67 bytes) 000577 1729089107.264111 mapwrite 0x03b970 thru 0x03c989 (0x101a bytes) 000578 1729089107.282316 read 0x02e347 thru 0x0346e5 (0x639f bytes) 000579 1729089107.282830 read 0x025afe thru 0x03475a (0xec5d bytes) 000580 1729089107.284946 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x776:0x0]: Invalid argument (22) 000581 1729089107.350724 fallocate 0x013b6f thru 0x01ffb9 (0xc44b bytes) 000582 1729089107.353409 read 0x0141f8 thru 0x01e338 (0xa141 bytes) 000583 1729089107.356867 trunc from 0x03c98a to 0x033340 (0x9649 bytes) 000585 1729089107.363264 fallocate 0x002369 thru 0x0065c7 (0x425f bytes) 000586 1729089107.365381 mapread 0x00f50b thru 0x00fa84 (0x57a bytes) 000588 1729089107.368092 fallocate 0x01bcee thru 0x01efbe (0x32d1 bytes) 000589 1729089107.370035 read 0x007da8 thru 0x00b6cc (0x3925 bytes) 000590 1729089107.374641 mapread 0x028227 thru 0x02f1a8 (0x6f82 bytes) 000591 1729089107.375241 mapwrite 0x0331e1 thru 0x03de57 (0xac77 bytes) 000592 1729089107.395890 mapread 0x024632 thru 0x032146 (0xdb15 bytes) 000593 1729089107.396619 mapwrite 0x03d0a0 thru 0x03ffff (0x2f60 bytes) 000594 1729089107.415946 write 0x01bad6 thru 0x01e71a (0x2c45 bytes) 000595 1729089107.416796 trunc from 0x040000 to 0x000e75 (0x3f18a bytes) 000596 1729089107.426945 MIRROR_EXTEND to 4 mirrors 000597 1729089107.530116 write 0x0220f1 thru 0x02942c (0x733c bytes) 000600 1729089107.544529 trunc from 0x02942d to 0x03320b (0x9ddf bytes) 000604 1729089107.556913 fallocate 0x0186f4 thru 0x01c647 (0x3f54 bytes) 000605 1729089107.558993 mapread 0x030ab1 thru 0x03320b (0x275b bytes) 000606 1729089107.565028 trunc from 0x03320c to 0x0149d8 (0x1e833 bytes) 000607 1729089107.570272 write 0x007055 thru 0x00f597 (0x8543 bytes) 000608 1729089107.574406 mapwrite 0x02e714 thru 0x03d48a (0xed77 bytes) 000609 1729089107.599867 trunc from 0x03d48b to 0x03cc1c (0x86e bytes) 000610 1729089107.604912 mapread 0x00a26e thru 0x013f63 (0x9cf6 bytes) 000611 1729089107.608767 trunc from 0x03cc1d to 0x03d5c7 (0x9ab bytes) 000612 1729089107.614665 fallocate 0x0304d0 thru 0x03666f (0x61a0 bytes) 000614 1729089107.617903 mapwrite 0x033291 thru 0x03ede6 (0xbb56 bytes) 000615 1729089107.638663 MIRROR_EXTEND to 5 mirrors 000616 1729089107.778444 trunc from 0x03ede7 to 0x02a919 (0x144cd bytes) 000618 1729089107.798930 punch from 0x023d10 to 0x024903 (0xbf4 bytes) 000619 1729089107.801125 write 0x0326e5 thru 0x03eb25 (0xc441 bytes) 000621 1729089107.802617 mapwrite 0x01f885 thru 0x02e04b (0xe7c7 bytes) 000622 1729089107.822201 mapwrite 0x03a7f5 thru 0x03ffff (0x580b bytes) 000623 1729089107.839385 write 0x016363 thru 0x020b1f (0xa7bd bytes) 000625 1729089107.843464 mapread 0x03e0be thru 0x03ef35 (0xe78 bytes) 000626 1729089107.844120 fallocate 0x0257ba thru 0x02d3d7 (0x7c1e bytes) 000628 1729089107.849299 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x776:0x0]: Invalid argument (22) 000629 1729089107.915757 read 0x02cc8d thru 0x02dcaa (0x101e bytes) 000631 1729089107.919167 read 0x02d19c thru 0x031c34 (0x4a99 bytes) 000632 1729089107.926158 MIRROR_RESYNC 4 mirrors 000633 1729089108.185591 trunc from 0x040000 to 0x020603 (0x1f9fc bytes) 000634 1729089108.203151 write 0x01b16c thru 0x026cb8 (0xbb4d bytes) 000636 1729089108.206240 fallocate 0x00b7c9 thru 0x0143b9 (0x8bf1 bytes) 000637 1729089108.208346 punch from 0x01f19c to 0x026cb8 (0x7b1d bytes) 000638 1729089108.213607 mapwrite 0x021615 thru 0x02aec0 (0x98ac bytes) 000640 1729089108.237360 mapwrite 0x025136 thru 0x02b8db (0x67a6 bytes) 000641 1729089108.259639 mapwrite 0x02e1e5 thru 0x03a712 (0xc52e bytes) 000643 1729089108.281587 mapwrite 0x0145b9 thru 0x01bdae (0x77f6 bytes) 000644 1729089108.292014 punch from 0x017f52 to 0x020ca4 (0x8d53 bytes) 000645 1729089108.294401 mapread 0x02ef84 thru 0x0356d3 (0x6750 bytes) 000646 1729089108.295174 trunc from 0x03a713 to 0x02cf51 (0xd7c1 bytes) 000647 1729089108.301894 write 0x023074 thru 0x02f4a3 (0xc430 bytes) 000648 1729089108.303854 trunc from 0x02f4a4 to 0x013303 (0x1c1a0 bytes) 000650 1729089108.310945 punch from 0x0067a8 to 0x0079b7 (0x1210 bytes) 000651 1729089108.313033 mapwrite 0x02dd40 thru 0x03a59f (0xc860 bytes) 000652 1729089108.334457 read 0x016f09 thru 0x01aa7e (0x3b76 bytes) 000653 1729089108.337633 trunc from 0x03a5a0 to 0x01f08e (0x1b511 bytes) 000654 1729089108.342970 read 0x00bb8c thru 0x01b895 (0xfd0a bytes) 000655 1729089108.346324 write 0x02ec66 thru 0x03e150 (0xf4eb bytes) 000656 1729089108.347476 fallocate 0x01c795 thru 0x02bf9f (0xf80b bytes) 000657 1729089108.349909 fallocate 0x021a02 thru 0x030e52 (0xf451 bytes) 000658 1729089108.353585 MIRROR_RESYNC 4 mirrors 000659 1729089108.583187 read 0x038eb9 thru 0x03e150 (0x5298 bytes) 000660 1729089108.586817 read 0x01dbff thru 0x024314 (0x6716 bytes) 000661 1729089108.592351 read 0x01272c thru 0x0185b8 (0x5e8d bytes) 000662 1729089108.592646 mapwrite 0x011a1f thru 0x01fe2e (0xe410 bytes) 000664 1729089108.633343 read 0x01e148 thru 0x029172 (0xb02b bytes) 000665 1729089108.633983 read 0x029173 thru 0x031ba4 (0x8a32 bytes) 000666 1729089108.634326 write 0x03fdae thru 0x03ffff (0x252 bytes) 000667 1729089108.634959 punch from 0x004d44 to 0x00f38f (0xa64c bytes) 000668 1729089108.638030 write 0x0187a3 thru 0x019039 (0x897 bytes) 000669 1729089108.638596 read 0x009a5b thru 0x00ab3f (0x10e5 bytes) 000670 1729089108.641932 read 0x0298e1 thru 0x02c194 (0x28b4 bytes) 000672 1729089108.642645 write 0x00dc07 thru 0x019d55 (0xc14f bytes) 000675 1729089108.644199 trunc from 0x040000 to 0x03e819 (0x17e6 bytes) 000676 1729089108.657793 fallocate 0x02cb96 thru 0x03c27e (0xf6e9 bytes) 000678 1729089108.660510 mapread 0x00f44d thru 0x01dae7 (0xe69b bytes) 000679 1729089108.663407 MIRROR_EXTEND to 5 mirrors 000680 1729089108.814477 trunc from 0x03e81a to 0x028bf7 (0x15c22 bytes) 000681 1729089108.839920 write 0x006d2c thru 0x00a820 (0x3af5 bytes) 000682 1729089108.843987 read 0x006385 thru 0x00fe97 (0x9b13 bytes) 000683 1729089108.848623 write 0x014b67 thru 0x01ef11 (0xa3ab bytes) 000684 1729089108.854622 mapwrite 0x035e22 thru 0x0393aa (0x3589 bytes) 000685 1729089108.877890 mapread 0x00fdfc thru 0x01c605 (0xc80a bytes) 000686 1729089108.878577 mapread 0x020222 thru 0x0230d2 (0x2eb1 bytes) 000687 1729089108.883102 fallocate 0x00a1b4 thru 0x017a1b (0xd868 bytes) 000689 1729089108.886760 punch from 0x00459d to 0x00a66e (0x60d2 bytes) 000690 1729089108.890997 MIRROR_EXTEND to 6 mirrors 000691 1729089109.065460 read 0x02c3cc thru 0x0378ac (0xb4e1 bytes) 000692 1729089109.070176 mapwrite 0x038cf7 thru 0x03ce82 (0x418c bytes) 000693 1729089109.122150 fallocate 0x03323c thru 0x038502 (0x52c7 bytes) 000694 1729089109.124810 punch from 0x02c4c3 to 0x036ffb (0xab39 bytes) 000695 1729089109.127327 mapwrite 0x004d3d thru 0x014cba (0xff7e bytes) 000696 1729089109.145780 fallocate 0x002d69 thru 0x00aa4e (0x7ce6 bytes) 000697 1729089109.148938 trunc from 0x03ce83 to 0x01a066 (0x22e1c bytes) 000698 1729089109.161852 MIRROR_EXTEND to 7 mirrors 000700 1729089109.339452 mapwrite 0x0290e1 thru 0x031f85 (0x8ea5 bytes) 000701 1729089109.371899 mapwrite 0x039049 thru 0x03ec2c (0x5be4 bytes) 000702 1729089109.394010 fallocate 0x0024e2 thru 0x011452 (0xef71 bytes) 000703 1729089109.396113 read 0x01a20e thru 0x02940f (0xf202 bytes) 000704 1729089109.401892 MIRROR_RESYNC 7 mirrors 000705 1729089109.871493 read 0x022dce thru 0x02c347 (0x957a bytes) 000706 1729089109.875041 trunc from 0x03ec2d to 0x0202ee (0x1e93e bytes) 000707 1729089109.893785 mapwrite 0x001eb3 thru 0x00f1c4 (0xd312 bytes) 000708 1729089109.915428 fallocate 0x02df86 thru 0x03af28 (0xcfa3 bytes) 000709 1729089109.917961 trunc from 0x0202ef to 0x00b137 (0x151b7 bytes) 000710 1729089109.927235 mapwrite 0x01d693 thru 0x02c338 (0xeca6 bytes) 000711 1729089109.951669 read 0x025197 thru 0x02c338 (0x71a2 bytes) 000712 1729089109.954216 MIRROR_RESYNC 7 mirrors 000714 1729089110.249933 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x776:0x0]: Invalid argument (22) 000716 1729089110.344836 mapread 0x00a396 thru 0x00e65e (0x42c9 bytes) 000717 1729089110.351474 write 0x028bf9 thru 0x037ad2 (0xeeda bytes) 000718 1729089110.370226 punch from 0x01b14c to 0x0251ec (0xa0a1 bytes) 000719 1729089110.372707 mapread 0x02e332 thru 0x0308fb (0x25ca bytes) 000720 1729089110.375402 MIRROR_RESYNC 6 mirrors 000722 1729089110.597737 trunc from 0x037ad3 to 0x00c517 (0x2b5bb bytes) 000723 1729089110.615880 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x776:0x0]: Invalid argument (22) 000724 1729089110.679100 fallocate 0x02ad2d thru 0x032488 (0x775c bytes) 000725 1729089110.680995 punch from 0x00860e to 0x00a627 (0x201a bytes) 000727 1729089110.683236 trunc from 0x00c518 to 0x00fbd5 (0x36be bytes) 000728 1729089110.688674 trunc from 0x00fbd6 to 0x02a69c (0x1aac7 bytes) 000729 1729089110.692372 trunc from 0x02a69d to 0x023f46 (0x6756 bytes) 000730 1729089110.696293 fallocate 0x00ea4b thru 0x0122dc (0x3892 bytes) 000731 1729089110.698023 fallocate 0x018dc4 thru 0x019065 (0x2a2 bytes) 000732 1729089110.699658 mapread 0x0205d9 thru 0x023f46 (0x396e bytes) 000733 1729089110.706282 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x776:0x0]: Invalid argument (22) 000734 1729089110.775682 fallocate 0x0364ae thru 0x0376f2 (0x1245 bytes) 000735 1729089110.778248 write 0x03d2f1 thru 0x03ddea (0xafa bytes) 000736 1729089110.782622 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x776:0x0]: Invalid argument (22) 000737 1729089110.846960 MIRROR_RESYNC 3 mirrors 000738 1729089111.002553 fallocate 0x037b32 thru 0x03ffff (0x84ce bytes) 000739 1729089111.015958 MIRROR_RESYNC 3 mirrors 000740 1729089111.204603 punch from 0x015f3b to 0x025f16 (0xffdc bytes) 000741 1729089111.216167 write 0x02fa1d thru 0x03e555 (0xeb39 bytes) 000742 1729089111.219063 write 0x032609 thru 0x037b14 (0x550c bytes) 000743 1729089111.219769 read 0x0253ea thru 0x02760d (0x2224 bytes) 000744 1729089111.222116 fallocate 0x023505 thru 0x031402 (0xdefe bytes) 000746 1729089111.224669 read 0x03bd6e thru 0x03e555 (0x27e8 bytes) 000747 1729089111.225046 trunc from 0x03e556 to 0x02302f (0x1b526 bytes) 000750 1729089111.232513 mapread 0x01ecc6 thru 0x02302f (0x436a bytes) 000751 1729089111.236116 read 0x0107b7 thru 0x01aaf3 (0xa33d bytes) 000752 1729089111.236476 punch from 0x000e33 to 0x007a5e (0x6c2c bytes) 000753 1729089111.238573 mapwrite 0x01d0d0 thru 0x027877 (0xa7a8 bytes) 000755 1729089111.255415 punch from 0x008676 to 0x0150bc (0xca47 bytes) 000756 1729089111.257524 mapwrite 0x03bb40 thru 0x03ffff (0x44c0 bytes) 000757 1729089111.273126 mapread 0x03e7a5 thru 0x03ffff (0x185b bytes) 000758 1729089111.273702 mapread 0x01b5e5 thru 0x01ec22 (0x363e bytes) 000760 1729089111.274548 write 0x033a58 thru 0x03931c (0x58c5 bytes) 000761 1729089111.275312 mapread 0x0389b2 thru 0x03ab9a (0x21e9 bytes) 000762 1729089111.275732 mapread 0x0396bc thru 0x03f2d6 (0x5c1b bytes) 000763 1729089111.276232 punch from 0x02d4fc to 0x02ea94 (0x1599 bytes) 000764 1729089111.278473 mapwrite 0x0295e5 thru 0x030e32 (0x784e bytes) 000766 1729089111.289210 trunc from 0x040000 to 0x024539 (0x1bac6 bytes) 000767 1729089111.295822 fallocate 0x03e540 thru 0x03ffff (0x1ac0 bytes) 000769 1729089111.298131 fallocate 0x02ba00 thru 0x02e76f (0x2d70 bytes) 000770 1729089111.300019 read 0x013593 thru 0x017edd (0x494b bytes) 000772 1729089111.304089 trunc from 0x040000 to 0x00e798 (0x31867 bytes) 000774 1729089111.310693 write 0x03b105 thru 0x03ffff (0x4efb bytes) 000775 1729089111.311497 mapwrite 0x0241d7 thru 0x02e14b (0x9f75 bytes) 000776 1729089111.335802 mapwrite 0x00d1d1 thru 0x00d5bb (0x3eb bytes) 000777 1729089111.341788 mapread 0x033a2a thru 0x03ffff (0xc5d6 bytes) 000778 1729089111.355306 write 0x0120f2 thru 0x015661 (0x3570 bytes) 000779 1729089111.361894 MIRROR_EXTEND to 4 mirrors 000780 1729089111.556673 punch from 0x01ef49 to 0x02326e (0x4326 bytes) 000781 1729089111.569434 trunc from 0x040000 to 0x001d70 (0x3e28f bytes) 000782 1729089111.582956 mapwrite 0x011135 thru 0x013e22 (0x2cee bytes) 000784 1729089111.607361 mapwrite 0x02eb36 thru 0x0391ce (0xa699 bytes) 000786 1729089111.635547 fallocate 0x01106b thru 0x01abe1 (0x9b77 bytes) 000787 1729089111.640844 MIRROR_EXTEND to 5 mirrors 000788 1729089111.803806 fallocate 0x02cd77 thru 0x02e34e (0x15d8 bytes) 000789 1729089111.817237 read 0x036389 thru 0x0391ce (0x2e46 bytes) 000790 1729089111.821527 fallocate 0x028b0c thru 0x032ece (0xa3c3 bytes) 000791 1729089111.824029 fallocate 0x005631 thru 0x01324e (0xdc1e bytes) 000792 1729089111.826134 write 0x0186ed thru 0x01f619 (0x6f2d bytes) 000793 1729089111.834933 MIRROR_RESYNC 5 mirrors 000794 1729089112.147788 mapread 0x01c467 thru 0x01fb3b (0x36d5 bytes) 000795 1729089112.154406 mapwrite 0x01d231 thru 0x02994b (0xc71b bytes) 000796 1729089112.197160 write 0x02977c thru 0x0308eb (0x7170 bytes) 000797 1729089112.199967 fallocate 0x03907f thru 0x03b4a9 (0x242b bytes) 000799 1729089112.202352 fallocate 0x03ee39 thru 0x03ffff (0x11c7 bytes) 000800 1729089112.207200 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x776:0x0]: Invalid argument (22) 000802 1729089112.300934 write 0x009510 thru 0x00e09e (0x4b8f bytes) 000803 1729089112.306723 write 0x03d9d1 thru 0x03ffff (0x262f bytes) 000805 1729089112.311389 mapread 0x02a15c thru 0x02c0d5 (0x1f7a bytes) 000806 1729089112.315206 trunc from 0x040000 to 0x01d555 (0x22aaa bytes) 000807 1729089112.324143 MIRROR_EXTEND to 5 mirrors 000808 1729089112.520034 punch from 0x00dcf0 to 0x01c98c (0xec9d bytes) 000811 1729089112.535882 write 0x00a1fc thru 0x01385d (0x9662 bytes) 000813 1729089112.542805 read 0x01ab5f thru 0x01d555 (0x29f7 bytes) 000814 1729089112.545397 trunc from 0x01d556 to 0x0053b2 (0x181a3 bytes) 000815 1729089112.552928 punch from 0x003194 to 0x0053b2 (0x221f bytes) 000816 1729089112.555002 mapwrite 0x03e7fe thru 0x03ffff (0x1802 bytes) 000817 1729089112.571587 punch from 0x009c9d to 0x013c17 (0x9f7b bytes) 000820 1729089112.574691 write 0x01e6fc thru 0x02bf96 (0xd89b bytes) 000821 1729089112.580516 read 0x01fd28 thru 0x028ac2 (0x8d9b bytes) 000823 1729089112.581341 mapread 0x03500f thru 0x03ffff (0xaff1 bytes) 000824 1729089112.586229 fallocate 0x034477 thru 0x03d2b6 (0x8e40 bytes) 000825 1729089112.589043 trunc from 0x040000 to 0x00e315 (0x31cea bytes) 000826 1729089112.596012 trunc from 0x00e316 to 0x035af6 (0x277e1 bytes) 000827 1729089112.602016 mapread 0x017e89 thru 0x01859f (0x717 bytes) 000828 1729089112.606252 mapread 0x00b2ed thru 0x0101ad (0x4ec1 bytes) 000830 1729089112.611514 mapwrite 0x001a0b thru 0x00c660 (0xac56 bytes) 000831 1729089112.624754 read 0x032e22 thru 0x035af6 (0x2cd5 bytes) 000832 1729089112.630742 write 0x026b0e thru 0x02cebd (0x63b0 bytes) 000834 1729089112.631912 mapwrite 0x006ac6 thru 0x00cb36 (0x6071 bytes) 000835 1729089112.644522 MIRROR_EXTEND to 6 mirrors 000836 1729089112.835150 trunc from 0x035af7 to 0x00d7bc (0x2833a bytes) 000838 1729089112.865290 write 0x025aa2 thru 0x0356e3 (0xfc42 bytes) 000839 1729089112.866617 fallocate 0x00b75a thru 0x00f705 (0x3fac bytes) 000840 1729089112.869386 mapwrite 0x004fd7 thru 0x00f2a4 (0xa2ce bytes) 000843 1729089112.892605 trunc from 0x0356e4 to 0x02ba3a (0x9ca9 bytes) 000844 1729089112.901929 fallocate 0x03b758 thru 0x03ffff (0x48a8 bytes) 000845 1729089112.905007 punch from 0x000eda to 0x0059ea (0x4b11 bytes) 000846 1729089112.908323 trunc from 0x02ba3b to 0x001742 (0x2a2f8 bytes) 000847 1729089112.918070 MIRROR_RESYNC 6 mirrors 000848 1729089113.180692 trunc from 0x001743 to 0x039fcd (0x3888b bytes) 000849 1729089113.198322 write 0x024bd3 thru 0x0308c2 (0xbcf0 bytes) 000850 1729089113.203341 read 0x038b09 thru 0x039fcd (0x14c5 bytes) 000851 1729089113.205392 trunc from 0x039fce to 0x01fdc8 (0x1a205 bytes) 000852 1729089113.213992 mapwrite 0x01769b thru 0x022f4d (0xb8b3 bytes) 000853 1729089113.239647 mapread 0x0063dc thru 0x01387c (0xd4a1 bytes) 000854 1729089113.247497 trunc from 0x022f4e to 0x00a4cd (0x18a80 bytes) 000855 1729089113.255037 write 0x016959 thru 0x01828b (0x1933 bytes) 000856 1729089113.255834 trunc from 0x01828c to 0x008adb (0xf7b0 bytes) 000857 1729089113.263746 fallocate 0x030759 thru 0x0333db (0x2c83 bytes) 000858 1729089113.268339 MIRROR_EXTEND to 7 mirrors 000859 1729089113.427244 fallocate 0x026826 thru 0x02b1a8 (0x4983 bytes) 000860 1729089113.442022 trunc from 0x0333dc to 0x019c6a (0x19771 bytes) 000861 1729089113.458234 mapread 0x0123b3 thru 0x019c6a (0x78b8 bytes) 000862 1729089113.465906 mapread 0x0010d2 thru 0x00ad57 (0x9c86 bytes) 000864 1729089113.472309 trunc from 0x019c6b to 0x0130c8 (0x6ba2 bytes) 000865 1729089113.478378 mapread 0x001160 thru 0x010856 (0xf6f7 bytes) 000867 1729089113.479444 punch from 0x000a54 to 0x010882 (0xfe2f bytes) 000868 1729089113.481988 trunc from 0x0130c9 to 0x00a636 (0x8a92 bytes) 000870 1729089113.491794 mapread 0x0031c6 thru 0x00a636 (0x7471 bytes) 000871 1729089113.496259 trunc from 0x00a637 to 0x006697 (0x3f9f bytes) 000872 1729089113.502065 read 0x0059ff thru 0x006697 (0xc99 bytes) 000873 1729089113.502519 mapread 0x002eef thru 0x002f72 (0x84 bytes) 000874 1729089113.504933 mapread 0x004cb7 thru 0x006697 (0x19e1 bytes) 000875 1729089113.505330 fallocate 0x00aac5 thru 0x01553a (0xaa76 bytes) 000876 1729089113.507531 write 0x01e420 thru 0x022346 (0x3f27 bytes) 000877 1729089113.508400 mapread 0x00aaeb thru 0x00c809 (0x1d1f bytes) 000878 1729089113.512203 read 0x00d9b7 thru 0x00e88a (0xed4 bytes) 000879 1729089113.512679 trunc from 0x022347 to 0x011763 (0x10be3 bytes) 000880 1729089113.519485 mapread 0x00deae thru 0x011763 (0x38b6 bytes) 000881 1729089113.520487 write 0x007591 thru 0x009c69 (0x26d9 bytes) 000882 1729089113.521250 trunc from 0x011764 to 0x00d384 (0x43df bytes) 000884 1729089113.529263 read 0x00bed1 thru 0x00d384 (0x14b4 bytes) 000885 1729089113.529703 read 0x008fb6 thru 0x00ca64 (0x3aaf bytes) 000886 1729089113.530074 fallocate 0x01637f thru 0x022906 (0xc588 bytes) 000887 1729089113.532868 read 0x002e71 thru 0x00690a (0x3a9a bytes) 000889 1729089113.535553 MIRROR_RESYNC 7 mirrors 000890 1729089113.909832 write 0x014bab thru 0x0186c8 (0x3b1e bytes) 000891 1729089113.929478 fallocate 0x0205b1 thru 0x024bfd (0x464d bytes) 000892 1729089113.932012 read 0x01744b thru 0x01b143 (0x3cf9 bytes) 000893 1729089113.934912 write 0x034aa2 thru 0x037dc6 (0x3325 bytes) 000894 1729089113.936083 mapread 0x032f73 thru 0x037dc6 (0x4e54 bytes) 000895 1729089113.940396 trunc from 0x037dc7 to 0x01fcee (0x180d8 bytes) 000896 1729089113.957103 write 0x0086f8 thru 0x0137a6 (0xb0af bytes) 000897 1729089113.963100 fallocate 0x03b60d thru 0x03ffff (0x49f3 bytes) 000899 1729089113.965885 trunc from 0x040000 to 0x02eea3 (0x1115c bytes) 000901 1729089113.973511 punch from 0x00fba0 to 0x0186be (0x8b1f bytes) 000902 1729089113.980774 write 0x00cab1 thru 0x01a9a6 (0xdef6 bytes) 000903 1729089113.981968 mapread 0x02c8f6 thru 0x02eea3 (0x25ae bytes) 000905 1729089113.987470 write 0x00b1ea thru 0x013762 (0x8579 bytes) 000906 1729089113.988376 read 0x02010f thru 0x029cd2 (0x9bc4 bytes) 000907 1729089113.992729 write 0x015f22 thru 0x019f35 (0x4014 bytes) 000908 1729089113.994246 write 0x03cd8f thru 0x03ffff (0x3271 bytes) 000910 1729089113.995216 trunc from 0x040000 to 0x03899e (0x7661 bytes) 000911 1729089114.007707 MIRROR_RESYNC 7 mirrors 000912 1729089114.486651 write 0x0104d6 thru 0x014609 (0x4134 bytes) 000913 1729089114.505308 mapwrite 0x00dc03 thru 0x017f2f (0xa32d bytes) 000914 1729089114.521457 fallocate 0x03d6a6 thru 0x03ffff (0x295a bytes) 000915 1729089114.526201 MIRROR_RESYNC 7 mirrors 000916 1729089114.875995 write 0x00c3ba thru 0x01215c (0x5da3 bytes) 000917 1729089114.892507 mapread 0x01b116 thru 0x026097 (0xaf82 bytes) 000918 1729089114.899229 fallocate 0x007677 thru 0x009fd6 (0x2960 bytes) 000919 1729089114.901640 trunc from 0x040000 to 0x015e27 (0x2a1d8 bytes) 000920 1729089114.924032 mapread 0x002142 thru 0x00d4b2 (0xb371 bytes) 000921 1729089114.933437 write 0x00e312 thru 0x01bbe8 (0xd8d7 bytes) 000922 1729089114.934497 trunc from 0x01bbe9 to 0x037ffd (0x1c415 bytes) 000923 1729089114.942287 mapwrite 0x032319 thru 0x03ffff (0xdce7 bytes) 000924 1729089114.973596 mapread 0x0311ce thru 0x031c7d (0xab0 bytes) 000925 1729089114.976594 MIRROR_RESYNC 7 mirrors 000927 1729089115.378542 read 0x00204f thru 0x00cfe3 (0xaf95 bytes) 000928 1729089115.382828 fallocate 0x0025f5 thru 0x00aa20 (0x842c bytes) 000929 1729089115.397791 punch from 0x00f7cc to 0x015377 (0x5bac bytes) 000931 1729089115.400862 read 0x015424 thru 0x01cf0c (0x7ae9 bytes) 000934 1729089115.407326 write 0x0019e2 thru 0x00b5d6 (0x9bf5 bytes) 000935 1729089115.410224 write 0x03e2a4 thru 0x03ffff (0x1d5c bytes) 000936 1729089115.410918 trunc from 0x040000 to 0x02f60c (0x109f3 bytes) 000937 1729089115.418381 punch from 0x00b3ae to 0x00c0cc (0xd1f bytes) 000938 1729089115.423848 punch from 0x000b19 to 0x00b416 (0xa8fe bytes) 000940 1729089115.426442 mapwrite 0x023beb thru 0x023e98 (0x2ae bytes) 000941 1729089115.435914 trunc from 0x02f60d to 0x0048d9 (0x2ad33 bytes) 000942 1729089115.447655 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x776:0x0]: Invalid argument (22) 000943 1729089115.526597 trunc from 0x0048da to 0x00b481 (0x6ba8 bytes) 000944 1729089115.534258 mapwrite 0x0083f7 thru 0x013048 (0xac52 bytes) 000945 1729089115.562866 trunc from 0x013049 to 0x02abbc (0x17b74 bytes) 000946 1729089115.567747 mapwrite 0x0395ee thru 0x03ae71 (0x1884 bytes) 000948 1729089115.585737 write 0x01b2ef thru 0x029c70 (0xe982 bytes) 000949 1729089115.592200 mapwrite 0x032473 thru 0x0328d4 (0x462 bytes) 000950 1729089115.600989 write 0x017d19 thru 0x01c84c (0x4b34 bytes) 000951 1729089115.603815 MIRROR_EXTEND to 7 mirrors 000952 1729089115.786399 write 0x03d18f thru 0x03ffff (0x2e71 bytes) 000953 1729089115.801913 mapread 0x00a62d thru 0x0106dd (0x60b1 bytes) 000954 1729089115.810516 mapread 0x000426 thru 0x001d8b (0x1966 bytes) 000955 1729089115.816482 trunc from 0x040000 to 0x01e29c (0x21d63 bytes) 000956 1729089115.830096 write 0x0212b7 thru 0x0271fc (0x5f46 bytes) 000958 1729089115.831366 write 0x0382d9 thru 0x03ba23 (0x374b bytes) 000959 1729089115.834775 MIRROR_RESYNC 7 mirrors 000960 1729089116.527157 mapread 0x01e596 thru 0x02de7a (0xf8e5 bytes) 000961 1729089116.537760 read 0x02eb65 thru 0x02f76e (0xc0a bytes) 000963 1729089116.541078 MIRROR_EXTEND to 8 mirrors 000964 1729089116.758481 read 0x009480 thru 0x00a6ec (0x126d bytes) 000965 1729089116.761819 trunc from 0x03ba24 to 0x014dfe (0x26c25 bytes) 000966 1729089116.779301 trunc from 0x014dff to 0x037f8a (0x2318c bytes) 000967 1729089116.787773 read 0x030762 thru 0x037f8a (0x7829 bytes) 000968 1729089116.792721 mapwrite 0x012182 thru 0x015ff0 (0x3e6f bytes) 000969 1729089116.801684 mapread 0x0173ec thru 0x019797 (0x23ac bytes) 000970 1729089116.802233 punch from 0x0061f3 to 0x01098a (0xa798 bytes) 000971 1729089116.805797 read 0x022e62 thru 0x02fd41 (0xcee0 bytes) 000972 1729089116.806465 trunc from 0x037f8b to 0x0095da (0x2e9b0 bytes) 000973 1729089116.816097 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x776:0x0]: Invalid argument (22) 000975 1729089116.892103 write 0x03421e thru 0x0347bd (0x5a0 bytes) 000977 1729089116.894835 punch from 0x010e77 to 0x01df32 (0xd0bc bytes) 000978 1729089116.896618 read 0x006a96 thru 0x009e7d (0x33e8 bytes) 000979 1729089116.898658 mapwrite 0x01f830 thru 0x02aa82 (0xb253 bytes) 000980 1729089116.918270 read 0x00fccb thru 0x01582f (0x5b65 bytes) 000982 1729089116.922298 punch from 0x0261de to 0x029aab (0x38ce bytes) 000984 1729089116.924444 mapwrite 0x03a2bb thru 0x03b443 (0x1189 bytes) 000985 1729089116.938606 read 0x01d6af thru 0x02a6ff (0xd051 bytes) 000986 1729089116.943046 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x776:0x0]: Invalid argument (22) 000987 1729089117.015031 write 0x03f05c thru 0x03ffff (0xfa4 bytes) 000988 1729089117.016989 punch from 0x01c66b to 0x025bec (0x9582 bytes) 000989 1729089117.020756 read 0x004501 thru 0x013b2b (0xf62b bytes) 000990 1729089117.024717 mapread 0x0379aa thru 0x03ea3d (0x7094 bytes) 000991 1729089117.032004 trunc from 0x040000 to 0x004808 (0x3b7f7 bytes) 000992 1729089117.038961 read 0x0042f5 thru 0x004808 (0x514 bytes) 000995 1729089117.040056 punch from 0x00294f to 0x004808 (0x1eba bytes) 000996 1729089117.042307 mapwrite 0x014775 thru 0x017e2b (0x36b7 bytes) 000999 1729089117.062418 trunc from 0x017e2c to 0x012831 (0x55fa bytes) 001000 1729089117.068881 fallocate 0x02fbaf thru 0x033c58 (0x40aa bytes) All operations completed A-OK! PASS 70a (23s) == sanity-flr test 200a: stress test ===================== 10:31:58 (1729089118) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre2 Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2795456 .. Extending file size to 6060512 .. done Extending file size to 3073536 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7055456 .. Extending file size to 6153216 .. failed Extending file size to 7665664 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7417472 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8725120 .. Extending file size to 8277024 .. Extending file size to 8780928 .. Extending file size to 8639840 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9163584 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8746528 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8901792 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9211552 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 2814 2815 2817 2818 2820 done 192.168.202.133@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg233-client.virtnet /mnt/lustre2 (opts:) 192.168.202.133@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg233-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (67s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 10:33:05 (1729089185) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre2 Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1275584 .. Extending file size to 4856256 .. done Extending file size to 7366240 .. Extending file size to 6778368 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7823776 .. Extending file size to 8956032 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8271616 .. Extending file size to 8450688 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8603744 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8604800 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8621696 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 5226 5227 5228 5229 5233 192.168.202.133@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg233-client.virtnet /mnt/lustre2 (opts:) 192.168.202.133@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg233-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 10:34:08 (1729089248) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2496 1285192 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2816 1284872 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 21232 3585788 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 15524 3591496 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 36756 7177284 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2496 1285192 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2816 1284872 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 21232 3585788 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 15524 3591496 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 36756 7177284 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0397487 s, 52.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0595026 s, 35.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.133@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg233-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 10:34:13 (1729089253) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0332184 s, 63.1 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 10:34:15 (1729089255) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5e:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaa:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaa:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xa7:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 10:34:17 (1729089257) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0272372 s, 77.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x948:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x946:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x947:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 10:34:19 (1729089259) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0284447 s, 73.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x949:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x94a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x948:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x949:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x94a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x948:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x949:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 10:34:21 (1729089261) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0262199 s, 80.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x94b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x94a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x94b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x94a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x94b:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 10:34:44 (1729089284) Waiting for MDT destroys to complete oleg233-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg233-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg233-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.017109 s, 61.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x94c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x94d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x94c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x94c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x94d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x94c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x94d:0x0] } PASS 204d (30s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 10:35:15 (1729089315) Waiting for MDT destroys to complete oleg233-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg233-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg233-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0330477 s, 63.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (31s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 10:35:46 (1729089346) Waiting for MDT destroys to complete oleg233-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg233-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg233-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool oleg233-server: Pool lustre.test_204f created Adding targets to pool oleg233-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg233-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0281809 s, 74.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg233-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg233-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg233-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' Updated after 2s: want 'foo' got 'foo' PASS 204f (48s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 10:36:34 (1729089394) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 10:36:36 (1729089396) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 10:36:38 (1729089398) oleg233-server: Pool lustre.test_206 created Adding targets to pool oleg233-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg233-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg233-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg233-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg233-server: Pool lustre.test_206 destroyed PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 10:36:51 (1729089411) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0213438 s, 49.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0137758 s, 76.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x3e1bc04e Mirror 3: 0x3e1bc04e PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 10:36:54 (1729089414) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 10:36:55 (1729089415) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 10:36:56 (1729089416) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0134375 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2490000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xc80000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xc80000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xc80000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1410000 Read this (4 bytes): BBBB 192.168.202.133@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg233-client.virtnet /mnt/lustre2 (opts:) PASS 209a (15s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 10:37:11 (1729089431) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00636184 s, 1.3 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 10:37:14 (1729089434) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0181125 s, 57.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7b lmm_fid: [0x200000405:0x7b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7b lmm_fid: [0x200000405:0x7b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb7:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611259 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-10-16 10:37:14.000000000 -0400 Modify: 2024-10-16 10:37:14.000000000 -0400 Change: 2024-10-16 10:37:15.000000000 -0400 Birth: - PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 10:37:16 (1729089436) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0121676 s, 86.2 MB/s oleg233-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg233-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg233-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg233-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 oleg233-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg233-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg233-server: O/280000401/d21/b5: File not found by ext2_lookup PASS 210b (10s) == sanity-flr test complete, duration 737 sec ============ 10:37:26 (1729089446) === sanity-flr: start cleanup 10:37:27 (1729089447) === === sanity-flr: finish cleanup 10:37:30 (1729089450) ===