-----============= acceptance-small: sanity-flr ============----- Thu Jun 5 07:06:11 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg323-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 07:06:24 (1749121584) === oleg323-client.virtnet: executing check_config_client /mnt/lustre oleg323-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg323-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012b895000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012b895000.idle_timeout=debug disable quota as required oleg323-server: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 07:06:37 (1749121597) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7262] == sanity-flr test 0a: lfs mirror create with -N option == 07:06:38 (1749121598) PASS 0a (7s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 07:06:45 (1749121605) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 07:06:46 (1749121606) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 07:06:47 (1749121607) PASS 0d (9s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 07:06:56 (1749121616) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 07:06:57 (1749121617) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 07:06:59 (1749121619) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88012b895000.stats=clear osc.lustre-OST0001-osc-ffff88012b895000.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 07:07:02 (1749121622) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (5s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 07:07:07 (1749121627) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 07:07:11 (1749121631) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 07:07:13 (1749121633) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 07:07:17 (1749121637) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 07:07:20 (1749121640) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 07:07:21 (1749121641) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 07:07:25 (1749121645) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 07:07:29 (1749121649) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 07:07:32 (1749121652) 17+0 records in 17+0 records out 17825792 bytes (18 MB) copied, 1.17809 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0822114 s, 12.8 MB/s PASS 21 (5s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 07:07:37 (1749121657) 11+0 records in 11+0 records out 11534336 bytes (12 MB) copied, 0.0804638 s, 143 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 07:07:40 (1749121660) Stopping /mnt/lustre-ost1 (opts:) on oleg323-server oleg323-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012b895000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012b895000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg323-server: oleg323-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg323-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg323-server oleg323-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012b895000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012b895000.ost_server_uuid in DISCONN state after 4 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg323-server: oleg323-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg323-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (26s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 07:08:06 (1749121686) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.238275 s, 44.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg323-server oleg323-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012b895000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012b895000.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg323-server: oleg323-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg323-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (14s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 07:08:20 (1749121700) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg323-server oleg323-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012b895000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012b895000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg323-server: oleg323-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg323-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg323-server oleg323-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012b895000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012b895000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg323-server: oleg323-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg323-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (26s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 07:08:46 (1749121726) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.927593 s, 56.5 MB/s -rw-r--r-- 1 root root 52428800 Jun 5 07:08 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.323s user 0m0.000s sys 0m0.118s Stopping /mnt/lustre-ost1 (opts:) on oleg323-server oleg323-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012b895000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012b895000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... real 0m0.375s user 0m0.002s sys 0m0.115s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg323-server: oleg323-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg323-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg323-server oleg323-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012b895000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012b895000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost1)... real 0m0.371s user 0m0.000s sys 0m0.117s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg323-server: oleg323-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg323-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg323-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012b895000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012b895000.ost_server_uuid in FULL state after 0 sec PASS 33b (26s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 07:09:12 (1749121752) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (0s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 07:09:13 (1749121753) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 07:09:14 (1749121754) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 07:09:15 (1749121755) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 07:09:17 (1749121757) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.125344 s, 100 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 07:09:20 (1749121760) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00934082 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100247 s, 105 MB/s PASS 36b (1s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 07:09:21 (1749121761) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 07:09:24 (1749121764) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000873909 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000582217 s, 1.8 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00071424 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00306142 s, 334 kB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 07:09:27 (1749121767) create test files with size 15745985 .. 5491a43937f56eac9922b52700a2a681 - d6b55c5791b5486b21bcdd87c929e9d1 - 76c09a24777e1747b1ee6bf200da2dc8 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg323-client.virtnet /mnt/lustre (opts:) Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre PASS 37 (7s) == sanity-flr test 38: resync ============================ 07:09:34 (1749121774) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 07:09:37 (1749121777) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 07:09:39 (1749121779) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0396557 s, 26.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108636 s, 96.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 07:09:41 (1749121781) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0400598 s, 52.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.141395 s, 29.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0615637 s, 34.1 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (4s) == sanity-flr test 42: lfs mirror verify ================= 07:09:45 (1749121785) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 07:09:46 (1749121786) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 07:09:47 (1749121787) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00980605 s, 107 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 07:09:49 (1749121789) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 07:09:50 (1749121790) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4e:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 07:09:53 (1749121793) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0807661 s, 130 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg323-client.virtnet /mnt/lustre (opts:) Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (22s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 07:10:15 (1749121815) llite.lustre-ffff8800a2b24000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0791941 s, 132 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg323-client.virtnet /mnt/lustre (opts:) Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff88012bc8e800.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 07:10:19 (1749121819) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (10s) == sanity-flr test 46: Verify setstripe --copy option ==== 07:10:29 (1749121829) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (10s) == sanity-flr test 47: Verify mirror obj alloc =========== 07:10:39 (1749121839) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 07:10:40 (1749121840) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.095513 s, 32.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0727318 s, 43.3 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 9a9a5a9cb71c22387cacd69d5a8367da - mirror 1 checksum: 81da09dc6c09029815d76b85e4834a4f - mirror 2 checksum: 9a9a5a9cb71c22387cacd69d5a8367da - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jun 5 07:10:42 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jun 5 07:10:42 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 9a9a5a9cb71c22387cacd69d5a8367da - mirror 1 checksum: 81da09dc6c09029815d76b85e4834a4f - mirror 2 checksum: 81da09dc6c09029815d76b85e4834a4f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jun 5 07:10:42 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jun 5 07:10:42 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } PASS 48 (4s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 07:10:44 (1749121844) Failing mds1 on oleg323-server Stopping /mnt/lustre-mds1 (opts:) on oleg323-server 07:10:46 (1749121846) shut down facet: mds1 facet_host: oleg323-server facet_failover_host: oleg323-server Failover mds1 to oleg323-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg323-server: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-MDT0000 07:11:00 (1749121860) targets are mounted 07:11:00 (1749121860) facet_failover done oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0180428 s, 58.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 50A (22s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 07:11:06 (1749121866) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 07:11:07 (1749121867) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 07:11:08 (1749121868) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00478144 s, 3.4 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 07:11:11 (1749121871) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 07:11:12 (1749121872) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 07:11:13 (1749121873) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (7s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 07:11:20 (1749121880) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 07:11:40 (1749121900) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 07:11:58 (1749121918) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 07:12:16 (1749121936) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 07:12:18 (1749121938) mirror create pid 22885 mirror split pid 22886 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4948: 22885 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4948: 22886 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 07:13:20 (1749122000) Seed set to 2001 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1749122001.186798 write 0x00f1f2 thru 0x01df81 (0xed90 bytes) 000002 1749122001.190423 mapread 0x00078a thru 0x003c57 (0x34ce bytes) 000003 1749122001.199175 mapwrite 0x015bcc thru 0x025493 (0xf8c8 bytes) 000004 1749122001.264171 write 0x02a731 thru 0x02fb1f (0x53ef bytes) 000005 1749122001.266706 MIRROR_EXTEND to 2 mirrors 000006 1749122001.403044 mapread 0x00e910 thru 0x011cf3 (0x33e4 bytes) 000009 1749122001.414776 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000010 1749122001.456568 write 0x02f504 thru 0x03c60d (0xd10a bytes) 000011 1749122001.461807 read 0x030047 thru 0x038a91 (0x8a4b bytes) 000013 1749122001.463614 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x16180 000014 1749122001.640374 trunc from 0x03c60e to 0x01617f (0x2648e bytes) 000015 1749122001.670351 write 0x00650e thru 0x007956 (0x1449 bytes) 000017 1749122001.675643 MIRROR_EXTEND to 3 mirrors 000018 1749122001.805146 write 0x030437 thru 0x03fbea (0xf7b4 bytes) 000019 1749122001.827239 mapwrite 0x02a545 thru 0x02ec0f (0x46cb bytes) 000021 1749122001.857128 read 0x00e73d thru 0x00ff57 (0x181b bytes) 000022 1749122001.859761 write 0x01367a thru 0x0148a1 (0x1228 bytes) 000025 1749122001.864038 write 0x029461 thru 0x037e72 (0xea12 bytes) truncating to largest ever: 0x33bd8 000027 1749122001.866281 trunc from 0x03fbeb to 0x033bd7 (0xc013 bytes) 000028 1749122001.879077 mapwrite 0x034b40 thru 0x03c42f (0x78f0 bytes) truncating to largest ever: 0x3ee94 000030 1749122001.916567 trunc from 0x03c430 to 0x03ee93 (0x2a64 bytes) 000031 1749122001.922623 mapwrite 0x025d73 thru 0x027850 (0x1ade bytes) 000032 1749122001.946003 read 0x02fb8d thru 0x03d607 (0xda7b bytes) 000034 1749122001.950390 mapread 0x03dab5 thru 0x03ee93 (0x13df bytes) 000036 1749122001.951396 read 0x01d2e3 thru 0x022114 (0x4e32 bytes) 000037 1749122001.954717 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000039 1749122002.014872 read 0x0326fe thru 0x033248 (0xb4b bytes) 000040 1749122002.017335 trunc from 0x03ee94 to 0x03aeb7 (0x3fdc bytes) 000041 1749122002.025146 trunc from 0x03aeb8 to 0x01675a (0x2475d bytes) 000045 1749122002.037343 MIRROR_RESYNC 2 mirrors 000046 1749122002.163638 mapwrite 0x00eb01 thru 0x01cbf2 (0xe0f2 bytes) 000051 1749122002.221171 mapwrite 0x032333 thru 0x03ffff (0xdccd bytes) 000053 1749122002.251499 MIRROR_EXTEND to 3 mirrors 000054 1749122002.409074 MIRROR_RESYNC 3 mirrors 000056 1749122002.568257 write 0x02177d thru 0x02dd92 (0xc616 bytes) 000057 1749122002.598293 mapwrite 0x0029ca thru 0x009fed (0x7624 bytes) 000058 1749122002.636047 MIRROR_RESYNC 3 mirrors 000059 1749122002.795537 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000060 1749122002.875305 mapread 0x014688 thru 0x01f32c (0xaca5 bytes) 000061 1749122002.887908 write 0x02f8d8 thru 0x03e011 (0xe73a bytes) 000064 1749122002.932499 trunc from 0x040000 to 0x025cd0 (0x1a32f bytes) 000065 1749122002.954887 mapread 0x007292 thru 0x015009 (0xdd78 bytes) 000071 1749122002.966357 trunc from 0x025cd1 to 0x02470e (0x15c2 bytes) 000073 1749122002.972233 read 0x0245fb thru 0x02470e (0x114 bytes) 000074 1749122002.976680 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000075 1749122003.021960 mapwrite 0x036441 thru 0x03ea29 (0x85e9 bytes) 000077 1749122003.068734 read 0x023b9e thru 0x0281aa (0x460d bytes) 000078 1749122003.072911 mapread 0x019d84 thru 0x01f4b4 (0x5731 bytes) 000080 1749122003.077274 write 0x01f680 thru 0x02ec7f (0xf600 bytes) 000081 1749122003.080571 mapwrite 0x007021 thru 0x00cc83 (0x5c63 bytes) 000082 1749122003.116172 mapread 0x01e99a thru 0x0294e3 (0xab4a bytes) 000083 1749122003.116860 mapread 0x035e91 thru 0x03ea29 (0x8b99 bytes) 000084 1749122003.117281 trunc from 0x03ea2a to 0x03148d (0xd59c bytes) 000089 1749122003.126801 mapread 0x018306 thru 0x0215ff (0x92fa bytes) 000091 1749122003.128117 mapwrite 0x0367fc thru 0x03e762 (0x7f67 bytes) 000095 1749122003.174018 write 0x009820 thru 0x0174e5 (0xdcc6 bytes) 000098 1749122003.179830 MIRROR_EXTEND to 2 mirrors 000099 1749122003.361642 mapwrite 0x03104d thru 0x03cacd (0xba81 bytes) 000100 1749122003.440840 read 0x00ecd0 thru 0x01807f (0x93b0 bytes) 000102 1749122003.444111 mapwrite 0x03010a thru 0x038c06 (0x8afd bytes) 000103 1749122003.465467 read 0x00d75d thru 0x015114 (0x79b8 bytes) 000104 1749122003.472068 MIRROR_RESYNC 2 mirrors 000105 1749122003.610699 write 0x013147 thru 0x022315 (0xf1cf bytes) 000106 1749122003.634895 read 0x02b892 thru 0x02bd73 (0x4e2 bytes) 000109 1749122003.638077 trunc from 0x03e763 to 0x0122dc (0x2c486 bytes) 000110 1749122003.651350 read 0x00230d thru 0x01166b (0xf35f bytes) 000112 1749122003.655919 write 0x0255f9 thru 0x03350d (0xdf15 bytes) 000113 1749122003.659403 MIRROR_RESYNC 2 mirrors 000114 1749122003.824283 mapread 0x01c47c thru 0x0232b8 (0x6e3d bytes) 000120 1749122003.841152 trunc from 0x03350e to 0x03739d (0x3e90 bytes) 000121 1749122003.880904 write 0x00e734 thru 0x01daa1 (0xf36e bytes) 000122 1749122003.886465 mapread 0x01b08a thru 0x0268df (0xb856 bytes) 000123 1749122003.896272 read 0x0155eb thru 0x01c34b (0x6d61 bytes) 000124 1749122003.898914 MIRROR_EXTEND to 3 mirrors 000128 1749122004.137584 mapread 0x022c2c thru 0x031f05 (0xf2da bytes) 000129 1749122004.161841 mapwrite 0x039974 thru 0x03ffff (0x668c bytes) 000130 1749122004.216002 MIRROR_EXTEND to 4 mirrors 000131 1749122004.376030 mapread 0x03c7f0 thru 0x03ffff (0x3810 bytes) 000132 1749122004.386851 write 0x02e25c thru 0x03db70 (0xf915 bytes) 000135 1749122004.416091 read 0x005170 thru 0x0070f5 (0x1f86 bytes) 000136 1749122004.420678 write 0x033313 thru 0x03ffff (0xcced bytes) 000137 1749122004.421638 mapread 0x034ca0 thru 0x03a6b6 (0x5a17 bytes) 000140 1749122004.423840 trunc from 0x040000 to 0x03d834 (0x27cb bytes) 000141 1749122004.442874 mapwrite 0x029307 thru 0x031d14 (0x8a0e bytes) 000143 1749122004.473471 write 0x0151f5 thru 0x0173c0 (0x21cc bytes) 000144 1749122004.477786 write 0x023995 thru 0x02ab8a (0x71f6 bytes) 000146 1749122004.481919 mapread 0x01b028 thru 0x0242f9 (0x92d2 bytes) 000147 1749122004.482538 trunc from 0x03d835 to 0x00bc3b (0x31bf9 bytes) 000148 1749122004.489760 trunc from 0x00bc3c to 0x02ab76 (0x1ef3b bytes) 000149 1749122004.494086 read 0x018c88 thru 0x020b6f (0x7ee8 bytes) 000150 1749122004.497517 mapwrite 0x015fe2 thru 0x01e963 (0x8982 bytes) 000151 1749122004.520007 write 0x02f5f3 thru 0x03eb89 (0xf597 bytes) 000152 1749122004.522753 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000153 1749122004.600956 write 0x028f5d thru 0x02f7de (0x6882 bytes) 000155 1749122004.605255 read 0x020cb5 thru 0x030b65 (0xfeb1 bytes) 000158 1749122004.610085 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000159 1749122004.677180 MIRROR_EXTEND to 3 mirrors 000160 1749122004.832458 trunc from 0x03eb8a to 0x03754a (0x763f bytes) 000162 1749122004.856679 MIRROR_EXTEND to 4 mirrors 000163 1749122004.988464 write 0x024b12 thru 0x026c62 (0x2151 bytes) 000164 1749122005.014294 mapread 0x020937 thru 0x02bfa1 (0xb66b bytes) 000167 1749122005.023956 MIRROR_EXTEND to 5 mirrors 000169 1749122005.144739 mapread 0x00d85c thru 0x010353 (0x2af8 bytes) 000170 1749122005.150081 write 0x00c912 thru 0x01219c (0x588b bytes) 000171 1749122005.171964 mapread 0x009889 thru 0x0126db (0x8e53 bytes) 000173 1749122005.175031 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000174 1749122005.234839 write 0x023b91 thru 0x02f631 (0xbaa1 bytes) 000175 1749122005.239616 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000176 1749122005.304501 MIRROR_RESYNC 3 mirrors 000177 1749122005.500111 mapwrite 0x02ac5f thru 0x037928 (0xccca bytes) 000179 1749122005.568402 mapread 0x00b4dc thru 0x01aba7 (0xf6cc bytes) 000185 1749122005.578783 trunc from 0x037929 to 0x00714e (0x307da bytes) truncating to largest ever: 0x3f389 000186 1749122005.588515 trunc from 0x00714f to 0x03f388 (0x3823a bytes) 000189 1749122005.595835 trunc from 0x03f389 to 0x023724 (0x1bc64 bytes) 000191 1749122005.603696 trunc from 0x023725 to 0x0151fb (0xe529 bytes) 000192 1749122005.610814 MIRROR_RESYNC 3 mirrors 000193 1749122005.775897 mapread 0x001d31 thru 0x00df39 (0xc209 bytes) 000194 1749122005.789971 mapread 0x002fe9 thru 0x006736 (0x374e bytes) 000195 1749122005.790360 mapread 0x00c2d9 thru 0x0140ee (0x7e16 bytes) 000198 1749122005.792764 mapread 0x0144c9 thru 0x0151fb (0xd33 bytes) 000199 1749122005.793147 mapwrite 0x027e7c thru 0x02e285 (0x640a bytes) 000201 1749122005.845633 mapwrite 0x0005f8 thru 0x00b6c9 (0xb0d2 bytes) 000202 1749122005.873754 mapwrite 0x02a78b thru 0x02b185 (0x9fb bytes) 000204 1749122005.897819 trunc from 0x02e286 to 0x037134 (0x8eaf bytes) 000205 1749122005.909284 MIRROR_RESYNC 3 mirrors 000206 1749122006.104576 read 0x00bb8b thru 0x00ed54 (0x31ca bytes) 000208 1749122006.108094 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000209 1749122006.180582 trunc from 0x037135 to 0x008943 (0x2e7f1 bytes) 000210 1749122006.208417 mapread 0x0040cc thru 0x008943 (0x4878 bytes) 000211 1749122006.215298 mapread 0x00113a thru 0x008943 (0x780a bytes) 000212 1749122006.215964 mapwrite 0x00785b thru 0x00c5b6 (0x4d5c bytes) 000213 1749122006.248237 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000214 1749122006.287890 mapread 0x0022a9 thru 0x009def (0x7b47 bytes) 000218 1749122006.295156 write 0x03c587 thru 0x03dc80 (0x16fa bytes) 000220 1749122006.296222 write 0x028328 thru 0x02f8f9 (0x75d2 bytes) 000222 1749122006.301249 trunc from 0x03dc81 to 0x035c88 (0x7ff8 bytes) 000223 1749122006.307472 read 0x01ee08 thru 0x027326 (0x851f bytes) 000224 1749122006.309750 trunc from 0x035c89 to 0x00f090 (0x26bf8 bytes) 000226 1749122006.315646 read 0x002b05 thru 0x009337 (0x6833 bytes) 000228 1749122006.316407 read 0x001513 thru 0x005f3d (0x4a2b bytes) 000229 1749122006.316778 write 0x007c38 thru 0x010241 (0x860a bytes) 000230 1749122006.317597 mapwrite 0x03b9ae thru 0x03ffff (0x4652 bytes) 000231 1749122006.352276 read 0x01b5ed thru 0x027088 (0xba9c bytes) 000234 1749122006.354324 trunc from 0x040000 to 0x0246bb (0x1b944 bytes) 000235 1749122006.361894 MIRROR_EXTEND to 2 mirrors 000239 1749122006.494285 mapwrite 0x01e2a3 thru 0x021dfb (0x3b59 bytes) 000241 1749122006.550583 write 0x0355d6 thru 0x0399a6 (0x43d1 bytes) 000242 1749122006.551531 write 0x03e227 thru 0x03ffff (0x1dd9 bytes) 000244 1749122006.552711 read 0x00ca8a thru 0x015b30 (0x90a7 bytes) 000245 1749122006.556084 mapwrite 0x026e63 thru 0x029d5b (0x2ef9 bytes) 000248 1749122006.583920 mapread 0x004101 thru 0x009156 (0x5056 bytes) 000250 1749122006.589378 mapwrite 0x00dae6 thru 0x010be9 (0x3104 bytes) 000251 1749122006.610106 trunc from 0x040000 to 0x001da3 (0x3e25c bytes) 000252 1749122006.630349 trunc from 0x001da4 to 0x011f6f (0x101cc bytes) 000253 1749122006.635302 trunc from 0x011f70 to 0x01dcd2 (0xbd63 bytes) 000254 1749122006.641375 mapread 0x01d407 thru 0x01dcd2 (0x8cc bytes) 000255 1749122006.644763 write 0x022b33 thru 0x025b8b (0x3059 bytes) 000256 1749122006.645486 read 0x0159ce thru 0x02537f (0xf9b2 bytes) 000258 1749122006.651901 mapwrite 0x000fa7 thru 0x0016c5 (0x71f bytes) 000259 1749122006.668322 mapread 0x02173f thru 0x025b8b (0x444d bytes) 000260 1749122006.668834 trunc from 0x025b8c to 0x01468e (0x114fd bytes) 000261 1749122006.673388 trunc from 0x01468f to 0x01999e (0x5310 bytes) 000262 1749122006.677801 mapread 0x00427c thru 0x0111b5 (0xcf3a bytes) 000264 1749122006.680508 MIRROR_RESYNC 2 mirrors 000266 1749122006.805388 read 0x015b27 thru 0x01999e (0x3e78 bytes) 000267 1749122006.807408 read 0x0171ca thru 0x01999e (0x27d5 bytes) 000268 1749122006.807638 mapwrite 0x02f5d6 thru 0x03d5b1 (0xdfdc bytes) 000274 1749122006.876020 read 0x00ae30 thru 0x013802 (0x89d3 bytes) 000275 1749122006.876487 mapwrite 0x0096c5 thru 0x011976 (0x82b2 bytes) 000277 1749122006.899244 mapread 0x03bac4 thru 0x03c552 (0xa8f bytes) 000278 1749122006.899830 mapread 0x00be50 thru 0x016f65 (0xb116 bytes) 000281 1749122006.901103 mapwrite 0x02c383 thru 0x02c530 (0x1ae bytes) 000284 1749122006.920177 mapwrite 0x03e3b5 thru 0x03ffff (0x1c4b bytes) 000285 1749122006.946997 mapwrite 0x01dad1 thru 0x028d41 (0xb271 bytes) 000287 1749122006.973055 MIRROR_EXTEND to 3 mirrors 000289 1749122007.159519 write 0x039c2a thru 0x03ffff (0x63d6 bytes) 000290 1749122007.181520 mapwrite 0x0305fd thru 0x032034 (0x1a38 bytes) 000293 1749122007.210512 mapread 0x02ead5 thru 0x03bab8 (0xcfe4 bytes) 000294 1749122007.213037 MIRROR_EXTEND to 4 mirrors 000296 1749122007.396570 mapwrite 0x023264 thru 0x024fd7 (0x1d74 bytes) 000298 1749122007.473297 mapwrite 0x0207fb thru 0x02da2a (0xd230 bytes) 000299 1749122007.503878 read 0x024d70 thru 0x02b21c (0x64ad bytes) 000300 1749122007.504334 mapread 0x038441 thru 0x03ffff (0x7bbf bytes) 000301 1749122007.508259 mapread 0x024711 thru 0x032740 (0xe030 bytes) 000305 1749122007.510978 read 0x00adab thru 0x00c690 (0x18e6 bytes) 000307 1749122007.518202 MIRROR_EXTEND to 5 mirrors 000309 1749122007.662828 write 0x03f6d6 thru 0x03ffff (0x92a bytes) 000310 1749122007.688320 MIRROR_EXTEND to 6 mirrors 000313 1749122007.854643 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000317 1749122007.936836 read 0x03e394 thru 0x03ffff (0x1c6c bytes) 000320 1749122007.941367 MIRROR_EXTEND to 6 mirrors 000321 1749122008.105158 mapread 0x03c645 thru 0x03ffff (0x39bb bytes) 000323 1749122008.114263 mapwrite 0x026656 thru 0x02e2b5 (0x7c60 bytes) 000325 1749122008.177007 write 0x039342 thru 0x03ffff (0x6cbe bytes) 000326 1749122008.180317 write 0x015caa thru 0x025872 (0xfbc9 bytes) 000330 1749122008.185321 mapread 0x02892a thru 0x02dc4e (0x5325 bytes) 000331 1749122008.185901 write 0x0111b4 thru 0x01223c (0x1089 bytes) 000332 1749122008.186509 mapwrite 0x035449 thru 0x03ffff (0xabb7 bytes) 000333 1749122008.209377 mapread 0x025b2e thru 0x02a8f1 (0x4dc4 bytes) 000335 1749122008.210340 mapread 0x02bad9 thru 0x02fe9a (0x43c2 bytes) 000336 1749122008.212467 mapread 0x0358e9 thru 0x0381e7 (0x28ff bytes) 000338 1749122008.213344 mapread 0x03418b thru 0x037aba (0x3930 bytes) 000339 1749122008.213901 mapwrite 0x036848 thru 0x03d83a (0x6ff3 bytes) 000342 1749122008.229317 mapwrite 0x01af62 thru 0x021899 (0x6938 bytes) 000343 1749122008.243575 mapwrite 0x016baa thru 0x02297b (0xbdd2 bytes) 000344 1749122008.265128 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000345 1749122008.332446 MIRROR_RESYNC 5 mirrors 000346 1749122008.578748 mapread 0x02ddbe thru 0x03a601 (0xc844 bytes) 000348 1749122008.592705 mapread 0x0176be thru 0x01865d (0xfa0 bytes) 000349 1749122008.595882 mapwrite 0x019472 thru 0x01f1ec (0x5d7b bytes) 000351 1749122008.651766 mapread 0x01f88e thru 0x02e9a4 (0xf117 bytes) 000352 1749122008.654961 read 0x03b031 thru 0x03ffff (0x4fcf bytes) 000355 1749122008.658759 read 0x015ead thru 0x024dff (0xef53 bytes) 000357 1749122008.666625 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000359 1749122008.726143 write 0x00d189 thru 0x019576 (0xc3ee bytes) 000360 1749122008.730718 write 0x01b9e2 thru 0x01d6d5 (0x1cf4 bytes) 000361 1749122008.733817 read 0x02a0d5 thru 0x02c6e0 (0x260c bytes) 000362 1749122008.735627 read 0x00575d thru 0x00c5e7 (0x6e8b bytes) 000364 1749122008.740001 mapread 0x0057e6 thru 0x00bd0e (0x6529 bytes) 000366 1749122008.742346 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000367 1749122008.814391 read 0x014abc thru 0x015c8b (0x11d0 bytes) 000368 1749122008.816756 trunc from 0x040000 to 0x02fb79 (0x10486 bytes) 000369 1749122008.821540 mapread 0x0185d8 thru 0x01ea69 (0x6492 bytes) 000372 1749122008.830747 mapread 0x01e18f thru 0x02b2fc (0xd16e bytes) 000373 1749122008.837734 MIRROR_EXTEND to 4 mirrors 000374 1749122008.973318 write 0x03315f thru 0x035410 (0x22b2 bytes) 000375 1749122008.998336 write 0x036724 thru 0x03fa69 (0x9346 bytes) truncating to largest ever: 0x3f6b2 000379 1749122009.000248 trunc from 0x03fa6a to 0x03f6b1 (0x3b8 bytes) 000380 1749122009.011179 mapwrite 0x001cad thru 0x00ba69 (0x9dbd bytes) 000381 1749122009.043420 write 0x03cdb7 thru 0x03ffff (0x3249 bytes) 000382 1749122009.044180 read 0x037142 thru 0x03bbc5 (0x4a84 bytes) 000383 1749122009.044887 mapread 0x027ec4 thru 0x033ecb (0xc008 bytes) 000384 1749122009.054332 trunc from 0x040000 to 0x00ebb3 (0x3144c bytes) 000386 1749122009.060305 mapwrite 0x03481a thru 0x03a788 (0x5f6f bytes) 000387 1749122009.095056 trunc from 0x03a789 to 0x018407 (0x22381 bytes) 000388 1749122009.107480 trunc from 0x018408 to 0x013fcf (0x4438 bytes) 000389 1749122009.114765 mapwrite 0x032299 thru 0x034985 (0x26ed bytes) 000390 1749122009.164212 mapread 0x016a63 thru 0x0246f6 (0xdc94 bytes) 000391 1749122009.172790 write 0x00c2df thru 0x017e87 (0xbba9 bytes) 000394 1749122009.174372 write 0x027a84 thru 0x034e44 (0xd3c1 bytes) 000396 1749122009.175825 read 0x00393d thru 0x00d9f1 (0xa0b5 bytes) 000398 1749122009.176805 read 0x030fa1 thru 0x034e44 (0x3ea4 bytes) 000399 1749122009.180510 MIRROR_RESYNC 4 mirrors 000401 1749122009.406448 mapread 0x01c58f thru 0x01e314 (0x1d86 bytes) 000402 1749122009.416200 mapwrite 0x03a184 thru 0x03ffff (0x5e7c bytes) 000404 1749122009.480062 MIRROR_RESYNC 4 mirrors 000405 1749122009.688128 mapwrite 0x017ff0 thru 0x024de4 (0xcdf5 bytes) 000406 1749122009.761636 mapwrite 0x03e0ba thru 0x03ffff (0x1f46 bytes) 000407 1749122009.791454 mapwrite 0x018038 thru 0x026781 (0xe74a bytes) 000408 1749122009.811981 trunc from 0x040000 to 0x0183f3 (0x27c0c bytes) 000409 1749122009.827341 read 0x00dfcd thru 0x015148 (0x717c bytes) 000411 1749122009.830171 mapwrite 0x006b04 thru 0x0162cc (0xf7c9 bytes) 000412 1749122009.856621 trunc from 0x0183f4 to 0x01333b (0x50b8 bytes) 000413 1749122009.862557 mapread 0x00fc7e thru 0x01333b (0x36be bytes) 000415 1749122009.863537 trunc from 0x01333c to 0x0065cf (0xcd6c bytes) 000416 1749122009.869564 MIRROR_EXTEND to 5 mirrors 000417 1749122009.996933 write 0x03d14b thru 0x03ffff (0x2eb5 bytes) 000419 1749122010.020117 trunc from 0x040000 to 0x037b18 (0x84e7 bytes) 000420 1749122010.034385 mapread 0x028679 thru 0x03110b (0x8a93 bytes) 000421 1749122010.047062 read 0x0236fd thru 0x02397a (0x27e bytes) 000423 1749122010.047776 mapread 0x019cab thru 0x023824 (0x9b7a bytes) 000424 1749122010.059288 write 0x0380dd thru 0x03ffff (0x7f23 bytes) 000426 1749122010.060246 read 0x0114b7 thru 0x018dc7 (0x7911 bytes) 000427 1749122010.067519 MIRROR_EXTEND to 6 mirrors 000428 1749122010.219771 mapwrite 0x018ead thru 0x0275cb (0xe71f bytes) 000429 1749122010.285151 mapread 0x002439 thru 0x00b143 (0x8d0b bytes) 000430 1749122010.289718 write 0x01c612 thru 0x02c226 (0xfc15 bytes) 000431 1749122010.291890 trunc from 0x040000 to 0x00369b (0x3c964 bytes) 000432 1749122010.303080 trunc from 0x00369c to 0x0385ca (0x34f2f bytes) 000435 1749122010.309675 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000436 1749122010.377531 MIRROR_EXTEND to 6 mirrors 000437 1749122010.514883 read 0x00d49e thru 0x018fb4 (0xbb17 bytes) 000438 1749122010.520451 write 0x0144a7 thru 0x01b5e7 (0x7141 bytes) 000439 1749122010.547240 MIRROR_RESYNC 6 mirrors 000440 1749122010.808952 read 0x00e8b3 thru 0x01ce53 (0xe5a1 bytes) 000443 1749122010.812419 mapread 0x0235ed thru 0x029ebe (0x68d2 bytes) 000444 1749122010.820807 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000446 1749122010.905412 trunc from 0x0385cb to 0x007caf (0x3091b bytes) 000447 1749122010.932113 trunc from 0x007cb0 to 0x02ac0d (0x22f5e bytes) 000448 1749122010.939729 mapread 0x029c2c thru 0x02ac0d (0xfe2 bytes) 000449 1749122010.943797 trunc from 0x02ac0e to 0x0314cc (0x68bf bytes) 000451 1749122010.949585 mapwrite 0x0289a2 thru 0x03547d (0xcadc bytes) 000452 1749122010.986202 write 0x022a3c thru 0x0248bf (0x1e84 bytes) 000455 1749122010.987314 mapwrite 0x0162f2 thru 0x01a00e (0x3d1d bytes) 000457 1749122011.005101 trunc from 0x03547e to 0x0346db (0xda2 bytes) 000458 1749122011.009972 read 0x0330ea thru 0x0346db (0x15f2 bytes) 000459 1749122011.011969 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000460 1749122011.084522 read 0x00d6d1 thru 0x017308 (0x9c38 bytes) 000461 1749122011.087599 mapwrite 0x03ec89 thru 0x03ffff (0x1377 bytes) 000463 1749122011.111978 trunc from 0x040000 to 0x012fd3 (0x2d02c bytes) 000464 1749122011.116746 trunc from 0x012fd4 to 0x026b2b (0x13b58 bytes) 000465 1749122011.121952 mapread 0x00f86a thru 0x019987 (0xa11e bytes) 000467 1749122011.128706 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000468 1749122011.197245 mapwrite 0x0111a2 thru 0x01d0ab (0xbf0a bytes) 000470 1749122011.226441 mapwrite 0x005822 thru 0x0142ce (0xeaad bytes) 000472 1749122011.245585 mapwrite 0x01134a thru 0x01842d (0x70e4 bytes) 000473 1749122011.260496 trunc from 0x026b2c to 0x02e911 (0x7de6 bytes) 000474 1749122011.266935 MIRROR_RESYNC 3 mirrors 000476 1749122011.402180 read 0x01585a thru 0x01b136 (0x58dd bytes) 000478 1749122011.404803 read 0x00bc5f thru 0x01b96b (0xfd0d bytes) 000479 1749122011.407644 mapread 0x0261e9 thru 0x02da4b (0x7863 bytes) 000480 1749122011.408047 read 0x020b2f thru 0x022485 (0x1957 bytes) 000481 1749122011.409487 MIRROR_RESYNC 3 mirrors 000482 1749122011.458517 MIRROR_RESYNC 3 mirrors 000485 1749122011.509646 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000486 1749122011.566707 read 0x02396b thru 0x02e911 (0xafa7 bytes) 000488 1749122011.570334 trunc from 0x02e912 to 0x017bff (0x16d12 bytes) 000492 1749122011.603426 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000493 1749122011.639800 mapwrite 0x01639f thru 0x01adf4 (0x4a56 bytes) 000494 1749122011.664387 mapwrite 0x0222e5 thru 0x022bfa (0x916 bytes) 000495 1749122011.684146 write 0x010586 thru 0x01d342 (0xcdbd bytes) 000498 1749122011.688820 mapwrite 0x01de7c thru 0x023230 (0x53b5 bytes) 000499 1749122011.711360 write 0x0073e4 thru 0x007c6f (0x88c bytes) 000503 1749122011.715622 mapwrite 0x02556b thru 0x02aa8c (0x5522 bytes) 000505 1749122011.745323 trunc from 0x02aa8d to 0x01ff82 (0xab0a bytes) 000508 1749122011.749946 write 0x037727 thru 0x03a5c6 (0x2ea0 bytes) 000509 1749122011.750792 trunc from 0x03a5c7 to 0x013805 (0x26dc1 bytes) 000510 1749122011.755799 trunc from 0x013806 to 0x03370d (0x1ff08 bytes) 000511 1749122011.759842 write 0x03ea5a thru 0x03ffff (0x15a6 bytes) 000513 1749122011.760922 read 0x003cdd thru 0x0077d5 (0x3af9 bytes) 000514 1749122011.761387 mapread 0x00bf90 thru 0x011e0c (0x5e7d bytes) 000515 1749122011.762008 mapwrite 0x0343de thru 0x0387db (0x43fe bytes) 000516 1749122011.781633 read 0x01874a thru 0x0226a9 (0x9f60 bytes) 000519 1749122011.787119 mapread 0x03c3ad thru 0x03ffff (0x3c53 bytes) 000520 1749122011.787731 trunc from 0x040000 to 0x00933e (0x36cc1 bytes) 000522 1749122011.792705 trunc from 0x00933f to 0x01615e (0xce20 bytes) 000525 1749122011.797289 mapwrite 0x014aa9 thru 0x023069 (0xe5c1 bytes) 000528 1749122011.841541 mapread 0x017ae6 thru 0x01f652 (0x7b6d bytes) 000529 1749122011.842159 read 0x007b16 thru 0x00e6f9 (0x6be4 bytes) 000531 1749122011.845419 mapwrite 0x015286 thru 0x0245e4 (0xf35f bytes) 000532 1749122011.873398 mapwrite 0x0255fb thru 0x02e998 (0x939e bytes) 000533 1749122011.901733 trunc from 0x02e999 to 0x02d89b (0x10fd bytes) 000534 1749122011.906388 mapread 0x0085e7 thru 0x00f8e7 (0x7301 bytes) 000541 1749122011.908885 write 0x01a8c4 thru 0x0281e0 (0xd91d bytes) 000542 1749122011.909705 mapwrite 0x026229 thru 0x02ad0a (0x4ae2 bytes) 000544 1749122011.928736 write 0x02eae1 thru 0x03544c (0x696c bytes) 000545 1749122011.930990 MIRROR_EXTEND to 2 mirrors 000547 1749122012.083575 write 0x02f2fa thru 0x03c4f1 (0xd1f8 bytes) 000550 1749122012.113206 MIRROR_EXTEND to 3 mirrors 000553 1749122012.291321 mapread 0x006cc8 thru 0x015179 (0xe4b2 bytes) 000555 1749122012.306831 read 0x019812 thru 0x0223d8 (0x8bc7 bytes) 000556 1749122012.312585 mapread 0x037326 thru 0x039070 (0x1d4b bytes) 000558 1749122012.323718 mapwrite 0x03bc0b thru 0x03ffff (0x43f5 bytes) 000559 1749122012.382386 write 0x02ddba thru 0x031ecc (0x4113 bytes) 000567 1749122012.390326 read 0x03de08 thru 0x03ffff (0x21f8 bytes) 000569 1749122012.391135 mapread 0x0361a5 thru 0x03ffff (0x9e5b bytes) 000570 1749122012.392084 trunc from 0x040000 to 0x03dc22 (0x23dd bytes) 000571 1749122012.399787 mapread 0x008418 thru 0x01426b (0xbe54 bytes) 000572 1749122012.406134 read 0x01f484 thru 0x02f18e (0xfd0b bytes) 000573 1749122012.407596 write 0x00554a thru 0x009e89 (0x4940 bytes) 000574 1749122012.408777 mapread 0x00c6d5 thru 0x011ca2 (0x55ce bytes) 000578 1749122012.412356 MIRROR_EXTEND to 4 mirrors 000579 1749122012.570728 MIRROR_EXTEND to 5 mirrors 000580 1749122012.754395 mapwrite 0x017f9c thru 0x01a6f5 (0x275a bytes) 000581 1749122012.823364 MIRROR_EXTEND to 6 mirrors 000583 1749122012.965777 mapread 0x0065b4 thru 0x00aa97 (0x44e4 bytes) 000586 1749122012.978222 MIRROR_RESYNC 6 mirrors 000587 1749122013.238461 trunc from 0x03dc23 to 0x00c805 (0x3141d bytes) 000591 1749122013.264287 mapread 0x00138e thru 0x00b6a6 (0xa319 bytes) 000593 1749122013.273673 mapread 0x006132 thru 0x00a29e (0x416d bytes) 000595 1749122013.274627 mapwrite 0x026e54 thru 0x030ce9 (0x9e96 bytes) 000596 1749122013.321320 read 0x025689 thru 0x027325 (0x1c9d bytes) 000597 1749122013.324313 MIRROR_RESYNC 6 mirrors 000598 1749122013.568858 read 0x014159 thru 0x0179a4 (0x384c bytes) 000599 1749122013.570982 trunc from 0x030cea to 0x0062f6 (0x2a9f3 bytes) 000601 1749122013.592110 mapwrite 0x039bf2 thru 0x03ffff (0x640e bytes) 000602 1749122013.623617 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000603 1749122013.692914 read 0x023c15 thru 0x02966d (0x5a59 bytes) 000605 1749122013.696155 trunc from 0x040000 to 0x036478 (0x9b87 bytes) 000607 1749122013.704642 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000609 1749122013.766940 mapread 0x024a6a thru 0x028fad (0x4544 bytes) 000612 1749122013.774618 read 0x0029df thru 0x00e1d1 (0xb7f3 bytes) 000614 1749122013.779458 mapwrite 0x026004 thru 0x02aea0 (0x4e9d bytes) 000615 1749122013.798775 write 0x00e76a thru 0x00f2d8 (0xb6f bytes) 000619 1749122013.802689 write 0x00c98c thru 0x010aeb (0x4160 bytes) 000621 1749122013.803470 trunc from 0x036479 to 0x02c825 (0x9c53 bytes) 000622 1749122013.808458 read 0x023c3e thru 0x02c825 (0x8be8 bytes) 000624 1749122013.809255 mapread 0x00be10 thru 0x00f5eb (0x37dc bytes) 000626 1749122013.811919 MIRROR_EXTEND to 5 mirrors 000631 1749122013.951383 mapread 0x0250da thru 0x028698 (0x35bf bytes) 000632 1749122013.957521 write 0x030c00 thru 0x03235c (0x175d bytes) 000633 1749122013.978403 write 0x035065 thru 0x03b9db (0x6977 bytes) 000634 1749122013.979211 read 0x02e38b thru 0x03559c (0x7212 bytes) 000636 1749122013.983716 MIRROR_EXTEND to 6 mirrors 000638 1749122014.123629 mapwrite 0x00b615 thru 0x010e18 (0x5804 bytes) 000639 1749122014.200219 write 0x035d29 thru 0x03ffff (0xa2d7 bytes) 000640 1749122014.203743 write 0x0113a5 thru 0x017049 (0x5ca5 bytes) 000641 1749122014.206114 write 0x031f47 thru 0x036ea5 (0x4f5f bytes) 000646 1749122014.209197 read 0x00abe5 thru 0x010a98 (0x5eb4 bytes) 000648 1749122014.210294 mapwrite 0x003b32 thru 0x0067a8 (0x2c77 bytes) 000651 1749122014.239779 trunc from 0x040000 to 0x0056cf (0x3a930 bytes) 000652 1749122014.249691 MIRROR_EXTEND to 7 mirrors 000655 1749122014.404226 mapwrite 0x02ebfc thru 0x037161 (0x8566 bytes) 000657 1749122014.455531 read 0x019f82 thru 0x020975 (0x69f4 bytes) 000659 1749122014.459609 trunc from 0x037162 to 0x01739a (0x1fdc7 bytes) 000660 1749122014.465052 mapread 0x013fb5 thru 0x01739a (0x33e6 bytes) 000663 1749122014.469395 write 0x018c0e thru 0x021106 (0x84f9 bytes) 000664 1749122014.470292 mapwrite 0x00ab59 thru 0x00bf04 (0x13ac bytes) 000666 1749122014.493771 read 0x002461 thru 0x00eca8 (0xc848 bytes) 000670 1749122014.497404 read 0x00be76 thru 0x014af5 (0x8c80 bytes) 000671 1749122014.499028 mapwrite 0x000c80 thru 0x00abfe (0x9f7f bytes) 000672 1749122014.518928 mapread 0x013168 thru 0x021106 (0xdf9f bytes) 000673 1749122014.519569 mapread 0x01b9aa thru 0x021106 (0x575d bytes) 000675 1749122014.520519 write 0x035be2 thru 0x0385c3 (0x29e2 bytes) 000680 1749122014.522305 trunc from 0x0385c4 to 0x0035fc (0x34fc7 bytes) 000682 1749122014.527986 mapread 0x0000e1 thru 0x0035fc (0x351c bytes) 000683 1749122014.528714 read 0x003335 thru 0x0035fc (0x2c8 bytes) 000685 1749122014.529371 read 0x001847 thru 0x0035fc (0x1db6 bytes) 000686 1749122014.529701 read 0x00002c thru 0x0035fc (0x35d1 bytes) 000687 1749122014.532122 MIRROR_EXTEND to 8 mirrors 000689 1749122014.648043 MIRROR_RESYNC 8 mirrors 000692 1749122014.906207 mapwrite 0x038ad4 thru 0x03c01c (0x3549 bytes) 000694 1749122014.962425 mapread 0x0163cb thru 0x021155 (0xad8b bytes) 000696 1749122014.969588 mapwrite 0x00940f thru 0x00b9ae (0x25a0 bytes) 000697 1749122015.001758 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000700 1749122015.062975 mapread 0x0204ea thru 0x02b8ca (0xb3e1 bytes) 000701 1749122015.071483 mapwrite 0x013272 thru 0x01f4d6 (0xc265 bytes) 000705 1749122015.098376 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000710 1749122015.165080 mapwrite 0x01bf89 thru 0x022161 (0x61d9 bytes) 000711 1749122015.192038 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000712 1749122015.256518 mapwrite 0x0203d8 thru 0x0301af (0xfdd8 bytes) 000716 1749122015.288187 mapread 0x032ee6 thru 0x03b177 (0x8292 bytes) 000720 1749122015.292300 read 0x021f6c thru 0x024f48 (0x2fdd bytes) 000721 1749122015.292604 mapwrite 0x037055 thru 0x03ffff (0x8fab bytes) 000722 1749122015.318925 MIRROR_RESYNC 5 mirrors 000724 1749122015.510532 mapwrite 0x019cdc thru 0x020a7f (0x6da4 bytes) 000725 1749122015.567787 MIRROR_EXTEND to 6 mirrors 000727 1749122015.687084 trunc from 0x040000 to 0x0012c0 (0x3ed3f bytes) 000728 1749122015.713362 write 0x028f8c thru 0x038162 (0xf1d7 bytes) 000729 1749122015.714528 read 0x00a2dc thru 0x01064a (0x636f bytes) 000730 1749122015.717246 read 0x01fd4c thru 0x020171 (0x426 bytes) 000732 1749122015.720942 read 0x034698 thru 0x038162 (0x3acb bytes) 000733 1749122015.721374 trunc from 0x038163 to 0x011bb0 (0x265b2 bytes) 000735 1749122015.728946 read 0x00fbd9 thru 0x011bb0 (0x1fd8 bytes) 000739 1749122015.730044 mapwrite 0x00f1a4 thru 0x0116fe (0x255b bytes) 000740 1749122015.747346 mapread 0x007c12 thru 0x010963 (0x8d52 bytes) 000743 1749122015.748413 write 0x01d8d3 thru 0x026c05 (0x9333 bytes) 000745 1749122015.749422 mapread 0x0132bb thru 0x01e1bb (0xaf01 bytes) 000746 1749122015.753197 write 0x01711c thru 0x017e44 (0xd29 bytes) 000747 1749122015.754850 write 0x00d8e9 thru 0x012828 (0x4f40 bytes) 000753 1749122015.756939 trunc from 0x026c06 to 0x026239 (0x9cc bytes) 000754 1749122015.762478 read 0x023c9d thru 0x026239 (0x259d bytes) 000755 1749122015.762945 trunc from 0x02623a to 0x0145c3 (0x11c76 bytes) 000756 1749122015.769046 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000757 1749122015.830061 read 0x01010d thru 0x0105ba (0x4ae bytes) 000758 1749122015.832939 MIRROR_RESYNC 5 mirrors 000760 1749122015.985542 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000765 1749122016.049542 mapwrite 0x01e46e thru 0x028f00 (0xaa93 bytes) 000766 1749122016.086871 trunc from 0x028f01 to 0x016107 (0x12df9 bytes) 000768 1749122016.093118 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000769 1749122016.143340 write 0x016ae6 thru 0x016e1a (0x335 bytes) 000770 1749122016.145005 write 0x0379ff thru 0x03ffff (0x8601 bytes) 000771 1749122016.145688 mapwrite 0x033309 thru 0x0380de (0x4dd6 bytes) 000772 1749122016.164441 mapwrite 0x0050f4 thru 0x00c7ed (0x76fa bytes) 000773 1749122016.183459 write 0x0269fc thru 0x02cf1f (0x6524 bytes) 000774 1749122016.186869 read 0x00b996 thru 0x015920 (0x9f8b bytes) 000775 1749122016.190492 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000776 1749122016.237694 trunc from 0x040000 to 0x013357 (0x2cca8 bytes) 000778 1749122016.243133 MIRROR_EXTEND to 3 mirrors 000780 1749122016.337091 mapread 0x010393 thru 0x013221 (0x2e8f bytes) 000781 1749122016.342818 write 0x03f331 thru 0x03ffff (0xccf bytes) 000782 1749122016.356051 trunc from 0x040000 to 0x006d65 (0x3929a bytes) 000783 1749122016.363278 write 0x000af5 thru 0x008ace (0x7fda bytes) 000784 1749122016.365708 read 0x00545f thru 0x008ace (0x3670 bytes) 000786 1749122016.366387 trunc from 0x008acf to 0x008cd1 (0x203 bytes) 000787 1749122016.369757 write 0x003f5b thru 0x00f374 (0xb41a bytes) 000789 1749122016.370791 mapwrite 0x031918 thru 0x03a129 (0x8812 bytes) 000795 1749122016.395237 trunc from 0x03a12a to 0x00a30e (0x2fe1b bytes) 000796 1749122016.399123 write 0x00aa75 thru 0x00c16a (0x16f6 bytes) 000797 1749122016.399787 write 0x010839 thru 0x01bb80 (0xb348 bytes) 000798 1749122016.400601 trunc from 0x01bb81 to 0x0271cf (0xb64f bytes) 000799 1749122016.404394 read 0x01a134 thru 0x01bf1f (0x1dec bytes) 000801 1749122016.406330 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000802 1749122016.454176 mapread 0x01a231 thru 0x01e823 (0x45f3 bytes) 000805 1749122016.459558 mapwrite 0x02591c thru 0x0323fc (0xcae1 bytes) 000806 1749122016.482331 mapwrite 0x014045 thru 0x018821 (0x47dd bytes) 000811 1749122016.500284 write 0x000c28 thru 0x0050de (0x44b7 bytes) 000813 1749122016.505408 MIRROR_RESYNC 2 mirrors 000816 1749122016.605162 MIRROR_RESYNC 2 mirrors 000817 1749122016.640409 mapread 0x01555a thru 0x015b60 (0x607 bytes) 000819 1749122016.642312 write 0x01b279 thru 0x02584b (0xa5d3 bytes) 000824 1749122016.662491 mapwrite 0x016292 thru 0x020808 (0xa577 bytes) 000827 1749122016.691369 MIRROR_EXTEND to 3 mirrors 000829 1749122016.796006 read 0x030352 thru 0x0323fc (0x20ab bytes) 000830 1749122016.799307 mapwrite 0x03514e thru 0x0359a1 (0x854 bytes) 000832 1749122016.840779 trunc from 0x0359a2 to 0x00bb93 (0x29e0e bytes) 000833 1749122016.845509 read 0x006d23 thru 0x00bb93 (0x4e71 bytes) 000834 1749122016.849184 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000837 1749122016.893950 MIRROR_RESYNC 2 mirrors 000840 1749122016.990318 read 0x00325f thru 0x00bb93 (0x8935 bytes) 000842 1749122016.992473 write 0x02c419 thru 0x02dbeb (0x17d3 bytes) 000843 1749122017.008982 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000846 1749122017.053170 MIRROR_EXTEND to 2 mirrors 000847 1749122017.176575 trunc from 0x02dbec to 0x028ce7 (0x4f04 bytes) 000848 1749122017.194964 write 0x029b36 thru 0x0347e1 (0xacac bytes) 000851 1749122017.196271 mapwrite 0x029b3d thru 0x038c31 (0xf0f5 bytes) 000852 1749122017.222183 trunc from 0x038c32 to 0x0172f6 (0x2193b bytes) 000853 1749122017.226677 mapwrite 0x032091 thru 0x038f61 (0x6ed1 bytes) 000855 1749122017.250014 mapwrite 0x01a9d7 thru 0x01fc6f (0x5299 bytes) 000856 1749122017.268645 write 0x00c03a thru 0x00f024 (0x2feb bytes) 000857 1749122017.275052 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x508:0x0]: Invalid argument (22) 000858 1749122017.320830 trunc from 0x038f62 to 0x02366c (0x158f5 bytes) 000862 1749122017.325582 mapwrite 0x038d54 thru 0x03e9d3 (0x5c80 bytes) 000866 1749122017.350384 write 0x0265ce thru 0x0332a2 (0xccd5 bytes) 000868 1749122017.353897 mapwrite 0x039187 thru 0x03ffff (0x6e79 bytes) 000869 1749122017.372554 read 0x03d757 thru 0x03ffff (0x28a9 bytes) 000876 1749122017.375184 trunc from 0x040000 to 0x00a99d (0x35662 bytes) 000877 1749122017.380488 mapread 0x00641d thru 0x00a99d (0x4581 bytes) 000878 1749122017.383903 trunc from 0x00a99e to 0x038a52 (0x2e0b5 bytes) 000884 1749122017.390612 mapread 0x0153ba thru 0x018a20 (0x3667 bytes) 000886 1749122017.395001 mapread 0x00ca0f thru 0x011fd2 (0x55c4 bytes) 000887 1749122017.398817 read 0x01caed thru 0x02b3dd (0xe8f1 bytes) 000890 1749122017.404191 mapread 0x0023c1 thru 0x00ca55 (0xa695 bytes) 000891 1749122017.404696 mapwrite 0x02b84c thru 0x039232 (0xd9e7 bytes) 000892 1749122017.431297 mapwrite 0x03a51b thru 0x03c803 (0x22e9 bytes) 000893 1749122017.455757 mapwrite 0x032bee thru 0x034a31 (0x1e44 bytes) 000895 1749122017.471547 write 0x016cd3 thru 0x021fd6 (0xb304 bytes) 000896 1749122017.472418 read 0x007a99 thru 0x00c963 (0x4ecb bytes) 000897 1749122017.472756 mapwrite 0x02af12 thru 0x0348f0 (0x99df bytes) 000899 1749122017.489936 trunc from 0x03c804 to 0x02564c (0x171b7 bytes) 000900 1749122017.495748 write 0x03d20c thru 0x03ffff (0x2df4 bytes) 000901 1749122017.496446 mapwrite 0x00df98 thru 0x01a5a1 (0xc60a bytes) 000903 1749122017.521493 trunc from 0x040000 to 0x03c219 (0x3de6 bytes) 000905 1749122017.529452 mapread 0x00af28 thru 0x011ccb (0x6da4 bytes) 000908 1749122017.530318 write 0x02dc56 thru 0x03b266 (0xd611 bytes) 000909 1749122017.535925 mapread 0x00382d thru 0x008156 (0x492a bytes) 000912 1749122017.537221 trunc from 0x03c21a to 0x03204f (0xa1ca bytes) 000913 1749122017.542347 read 0x00cac4 thru 0x00fb35 (0x3072 bytes) 000919 1749122017.544562 trunc from 0x032050 to 0x015f0c (0x1c143 bytes) 000925 1749122017.551496 mapwrite 0x03d708 thru 0x03ffff (0x28f8 bytes) 000927 1749122017.580460 mapwrite 0x00b8ba thru 0x0131a1 (0x78e8 bytes) 000928 1749122017.600700 mapwrite 0x00df43 thru 0x01dec9 (0xff87 bytes) 000929 1749122017.621538 mapwrite 0x022e8e thru 0x0265ab (0x371e bytes) 000931 1749122017.636175 write 0x02d0b6 thru 0x030855 (0x37a0 bytes) 000932 1749122017.636808 mapread 0x00c231 thru 0x019d3b (0xdb0b bytes) 000933 1749122017.637285 mapread 0x00da9c thru 0x014750 (0x6cb5 bytes) 000934 1749122017.637758 mapread 0x03604d thru 0x036b35 (0xae9 bytes) 000935 1749122017.638213 trunc from 0x040000 to 0x03ea2a (0x15d5 bytes) 000936 1749122017.643140 mapread 0x006e0d thru 0x016773 (0xf967 bytes) 000937 1749122017.643876 trunc from 0x03ea2b to 0x0258cc (0x1915e bytes) 000940 1749122017.649199 mapwrite 0x00cf58 thru 0x01bec3 (0xef6c bytes) 000941 1749122017.670311 mapread 0x00f268 thru 0x0131ce (0x3f67 bytes) 000942 1749122017.670919 trunc from 0x0258cd to 0x00409b (0x21831 bytes) 000944 1749122017.676243 mapread 0x00389f thru 0x00409b (0x7fd bytes) 000946 1749122017.677191 write 0x03d1a6 thru 0x03ffff (0x2e5a bytes) 000947 1749122017.677945 read 0x02a8a8 thru 0x032225 (0x797e bytes) 000949 1749122017.681969 mapread 0x0106e0 thru 0x013cf4 (0x3615 bytes) 000955 1749122017.683897 mapread 0x0059cb thru 0x00f39d (0x99d3 bytes) 000958 1749122017.685016 mapread 0x01cf26 thru 0x01edc3 (0x1e9e bytes) 000960 1749122017.686017 write 0x03027a thru 0x03e0c4 (0xde4b bytes) 000962 1749122017.686915 trunc from 0x040000 to 0x01a66c (0x25993 bytes) 000964 1749122017.692423 read 0x0103c9 thru 0x019f1b (0x9b53 bytes) 000967 1749122017.693469 mapwrite 0x03d7c0 thru 0x03ffff (0x2840 bytes) 000971 1749122017.722835 MIRROR_EXTEND to 2 mirrors 000972 1749122017.851363 read 0x03abeb thru 0x03d54c (0x2962 bytes) 000973 1749122017.854212 mapwrite 0x017652 thru 0x01aaf8 (0x34a7 bytes) 000974 1749122017.906101 read 0x015295 thru 0x01d197 (0x7f03 bytes) 000975 1749122017.908137 MIRROR_RESYNC 2 mirrors 000976 1749122018.010993 trunc from 0x040000 to 0x019cc0 (0x2633f bytes) 000977 1749122018.030142 read 0x00a59b thru 0x018073 (0xdad9 bytes) 000979 1749122018.033175 mapwrite 0x035bb1 thru 0x03ae75 (0x52c5 bytes) 000982 1749122018.058475 write 0x0187a6 thru 0x01faf7 (0x7352 bytes) 000984 1749122018.060782 mapwrite 0x0225a9 thru 0x03024f (0xdca7 bytes) 000986 1749122018.078173 write 0x01d216 thru 0x0285fd (0xb3e8 bytes) 000987 1749122018.078875 read 0x01b0c0 thru 0x02b072 (0xffb3 bytes) 000989 1749122018.079544 read 0x0229a4 thru 0x026b67 (0x41c4 bytes) 000990 1749122018.079961 mapwrite 0x036db2 thru 0x03ffff (0x924e bytes) 000995 1749122018.101976 mapread 0x02ab5b thru 0x039083 (0xe529 bytes) 000996 1749122018.102525 mapread 0x025182 thru 0x0268ba (0x1739 bytes) 000997 1749122018.102911 trunc from 0x040000 to 0x01ed3c (0x212c3 bytes) 000998 1749122018.108120 write 0x01c694 thru 0x028e5f (0xc7cc bytes) 000999 1749122018.109218 mapwrite 0x0143c1 thru 0x014dd6 (0xa16 bytes) 001000 1749122018.124436 write 0x00262c thru 0x0097fb (0x71d0 bytes) All operations completed A-OK! PASS 70a (19s) == sanity-flr test 71: check mirror extend parent fid ==== 07:13:39 (1749122019) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 200a: stress test ===================== 07:13:40 (1749122020) Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre2 Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 1019968 .. Extending file size to 5529664 .. failed Extending file size to 8428640 .. Extending file size to 7883648 .. Extending file size to 8815296 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7994272 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8309856 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8597056 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8697632 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8849312 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8935936 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9303520 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9136512 .. done Extending file size to 9194624 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 27162 27163 27164 27165 27169 192.168.203.123@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg323-client.virtnet /mnt/lustre2 (opts:) 192.168.203.123@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg323-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (65s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 07:14:45 (1749122085) Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre2 Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5616672 .. Extending file size to 6900736 .. failed Extending file size to 8182976 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8181280 .. failed Extending file size to 9299520 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8453824 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9229056 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 30922 30923 30924 30926 30929 192.168.203.123@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg323-client.virtnet /mnt/lustre2 (opts:) 192.168.203.123@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg323-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 07:15:48 (1749122148) Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4608 2203776 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4608 2203776 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0402994 s, 52.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.13118 s, 16.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.123@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg323-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 07:15:54 (1749122154) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0352638 s, 59.5 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 07:15:56 (1749122156) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x559:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x636:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x636:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x637:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x631:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 07:15:58 (1749122158) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0366314 s, 57.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x638:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x632:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x633:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 07:16:00 (1749122160) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0459499 s, 45.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x639:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x634:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x639:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x634:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x635:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 07:16:02 (1749122162) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0312027 s, 67.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x636:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x636:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x637:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 07:16:26 (1749122186) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg323-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0239787 s, 43.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x639:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x638:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x639:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x638:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63d:0x0] } PASS 204d (43s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 07:17:09 (1749122229) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg323-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x63a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x63b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0347478 s, 60.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x63a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x63b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x63a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x63b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 07:17:55 (1749122275) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg323-server mds-ost sync done. Creating new pool test_204f oleg323-server: Pool lustre.test_204f created Adding targets to pool oleg323-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg323-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0252889 s, 82.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x63c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x63c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg323-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg323-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg323-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (56s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 07:18:51 (1749122331) PASS 205a (1s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 07:18:52 (1749122332) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 07:18:54 (1749122334) oleg323-server: Pool lustre.test_206 created Adding targets to pool oleg323-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg323-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg323-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg323-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg323-server: Pool lustre.test_206 destroyed PASS 206 (11s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 07:19:05 (1749122345) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0187574 s, 55.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108833 s, 96.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xb2d8ecec Mirror 3: 0xb2d8ecec PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 07:19:08 (1749122348) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (0s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 07:19:08 (1749122348) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 07:19:09 (1749122349) Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0116369 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1dd0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x20c0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x20c0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x20c0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x24b0000 Read this (4 bytes): BBBB 192.168.203.123@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg323-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 07:19:22 (1749122362) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00278866 s, 2.9 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 07:19:24 (1749122364) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00807335 s, 130 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x57e lmm_fid: [0x200000404:0x57e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x57e lmm_fid: [0x200000404:0x57e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x649:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835326 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-06-05 07:19:24.000000000 -0400 Modify: 2025-06-05 07:19:24.000000000 -0400 Change: 2025-06-05 07:19:24.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 07:19:26 (1749122366) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (0s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 07:19:26 (1749122366) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0978394 s, 419 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0156984 s, 261 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 5 07:19 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x580:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 5 07:19 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 796 sec ============ 07:19:28 (1749122368) === sanity-flr: start cleanup 07:19:28 (1749122368) === === sanity-flr: finish cleanup 07:19:30 (1749122370) ===