-----============= acceptance-small: sanity-flr ============----- Sun Apr 13 05:07:20 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg326-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 05:07:25 (1744535245) === oleg326-client.virtnet: executing check_config_client /mnt/lustre oleg326-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg326-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012a5b3000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012a5b3000.idle_timeout=debug disable quota as required oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 05:07:32 (1744535252) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f6998] == sanity-flr test 0a: lfs mirror create with -N option == 05:07:32 (1744535252) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 05:07:35 (1744535255) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 05:07:36 (1744535256) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 05:07:37 (1744535257) PASS 0d (4s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 05:07:41 (1744535261) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 05:07:42 (1744535262) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 05:07:43 (1744535263) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88012a5b3000.stats=clear osc.lustre-OST0001-osc-ffff88012a5b3000.stats=clear PASS 0g (1s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 05:07:45 (1744535265) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 05:07:47 (1744535267) PASS 0j (1s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 05:07:48 (1744535268) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 05:07:49 (1744535269) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 05:07:51 (1744535271) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 05:07:53 (1744535273) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (0s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 05:07:53 (1744535273) lfs extend: cannot get the last init comp error: lfs extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 05:07:55 (1744535275) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (1s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 05:07:57 (1744535277) PASS 7 (1s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 05:07:58 (1744535278) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0633403 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0319345 s, 32.8 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 05:08:00 (1744535280) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB) copied, 0.0517821 s, 182 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 05:08:02 (1744535282) Stopping /mnt/lustre-ost1 (opts:) on oleg326-server oleg326-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a5b3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a5b3000.ost_server_uuid in DISCONN state after 2 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg326-server: oleg326-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg326-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg326-server oleg326-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a5b3000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a5b3000.ost_server_uuid in DISCONN state after 0 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg326-server: oleg326-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg326-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (20s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 05:08:22 (1744535302) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.129202 s, 64.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg326-server oleg326-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a5b3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a5b3000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg326-server: oleg326-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg326-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (9s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 05:08:31 (1744535311) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg326-server oleg326-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a5b3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a5b3000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg326-server: oleg326-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg326-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg326-server oleg326-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a5b3000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a5b3000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg326-server: oleg326-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg326-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (16s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 05:08:47 (1744535327) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.737893 s, 71.1 MB/s -rw-r--r-- 1 root root 52428800 Apr 13 05:08 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.192s user 0m0.001s sys 0m0.107s Stopping /mnt/lustre-ost1 (opts:) on oleg326-server oleg326-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a5b3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a5b3000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.206s user 0m0.000s sys 0m0.106s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg326-server: oleg326-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg326-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg326-server oleg326-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a5b3000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a5b3000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.229s user 0m0.001s sys 0m0.111s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg326-server: oleg326-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg326-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg326-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012a5b3000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a5b3000.ost_server_uuid in FULL state after 0 sec PASS 33b (16s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 05:09:04 (1744535344) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (0s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 05:09:04 (1744535344) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 05:09:05 (1744535345) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 05:09:06 (1744535346) PASS 35 (1s) == sanity-flr test 36a: write to mirrored files ========== 05:09:07 (1744535347) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.097218 s, 129 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 05:09:10 (1744535350) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00675246 s, 155 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00780287 s, 134 MB/s PASS 36b (1s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 05:09:11 (1744535351) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 05:09:14 (1744535354) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000539957 s, 1.9 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000575608 s, 1.8 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000602216 s, 1.7 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000600335 s, 1.7 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 05:09:16 (1744535356) create test files with size 15731244 .. 42d98e81ac61e3179f5556c57941a01d - a2b84ccc42cf511499ac7ed0ec919b4b - cc4f9b1574760ac19cad6d8909a5baa9 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg326-client.virtnet /mnt/lustre (opts:) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre PASS 37 (4s) == sanity-flr test 38: resync ============================ 05:09:20 (1744535360) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 05:09:23 (1744535363) PASS 39 (1s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 05:09:25 (1744535365) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0237658 s, 44.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00950378 s, 110 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (1s) == sanity-flr test 41: lfs mirror resync check =========== 05:09:26 (1744535366) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0264088 s, 79.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0821612 s, 51.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0378145 s, 55.5 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 05:09:29 (1744535369) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 05:09:30 (1744535370) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (0s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 05:09:30 (1744535370) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107444 s, 97.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 05:09:32 (1744535372) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 05:09:33 (1744535373) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (1s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 05:09:35 (1744535375) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0605155 s, 173 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg326-client.virtnet /mnt/lustre (opts:) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (24s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 05:09:59 (1744535399) llite.lustre-ffff88012a54d000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0600282 s, 175 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg326-client.virtnet /mnt/lustre (opts:) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff88012ff8b000.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 05:10:02 (1744535402) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (7s) == sanity-flr test 46: Verify setstripe --copy option ==== 05:10:09 (1744535409) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (6s) == sanity-flr test 47: Verify mirror obj alloc =========== 05:10:15 (1744535415) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 05:10:16 (1744535416) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0607681 s, 51.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0439213 s, 71.6 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: c8aea88b05fe362b2720a3cb402341b0 - mirror 1 checksum: 245226bba6bdca216715e438988b6e24 - mirror 2 checksum: c8aea88b05fe362b2720a3cb402341b0 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Apr 13 05:10:17 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Apr 13 05:10:17 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: c8aea88b05fe362b2720a3cb402341b0 - mirror 1 checksum: 245226bba6bdca216715e438988b6e24 - mirror 2 checksum: 245226bba6bdca216715e438988b6e24 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Apr 13 05:10:17 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Apr 13 05:10:17 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 48 (2s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 05:10:18 (1744535418) Failing mds1 on oleg326-server Stopping /mnt/lustre-mds1 (opts:) on oleg326-server 05:10:20 (1744535420) shut down facet: mds1 facet_host: oleg326-server facet_failover_host: oleg326-server Failover mds1 to oleg326-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-MDT0000 05:10:32 (1744535432) targets are mounted 05:10:32 (1744535432) facet_failover done oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00964911 s, 109 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } PASS 50A (19s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 05:10:37 (1744535437) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 05:10:38 (1744535438) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 05:10:39 (1744535439) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00219369 s, 7.5 MB/s PASS 50c (1s) == sanity-flr test 50d: mirror rsync keep holes ========== 05:10:40 (1744535440) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 05:10:41 (1744535441) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 05:10:42 (1744535442) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (4s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 05:10:46 (1744535446) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (17s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 05:11:03 (1744535463) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 05:11:20 (1744535480) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 05:11:37 (1744535497) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (1s) == sanity-flr test 70: mirror create and split race ====== 05:11:38 (1744535498) mirror create pid 22393 mirror split pid 22394 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7567: 22394 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 22393 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 05:12:40 (1744535560) Seed set to 5560 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1744535560.546389 MIRROR_EXTEND to 2 mirrors 000005 1744535560.603293 write 0x01ac3a thru 0x02791d (0xcce4 bytes) truncating to largest ever: 0x1dbdd 000006 1744535560.615752 trunc from 0x02791e to 0x01dbdc (0x9d41 bytes) 000008 1744535560.623704 trunc from 0x01dbdd to 0x00ae18 (0x12dc4 bytes) truncating to largest ever: 0x2de11 000010 1744535560.628091 trunc from 0x00ae19 to 0x02de10 (0x22ff8 bytes) 000011 1744535560.633832 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x932:0x0]: Invalid argument (22) 000012 1744535560.663803 mapwrite 0x0228ff thru 0x0235a2 (0xca4 bytes) 000013 1744535560.682859 trunc from 0x02de11 to 0x0207ea (0xd626 bytes) 000015 1744535560.686880 mapwrite 0x00ad0e thru 0x0111c1 (0x64b4 bytes) 000016 1744535560.705036 trunc from 0x0207eb to 0x01314d (0xd69d bytes) 000018 1744535560.708943 mapread 0x00755a thru 0x009500 (0x1fa7 bytes) 000020 1744535560.709944 read 0x007694 thru 0x00d1fd (0x5b6a bytes) 000021 1744535560.710239 mapread 0x010cd6 thru 0x01314d (0x2478 bytes) truncating to largest ever: 0x36648 000023 1744535560.712392 trunc from 0x01314e to 0x036647 (0x234fa bytes) 000028 1744535560.717310 trunc from 0x036648 to 0x02984b (0xcdfc bytes) 000029 1744535560.720610 mapread 0x019533 thru 0x020b49 (0x7617 bytes) 000030 1744535560.723355 read 0x01672f thru 0x0232cd (0xcb9f bytes) 000031 1744535560.725526 mapread 0x0086cc thru 0x00b8a0 (0x31d5 bytes) 000032 1744535560.726051 trunc from 0x02984c to 0x0078a9 (0x21fa2 bytes) 000034 1744535560.730124 read 0x0032c0 thru 0x003b0d (0x84e bytes) 000035 1744535560.730489 mapwrite 0x00ff42 thru 0x01edd7 (0xee96 bytes) 000037 1744535560.750073 trunc from 0x01edd8 to 0x01a577 (0x4860 bytes) 000039 1744535560.753818 mapread 0x002c39 thru 0x010cfa (0xe0c2 bytes) 000040 1744535560.754435 write 0x03dba7 thru 0x03ffff (0x2459 bytes) 000041 1744535560.755030 read 0x02c312 thru 0x0307be (0x44ad bytes) 000042 1744535560.757659 write 0x039171 thru 0x03ffff (0x6e8f bytes) 000043 1744535560.758151 write 0x0232df thru 0x0303ab (0xd0cd bytes) 000045 1744535560.759114 read 0x03923f thru 0x03ef57 (0x5d19 bytes) 000051 1744535560.760972 write 0x026583 thru 0x02e2b6 (0x7d34 bytes) 000053 1744535560.761846 read 0x029a0b thru 0x02ef74 (0x556a bytes) 000054 1744535560.762178 write 0x009582 thru 0x017340 (0xddbf bytes) 000055 1744535560.762914 mapread 0x023e1f thru 0x02e2d6 (0xa4b8 bytes) 000056 1744535560.763519 mapread 0x033616 thru 0x03ffff (0xc9ea bytes) 000060 1744535560.764867 read 0x02b175 thru 0x0329a9 (0x7835 bytes) 000061 1744535560.765156 mapread 0x033310 thru 0x033c2a (0x91b bytes) 000062 1744535560.765591 write 0x01d43a thru 0x023eb1 (0x6a78 bytes) 000063 1744535560.766032 read 0x037d8b thru 0x03d527 (0x579d bytes) 000064 1744535560.766273 write 0x01f35e thru 0x01f9af (0x652 bytes) 000065 1744535560.766522 write 0x013354 thru 0x019118 (0x5dc5 bytes) 000066 1744535560.767025 trunc from 0x040000 to 0x02f885 (0x1077a bytes) 000068 1744535560.771036 write 0x036dbd thru 0x03f243 (0x8487 bytes) 000070 1744535560.772174 mapwrite 0x025d3b thru 0x02ff0a (0xa1d0 bytes) 000074 1744535560.787653 mapread 0x020e62 thru 0x0215cd (0x76c bytes) 000076 1744535560.788668 trunc from 0x03f244 to 0x018305 (0x26f3e bytes) 000079 1744535560.793216 trunc from 0x018306 to 0x0280af (0xfdaa bytes) 000080 1744535560.796313 read 0x02359e thru 0x0280af (0x4b12 bytes) 000083 1744535560.800111 read 0x025b93 thru 0x0280af (0x251d bytes) 000086 1744535560.801018 read 0x00507b thru 0x00bdbb (0x6d41 bytes) 000089 1744535560.801775 read 0x020de5 thru 0x0280af (0x72cb bytes) 000090 1744535560.802075 mapread 0x01c43b thru 0x0280af (0xbc75 bytes) 000091 1744535560.802533 mapwrite 0x026db4 thru 0x0273a8 (0x5f5 bytes) 000096 1744535560.816498 MIRROR_EXTEND to 2 mirrors 000097 1744535560.903738 MIRROR_RESYNC 2 mirrors 000100 1744535560.934878 write 0x01163d thru 0x01f676 (0xe03a bytes) 000103 1744535560.951314 mapread 0x00fdcb thru 0x015480 (0x56b6 bytes) 000104 1744535560.954234 MIRROR_RESYNC 2 mirrors 000111 1744535561.044790 mapwrite 0x00f115 thru 0x018eb2 (0x9d9e bytes) 000112 1744535561.086935 write 0x0340d9 thru 0x034cb2 (0xbda bytes) 000114 1744535561.087758 trunc from 0x034cb3 to 0x01dd45 (0x16f6d bytes) 000117 1744535561.093551 MIRROR_EXTEND to 3 mirrors 000120 1744535561.197355 read 0x01473f thru 0x01c387 (0x7c49 bytes) 000121 1744535561.200893 write 0x003c4e thru 0x007d6f (0x4122 bytes) 000123 1744535561.218928 read 0x00e36b thru 0x013a73 (0x5709 bytes) 000124 1744535561.222638 MIRROR_RESYNC 3 mirrors 000125 1744535561.336592 mapwrite 0x025fc1 thru 0x02a439 (0x4479 bytes) 000127 1744535561.374558 mapread 0x013090 thru 0x01353f (0x4b0 bytes) 000129 1744535561.377435 read 0x01fcbb thru 0x0257a9 (0x5aef bytes) 000131 1744535561.379190 write 0x03fdac thru 0x03ffff (0x254 bytes) 000132 1744535561.379841 mapread 0x0197d3 thru 0x02354a (0x9d78 bytes) 000134 1744535561.381939 mapwrite 0x02e260 thru 0x039892 (0xb633 bytes) 000135 1744535561.398401 read 0x01aeef thru 0x0213a8 (0x64ba bytes) 000137 1744535561.398930 mapwrite 0x0050c6 thru 0x0111b5 (0xc0f0 bytes) 000138 1744535561.415945 write 0x02e8a9 thru 0x037173 (0x88cb bytes) 000141 1744535561.417304 write 0x016395 thru 0x022765 (0xc3d1 bytes) 000143 1744535561.418362 trunc from 0x040000 to 0x01c81d (0x237e2 bytes) 000144 1744535561.423065 read 0x0140df thru 0x01c81d (0x873f bytes) 000145 1744535561.423480 mapwrite 0x00ab55 thru 0x017e65 (0xd311 bytes) 000147 1744535561.438473 mapwrite 0x0261dd thru 0x034f87 (0xedab bytes) 000148 1744535561.458324 write 0x01ddb7 thru 0x0255a4 (0x77ee bytes) 000152 1744535561.459711 mapread 0x028685 thru 0x034506 (0xbe82 bytes) 000154 1744535561.460607 read 0x01ec8b thru 0x025529 (0x689f bytes) 000155 1744535561.460918 read 0x01ede8 thru 0x024d0f (0x5f28 bytes) 000159 1744535561.462145 write 0x0079f8 thru 0x00ee1e (0x7427 bytes) 000162 1744535561.463354 mapwrite 0x02e659 thru 0x03b212 (0xcbba bytes) 000163 1744535561.483592 read 0x020ae7 thru 0x0222f0 (0x180a bytes) 000164 1744535561.485151 MIRROR_EXTEND to 4 mirrors 000168 1744535561.580473 trunc from 0x03b213 to 0x020f60 (0x1a2b2 bytes) 000171 1744535561.608312 mapwrite 0x01dd42 thru 0x01f70a (0x19c9 bytes) 000172 1744535561.626546 read 0x016fe4 thru 0x020f60 (0x9f7d bytes) 000173 1744535561.628212 mapwrite 0x036a39 thru 0x03af6c (0x4534 bytes) 000176 1744535561.647894 read 0x002185 thru 0x009c89 (0x7b05 bytes) truncating to largest ever: 0x37458 000177 1744535561.650769 trunc from 0x03af6d to 0x037457 (0x3b15 bytes) 000181 1744535561.655083 write 0x03c458 thru 0x03ffff (0x3ba8 bytes) 000183 1744535561.655791 mapwrite 0x0208dc thru 0x030611 (0xfd36 bytes) 000185 1744535561.671670 MIRROR_EXTEND to 5 mirrors 000187 1744535561.765245 mapwrite 0x002404 thru 0x004b1c (0x2719 bytes) 000193 1744535561.812943 mapwrite 0x037090 thru 0x03ffff (0x8f70 bytes) 000194 1744535561.831360 MIRROR_EXTEND to 6 mirrors 000195 1744535561.928876 mapwrite 0x0182ae thru 0x021ac6 (0x9819 bytes) 000197 1744535561.971971 mapread 0x02917d thru 0x030802 (0x7686 bytes) 000198 1744535561.974287 read 0x013f4b thru 0x015f17 (0x1fcd bytes) 000199 1744535561.974649 mapwrite 0x021351 thru 0x022319 (0xfc9 bytes) 000201 1744535561.984642 MIRROR_EXTEND to 7 mirrors 000202 1744535562.089592 mapwrite 0x02367c thru 0x028922 (0x52a7 bytes) 000203 1744535562.135417 trunc from 0x040000 to 0x030cf3 (0xf30c bytes) 000204 1744535562.139507 mapread 0x01318f thru 0x014b31 (0x19a3 bytes) 000206 1744535562.142363 write 0x025bff thru 0x032fa7 (0xd3a9 bytes) 000207 1744535562.143146 write 0x028c0a thru 0x033fe2 (0xb3d9 bytes) 000211 1744535562.144826 mapread 0x011b7c thru 0x01dba4 (0xc029 bytes) 000212 1744535562.148037 MIRROR_EXTEND to 8 mirrors 000213 1744535562.254120 MIRROR_EXTEND to 9 mirrors 000216 1744535562.352119 MIRROR_RESYNC 9 mirrors 000217 1744535562.561597 mapread 0x0071f7 thru 0x00d127 (0x5f31 bytes) 000218 1744535562.568167 mapread 0x00f963 thru 0x01eac9 (0xf167 bytes) 000219 1744535562.571510 mapread 0x0234ca thru 0x02bebd (0x89f4 bytes) 000220 1744535562.574311 write 0x0211a2 thru 0x0222ae (0x110d bytes) 000221 1744535562.591082 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x932:0x0]: Invalid argument (22) 000225 1744535562.637433 mapwrite 0x014b10 thru 0x018298 (0x3789 bytes) 000226 1744535562.657859 mapread 0x0253a7 thru 0x02c18e (0x6de8 bytes) 000227 1744535562.660662 write 0x018ab6 thru 0x024c2a (0xc175 bytes) 000231 1744535562.663242 write 0x00d7a0 thru 0x01968e (0xbeef bytes) 000236 1744535562.666933 write 0x017f4e thru 0x0183f8 (0x4ab bytes) 000237 1744535562.667551 trunc from 0x033fe3 to 0x0023a1 (0x31c41 bytes) 000238 1744535562.674890 mapread 0x001a07 thru 0x0023a1 (0x99b bytes) 000240 1744535562.676993 read 0x000d12 thru 0x0023a1 (0x1690 bytes) 000241 1744535562.678301 write 0x03067e thru 0x0369fe (0x6381 bytes) 000243 1744535562.679239 mapwrite 0x024142 thru 0x02f75a (0xb619 bytes) 000244 1744535562.697201 MIRROR_EXTEND to 9 mirrors 000245 1744535562.798416 mapread 0x02d32e thru 0x033976 (0x6649 bytes) 000248 1744535562.807201 mapread 0x006d80 thru 0x008f9d (0x221e bytes) 000249 1744535562.811157 read 0x01ea37 thru 0x024c73 (0x623d bytes) 000252 1744535562.816120 mapwrite 0x01193f thru 0x01523b (0x38fd bytes) 000253 1744535562.864920 read 0x01e0f7 thru 0x020002 (0x1f0c bytes) 000256 1744535562.868486 mapwrite 0x0194ce thru 0x025994 (0xc4c7 bytes) 000257 1744535562.884436 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x932:0x0]: Invalid argument (22) 000258 1744535562.932907 mapwrite 0x01fbb2 thru 0x027742 (0x7b91 bytes) 000265 1744535562.952281 trunc from 0x0369ff to 0x01732f (0x1f6cf bytes) 000266 1744535562.960634 read 0x010bb8 thru 0x01732f (0x6778 bytes) 000267 1744535562.964262 MIRROR_RESYNC 8 mirrors 000268 1744535563.188714 write 0x0229e6 thru 0x024df0 (0x240b bytes) 000269 1744535563.202107 trunc from 0x024df1 to 0x00dbf9 (0x171f7 bytes) 000275 1744535563.208087 trunc from 0x00dbfa to 0x00e72e (0xb35 bytes) 000277 1744535563.212796 write 0x022c26 thru 0x02c5e8 (0x99c3 bytes) 000278 1744535563.213524 write 0x0061dc thru 0x006f7f (0xda4 bytes) 000281 1744535563.216781 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x932:0x0]: Invalid argument (22) 000282 1744535563.272128 mapwrite 0x027c99 thru 0x036461 (0xe7c9 bytes) 000283 1744535563.303513 read 0x0310ad thru 0x036461 (0x53b5 bytes) 000285 1744535563.304313 write 0x00eeb5 thru 0x01e0c7 (0xf213 bytes) 000289 1744535563.309690 write 0x015a08 thru 0x018693 (0x2c8c bytes) 000292 1744535563.310928 write 0x03026d thru 0x039342 (0x90d6 bytes) 000294 1744535563.311962 read 0x0143fe thru 0x01a8e0 (0x64e3 bytes) 000295 1744535563.312238 write 0x02f274 thru 0x035de9 (0x6b76 bytes) 000296 1744535563.312853 write 0x00b22c thru 0x01ae7d (0xfc52 bytes) 000297 1744535563.313800 write 0x00fb97 thru 0x01c0d9 (0xc543 bytes) 000298 1744535563.314586 read 0x01b234 thru 0x027a2e (0xc7fb bytes) 000299 1744535563.318546 trunc from 0x039343 to 0x03123b (0x8107 bytes) 000300 1744535563.323307 write 0x015fe9 thru 0x02550c (0xf524 bytes) 000304 1744535563.324922 mapread 0x0135df thru 0x01fddc (0xc7fe bytes) 000306 1744535563.325658 mapread 0x0045b0 thru 0x00994c (0x539d bytes) 000307 1744535563.327596 MIRROR_RESYNC 7 mirrors 000308 1744535563.529416 read 0x02be34 thru 0x03123b (0x5408 bytes) 000309 1744535563.531802 mapwrite 0x00e13a thru 0x00f361 (0x1228 bytes) 000310 1744535563.577454 read 0x02fa06 thru 0x03123b (0x1836 bytes) 000313 1744535563.581608 mapwrite 0x020da5 thru 0x02651a (0x5776 bytes) 000315 1744535563.596718 mapwrite 0x01e8ac thru 0x026fe3 (0x8738 bytes) 000316 1744535563.609733 write 0x034d5c thru 0x03d54c (0x87f1 bytes) 000320 1744535563.611096 mapread 0x000e7a thru 0x01052f (0xf6b6 bytes) 000322 1744535563.613678 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x932:0x0]: Invalid argument (22) 000323 1744535563.666946 write 0x032556 thru 0x039e66 (0x7911 bytes) 000325 1744535563.670314 read 0x02a2d2 thru 0x02b7e2 (0x1511 bytes) 000326 1744535563.671977 mapwrite 0x026fc9 thru 0x0346c4 (0xd6fc bytes) 000330 1744535563.693656 MIRROR_EXTEND to 7 mirrors 000331 1744535563.792821 mapread 0x0263f4 thru 0x034ce6 (0xe8f3 bytes) 000332 1744535563.801231 trunc from 0x03d54d to 0x013018 (0x2a534 bytes) 000334 1744535563.823003 mapread 0x00674e thru 0x008ab7 (0x236a bytes) 000335 1744535563.827739 MIRROR_EXTEND to 8 mirrors 000336 1744535563.930794 mapread 0x004bb8 thru 0x006c82 (0x20cb bytes) 000338 1744535563.939034 mapwrite 0x01cee8 thru 0x02096f (0x3a88 bytes) 000339 1744535563.978736 mapread 0x000f69 thru 0x00cf7c (0xc014 bytes) 000340 1744535563.983980 trunc from 0x020970 to 0x01be97 (0x4ad8 bytes) 000341 1744535563.988835 mapread 0x006af2 thru 0x012671 (0xbb80 bytes) 000342 1744535563.989305 mapread 0x007644 thru 0x00ed4e (0x770b bytes) 000345 1744535563.990393 read 0x018aec thru 0x01be97 (0x33ac bytes) 000346 1744535563.990753 write 0x004473 thru 0x00945a (0x4fe8 bytes) 000350 1744535563.993436 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x932:0x0]: Invalid argument (22) 000351 1744535564.046954 mapread 0x016a16 thru 0x01be97 (0x5482 bytes) 000353 1744535564.053378 MIRROR_RESYNC 7 mirrors 000354 1744535564.253802 mapread 0x00ed72 thru 0x014663 (0x58f2 bytes) 000355 1744535564.258237 mapwrite 0x013286 thru 0x0144b9 (0x1234 bytes) 000356 1744535564.300286 mapwrite 0x012e05 thru 0x017dd1 (0x4fcd bytes) 000357 1744535564.312864 read 0x0101a8 thru 0x011126 (0xf7f bytes) 000359 1744535564.313473 trunc from 0x01be98 to 0x01179e (0xa6f9 bytes) 000360 1744535564.318753 read 0x0009a9 thru 0x003d5c (0x33b4 bytes) 000364 1744535564.322179 write 0x019266 thru 0x01a2b6 (0x1051 bytes) 000365 1744535564.324238 MIRROR_EXTEND to 8 mirrors 000366 1744535564.417833 trunc from 0x01a2b7 to 0x028a15 (0xe75f bytes) 000367 1744535564.437822 mapread 0x01decf thru 0x025e7d (0x7faf bytes) 000369 1744535564.442785 mapread 0x0237d5 thru 0x028a15 (0x5241 bytes) 000370 1744535564.444367 MIRROR_RESYNC 8 mirrors 000371 1744535564.657356 mapread 0x00044a thru 0x0097c5 (0x937c bytes) 000372 1744535564.662527 mapread 0x01ec9b thru 0x028a15 (0x9d7b bytes) 000375 1744535564.666096 mapwrite 0x01726b thru 0x01b501 (0x4297 bytes) 000377 1744535564.709962 write 0x02dbe0 thru 0x03b7ca (0xdbeb bytes) 000378 1744535564.710855 read 0x029449 thru 0x02d91e (0x44d6 bytes) 000379 1744535564.712666 mapwrite 0x036805 thru 0x036e34 (0x630 bytes) 000380 1744535564.726191 read 0x008aaf thru 0x0136bf (0xac11 bytes) 000381 1744535564.730163 read 0x0267fa thru 0x03598c (0xf193 bytes) 000384 1744535564.731399 write 0x002ae5 thru 0x005a0e (0x2f2a bytes) 000385 1744535564.733551 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x932:0x0]: Invalid argument (22) 000386 1744535564.782232 mapread 0x02c342 thru 0x02facc (0x378b bytes) 000388 1744535564.788392 mapwrite 0x013530 thru 0x016148 (0x2c19 bytes) 000389 1744535564.803667 mapwrite 0x02bd68 thru 0x03823c (0xc4d5 bytes) 000390 1744535564.818395 read 0x0057fb thru 0x008173 (0x2979 bytes) 000392 1744535564.820316 mapwrite 0x02ed13 thru 0x030e8b (0x2179 bytes) 000394 1744535564.832535 MIRROR_EXTEND to 8 mirrors 000396 1744535564.934242 mapwrite 0x03c79c thru 0x03ffff (0x3864 bytes) 000398 1744535564.972834 mapwrite 0x001d34 thru 0x008c19 (0x6ee6 bytes) 000399 1744535564.987055 read 0x026d55 thru 0x034d2b (0xdfd7 bytes) 000403 1744535564.990585 mapread 0x02d69d thru 0x032c46 (0x55aa bytes) 000404 1744535564.991133 read 0x002837 thru 0x007048 (0x4812 bytes) 000407 1744535564.992100 mapwrite 0x0160b7 thru 0x024392 (0xe2dc bytes) 000408 1744535565.009977 mapwrite 0x025ea5 thru 0x030a8f (0xabeb bytes) 000409 1744535565.025599 mapwrite 0x0087da thru 0x013e0f (0xb636 bytes) 000410 1744535565.041617 mapread 0x01d009 thru 0x0216e0 (0x46d8 bytes) 000411 1744535565.042242 mapread 0x00e67c thru 0x00ffc9 (0x194e bytes) 000412 1744535565.044343 MIRROR_EXTEND to 9 mirrors 000413 1744535565.145647 mapread 0x001a05 thru 0x011425 (0xfa21 bytes) 000418 1744535565.157199 read 0x02d9fa thru 0x032e3e (0x5445 bytes) 000419 1744535565.160138 trunc from 0x040000 to 0x01a048 (0x25fb7 bytes) 000420 1744535565.179860 MIRROR_EXTEND to 10 mirrors 000421 1744535565.272522 trunc from 0x01a049 to 0x001501 (0x18b47 bytes) 000422 1744535565.293225 mapread 0x00041d thru 0x001501 (0x10e5 bytes) 000424 1744535565.296232 write 0x037fcf thru 0x03ffb0 (0x7fe2 bytes) 000426 1744535565.298523 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x932:0x0]: Invalid argument (22) 000428 1744535565.354009 mapread 0x0031bd thru 0x012cf3 (0xfb37 bytes) 000429 1744535565.360982 mapread 0x00cae6 thru 0x00fdc5 (0x32e0 bytes) 000432 1744535565.361862 read 0x0390dc thru 0x03d887 (0x47ac bytes) 000437 1744535565.365656 read 0x025eb0 thru 0x02fe06 (0x9f57 bytes) 000441 1744535565.369771 read 0x038ea0 thru 0x03edca (0x5f2b bytes) 000442 1744535565.370200 trunc from 0x03ffb1 to 0x0230a2 (0x1cf0e bytes) 000445 1744535565.376395 MIRROR_RESYNC 9 mirrors 000449 1744535565.610491 read 0x0077c9 thru 0x010310 (0x8b48 bytes) 000451 1744535565.612711 read 0x00f7d6 thru 0x01ad2f (0xb55a bytes) 000452 1744535565.615696 MIRROR_EXTEND to 10 mirrors 000454 1744535565.722801 trunc from 0x0230a3 to 0x029d01 (0x6c5f bytes) 000457 1744535565.741366 MIRROR_EXTEND to 11 mirrors 000458 1744535565.837826 write 0x000aae thru 0x00b2e5 (0xa838 bytes) 000459 1744535565.856268 MIRROR_RESYNC 11 mirrors 000460 1744535566.152985 trunc from 0x029d02 to 0x00b3a1 (0x1e960 bytes) 000462 1744535566.170121 read 0x0064e5 thru 0x0091e0 (0x2cfc bytes) 000464 1744535566.172225 trunc from 0x00b3a2 to 0x02384b (0x184aa bytes) 000465 1744535566.178311 MIRROR_RESYNC 11 mirrors truncating to largest ever: 0x3c01e 000467 1744535566.433619 trunc from 0x02384c to 0x03c01d (0x187d2 bytes) 000468 1744535566.451903 mapread 0x01044d thru 0x011e76 (0x1a2a bytes) 000469 1744535566.454311 mapwrite 0x0000c7 thru 0x00542d (0x5367 bytes) 000471 1744535566.471317 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x932:0x0]: Invalid argument (22) 000472 1744535566.517592 write 0x018ec6 thru 0x02357c (0xa6b7 bytes) 000473 1744535566.520440 read 0x01b522 thru 0x02701d (0xbafc bytes) 000475 1744535566.522477 write 0x02ec78 thru 0x03e766 (0xfaef bytes) 000477 1744535566.525745 trunc from 0x03e767 to 0x00be4d (0x32919 bytes) 000478 1744535566.529908 mapread 0x005a88 thru 0x00b507 (0x5a80 bytes) 000479 1744535566.532502 trunc from 0x00be4e to 0x0306aa (0x2485d bytes) 000480 1744535566.535905 write 0x015a5e thru 0x0233bb (0xd95e bytes) 000482 1744535566.540203 write 0x03de3e thru 0x03ffff (0x21c2 bytes) 000485 1744535566.541469 read 0x00f93d thru 0x0177ed (0x7eb1 bytes) 000486 1744535566.541776 read 0x00d1e3 thru 0x012016 (0x4e34 bytes) 000493 1744535566.545817 write 0x028e55 thru 0x0337ea (0xa996 bytes) 000494 1744535566.546553 trunc from 0x040000 to 0x02ff1e (0x100e1 bytes) 000495 1744535566.550594 mapwrite 0x03892d thru 0x03ffff (0x76d3 bytes) 000496 1744535566.577090 mapwrite 0x005838 thru 0x00cba3 (0x736c bytes) 000497 1744535566.590078 read 0x0371c6 thru 0x038c3d (0x1a78 bytes) 000498 1744535566.593498 mapread 0x0102e8 thru 0x019c46 (0x995f bytes) 000499 1744535566.593978 trunc from 0x040000 to 0x03a715 (0x58ea bytes) 000502 1744535566.597907 trunc from 0x03a716 to 0x032881 (0x7e94 bytes) 000507 1744535566.601663 read 0x010004 thru 0x011951 (0x194e bytes) 000508 1744535566.601960 read 0x02a7d8 thru 0x02e7b1 (0x3fda bytes) 000510 1744535566.602421 read 0x021c12 thru 0x02df43 (0xc332 bytes) 000512 1744535566.604041 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x932:0x0]: Invalid argument (22) 000513 1744535566.653912 read 0x02ebc1 thru 0x032881 (0x3cc1 bytes) 000514 1744535566.657181 MIRROR_EXTEND to 10 mirrors 000515 1744535566.754719 MIRROR_RESYNC 10 mirrors 000518 1744535566.977954 mapwrite 0x02413c thru 0x0268fc (0x27c1 bytes) 000520 1744535567.022985 mapwrite 0x0123af thru 0x018fa3 (0x6bf5 bytes) 000524 1744535567.038781 trunc from 0x032882 to 0x001824 (0x3105d bytes) 000525 1744535567.047314 mapread 0x001691 thru 0x001824 (0x194 bytes) 000527 1744535567.049304 read 0x0015b6 thru 0x001824 (0x26f bytes) 000528 1744535567.049800 mapwrite 0x03a39c thru 0x03ffff (0x5c64 bytes) 000529 1744535567.068399 write 0x00b527 thru 0x00f410 (0x3eea bytes) 000531 1744535567.071228 write 0x02f31a thru 0x03667f (0x7366 bytes) 000532 1744535567.074842 MIRROR_EXTEND to 11 mirrors 000534 1744535567.186680 trunc from 0x040000 to 0x02c49e (0x13b61 bytes) 000540 1744535567.210009 MIRROR_EXTEND to 12 mirrors 000541 1744535567.302120 mapread 0x02b24f thru 0x02c49e (0x1250 bytes) 000542 1744535567.305131 mapread 0x02108d thru 0x027c35 (0x6ba9 bytes) 000546 1744535567.313443 MIRROR_EXTEND to 13 mirrors 000549 1744535567.415165 trunc from 0x02c49f to 0x03a859 (0xe3bb bytes) 000550 1744535567.438130 mapwrite 0x03ea42 thru 0x03ffff (0x15be bytes) 000552 1744535567.457991 write 0x0217d5 thru 0x023c8b (0x24b7 bytes) 000553 1744535567.460934 read 0x0168f9 thru 0x0188a6 (0x1fae bytes) 000555 1744535567.462683 trunc from 0x040000 to 0x022ed9 (0x1d126 bytes) 000556 1744535567.466388 trunc from 0x022eda to 0x0318c1 (0xe9e8 bytes) 000557 1744535567.469738 read 0x00f6c8 thru 0x01d6c1 (0xdffa bytes) 000558 1744535567.472863 write 0x013070 thru 0x01e97e (0xb90f bytes) 000559 1744535567.473597 mapread 0x01d443 thru 0x01fd11 (0x28cf bytes) 000561 1744535567.475649 MIRROR_EXTEND to 14 mirrors 000563 1744535567.575686 mapwrite 0x02b9f2 thru 0x030da0 (0x53af bytes) 000565 1744535567.622275 mapwrite 0x035318 thru 0x03e87c (0x9565 bytes) 000567 1744535567.651678 MIRROR_EXTEND to 15 mirrors 000568 1744535567.747643 trunc from 0x03e87d to 0x01a490 (0x243ec bytes) 000570 1744535567.769037 mapread 0x0176db thru 0x019a40 (0x2366 bytes) 000573 1744535567.774265 mapread 0x00d877 thru 0x00f4c3 (0x1c4d bytes) 000574 1744535567.777693 mapread 0x00dcfe thru 0x01a490 (0xc793 bytes) 000576 1744535567.779858 MIRROR_SPLIT mirror 13 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x932:0x0]: Invalid argument (22) 000577 1744535567.827789 trunc from 0x01a491 to 0x01a7e0 (0x350 bytes) 000579 1744535567.831383 trunc from 0x01a7e1 to 0x023646 (0x8e66 bytes) 000580 1744535567.834750 read 0x020619 thru 0x023646 (0x302e bytes) truncating to largest ever: 0x3d875 000581 1744535567.836596 trunc from 0x023647 to 0x03d874 (0x1a22e bytes) 000582 1744535567.839688 read 0x02bf72 thru 0x02ea94 (0x2b23 bytes) 000583 1744535567.842906 mapwrite 0x0299ac thru 0x02cc3a (0x328f bytes) 000584 1744535567.856266 trunc from 0x03d875 to 0x03b838 (0x203c bytes) 000585 1744535567.860640 read 0x012678 thru 0x017d46 (0x56cf bytes) 000586 1744535567.860966 mapread 0x0111e8 thru 0x013a0e (0x2827 bytes) 000587 1744535567.861436 write 0x02a5ac thru 0x02e77c (0x41d1 bytes) 000591 1744535567.863993 MIRROR_RESYNC 14 mirrors 000592 1744535568.208671 read 0x001665 thru 0x008bc8 (0x7564 bytes) 000596 1744535568.211299 trunc from 0x03b839 to 0x03cbe0 (0x13a8 bytes) 000597 1744535568.228312 read 0x00cc69 thru 0x018f26 (0xc2be bytes) 000598 1744535568.231908 mapread 0x025789 thru 0x029148 (0x39c0 bytes) 000599 1744535568.232323 mapwrite 0x03db12 thru 0x03ffff (0x24ee bytes) 000600 1744535568.251911 MIRROR_RESYNC 14 mirrors 000603 1744535568.582728 read 0x01051f thru 0x018efd (0x89df bytes) 000605 1744535568.585146 write 0x007a33 thru 0x0134d0 (0xba9e bytes) 000607 1744535568.603095 mapread 0x0023b6 thru 0x007384 (0x4fcf bytes) 000608 1744535568.605408 write 0x00a44b thru 0x00b81d (0x13d3 bytes) truncating to largest ever: 0x3e38d 000609 1744535568.605917 trunc from 0x040000 to 0x03e38c (0x1c73 bytes) 000611 1744535568.615842 mapread 0x003619 thru 0x0096b3 (0x609b bytes) 000612 1744535568.616526 write 0x00f1e0 thru 0x014832 (0x5653 bytes) 000613 1744535568.620509 MIRROR_SPLIT mirror 10 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x932:0x0]: Invalid argument (22) 000617 1744535568.679770 trunc from 0x03e38d to 0x0018ec (0x3caa0 bytes) 000619 1744535568.683975 mapread 0x00043b thru 0x0018ec (0x14b2 bytes) 000622 1744535568.687808 read 0x00098a thru 0x0018ec (0xf63 bytes) 000623 1744535568.688194 mapwrite 0x019c6d thru 0x01f69b (0x5a2f bytes) 000625 1744535568.710210 trunc from 0x01f69c to 0x0170f2 (0x85a9 bytes) 000626 1744535568.714218 trunc from 0x0170f3 to 0x03a6d6 (0x235e4 bytes) 000627 1744535568.717541 trunc from 0x03a6d7 to 0x021cee (0x189e8 bytes) 000629 1744535568.722175 MIRROR_RESYNC 13 mirrors 000632 1744535569.027764 mapread 0x021051 thru 0x021cee (0xc9e bytes) 000633 1744535569.028077 mapwrite 0x01fb3b thru 0x02da5a (0xdf20 bytes) 000635 1744535569.067777 MIRROR_RESYNC 13 mirrors 000636 1744535569.348202 write 0x003371 thru 0x00cbfc (0x988c bytes) 000638 1744535569.363651 mapwrite 0x026a7f thru 0x036145 (0xf6c7 bytes) 000639 1744535569.391141 mapwrite 0x01345a thru 0x021cb6 (0xe85d bytes) 000643 1744535569.408505 mapwrite 0x006bd0 thru 0x012554 (0xb985 bytes) 000645 1744535569.421927 mapwrite 0x012184 thru 0x018db9 (0x6c36 bytes) 000648 1744535569.433196 mapread 0x00d55b thru 0x011054 (0x3afa bytes) 000650 1744535569.433929 mapwrite 0x00cb58 thru 0x01bb52 (0xeffb bytes) 000651 1744535569.444587 trunc from 0x036146 to 0x026f95 (0xf1b0 bytes) 000652 1744535569.448184 mapread 0x01f480 thru 0x01fd3c (0x8bd bytes) 000654 1744535569.448815 trunc from 0x026f96 to 0x006459 (0x20b3c bytes) 000656 1744535569.452937 mapwrite 0x031d9c thru 0x03ffff (0xe264 bytes) 000657 1744535569.473038 mapread 0x02d14c thru 0x036ba3 (0x9a58 bytes) 000658 1744535569.473519 mapread 0x037c8b thru 0x03ffff (0x8375 bytes) 000659 1744535569.473882 mapread 0x01c8e7 thru 0x026519 (0x9c33 bytes) 000660 1744535569.478171 mapread 0x019965 thru 0x027d50 (0xe3ec bytes) 000662 1744535569.479231 read 0x02ad15 thru 0x037464 (0xc750 bytes) 000663 1744535569.479719 trunc from 0x040000 to 0x032f5c (0xd0a3 bytes) 000665 1744535569.483784 mapwrite 0x019577 thru 0x01cd42 (0x37cc bytes) 000666 1744535569.494305 trunc from 0x032f5d to 0x0229c5 (0x10597 bytes) 000668 1744535569.498718 trunc from 0x0229c6 to 0x01a387 (0x863e bytes) 000670 1744535569.504447 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x932:0x0]: Invalid argument (22) 000673 1744535569.553756 MIRROR_SPLIT mirror 2 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x932:0x0]: Invalid argument (22) 000676 1744535569.600823 read 0x01040c thru 0x0162a3 (0x5e98 bytes) 000677 1744535569.605425 MIRROR_EXTEND to 12 mirrors 000678 1744535569.701383 trunc from 0x01a388 to 0x02a2b0 (0xff29 bytes) 000680 1744535569.720205 trunc from 0x02a2b1 to 0x022322 (0x7f8e bytes) 000681 1744535569.724347 read 0x0195f6 thru 0x022322 (0x8d2d bytes) 000683 1744535569.727073 mapread 0x010096 thru 0x015026 (0x4f91 bytes) 000685 1744535569.731327 read 0x01e883 thru 0x0208de (0x205c bytes) 000688 1744535569.733571 MIRROR_EXTEND to 13 mirrors 000689 1744535569.825054 mapread 0x0139d2 thru 0x01d6e3 (0x9d12 bytes) 000690 1744535569.832193 mapread 0x01781f thru 0x022322 (0xab04 bytes) 000692 1744535569.836226 mapread 0x000fc4 thru 0x010c57 (0xfc94 bytes) 000694 1744535569.843358 mapread 0x005624 thru 0x00d10e (0x7aeb bytes) 000696 1744535569.846491 trunc from 0x022323 to 0x02de63 (0xbb41 bytes) 000698 1744535569.866869 trunc from 0x02de64 to 0x00fc52 (0x1e211 bytes) 000699 1744535569.871232 mapwrite 0x019c01 thru 0x01d37f (0x377f bytes) 000700 1744535569.891661 read 0x0121fa thru 0x01b2cb (0x90d2 bytes) 000702 1744535569.892487 mapread 0x005bdb thru 0x0118c8 (0xbcee bytes) 000703 1744535569.898028 MIRROR_EXTEND to 14 mirrors 000704 1744535569.990867 write 0x031f09 thru 0x037f6c (0x6064 bytes) 000706 1744535570.006862 MIRROR_RESYNC 14 mirrors 000708 1744535570.333953 trunc from 0x037f6d to 0x019baf (0x1e3bd bytes) 000712 1744535570.353161 MIRROR_EXTEND to 15 mirrors 000713 1744535570.453880 trunc from 0x019bb0 to 0x00ccb0 (0xceff bytes) 000714 1744535570.469785 mapwrite 0x01a905 thru 0x027c3e (0xd33a bytes) 000715 1744535570.494585 trunc from 0x027c3f to 0x022f95 (0x4ca9 bytes) 000716 1744535570.497912 write 0x006b0a thru 0x00be4f (0x5346 bytes) 000719 1744535570.500890 mapread 0x021000 thru 0x022f95 (0x1f96 bytes) 000722 1744535570.501851 write 0x0351d7 thru 0x03b99b (0x67c5 bytes) 000723 1744535570.502483 trunc from 0x03b99c to 0x01127a (0x2a721 bytes) 000726 1744535570.506445 mapread 0x001a67 thru 0x00588b (0x3e25 bytes) 000727 1744535570.506850 read 0x00cf77 thru 0x01127a (0x4304 bytes) 000728 1744535570.509439 MIRROR_RESYNC 15 mirrors 000729 1744535570.823661 trunc from 0x01127b to 0x0336de (0x22464 bytes) 000730 1744535570.842879 MIRROR_EXTEND to 16 mirrors 000734 1744535570.943106 MIRROR_RESYNC 16 mirrors 000739 1744535571.296645 MIRROR_RESYNC 16 mirrors 000740 1744535571.372704 mapread 0x02de71 thru 0x02fd33 (0x1ec3 bytes) 000741 1744535571.377569 read 0x02b31c thru 0x03076a (0x544f bytes) 000742 1744535571.377927 mapread 0x02781f thru 0x029a82 (0x2264 bytes) 000745 1744535571.380259 mapread 0x008fba thru 0x00c605 (0x364c bytes) 000747 1744535571.383688 mapread 0x015424 thru 0x022d5e (0xd93b bytes) 000748 1744535571.387102 write 0x01fa71 thru 0x02be6d (0xc3fd bytes) 000750 1744535571.403619 read 0x01b323 thru 0x022771 (0x744f bytes) 000751 1744535571.403920 write 0x03aaa0 thru 0x03ffff (0x5560 bytes) 000753 1744535571.404843 mapwrite 0x035a01 thru 0x03931a (0x391a bytes) 000755 1744535571.421878 write 0x00fd21 thru 0x01c7eb (0xcacb bytes) 000756 1744535571.422983 write 0x01d9da thru 0x02a36a (0xc991 bytes) 000757 1744535571.423962 read 0x00f1d8 thru 0x01a0f8 (0xaf21 bytes) 000761 1744535571.425584 write 0x02a4e4 thru 0x0310b4 (0x6bd1 bytes) 000762 1744535571.427985 MIRROR_RESYNC 16 mirrors 000763 1744535571.795533 read 0x02728e thru 0x0280ae (0xe21 bytes) 000765 1744535571.797584 trunc from 0x040000 to 0x00034f (0x3fcb0 bytes) 000767 1744535571.821969 MIRROR_SPLIT mirror 6 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x932:0x0]: Invalid argument (22) 000769 1744535571.875663 mapwrite 0x02275b thru 0x030ca1 (0xe547 bytes) 000771 1744535571.898232 write 0x03d09b thru 0x03d464 (0x3ca bytes) 000773 1744535571.899300 trunc from 0x03d465 to 0x02ed03 (0xe761 bytes) 000774 1744535571.902940 mapwrite 0x00dfe0 thru 0x01c5ee (0xe60f bytes) 000775 1744535571.922190 MIRROR_SPLIT mirror 7 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x932:0x0]: Invalid argument (22) 000776 1744535571.970531 read 0x019d8d thru 0x028526 (0xe79a bytes) 000777 1744535571.973067 mapwrite 0x013e03 thru 0x01659f (0x279d bytes) 000778 1744535571.988990 write 0x004faf thru 0x007c86 (0x2cd8 bytes) 000779 1744535571.991586 trunc from 0x02ed04 to 0x001f87 (0x2cd7c bytes) 000780 1744535571.995441 trunc from 0x001f88 to 0x030744 (0x2e7bd bytes) 000781 1744535571.998661 write 0x014c6b thru 0x024116 (0xf4ac bytes) 000784 1744535572.002657 write 0x00b76f thru 0x01465d (0x8eef bytes) 000785 1744535572.005364 trunc from 0x030745 to 0x0070d5 (0x2966f bytes) 000786 1744535572.008895 read 0x00513d thru 0x0070d5 (0x1f99 bytes) 000788 1744535572.009481 mapread 0x002c13 thru 0x0070d5 (0x44c3 bytes) 000789 1744535572.013063 MIRROR_SPLIT mirror 4 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x932:0x0]: Invalid argument (22) 000791 1744535572.061236 MIRROR_SPLIT mirror 11 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x932:0x0]: Invalid argument (22) 000792 1744535572.108349 mapread 0x000ef7 thru 0x0013fb (0x505 bytes) 000794 1744535572.111090 read 0x001ae4 thru 0x0070d5 (0x55f2 bytes) 000795 1744535572.114185 read 0x0035db thru 0x0070d5 (0x3afb bytes) 000796 1744535572.114655 trunc from 0x0070d6 to 0x00c2ee (0x5219 bytes) 000798 1744535572.118761 mapread 0x001fb9 thru 0x00346e (0x14b6 bytes) 000801 1744535572.120700 MIRROR_EXTEND to 13 mirrors 000802 1744535572.214201 MIRROR_EXTEND to 14 mirrors 000804 1744535572.305797 read 0x000e94 thru 0x00c2ee (0xb45b bytes) 000806 1744535572.309658 mapwrite 0x02c9de thru 0x0300ba (0x36dd bytes) 000807 1744535572.346260 MIRROR_RESYNC 14 mirrors 000808 1744535572.678017 write 0x01531a thru 0x024450 (0xf137 bytes) 000809 1744535572.693879 mapwrite 0x031a47 thru 0x039e71 (0x842b bytes) 000810 1744535572.719071 MIRROR_SPLIT mirror 8 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x932:0x0]: Invalid argument (22) 000811 1744535572.768255 read 0x00e381 thru 0x017103 (0x8d83 bytes) 000812 1744535572.770552 read 0x00f37c thru 0x017282 (0x7f07 bytes) 000813 1744535572.770829 mapwrite 0x036e34 thru 0x03f4db (0x86a8 bytes) 000814 1744535572.793032 mapwrite 0x004e77 thru 0x008146 (0x32d0 bytes) 000815 1744535572.804671 MIRROR_SPLIT mirror 9 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x932:0x0]: Invalid argument (22) 000820 1744535572.855925 trunc from 0x03f4dc to 0x0393b9 (0x6122 bytes) 000822 1744535572.860268 read 0x01c8c0 thru 0x021421 (0x4b62 bytes) 000823 1744535572.862310 read 0x0006cb thru 0x00a19d (0x9ad3 bytes) 000824 1744535572.867594 MIRROR_RESYNC 12 mirrors 000827 1744535573.155408 read 0x022280 thru 0x030571 (0xe2f2 bytes) 000828 1744535573.157705 write 0x01de81 thru 0x01fdce (0x1f4e bytes) 000830 1744535573.174303 mapwrite 0x00636b thru 0x00fe5d (0x9af3 bytes) 000831 1744535573.191500 mapwrite 0x015b28 thru 0x01c0d6 (0x65af bytes) 000832 1744535573.205648 read 0x02ce5d thru 0x037b10 (0xacb4 bytes) 000834 1744535573.208614 mapread 0x00bc94 thru 0x016373 (0xa6e0 bytes) 000836 1744535573.209595 mapwrite 0x004ffe thru 0x0075a8 (0x25ab bytes) 000838 1744535573.220996 MIRROR_SPLIT mirror 4 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x932:0x0]: Invalid argument (22) 000839 1744535573.271121 mapread 0x01beb2 thru 0x0290bc (0xd20b bytes) 000841 1744535573.278241 write 0x014b2f thru 0x023b70 (0xf042 bytes) 000842 1744535573.281765 write 0x03950b thru 0x03ffff (0x6af5 bytes) 000844 1744535573.284512 mapread 0x009269 thru 0x010b8b (0x7923 bytes) 000845 1744535573.286919 trunc from 0x040000 to 0x01375f (0x2c8a0 bytes) 000846 1744535573.291282 read 0x007246 thru 0x01375f (0xc51a bytes) 000847 1744535573.291783 write 0x020609 thru 0x029bb1 (0x95a9 bytes) 000850 1744535573.293141 mapwrite 0x013975 thru 0x022c97 (0xf323 bytes) 000851 1744535573.312793 trunc from 0x029bb2 to 0x02ea4a (0x4e99 bytes) 000852 1744535573.316713 mapwrite 0x02a666 thru 0x034999 (0xa334 bytes) 000854 1744535573.339648 MIRROR_EXTEND to 12 mirrors 000856 1744535573.434208 trunc from 0x03499a to 0x02e506 (0x6493 bytes) 000857 1744535573.456871 mapread 0x009e51 thru 0x015032 (0xb1e2 bytes) 000858 1744535573.465218 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x932:0x0]: Invalid argument (22) 000860 1744535573.512551 read 0x0269ef thru 0x02cd46 (0x6358 bytes) 000862 1744535573.515112 read 0x01a22b thru 0x028c7a (0xea50 bytes) 000864 1744535573.518459 trunc from 0x02e507 to 0x013551 (0x1afb5 bytes) 000865 1744535573.523041 trunc from 0x013552 to 0x00aebf (0x8692 bytes) 000866 1744535573.526936 write 0x01c434 thru 0x024638 (0x8205 bytes) 000867 1744535573.527810 mapread 0x01a049 thru 0x024638 (0xa5f0 bytes) 000868 1744535573.530614 mapwrite 0x00049b thru 0x006262 (0x5dc8 bytes) 000869 1744535573.545091 mapwrite 0x032f08 thru 0x03eeb3 (0xbfac bytes) 000870 1744535573.568518 MIRROR_EXTEND to 12 mirrors 000872 1744535573.666296 write 0x00a142 thru 0x00e492 (0x4351 bytes) 000873 1744535573.682684 mapread 0x021a8e thru 0x024311 (0x2884 bytes) 000874 1744535573.686439 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x932:0x0]: Invalid argument (22) 000880 1744535573.741410 read 0x00ea05 thru 0x0153cf (0x69cb bytes) 000881 1744535573.746081 MIRROR_EXTEND to 12 mirrors 000882 1744535573.855771 write 0x000c48 thru 0x00400c (0x33c5 bytes) 000884 1744535573.872641 read 0x00b1ac thru 0x00d148 (0x1f9d bytes) 000886 1744535573.874348 mapread 0x016c5d thru 0x024171 (0xd515 bytes) 000889 1744535573.880991 mapwrite 0x01fcd0 thru 0x0279a9 (0x7cda bytes) 000890 1744535573.895204 trunc from 0x03eeb4 to 0x02a3e1 (0x14ad2 bytes) 000893 1744535573.904065 mapwrite 0x033597 thru 0x03ffff (0xca69 bytes) 000895 1744535573.926704 mapread 0x0350ea thru 0x03ffff (0xaf16 bytes) 000900 1744535573.930115 MIRROR_EXTEND to 13 mirrors 000903 1744535574.033170 mapread 0x0325de thru 0x033cb6 (0x16d9 bytes) 000904 1744535574.037553 mapwrite 0x0267c2 thru 0x026cc4 (0x503 bytes) 000906 1744535574.085279 read 0x02f189 thru 0x03bb64 (0xc9dc bytes) 000908 1744535574.087922 mapread 0x03b5a0 thru 0x03ffff (0x4a60 bytes) 000909 1744535574.089733 trunc from 0x040000 to 0x0192f3 (0x26d0c bytes) 000910 1744535574.097735 mapread 0x0018cc thru 0x0044a3 (0x2bd8 bytes) 000916 1744535574.101226 mapwrite 0x0196d4 thru 0x01ff0a (0x6837 bytes) 000917 1744535574.121058 MIRROR_SPLIT mirror 11 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x932:0x0]: Invalid argument (22) 000918 1744535574.166922 mapread 0x00c227 thru 0x01b361 (0xf13b bytes) 000920 1744535574.173595 trunc from 0x01ff0b to 0x0063b4 (0x19b56 bytes) 000922 1744535574.178674 MIRROR_SPLIT mirror 4 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x932:0x0]: Invalid argument (22) 000923 1744535574.230410 mapread 0x00535a thru 0x0063b4 (0x105b bytes) 000927 1744535574.234513 mapwrite 0x02f96f thru 0x036a42 (0x70d4 bytes) 000929 1744535574.255400 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x932:0x0]: Invalid argument (22) 000930 1744535574.300942 mapwrite 0x00e5d5 thru 0x012553 (0x3f7f bytes) 000932 1744535574.317465 trunc from 0x036a43 to 0x004313 (0x3272f bytes) 000933 1744535574.321141 mapread 0x002d27 thru 0x004313 (0x15ed bytes) 000937 1744535574.323798 trunc from 0x004314 to 0x00a637 (0x6324 bytes) 000938 1744535574.327363 trunc from 0x00a638 to 0x01da1a (0x133e3 bytes) 000939 1744535574.330648 read 0x008cb9 thru 0x00a2af (0x15f7 bytes) 000943 1744535574.334052 mapread 0x00d7ae thru 0x013593 (0x5de6 bytes) 000944 1744535574.336670 mapread 0x01bb74 thru 0x01da1a (0x1ea7 bytes) 000947 1744535574.338526 mapwrite 0x0159c6 thru 0x016f95 (0x15d0 bytes) 000950 1744535574.350125 write 0x01fc39 thru 0x0274c2 (0x788a bytes) 000953 1744535574.351275 trunc from 0x0274c3 to 0x02c7ba (0x52f8 bytes) 000955 1744535574.355316 write 0x00bf61 thru 0x010acf (0x4b6f bytes) 000957 1744535574.358765 trunc from 0x02c7bb to 0x037f6d (0xb7b3 bytes) 000960 1744535574.362416 trunc from 0x037f6e to 0x0049a2 (0x335cb bytes) 000962 1744535574.366886 MIRROR_EXTEND to 11 mirrors 000963 1744535574.453742 mapwrite 0x035217 thru 0x039c3f (0x4a29 bytes) 000964 1744535574.492450 mapread 0x0027ee thru 0x01024f (0xda62 bytes) 000965 1744535574.497241 mapwrite 0x0188dc thru 0x01d7bb (0x4ee0 bytes) 000966 1744535574.512211 trunc from 0x039c40 to 0x013fe0 (0x25c5f bytes) 000968 1744535574.517276 write 0x011784 thru 0x01ef05 (0xd782 bytes) 000969 1744535574.518008 write 0x018865 thru 0x019213 (0x9af bytes) 000974 1744535574.519249 write 0x01e7ef thru 0x024bc6 (0x63d8 bytes) 000976 1744535574.519951 write 0x011e05 thru 0x020573 (0xe76f bytes) 000977 1744535574.520736 read 0x00b6b5 thru 0x0199a5 (0xe2f1 bytes) 000982 1744535574.523324 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x932:0x0]: Invalid argument (22) 000983 1744535574.573136 write 0x006d4f thru 0x00faa2 (0x8d54 bytes) 000985 1744535574.575945 write 0x02fcc1 thru 0x03a8a3 (0xabe3 bytes) 000986 1744535574.576696 write 0x01e08b thru 0x029943 (0xb8b9 bytes) 000989 1744535574.580245 read 0x0321d0 thru 0x0368a7 (0x46d8 bytes) 000990 1744535574.582275 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x932:0x0]: Invalid argument (22) 000991 1744535574.632328 trunc from 0x03a8a4 to 0x02e493 (0xc410 bytes) 000992 1744535574.635799 mapread 0x027b5e thru 0x02e493 (0x6936 bytes) 000993 1744535574.640072 read 0x028267 thru 0x02e493 (0x622d bytes) 000994 1744535574.640439 read 0x019b37 thru 0x01fb15 (0x5fdf bytes) 000995 1744535574.642812 read 0x01d5b9 thru 0x0252c7 (0x7d0f bytes) 000997 1744535574.643279 mapwrite 0x004b79 thru 0x010757 (0xbbdf bytes) 001000 1744535574.659501 mapread 0x0298d1 thru 0x02e493 (0x4bc3 bytes) All operations completed A-OK! PASS 70a (15s) == sanity-flr test 71: check mirror extend parent fid ==== 05:12:55 (1744535575) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 200a: stress test ===================== 05:12:56 (1744535576) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre2 Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3886272 .. Extending file size to 2238816 .. Extending file size to 7092928 .. Extending file size to 6997696 .. done Extending file size to 8617568 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8462912 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8680256 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8641728 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9282016 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8723680 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8902464 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9096128 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9215584 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 30317 30318 30320 30322 30324 failed 192.168.203.126@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg326-client.virtnet /mnt/lustre2 (opts:) 192.168.203.126@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg326-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 05:14:00 (1744535640) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre2 Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5509184 .. Extending file size to 5028704 .. failed Extending file size to 7365152 .. Extending file size to 5545536 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8052064 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8171520 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8907840 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8488224 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8614656 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8948768 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8931680 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 9030560 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9119808 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 2141 2142 2144 2146 2148 192.168.203.126@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg326-client.virtnet /mnt/lustre2 (opts:) 192.168.203.126@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg326-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 05:15:03 (1744535703) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 5632 2202624 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 5632 2202624 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0281849 s, 74.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0866037 s, 24.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.126@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg326-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 05:15:07 (1744535707) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0202574 s, 104 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 05:15:09 (1744535709) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x980:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcea:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcea:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xceb:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0xced:0x0] } PASS 203 (1s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 05:15:10 (1744535710) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0305564 s, 68.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcec:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcee:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcef:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 05:15:12 (1744535712) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0284384 s, 73.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xced:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcee:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcf0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xced:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcee:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcf0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcf1:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 05:15:14 (1744535714) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0235841 s, 88.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcf2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcf2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcf3:0x0] } PASS 204c (22s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 05:15:36 (1744535736) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg326-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0192477 s, 54.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcf0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcf4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcf0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcf4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcf1:0x0] } PASS 204d (41s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 05:16:17 (1744535777) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg326-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcf2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcf3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0234851 s, 89.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcf2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcf3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcf6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcf2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcf3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcf6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (40s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 05:16:57 (1744535817) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg326-server mds-ost sync done. Creating new pool test_204f oleg326-server: Pool lustre.test_204f created Adding targets to pool oleg326-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg326-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0240494 s, 87.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcf4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcf7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcf4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcf7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg326-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg326-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg326-server: Pool lustre.test_204f destroyed PASS 204f (51s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 05:17:48 (1744535868) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 05:17:50 (1744535870) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 05:17:52 (1744535872) oleg326-server: Pool lustre.test_206 created Adding targets to pool oleg326-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg326-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg326-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg326-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg326-server: Pool lustre.test_206 destroyed PASS 206 (11s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 05:18:03 (1744535883) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0184337 s, 56.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00862468 s, 122 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xb29fa08a Mirror 3: 0xb29fa08a PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 05:18:05 (1744535885) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 05:18:06 (1744535886) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 05:18:07 (1744535887) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0180321 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1020000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1380000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1380000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1380000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xe80000 Read this (4 bytes): BBBB 192.168.203.126@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg326-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 05:18:20 (1744535900) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00252684 s, 3.2 MB/s the very first read read with warm cache PASS 209b (1s) == sanity-flr test 210a: handle broken mirrored lovea ==== 05:18:21 (1744535901) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00929972 s, 113 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x9a5 lmm_fid: [0x200000404:0x9a5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x9a5 lmm_fid: [0x200000404:0x9a5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd02:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322836389 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-13 05:18:22.000000000 -0400 Modify: 2025-04-13 05:18:22.000000000 -0400 Change: 2025-04-13 05:18:22.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 05:18:23 (1744535903) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 05:18:24 (1744535904) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0976456 s, 419 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0139615 s, 293 kB/s size after second write -rw-r--r-- 1 root root 4096 Apr 13 05:18 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9a7:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Apr 13 05:18 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 666 sec ============ 05:18:26 (1744535906) === sanity-flr: start cleanup 05:18:26 (1744535906) === === sanity-flr: finish cleanup 05:18:27 (1744535907) ===