-----============= acceptance-small: sanity-flr ============----- Fri Mar 28 19:06:38 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg216-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 19:06:42 (1743203202) === oleg216-client.virtnet: executing check_config_client /mnt/lustre oleg216-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg216-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b6069800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6069800.idle_timeout=debug disable quota as required oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 19:06:49 (1743203209) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7010] == sanity-flr test 0a: lfs mirror create with -N option == 19:06:50 (1743203210) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 19:06:53 (1743203213) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 19:06:54 (1743203214) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 19:06:55 (1743203215) PASS 0d (5s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 19:07:00 (1743203220) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (0s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 19:07:00 (1743203220) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 19:07:01 (1743203221) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b6069800.stats=clear osc.lustre-OST0001-osc-ffff8800b6069800.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 19:07:03 (1743203223) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 19:07:05 (1743203225) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 19:07:07 (1743203227) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 19:07:08 (1743203228) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 19:07:10 (1743203230) PASS 2 (1s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 19:07:11 (1743203231) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 19:07:12 (1743203232) lfs extend: cannot get the last init comp error: lfs extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 19:07:14 (1743203234) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (1s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 19:07:16 (1743203236) PASS 7 (1s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 19:07:17 (1743203237) 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.423799 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0373102 s, 28.1 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 19:07:20 (1743203240) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB) copied, 0.0545573 s, 173 MB/s PASS 22 (1s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 19:07:22 (1743203242) Stopping /mnt/lustre-ost1 (opts:) on oleg216-server oleg216-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6069800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6069800.ost_server_uuid in DISCONN state after 4 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg216-server: oleg216-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg216-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg216-server oleg216-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6069800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6069800.ost_server_uuid in DISCONN state after 3 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg216-server: oleg216-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg216-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (26s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 19:07:48 (1743203268) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0601193 s, 69.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg216-server oleg216-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6069800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6069800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg216-server: oleg216-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg216-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (9s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 19:07:57 (1743203277) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg216-server oleg216-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6069800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6069800.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg216-server: oleg216-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg216-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg216-server oleg216-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6069800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6069800.ost_server_uuid in DISCONN state after 4 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg216-server: oleg216-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg216-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (21s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 19:08:18 (1743203298) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.73054 s, 71.8 MB/s -rw-r--r-- 1 root root 52428800 Mar 28 19:08 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.228s user 0m0.000s sys 0m0.103s Stopping /mnt/lustre-ost1 (opts:) on oleg216-server oleg216-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6069800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6069800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.225s user 0m0.000s sys 0m0.099s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg216-server: oleg216-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg216-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg216-server oleg216-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6069800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6069800.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost1)... real 0m0.271s user 0m0.002s sys 0m0.112s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg216-server: oleg216-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg216-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg216-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b6069800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6069800.ost_server_uuid in FULL state after 0 sec PASS 33b (22s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 19:08:40 (1743203320) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 19:08:41 (1743203321) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 19:08:42 (1743203322) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 19:08:43 (1743203323) PASS 35 (1s) == sanity-flr test 36a: write to mirrored files ========== 19:08:44 (1743203324) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.121372 s, 104 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 19:08:47 (1743203327) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00971867 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0073611 s, 142 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 19:08:49 (1743203329) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 19:08:52 (1743203332) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000741492 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000742288 s, 1.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000630868 s, 1.6 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00112642 s, 909 kB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 19:08:54 (1743203334) create test files with size 15737333 .. c1758f14ecbc3eab73430baee215c09b - 28c6f363fc203a645539c5aeca0989bf - 8d8994e1a4ab0bf160ad2c93c519248e - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg216-client.virtnet /mnt/lustre (opts:) Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre PASS 37 (5s) == sanity-flr test 38: resync ============================ 19:08:59 (1743203339) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 19:09:02 (1743203342) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 19:09:04 (1743203344) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0199393 s, 52.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00926868 s, 113 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 19:09:06 (1743203346) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0243138 s, 86.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0854192 s, 49.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0374729 s, 56.0 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) == sanity-flr test 42: lfs mirror verify ================= 19:09:08 (1743203348) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 19:09:09 (1743203349) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 19:09:10 (1743203350) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0062379 s, 168 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (1s) == sanity-flr test 44a: lfs mirror split check =========== 19:09:11 (1743203351) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 19:09:12 (1743203352) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 19:09:14 (1743203354) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0533103 s, 197 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg216-client.virtnet /mnt/lustre (opts:) Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (24s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 19:09:38 (1743203378) llite.lustre-ffff8800aa7a7800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0625185 s, 168 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg216-client.virtnet /mnt/lustre (opts:) Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff88012ab2d000.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 19:09:41 (1743203381) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (6s) == sanity-flr test 46: Verify setstripe --copy option ==== 19:09:47 (1743203387) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (6s) == sanity-flr test 47: Verify mirror obj alloc =========== 19:09:53 (1743203393) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 19:09:54 (1743203394) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0514256 s, 61.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0406259 s, 77.4 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: a5a40fd6e951dd807624a90caaf24ca9 - mirror 1 checksum: 3a6e2edf199b50ccc09e99e73fe77e6e - mirror 2 checksum: a5a40fd6e951dd807624a90caaf24ca9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Mar 28 19:09:54 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Mar 28 19:09:54 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: a5a40fd6e951dd807624a90caaf24ca9 - mirror 1 checksum: 3a6e2edf199b50ccc09e99e73fe77e6e - mirror 2 checksum: 3a6e2edf199b50ccc09e99e73fe77e6e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Mar 28 19:09:54 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Mar 28 19:09:54 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 48 (2s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 19:09:56 (1743203396) Failing mds1 on oleg216-server Stopping /mnt/lustre-mds1 (opts:) on oleg216-server 19:10:02 (1743203402) shut down facet: mds1 facet_host: oleg216-server facet_failover_host: oleg216-server Failover mds1 to oleg216-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-MDT0000 19:10:15 (1743203415) targets are mounted 19:10:15 (1743203415) facet_failover done oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00903782 s, 116 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } PASS 50A (27s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 19:10:23 (1743203423) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (0s) == sanity-flr test 50b: mirror rsync handles sparseness == 19:10:23 (1743203423) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 19:10:24 (1743203424) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00175011 s, 9.4 MB/s PASS 50c (1s) == sanity-flr test 50d: mirror rsync keep holes ========== 19:10:25 (1743203425) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 19:10:26 (1743203426) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 19:10:27 (1743203427) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (4s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 19:10:31 (1743203431) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (17s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 19:10:48 (1743203448) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 19:11:05 (1743203465) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 19:11:22 (1743203482) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 19:11:24 (1743203484) mirror create pid 22548 mirror split pid 22549 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6748: 22549 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6748: 22549 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4884: 22548 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4884: 22549 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (61s) == sanity-flr test 70a: flr mode fsx test ================ 19:12:25 (1743203545) Seed set to 3546 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1743203546.237383 write 0x02753a thru 0x027dd7 (0x89e bytes) 000003 1743203546.240524 mapread 0x00c484 thru 0x019c11 (0xd78e bytes) 000011 1743203546.252858 MIRROR_EXTEND to 2 mirrors 000012 1743203546.357137 mapread 0x022c03 thru 0x026582 (0x3980 bytes) 000013 1743203546.362438 read 0x026c3e thru 0x027dd7 (0x119a bytes) 000015 1743203546.364683 read 0x016576 thru 0x023555 (0xcfe0 bytes) 000016 1743203546.367379 mapread 0x011241 thru 0x01a597 (0x9357 bytes) 000019 1743203546.370945 mapwrite 0x032d37 thru 0x03b5ab (0x8875 bytes) truncating to largest ever: 0x162ee 000021 1743203546.403124 trunc from 0x03b5ac to 0x0162ed (0x252be bytes) 000022 1743203546.408129 mapwrite 0x026f78 thru 0x0339b1 (0xca3a bytes) 000023 1743203546.439846 mapread 0x00f446 thru 0x019d2d (0xa8e8 bytes) 000024 1743203546.448825 trunc from 0x0339b2 to 0x00d1e5 (0x267cc bytes) 000026 1743203546.453275 MIRROR_EXTEND to 3 mirrors 000028 1743203546.551336 mapread 0x006bd6 thru 0x00d1e5 (0x6610 bytes) 000029 1743203546.556618 mapwrite 0x017f54 thru 0x0184a5 (0x552 bytes) 000031 1743203546.598992 write 0x004991 thru 0x0112ff (0xc96f bytes) 000032 1743203546.603211 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000035 1743203546.652771 read 0x015e8b thru 0x017264 (0x13da bytes) 000036 1743203546.654553 write 0x002afd thru 0x00c7e1 (0x9ce5 bytes) truncating to largest ever: 0x1e49e 000037 1743203546.657705 trunc from 0x0184a6 to 0x01e49d (0x5ff8 bytes) 000039 1743203546.663283 read 0x00cab9 thru 0x00edaf (0x22f7 bytes) 000042 1743203546.666406 mapread 0x00e888 thru 0x0138c1 (0x503a bytes) 000043 1743203546.667005 write 0x023258 thru 0x02e9d2 (0xb77b bytes) 000045 1743203546.669748 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000047 1743203546.704725 write 0x00df14 thru 0x01100d (0x30fa bytes) 000048 1743203546.707942 write 0x03a878 thru 0x03ffff (0x5788 bytes) 000049 1743203546.708732 write 0x0078f7 thru 0x00eae8 (0x71f2 bytes) 000058 1743203546.715283 mapread 0x038eca thru 0x03ffff (0x7136 bytes) truncating to largest ever: 0x22cce 000059 1743203546.719158 trunc from 0x040000 to 0x022ccd (0x1d332 bytes) 000061 1743203546.726006 mapread 0x0093d0 thru 0x00b938 (0x2569 bytes) 000063 1743203546.726949 write 0x031a51 thru 0x03a934 (0x8ee4 bytes) 000064 1743203546.729707 MIRROR_EXTEND to 2 mirrors 000065 1743203546.851092 MIRROR_RESYNC 2 mirrors 000066 1743203546.890069 mapread 0x0078c5 thru 0x00f0f0 (0x782c bytes) truncating to largest ever: 0x2fecf 000067 1743203546.895775 trunc from 0x03a935 to 0x02fece (0xaa66 bytes) 000069 1743203546.917062 read 0x018f02 thru 0x026da3 (0xdea2 bytes) 000070 1743203546.919643 trunc from 0x02fecf to 0x007047 (0x28e87 bytes) 000071 1743203546.923754 write 0x02c1e3 thru 0x03ba60 (0xf87e bytes) 000073 1743203546.925784 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000076 1743203546.958343 mapread 0x0299a5 thru 0x037ea7 (0xe503 bytes) 000077 1743203546.965196 write 0x012c71 thru 0x013e16 (0x11a6 bytes) 000079 1743203546.967695 write 0x03f52d thru 0x03fecf (0x9a3 bytes) 000082 1743203546.968858 read 0x00e108 thru 0x017c05 (0x9afe bytes) 000086 1743203546.970223 trunc from 0x03fed0 to 0x02a1e9 (0x15ce6 bytes) truncating to largest ever: 0x3929c 000087 1743203546.975064 trunc from 0x02a1ea to 0x03929b (0xf0b2 bytes) 000088 1743203546.978822 mapwrite 0x015586 thru 0x01a59b (0x5016 bytes) 000090 1743203546.995602 mapwrite 0x0288c9 thru 0x035598 (0xccd0 bytes) 000092 1743203547.015401 write 0x0097a7 thru 0x018d34 (0xf58e bytes) 000094 1743203547.018903 trunc from 0x03929c to 0x007b60 (0x3173b bytes) truncating to largest ever: 0x3c97a 000095 1743203547.022872 trunc from 0x007b61 to 0x03c979 (0x34e19 bytes) 000096 1743203547.026387 mapwrite 0x0138eb thru 0x022e15 (0xf52b bytes) 000098 1743203547.046956 write 0x02dcb3 thru 0x03a718 (0xca66 bytes) 000099 1743203547.051184 trunc from 0x03c97a to 0x01882b (0x2414e bytes) 000103 1743203547.055219 mapread 0x00ad7e thru 0x00f6bd (0x4940 bytes) 000104 1743203547.056888 mapread 0x0062ae thru 0x00d774 (0x74c7 bytes) 000105 1743203547.058177 MIRROR_EXTEND to 2 mirrors 000107 1743203547.158426 read 0x0172c6 thru 0x0187d1 (0x150c bytes) 000108 1743203547.161417 mapwrite 0x016bac thru 0x02313d (0xc592 bytes) 000109 1743203547.213245 trunc from 0x02313e to 0x00ba9a (0x176a3 bytes) 000112 1743203547.220188 mapwrite 0x001f55 thru 0x0060a9 (0x4155 bytes) 000113 1743203547.236560 write 0x039396 thru 0x03ae0b (0x1a76 bytes) 000114 1743203547.237199 trunc from 0x03ae0c to 0x012dbf (0x2804c bytes) 000115 1743203547.241571 mapread 0x008646 thru 0x00bf2f (0x38ea bytes) 000116 1743203547.242181 read 0x000542 thru 0x003d27 (0x37e6 bytes) 000118 1743203547.245039 mapwrite 0x02b7ec thru 0x02fad2 (0x42e7 bytes) 000119 1743203547.265199 trunc from 0x02fad3 to 0x025c3c (0x9e96 bytes) 000120 1743203547.269543 mapread 0x000a6c thru 0x005b50 (0x50e5 bytes) 000121 1743203547.270173 write 0x000915 thru 0x00aac5 (0xa1b1 bytes) 000122 1743203547.271003 write 0x01ab90 thru 0x0211cb (0x663c bytes) 000123 1743203547.271696 mapwrite 0x0240e1 thru 0x02ebee (0xab0e bytes) 000125 1743203547.307223 mapread 0x00e2bc thru 0x01ca25 (0xe76a bytes) 000126 1743203547.307978 trunc from 0x02ebef to 0x02379f (0xb44f bytes) 000128 1743203547.312827 mapwrite 0x0394df thru 0x03ffff (0x6b21 bytes) 000130 1743203547.337645 mapread 0x02c66b thru 0x03a63a (0xdfd0 bytes) 000135 1743203547.344850 mapwrite 0x02e775 thru 0x02fccc (0x1558 bytes) 000136 1743203547.357408 write 0x03635d thru 0x039ea0 (0x3b44 bytes) 000137 1743203547.357920 read 0x00d205 thru 0x015e22 (0x8c1e bytes) 000139 1743203547.358465 read 0x035d8a thru 0x03c687 (0x68fe bytes) 000145 1743203547.359680 mapwrite 0x00bd32 thru 0x01195d (0x5c2c bytes) 000147 1743203547.374120 trunc from 0x040000 to 0x02e2c5 (0x11d3a bytes) 000148 1743203547.378171 trunc from 0x02e2c6 to 0x00cfc0 (0x21305 bytes) 000150 1743203547.382895 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000151 1743203547.414637 mapread 0x00bb5a thru 0x00cfc0 (0x1467 bytes) 000152 1743203547.417620 mapwrite 0x02ecf4 thru 0x03136f (0x267c bytes) 000154 1743203547.437675 read 0x020d45 thru 0x02a11a (0x93d6 bytes) 000156 1743203547.441053 trunc from 0x031370 to 0x019636 (0x17d39 bytes) 000159 1743203547.445036 write 0x0188ad thru 0x01b262 (0x29b6 bytes) 000160 1743203547.447094 write 0x02cd7f thru 0x02dc1f (0xea1 bytes) 000163 1743203547.448052 mapwrite 0x033092 thru 0x034613 (0x1582 bytes) 000164 1743203547.469685 write 0x00ae24 thru 0x00e9e1 (0x3bbe bytes) 000168 1743203547.473399 mapread 0x0066ad thru 0x01289d (0xc1f1 bytes) 000169 1743203547.473996 trunc from 0x034614 to 0x00633d (0x2e2d6 bytes) 000171 1743203547.479269 mapwrite 0x019579 thru 0x028fec (0xfa74 bytes) 000172 1743203547.502891 trunc from 0x028fed to 0x03af68 (0x11f7c bytes) 000174 1743203547.507075 write 0x0321ef thru 0x034c8f (0x2aa1 bytes) 000176 1743203547.511227 MIRROR_EXTEND to 2 mirrors 000177 1743203547.644367 read 0x0328c3 thru 0x034736 (0x1e74 bytes) 000178 1743203547.646815 mapwrite 0x026668 thru 0x02ecf2 (0x868b bytes) 000179 1743203547.698087 MIRROR_EXTEND to 3 mirrors 000183 1743203547.816188 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000186 1743203547.867428 read 0x0218d2 thru 0x029f44 (0x8673 bytes) 000187 1743203547.869836 mapwrite 0x01244c thru 0x01d00f (0xabc4 bytes) 000189 1743203547.895525 mapwrite 0x02ecb3 thru 0x03d176 (0xe4c4 bytes) 000191 1743203547.929771 read 0x027864 thru 0x0305f3 (0x8d90 bytes) 000193 1743203547.930510 mapread 0x017546 thru 0x01ff18 (0x89d3 bytes) 000194 1743203547.931079 trunc from 0x03d177 to 0x01a72b (0x22a4b bytes) 000197 1743203547.936292 mapwrite 0x02c95e thru 0x02ee31 (0x24d4 bytes) 000199 1743203547.956561 mapwrite 0x03c8b8 thru 0x03ffff (0x3748 bytes) 000200 1743203547.980159 trunc from 0x040000 to 0x0194a5 (0x26b5a bytes) 000201 1743203547.984999 read 0x00575c thru 0x007d8a (0x262f bytes) 000202 1743203547.985408 trunc from 0x0194a6 to 0x015de7 (0x36be bytes) 000203 1743203547.992666 mapwrite 0x0147f2 thru 0x015e2f (0x163e bytes) 000204 1743203548.013632 mapwrite 0x029fcf thru 0x02cbf0 (0x2c22 bytes) 000205 1743203548.033925 mapwrite 0x0388c8 thru 0x03fa01 (0x713a bytes) 000206 1743203548.058117 mapread 0x02bfec thru 0x03777a (0xb78f bytes) truncating to largest ever: 0x3ffc0 000207 1743203548.058736 trunc from 0x03fa02 to 0x03ffbf (0x5be bytes) 000209 1743203548.063329 mapread 0x03c6f3 thru 0x03ffbf (0x38cd bytes) 000210 1743203548.063731 mapread 0x02dc7e thru 0x03d74b (0xface bytes) 000212 1743203548.064517 mapwrite 0x01c64b thru 0x023ca0 (0x7656 bytes) 000214 1743203548.079202 read 0x0182c3 thru 0x024964 (0xc6a2 bytes) 000215 1743203548.079646 mapread 0x003c83 thru 0x0054bf (0x183d bytes) 000216 1743203548.081772 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000217 1743203548.116646 mapwrite 0x0079e5 thru 0x0163de (0xe9fa bytes) 000220 1743203548.135847 write 0x03b6b1 thru 0x03ffff (0x494f bytes) 000224 1743203548.139286 write 0x01636e thru 0x018354 (0x1fe7 bytes) 000225 1743203548.139699 read 0x0043c1 thru 0x00c4e2 (0x8122 bytes) 000226 1743203548.139962 read 0x020d19 thru 0x02c439 (0xb721 bytes) 000227 1743203548.143087 trunc from 0x040000 to 0x0303ca (0xfc35 bytes) 000228 1743203548.147317 write 0x00a176 thru 0x00a6b8 (0x543 bytes) 000229 1743203548.147774 trunc from 0x0303cb to 0x025ca4 (0xa726 bytes) 000231 1743203548.151522 mapwrite 0x03bc98 thru 0x03ffff (0x4368 bytes) 000233 1743203548.185714 mapwrite 0x03a483 thru 0x03ffff (0x5b7d bytes) 000234 1743203548.206166 mapwrite 0x020401 thru 0x02333c (0x2f3c bytes) 000235 1743203548.220957 read 0x020de7 thru 0x02cc7a (0xbe94 bytes) 000237 1743203548.224765 mapread 0x01f389 thru 0x0292c2 (0x9f3a bytes) 000245 1743203548.227858 trunc from 0x040000 to 0x03de5a (0x21a5 bytes) 000246 1743203548.233594 MIRROR_EXTEND to 2 mirrors 000247 1743203548.336868 mapwrite 0x0206c9 thru 0x026c0e (0x6546 bytes) 000248 1743203548.389061 mapread 0x02684b thru 0x03550c (0xecc2 bytes) 000249 1743203548.394360 write 0x0117f9 thru 0x018947 (0x714f bytes) 000252 1743203548.397917 mapwrite 0x01f630 thru 0x025601 (0x5fd2 bytes) 000254 1743203548.415452 MIRROR_RESYNC 2 mirrors 000256 1743203548.510793 mapread 0x018b0a thru 0x01cac5 (0x3fbc bytes) 000257 1743203548.514872 mapread 0x032144 thru 0x038874 (0x6731 bytes) 000262 1743203548.520421 mapread 0x00a756 thru 0x012cb5 (0x8560 bytes) 000263 1743203548.526017 write 0x031270 thru 0x03d8fe (0xc68f bytes) 000264 1743203548.552896 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000266 1743203548.596679 trunc from 0x03de5b to 0x023cf9 (0x1a161 bytes) 000270 1743203548.604577 mapread 0x01ad92 thru 0x01c9c9 (0x1c38 bytes) 000271 1743203548.608959 mapwrite 0x029e1e thru 0x037bbb (0xdd9e bytes) 000272 1743203548.632852 mapwrite 0x01aade thru 0x01dd6e (0x3291 bytes) 000275 1743203548.647953 mapwrite 0x019867 thru 0x0254b4 (0xbc4e bytes) 000277 1743203548.664240 mapwrite 0x0078bf thru 0x008641 (0xd83 bytes) 000280 1743203548.678399 mapread 0x012f92 thru 0x01aa55 (0x7ac4 bytes) 000284 1743203548.681143 mapread 0x028d8a thru 0x035019 (0xc290 bytes) 000287 1743203548.682235 mapwrite 0x039a56 thru 0x03ffff (0x65aa bytes) 000288 1743203548.699367 mapread 0x035ead thru 0x037c01 (0x1d55 bytes) 000289 1743203548.700723 MIRROR_EXTEND to 2 mirrors 000290 1743203548.807737 mapread 0x003d11 thru 0x012f88 (0xf278 bytes) 000292 1743203548.817648 read 0x00b119 thru 0x00f1b2 (0x409a bytes) 000294 1743203548.819606 mapread 0x00e58f thru 0x013a1f (0x5491 bytes) 000296 1743203548.822490 write 0x03f31b thru 0x03ffff (0xce5 bytes) 000297 1743203548.839533 mapread 0x02d140 thru 0x03bf90 (0xee51 bytes) 000300 1743203548.845689 trunc from 0x040000 to 0x0283b4 (0x17c4b bytes) 000302 1743203548.857737 MIRROR_EXTEND to 3 mirrors 000304 1743203548.966975 mapread 0x01be73 thru 0x01c5ac (0x73a bytes) 000309 1743203548.976580 write 0x03cc03 thru 0x03ffff (0x33fd bytes) 000310 1743203548.992378 write 0x02600b thru 0x02cc00 (0x6bf6 bytes) 000311 1743203548.994954 write 0x020576 thru 0x02da9b (0xd526 bytes) 000312 1743203548.995711 write 0x01162d thru 0x01833a (0x6d0e bytes) 000314 1743203548.998460 trunc from 0x040000 to 0x039827 (0x67d8 bytes) 000315 1743203549.005908 trunc from 0x039828 to 0x010a10 (0x28e17 bytes) 000316 1743203549.009566 read 0x00b5cf thru 0x010a10 (0x5442 bytes) 000317 1743203549.010007 mapread 0x001e1b thru 0x002b48 (0xd2e bytes) 000319 1743203549.013476 trunc from 0x010a11 to 0x00c987 (0x4089 bytes) 000320 1743203549.017430 mapread 0x00b85f thru 0x00c987 (0x1129 bytes) 000321 1743203549.017801 mapread 0x001a23 thru 0x00c987 (0xaf65 bytes) 000322 1743203549.018315 write 0x01485f thru 0x01e345 (0x9ae7 bytes) 000323 1743203549.019294 trunc from 0x01e346 to 0x009ea6 (0x1449f bytes) 000324 1743203549.023626 write 0x02379d thru 0x03199e (0xe202 bytes) 000326 1743203549.027035 MIRROR_EXTEND to 4 mirrors 000327 1743203549.175721 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000328 1743203549.244751 trunc from 0x03199f to 0x02fbad (0x1df1 bytes) 000331 1743203549.261207 write 0x01c270 thru 0x01eb9b (0x292c bytes) 000335 1743203549.264537 mapread 0x015b42 thru 0x023bb0 (0xe06f bytes) 000336 1743203549.269957 mapwrite 0x023508 thru 0x02570c (0x2205 bytes) 000337 1743203549.284564 read 0x018715 thru 0x020900 (0x81ec bytes) 000338 1743203549.284899 trunc from 0x02fbae to 0x03d4ca (0xd91d bytes) 000339 1743203549.290590 trunc from 0x03d4cb to 0x01c047 (0x21483 bytes) 000341 1743203549.294771 trunc from 0x01c048 to 0x01ab1e (0x1529 bytes) 000342 1743203549.298602 mapread 0x012eb9 thru 0x016dff (0x3f47 bytes) 000343 1743203549.300979 mapwrite 0x03b5ff thru 0x03ffff (0x4a01 bytes) 000344 1743203549.321404 mapwrite 0x03574b thru 0x03ffff (0xa8b5 bytes) 000345 1743203549.335853 trunc from 0x040000 to 0x0075ee (0x38a11 bytes) 000346 1743203549.340128 trunc from 0x0075ef to 0x0228fc (0x1b30e bytes) 000351 1743203549.346693 MIRROR_EXTEND to 4 mirrors 000352 1743203549.466870 mapread 0x00e364 thru 0x015494 (0x7131 bytes) 000354 1743203549.477915 write 0x03a573 thru 0x03ffff (0x5a8d bytes) 000355 1743203549.499239 mapwrite 0x023bfd thru 0x02e520 (0xa924 bytes) 000356 1743203549.525135 mapwrite 0x011f11 thru 0x01c5f0 (0xa6e0 bytes) 000357 1743203549.545529 mapwrite 0x024803 thru 0x027e88 (0x3686 bytes) 000359 1743203549.557727 trunc from 0x040000 to 0x012353 (0x2dcac bytes) 000361 1743203549.562715 mapwrite 0x01ff92 thru 0x02e5dd (0xe64c bytes) 000362 1743203549.584586 MIRROR_EXTEND to 5 mirrors 000363 1743203549.690647 trunc from 0x02e5de to 0x02b3df (0x31fe bytes) 000364 1743203549.713016 read 0x0190ef thru 0x01ce8c (0x3d9e bytes) 000367 1743203549.715519 write 0x03ba33 thru 0x03ffff (0x45cd bytes) 000368 1743203549.716317 read 0x030feb thru 0x033765 (0x277b bytes) 000370 1743203549.720812 mapwrite 0x03930e thru 0x03ffff (0x6cf2 bytes) 000372 1743203549.743143 mapread 0x02c84b thru 0x02f478 (0x2c2e bytes) 000377 1743203549.745038 mapwrite 0x03b4e0 thru 0x03ffff (0x4b20 bytes) 000378 1743203549.759070 read 0x03aad2 thru 0x03db59 (0x3088 bytes) 000379 1743203549.759475 trunc from 0x040000 to 0x02a946 (0x156b9 bytes) 000380 1743203549.766437 trunc from 0x02a947 to 0x01d5ed (0xd359 bytes) 000381 1743203549.771243 MIRROR_EXTEND to 6 mirrors 000382 1743203549.863432 write 0x0256d7 thru 0x02f9b7 (0xa2e1 bytes) 000388 1743203549.879093 read 0x01c0af thru 0x028ba7 (0xcaf9 bytes) 000389 1743203549.881400 read 0x02472d thru 0x02f9b7 (0xb28b bytes) 000390 1743203549.881807 read 0x0229e6 thru 0x0258b9 (0x2ed4 bytes) 000391 1743203549.882032 mapwrite 0x01d56a thru 0x0245ef (0x7086 bytes) 000392 1743203549.896577 mapwrite 0x018159 thru 0x024c86 (0xcb2e bytes) 000393 1743203549.911796 read 0x017800 thru 0x0190bd (0x18be bytes) 000394 1743203549.913909 MIRROR_RESYNC 6 mirrors 000396 1743203550.122443 MIRROR_RESYNC 6 mirrors 000398 1743203550.178130 trunc from 0x02f9b8 to 0x030398 (0x9e1 bytes) 000400 1743203550.196552 mapread 0x007855 thru 0x00a457 (0x2c03 bytes) 000401 1743203550.201057 mapwrite 0x015a1c thru 0x01f33a (0x991f bytes) 000402 1743203550.223817 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000403 1743203550.269166 write 0x017f3b thru 0x0195c3 (0x1689 bytes) 000405 1743203550.271820 mapread 0x02a2d6 thru 0x030398 (0x60c3 bytes) 000406 1743203550.274076 trunc from 0x030399 to 0x020c90 (0xf708 bytes) 000408 1743203550.278065 write 0x039c4d thru 0x03ffff (0x63b3 bytes) 000409 1743203550.278589 read 0x03c33f thru 0x03ffff (0x3cc1 bytes) 000410 1743203550.278816 trunc from 0x040000 to 0x02ad1e (0x152e1 bytes) 000412 1743203550.281995 write 0x0240bc thru 0x03316e (0xf0b3 bytes) 000413 1743203550.285688 MIRROR_EXTEND to 6 mirrors 000415 1743203550.416373 MIRROR_EXTEND to 7 mirrors 000419 1743203550.523703 mapread 0x00992d thru 0x00f045 (0x5719 bytes) 000420 1743203550.528681 MIRROR_EXTEND to 8 mirrors 000422 1743203550.646140 read 0x02a3ef thru 0x03316e (0x8d80 bytes) 000424 1743203550.650382 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000425 1743203550.755056 mapread 0x0144b1 thru 0x023308 (0xee58 bytes) 000427 1743203550.772392 write 0x015f28 thru 0x025a55 (0xfb2e bytes) 000429 1743203550.794231 trunc from 0x03316f to 0x0212b2 (0x11ebc bytes) 000430 1743203550.801223 mapwrite 0x0290d8 thru 0x03619e (0xd0c7 bytes) 000432 1743203550.830058 mapwrite 0x01b656 thru 0x0299df (0xe38a bytes) 000433 1743203550.844535 read 0x00c974 thru 0x00ca5b (0xe8 bytes) 000435 1743203550.847575 read 0x003e6b thru 0x00f073 (0xb209 bytes) 000436 1743203550.850161 mapread 0x00bb3a thru 0x0140c6 (0x858d bytes) 000437 1743203550.850748 read 0x02bae3 thru 0x03434b (0x8869 bytes) 000438 1743203550.850986 trunc from 0x03619f to 0x00e663 (0x27b3b bytes) 000439 1743203550.855207 read 0x00477e thru 0x005ec0 (0x1743 bytes) 000440 1743203550.855669 trunc from 0x00e664 to 0x025801 (0x1719e bytes) 000441 1743203550.860619 write 0x01c909 thru 0x01ddf0 (0x14e8 bytes) 000442 1743203550.867281 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000443 1743203550.926838 write 0x03c90c thru 0x03ffff (0x36f4 bytes) 000445 1743203550.927909 write 0x009d38 thru 0x00d90e (0x3bd7 bytes) 000446 1743203550.930833 read 0x0167c1 thru 0x01b25f (0x4a9f bytes) 000447 1743203550.934918 MIRROR_EXTEND to 7 mirrors 000449 1743203551.080888 read 0x01a183 thru 0x01c05d (0x1edb bytes) 000454 1743203551.092924 mapread 0x03dd8c thru 0x03ffff (0x2274 bytes) 000456 1743203551.104630 write 0x006181 thru 0x00996f (0x37ef bytes) 000457 1743203551.126109 trunc from 0x040000 to 0x03cc4d (0x33b2 bytes) 000458 1743203551.132921 write 0x030351 thru 0x033d5e (0x3a0e bytes) 000459 1743203551.137756 mapwrite 0x012ebd thru 0x015248 (0x238c bytes) 000460 1743203551.158923 write 0x019c65 thru 0x0232eb (0x9687 bytes) 000462 1743203551.162287 mapread 0x0371dc thru 0x03cc4d (0x5a72 bytes) 000463 1743203551.165127 mapread 0x0253e2 thru 0x02d2ce (0x7eed bytes) 000464 1743203551.168640 mapwrite 0x02b629 thru 0x0334ed (0x7ec5 bytes) 000466 1743203551.192285 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000467 1743203551.243226 trunc from 0x03cc4e to 0x0285b8 (0x14695 bytes) 000468 1743203551.247452 mapread 0x00993c thru 0x00a10a (0x7cf bytes) 000472 1743203551.251600 read 0x00077a thru 0x00308c (0x2913 bytes) 000474 1743203551.254902 write 0x023734 thru 0x030957 (0xd224 bytes) 000476 1743203551.259521 mapwrite 0x01fe2c thru 0x02187d (0x1a52 bytes) 000477 1743203551.276624 write 0x00ec26 thru 0x0152a1 (0x667c bytes) 000479 1743203551.280304 read 0x016847 thru 0x01767e (0xe38 bytes) 000482 1743203551.284357 mapwrite 0x02ac67 thru 0x02d915 (0x2caf bytes) 000484 1743203551.298876 mapwrite 0x01e0f9 thru 0x025f2c (0x7e34 bytes) 000485 1743203551.313029 write 0x0019e4 thru 0x006c5b (0x5278 bytes) 000486 1743203551.313769 read 0x018b84 thru 0x025d02 (0xd17f bytes) 000487 1743203551.314157 read 0x00b800 thru 0x00ff86 (0x4787 bytes) 000488 1743203551.314431 mapwrite 0x02867b thru 0x034b7e (0xc504 bytes) 000490 1743203551.335585 mapwrite 0x01fe2b thru 0x0224c1 (0x2697 bytes) 000491 1743203551.347228 mapread 0x013fbe thru 0x0192f5 (0x5338 bytes) 000492 1743203551.347837 read 0x0336b5 thru 0x034b7e (0x14ca bytes) 000494 1743203551.348599 mapread 0x0126fe thru 0x017875 (0x5178 bytes) 000495 1743203551.350856 MIRROR_RESYNC 6 mirrors 000496 1743203551.573127 write 0x03d0b1 thru 0x03d791 (0x6e1 bytes) 000498 1743203551.587129 write 0x014c2c thru 0x01db34 (0x8f09 bytes) 000502 1743203551.590395 read 0x02c67f thru 0x02f69c (0x301e bytes) 000503 1743203551.592189 mapwrite 0x038789 thru 0x03c97a (0x41f2 bytes) 000505 1743203551.610125 read 0x02e1c7 thru 0x03756f (0x93a9 bytes) 000506 1743203551.613000 read 0x02d081 thru 0x031765 (0x46e5 bytes) 000508 1743203551.613643 read 0x03a5f7 thru 0x03d791 (0x319b bytes) 000512 1743203551.614941 mapread 0x02b945 thru 0x02f4fc (0x3bb8 bytes) 000513 1743203551.615439 mapread 0x0066d5 thru 0x00fc27 (0x9553 bytes) 000514 1743203551.615994 mapwrite 0x03c809 thru 0x03ffff (0x37f7 bytes) 000515 1743203551.641645 read 0x0062d5 thru 0x0066de (0x40a bytes) 000516 1743203551.643490 MIRROR_RESYNC 6 mirrors 000518 1743203551.834356 read 0x0038ea thru 0x00d73e (0x9e55 bytes) 000520 1743203551.836832 mapread 0x001521 thru 0x00fbd2 (0xe6b2 bytes) 000522 1743203551.841472 write 0x01e6e9 thru 0x02db04 (0xf41c bytes) 000523 1743203551.861270 MIRROR_EXTEND to 7 mirrors 000524 1743203551.986951 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000525 1743203552.056747 write 0x02f0a9 thru 0x0390b7 (0xa00f bytes) 000526 1743203552.072161 trunc from 0x040000 to 0x03c1d2 (0x3e2d bytes) 000527 1743203552.079166 write 0x027790 thru 0x02cc38 (0x54a9 bytes) 000528 1743203552.082840 MIRROR_EXTEND to 7 mirrors 000530 1743203552.196120 trunc from 0x03c1d3 to 0x011f4b (0x2a287 bytes) 000531 1743203552.214006 mapwrite 0x021a50 thru 0x025d4e (0x42ff bytes) 000532 1743203552.238844 write 0x02dc02 thru 0x03ae34 (0xd233 bytes) 000533 1743203552.239718 write 0x016f1b thru 0x023e3a (0xcf20 bytes) 000534 1743203552.242359 mapwrite 0x00bf07 thru 0x0111a3 (0x529d bytes) 000536 1743203552.258991 mapwrite 0x0051f8 thru 0x0084ca (0x32d3 bytes) 000537 1743203552.270553 mapwrite 0x012e4d thru 0x02156a (0xe71e bytes) 000538 1743203552.284588 mapread 0x017692 thru 0x02431c (0xcc8b bytes) 000539 1743203552.286407 MIRROR_EXTEND to 8 mirrors 000540 1743203552.379173 write 0x03e357 thru 0x03ffff (0x1ca9 bytes) 000543 1743203552.397803 mapwrite 0x034267 thru 0x037191 (0x2f2b bytes) 000544 1743203552.416941 MIRROR_EXTEND to 9 mirrors 000547 1743203552.516485 write 0x027f02 thru 0x02a8dc (0x29db bytes) 000550 1743203552.533927 mapread 0x03af8e thru 0x03f0ff (0x4172 bytes) 000551 1743203552.536818 trunc from 0x040000 to 0x000fe2 (0x3f01d bytes) 000552 1743203552.546076 mapwrite 0x00bb59 thru 0x0184c3 (0xc96b bytes) 000553 1743203552.569170 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000554 1743203552.624576 write 0x01a1b8 thru 0x01db92 (0x39db bytes) 000556 1743203552.625703 mapwrite 0x00b296 thru 0x013ee8 (0x8c53 bytes) 000557 1743203552.649564 mapwrite 0x03ef8f thru 0x03ffff (0x1071 bytes) 000558 1743203552.669680 mapread 0x035219 thru 0x03ffff (0xade7 bytes) 000560 1743203552.672795 mapread 0x00d7bc thru 0x0115b0 (0x3df5 bytes) 000561 1743203552.673500 write 0x020ff7 thru 0x021e3d (0xe47 bytes) 000562 1743203552.676398 write 0x030691 thru 0x03c703 (0xc073 bytes) 000563 1743203552.677125 mapread 0x0088a1 thru 0x0150a9 (0xc809 bytes) 000564 1743203552.677567 mapwrite 0x02e53e thru 0x03535e (0x6e21 bytes) 000565 1743203552.692389 read 0x00f061 thru 0x014fc3 (0x5f63 bytes) 000566 1743203552.692794 mapread 0x013d5c thru 0x019c95 (0x5f3a bytes) 000567 1743203552.696027 MIRROR_RESYNC 8 mirrors 000568 1743203552.941979 write 0x01b0a4 thru 0x024e5d (0x9dba bytes) 000569 1743203552.960301 trunc from 0x040000 to 0x01d4ae (0x22b51 bytes) 000571 1743203552.970577 MIRROR_EXTEND to 9 mirrors 000572 1743203553.069973 mapwrite 0x017348 thru 0x01e9e6 (0x769f bytes) 000573 1743203553.103072 read 0x00bfba thru 0x018e3b (0xce82 bytes) 000574 1743203553.106694 MIRROR_EXTEND to 10 mirrors 000575 1743203553.203034 trunc from 0x01e9e7 to 0x012c1e (0xbdc8 bytes) 000581 1743203553.227382 MIRROR_EXTEND to 11 mirrors 000584 1743203553.330233 trunc from 0x012c1f to 0x03847b (0x2585d bytes) 000585 1743203553.352241 mapread 0x02d90c thru 0x02eef9 (0x15ee bytes) 000587 1743203553.357223 mapread 0x03645e thru 0x03847b (0x201e bytes) 000592 1743203553.362762 trunc from 0x03847c to 0x03f72f (0x72b4 bytes) 000593 1743203553.369790 mapread 0x013dba thru 0x0169ad (0x2bf4 bytes) 000595 1743203553.373311 trunc from 0x03f730 to 0x036f85 (0x87aa bytes) 000600 1743203553.378857 mapread 0x026265 thru 0x03163d (0xb3d9 bytes) 000601 1743203553.383889 mapread 0x002cef thru 0x00c1d4 (0x94e6 bytes) 000604 1743203553.389562 MIRROR_EXTEND to 12 mirrors 000605 1743203553.497162 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000610 1743203553.569688 mapwrite 0x0181cc thru 0x0212e0 (0x9115 bytes) 000612 1743203553.620529 write 0x014c0f thru 0x014e4e (0x240 bytes) 000613 1743203553.621118 read 0x036f7b thru 0x036f85 (0xb bytes) 000615 1743203553.623551 mapread 0x02c6a4 thru 0x0329cd (0x632a bytes) 000616 1743203553.626629 mapwrite 0x021e6b thru 0x02dcf1 (0xbe87 bytes) 000619 1743203553.649471 MIRROR_EXTEND to 12 mirrors 000620 1743203553.752283 trunc from 0x036f86 to 0x02df7a (0x900b bytes) 000621 1743203553.772722 mapwrite 0x03ff16 thru 0x03ffff (0xea bytes) 000623 1743203553.793284 mapread 0x0216e7 thru 0x02fed1 (0xe7eb bytes) 000624 1743203553.800565 MIRROR_EXTEND to 13 mirrors 000626 1743203553.898881 MIRROR_RESYNC 13 mirrors 000627 1743203554.209113 MIRROR_RESYNC 13 mirrors 000629 1743203554.284319 mapread 0x03dc55 thru 0x03ffff (0x23ab bytes) 000631 1743203554.287758 mapread 0x03b039 thru 0x03ffff (0x4fc7 bytes) 000632 1743203554.288869 MIRROR_SPLIT mirror 4 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000633 1743203554.375042 MIRROR_EXTEND to 13 mirrors 000634 1743203554.500755 write 0x005cd4 thru 0x006da5 (0x10d2 bytes) 000635 1743203554.519976 MIRROR_EXTEND to 14 mirrors 000638 1743203554.621940 mapread 0x00f79a thru 0x019bbf (0xa426 bytes) 000639 1743203554.632178 trunc from 0x040000 to 0x02c1a1 (0x13e5e bytes) 000642 1743203554.651418 read 0x0155f1 thru 0x016f0d (0x191d bytes) 000643 1743203554.653693 mapwrite 0x032a7d thru 0x032fd3 (0x557 bytes) 000644 1743203554.672482 read 0x030216 thru 0x032fd3 (0x2dbe bytes) 000646 1743203554.673277 mapread 0x00ab78 thru 0x01a0f8 (0xf581 bytes) 000654 1743203554.679769 MIRROR_EXTEND to 15 mirrors 000656 1743203554.784459 write 0x004d9f thru 0x005be8 (0xe4a bytes) 000657 1743203554.801808 MIRROR_SPLIT mirror 9 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000658 1743203554.859840 mapread 0x014073 thru 0x016a7e (0x2a0c bytes) 000661 1743203554.866364 read 0x000d08 thru 0x00ec31 (0xdf2a bytes) 000662 1743203554.871760 MIRROR_EXTEND to 15 mirrors 000663 1743203554.975137 trunc from 0x032fd4 to 0x02a5e9 (0x89ea bytes) 000666 1743203555.002233 mapwrite 0x004007 thru 0x005a82 (0x1a7c bytes) 000668 1743203555.020363 trunc from 0x02a5ea to 0x02bfde (0x19f5 bytes) 000671 1743203555.025658 write 0x033722 thru 0x03e53c (0xae1b bytes) 000677 1743203555.027675 mapread 0x020535 thru 0x02b300 (0xadcc bytes) 000678 1743203555.031575 mapread 0x010798 thru 0x0125bc (0x1e25 bytes) 000679 1743203555.033508 read 0x0225da thru 0x02d1cd (0xabf4 bytes) 000680 1743203555.033807 write 0x02b7ab thru 0x03794c (0xc1a2 bytes) 000685 1743203555.035431 read 0x037de1 thru 0x039734 (0x1954 bytes) 000686 1743203555.035758 mapread 0x00cb72 thru 0x015091 (0x8520 bytes) 000687 1743203555.036180 mapread 0x024a52 thru 0x02c0c8 (0x7677 bytes) 000689 1743203555.036777 write 0x00dce5 thru 0x01095a (0x2c76 bytes) 000692 1743203555.037775 mapread 0x01f25b thru 0x028899 (0x963f bytes) 000693 1743203555.039918 MIRROR_SPLIT mirror 3 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000694 1743203555.096628 MIRROR_EXTEND to 15 mirrors 000696 1743203555.223982 read 0x02e864 thru 0x030586 (0x1d23 bytes) 000697 1743203555.228125 mapwrite 0x02fd0b thru 0x03d682 (0xd978 bytes) 000698 1743203555.277142 read 0x02f481 thru 0x035f16 (0x6a96 bytes) 000700 1743203555.277926 read 0x01e745 thru 0x0290fc (0xa9b8 bytes) 000701 1743203555.281137 read 0x025bcb thru 0x02b297 (0x56cd bytes) 000702 1743203555.281567 write 0x026c0a thru 0x02cee8 (0x62df bytes) 000703 1743203555.282307 trunc from 0x03e53d to 0x00f185 (0x2f3b7 bytes) 000705 1743203555.287761 write 0x001aa8 thru 0x004252 (0x27ab bytes) 000706 1743203555.288442 trunc from 0x00f186 to 0x0219eb (0x12866 bytes) 000708 1743203555.293730 MIRROR_SPLIT mirror 12 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000709 1743203555.345306 mapread 0x00697f thru 0x0161f1 (0xf873 bytes) 000711 1743203555.354520 mapread 0x01505c thru 0x020f38 (0xbedd bytes) 000713 1743203555.358025 read 0x0036df thru 0x013165 (0xfa87 bytes) 000717 1743203555.359270 mapread 0x0062d2 thru 0x011aa7 (0xb7d6 bytes) 000720 1743203555.360355 write 0x0295ef thru 0x02e3a1 (0x4db3 bytes) 000723 1743203555.361609 trunc from 0x02e3a2 to 0x0080e8 (0x262b9 bytes) 000725 1743203555.365956 read 0x004d3e thru 0x007baf (0x2e72 bytes) 000728 1743203555.366925 mapwrite 0x006982 thru 0x015fa8 (0xf627 bytes) 000729 1743203555.388375 trunc from 0x015fa9 to 0x038e08 (0x22e60 bytes) 000732 1743203555.392678 trunc from 0x038e09 to 0x0159b8 (0x23450 bytes) 000733 1743203555.396728 read 0x00cdab thru 0x010757 (0x39ad bytes) 000735 1743203555.397482 trunc from 0x0159b9 to 0x02880d (0x12e55 bytes) 000736 1743203555.400771 trunc from 0x02880e to 0x00b06e (0x1d79f bytes) 000738 1743203555.404375 mapread 0x000575 thru 0x00b06e (0xaafa bytes) 000740 1743203555.405125 mapread 0x009dad thru 0x00a883 (0xad7 bytes) 000743 1743203555.406126 mapwrite 0x01dd10 thru 0x024307 (0x65f8 bytes) 000744 1743203555.423450 mapwrite 0x034007 thru 0x0350da (0x10d4 bytes) 000746 1743203555.440401 trunc from 0x0350db to 0x02e4bd (0x6c1d bytes) 000749 1743203555.446585 write 0x01dd83 thru 0x025b40 (0x7dbe bytes) 000750 1743203555.447264 read 0x02be3a thru 0x02e4bd (0x2684 bytes) 000752 1743203555.447668 mapread 0x00c8b3 thru 0x016c0c (0xa35a bytes) 000754 1743203555.449617 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000755 1743203555.504547 MIRROR_SPLIT mirror 9 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000756 1743203555.550308 mapwrite 0x005306 thru 0x00e978 (0x9673 bytes) 000758 1743203555.571282 mapread 0x00974a thru 0x015ca1 (0xc558 bytes) 000759 1743203555.576311 MIRROR_SPLIT mirror 2 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000760 1743203555.624113 MIRROR_EXTEND to 12 mirrors 000761 1743203555.729557 write 0x020c2a thru 0x02c4b0 (0xb887 bytes) 000762 1743203555.748460 MIRROR_SPLIT mirror 5 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000763 1743203555.794666 mapread 0x0232c6 thru 0x02e4bd (0xb1f8 bytes) 000767 1743203555.804271 write 0x028cca thru 0x02c22a (0x3561 bytes) 000768 1743203555.805835 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000771 1743203555.853737 write 0x03962e thru 0x03ffff (0x69d2 bytes) 000772 1743203555.854571 write 0x01f2e2 thru 0x02877f (0x949e bytes) 000776 1743203555.858951 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000777 1743203555.908546 trunc from 0x040000 to 0x02f08a (0x10f75 bytes) 000778 1743203555.915205 mapwrite 0x006c8a thru 0x00ed4e (0x80c5 bytes) 000779 1743203555.936793 trunc from 0x02f08b to 0x01bcd4 (0x133b6 bytes) 000780 1743203555.942185 mapwrite 0x0047d8 thru 0x013cf2 (0xf51b bytes) 000783 1743203555.961155 mapread 0x00e1fa thru 0x017f0d (0x9d14 bytes) 000785 1743203555.961723 mapread 0x007caa thru 0x008f64 (0x12bb bytes) 000786 1743203555.962010 mapwrite 0x007655 thru 0x00e766 (0x7112 bytes) 000789 1743203555.973587 read 0x00db51 thru 0x018ca4 (0xb154 bytes) 000792 1743203555.974585 write 0x00f106 thru 0x010371 (0x126c bytes) 000793 1743203555.975000 mapread 0x00db4e thru 0x014292 (0x6745 bytes) 000794 1743203555.976588 MIRROR_RESYNC 9 mirrors 000795 1743203556.207971 trunc from 0x01bcd5 to 0x00b003 (0x10cd1 bytes) 000796 1743203556.222869 trunc from 0x00b004 to 0x018117 (0xd114 bytes) 000802 1743203556.227788 read 0x006d22 thru 0x015314 (0xe5f3 bytes) 000805 1743203556.231454 write 0x01cc35 thru 0x025cef (0x90bb bytes) 000806 1743203556.232172 read 0x010079 thru 0x01c202 (0xc18a bytes) 000808 1743203556.234092 read 0x00c311 thru 0x00fea0 (0x3b90 bytes) 000810 1743203556.234589 trunc from 0x025cf0 to 0x018623 (0xd6cc bytes) 000811 1743203556.237796 write 0x010b4d thru 0x01f87d (0xed31 bytes) 000812 1743203556.239739 MIRROR_EXTEND to 10 mirrors 000813 1743203556.351160 mapread 0x01ad9d thru 0x01f87d (0x4ae1 bytes) 000817 1743203556.359664 read 0x01bed1 thru 0x01f87d (0x39ad bytes) 000818 1743203556.360658 mapwrite 0x02c989 thru 0x03110b (0x4783 bytes) 000819 1743203556.395580 read 0x006f38 thru 0x00a1ba (0x3283 bytes) 000820 1743203556.398506 trunc from 0x03110c to 0x010df3 (0x20318 bytes) 000821 1743203556.402653 read 0x0027d5 thru 0x003749 (0xf75 bytes) 000823 1743203556.403385 mapwrite 0x01f033 thru 0x01f5ba (0x588 bytes) 000824 1743203556.419604 read 0x001905 thru 0x004476 (0x2b72 bytes) 000825 1743203556.419876 write 0x034324 thru 0x03528f (0xf6c bytes) 000827 1743203556.420529 trunc from 0x035290 to 0x008baa (0x2c6e5 bytes) 000828 1743203556.424039 read 0x00719e thru 0x008baa (0x1a0d bytes) 000829 1743203556.424325 mapwrite 0x02e893 thru 0x03c518 (0xdc86 bytes) 000830 1743203556.445397 MIRROR_EXTEND to 11 mirrors 000831 1743203556.537370 write 0x00a460 thru 0x0146c1 (0xa262 bytes) 000832 1743203556.554747 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000833 1743203556.609445 mapwrite 0x022db2 thru 0x02eb7d (0xbdcc bytes) 000834 1743203556.637748 trunc from 0x03c519 to 0x00ec26 (0x2d8f2 bytes) 000835 1743203556.646478 trunc from 0x00ec27 to 0x030ce7 (0x220c1 bytes) 000838 1743203556.650493 mapread 0x020bc3 thru 0x02115b (0x599 bytes) 000841 1743203556.654865 write 0x03079a thru 0x03aff7 (0xa85e bytes) 000842 1743203556.658935 MIRROR_EXTEND to 11 mirrors 000843 1743203556.798331 read 0x008e12 thru 0x010c4f (0x7e3e bytes) 000844 1743203556.801217 mapread 0x0239a4 thru 0x026a31 (0x308e bytes) 000848 1743203556.809209 mapwrite 0x027beb thru 0x032ec4 (0xb2da bytes) 000849 1743203556.853237 mapread 0x00bec5 thru 0x018ffa (0xd136 bytes) 000853 1743203556.859001 trunc from 0x03aff8 to 0x0326ad (0x894a bytes) 000854 1743203556.867141 read 0x010efe thru 0x016c16 (0x5d19 bytes) 000856 1743203556.867677 mapwrite 0x03555d thru 0x03a6b5 (0x5159 bytes) 000857 1743203556.887863 MIRROR_RESYNC 11 mirrors 000858 1743203557.152096 MIRROR_EXTEND to 12 mirrors 000861 1743203557.271551 mapwrite 0x00c075 thru 0x00eb10 (0x2a9c bytes) 000862 1743203557.313892 write 0x001a2e thru 0x003b7d (0x2150 bytes) 000863 1743203557.315871 read 0x010d0b thru 0x017c3f (0x6f35 bytes) 000864 1743203557.317063 mapread 0x01a6e4 thru 0x01cff0 (0x290d bytes) 000866 1743203557.319396 trunc from 0x03a6b6 to 0x022fe0 (0x176d5 bytes) 000871 1743203557.324684 read 0x00865e thru 0x0165d5 (0xdf78 bytes) 000873 1743203557.326290 MIRROR_EXTEND to 13 mirrors 000874 1743203557.410034 trunc from 0x022fe1 to 0x015740 (0xd8a0 bytes) 000875 1743203557.428383 trunc from 0x015741 to 0x01b58b (0x5e4b bytes) 000876 1743203557.431784 mapwrite 0x033359 thru 0x03550e (0x21b6 bytes) 000877 1743203557.449290 read 0x0342ea thru 0x03550e (0x1225 bytes) 000878 1743203557.449702 read 0x02e667 thru 0x032108 (0x3aa2 bytes) 000882 1743203557.451931 MIRROR_SPLIT mirror 11 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000886 1743203557.494907 mapread 0x02ad58 thru 0x031459 (0x6702 bytes) 000890 1743203557.501768 MIRROR_RESYNC 12 mirrors 000891 1743203557.780702 mapwrite 0x0120e1 thru 0x01a681 (0x85a1 bytes) 000892 1743203557.825047 write 0x002e66 thru 0x0033e0 (0x57b bytes) 000893 1743203557.827288 trunc from 0x03550f to 0x025713 (0xfdfb bytes) 000895 1743203557.835642 write 0x02cf52 thru 0x0311be (0x426d bytes) 000896 1743203557.836169 write 0x00e602 thru 0x0105ed (0x1fec bytes) 000897 1743203557.837656 MIRROR_EXTEND to 13 mirrors 000898 1743203557.922547 mapread 0x0022a2 thru 0x004939 (0x2698 bytes) 000899 1743203557.927658 mapwrite 0x03707f thru 0x03adeb (0x3d6d bytes) 000901 1743203557.958244 mapread 0x03967d thru 0x03adeb (0x176f bytes) 000902 1743203557.958736 write 0x0032ca thru 0x005635 (0x236c bytes) 000904 1743203557.961229 trunc from 0x03adec to 0x00cbf8 (0x2e1f3 bytes) 000905 1743203557.965731 read 0x009e94 thru 0x00cbf8 (0x2d65 bytes) 000906 1743203557.966007 mapwrite 0x01a8c7 thru 0x021908 (0x7042 bytes) 000910 1743203557.984167 trunc from 0x021909 to 0x009bc3 (0x17d45 bytes) 000914 1743203557.987883 mapwrite 0x02c9e8 thru 0x035f30 (0x9549 bytes) 000915 1743203558.004733 mapread 0x0198da thru 0x01a646 (0xd6d bytes) 000916 1743203558.007126 write 0x01b233 thru 0x01e180 (0x2f4e bytes) 000921 1743203558.008798 write 0x036d53 thru 0x03d4e7 (0x6795 bytes) 000922 1743203558.010545 MIRROR_EXTEND to 14 mirrors 000923 1743203558.109113 write 0x031307 thru 0x034c82 (0x397c bytes) 000924 1743203558.124663 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000925 1743203558.166707 mapread 0x01e045 thru 0x029099 (0xb055 bytes) 000926 1743203558.173342 mapwrite 0x022461 thru 0x0295e3 (0x7183 bytes) 000927 1743203558.187794 mapread 0x029c49 thru 0x02bc07 (0x1fbf bytes) 000928 1743203558.189721 MIRROR_SPLIT mirror 11 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000929 1743203558.232128 trunc from 0x03d4e8 to 0x0236d6 (0x19e11 bytes) 000930 1743203558.239095 MIRROR_RESYNC 12 mirrors 000931 1743203558.494423 mapread 0x000746 thru 0x00fada (0xf395 bytes) 000933 1743203558.500000 trunc from 0x0236d7 to 0x00c813 (0x16ec3 bytes) 000935 1743203558.521911 mapwrite 0x03b650 thru 0x03ffff (0x49b0 bytes) 000937 1743203558.540064 mapread 0x0176b2 thru 0x02727a (0xfbc9 bytes) 000938 1743203558.545414 mapread 0x00497c thru 0x010bf9 (0xc27e bytes) 000939 1743203558.547786 trunc from 0x040000 to 0x006df0 (0x3920f bytes) 000941 1743203558.552018 write 0x0275cb thru 0x02b12f (0x3b65 bytes) 000942 1743203558.554017 MIRROR_RESYNC 12 mirrors 000945 1743203558.807119 write 0x02858c thru 0x037858 (0xf2cd bytes) 000947 1743203558.819586 mapwrite 0x0143d7 thru 0x015da7 (0x19d1 bytes) 000949 1743203558.834671 read 0x034241 thru 0x037858 (0x3618 bytes) 000952 1743203558.835374 write 0x0070e0 thru 0x008279 (0x119a bytes) 000954 1743203558.837753 trunc from 0x037859 to 0x012ac0 (0x24d98 bytes) 000955 1743203558.844943 mapread 0x00bb68 thru 0x012ac0 (0x6f59 bytes) 000956 1743203558.845598 trunc from 0x012ac1 to 0x0300f5 (0x1d635 bytes) 000957 1743203558.848473 mapwrite 0x02c5f3 thru 0x02cbfe (0x60c bytes) 000958 1743203558.861873 mapwrite 0x00a6d2 thru 0x00f494 (0x4dc3 bytes) 000959 1743203558.873186 read 0x00581a thru 0x0154d5 (0xfcbc bytes) 000962 1743203558.876489 mapwrite 0x031aa2 thru 0x03eccb (0xd22a bytes) 000963 1743203558.897752 trunc from 0x03eccc to 0x0303bd (0xe90e bytes) 000964 1743203558.901065 read 0x01e342 thru 0x01fa2c (0x16eb bytes) 000965 1743203558.901411 write 0x000a4b thru 0x00b802 (0xadb8 bytes) 000970 1743203558.903240 write 0x022628 thru 0x0322cb (0xfca4 bytes) 000971 1743203558.904117 mapwrite 0x011f5d thru 0x016651 (0x46f5 bytes) 000973 1743203558.919333 MIRROR_SPLIT mirror 4 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000975 1743203558.959049 read 0x0257a7 thru 0x031a40 (0xc29a bytes) 000976 1743203558.961099 mapwrite 0x0270be thru 0x029680 (0x25c3 bytes) 000978 1743203558.971311 trunc from 0x0322cc to 0x013163 (0x1f168 bytes) 000979 1743203558.974894 write 0x016cae thru 0x0186c0 (0x1a13 bytes) 000980 1743203558.976716 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x838:0x0]: Invalid argument (22) 000984 1743203559.024371 write 0x004cc8 thru 0x0090b6 (0x43ef bytes) 000985 1743203559.026871 mapread 0x0075ca thru 0x0096b8 (0x20ef bytes) 000986 1743203559.027307 mapread 0x00b7a9 thru 0x01041b (0x4c73 bytes) 000987 1743203559.029606 read 0x005246 thru 0x009b8f (0x494a bytes) 000988 1743203559.030166 write 0x013590 thru 0x01c9b7 (0x9428 bytes) 000989 1743203559.030774 trunc from 0x01c9b8 to 0x02fb25 (0x1316e bytes) 000990 1743203559.034244 write 0x03407b thru 0x03baab (0x7a31 bytes) 000992 1743203559.035207 trunc from 0x03baac to 0x035bc5 (0x5ee6 bytes) 000996 1743203559.038965 mapread 0x00c7fa thru 0x00d63e (0xe45 bytes) 000997 1743203559.039494 write 0x022f57 thru 0x028b89 (0x5c33 bytes) 000999 1743203559.042181 read 0x03218c thru 0x03575b (0x35d0 bytes) All operations completed A-OK! PASS 70a (14s) == sanity-flr test 71: check mirror extend parent fid ==== 19:12:40 (1743203560) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 200a: stress test ===================== 19:12:40 (1743203560) Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre2 Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5804896 .. Extending file size to 3382624 .. Extending file size to 5855072 .. Extending file size to 9170496 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7021632 .. Extending file size to 8649376 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8728256 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8835680 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8928448 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9165216 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9249664 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 29570 29571 29573 29575 29577 192.168.202.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg216-client.virtnet /mnt/lustre2 (opts:) 192.168.202.116@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg216-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (63s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 19:13:43 (1743203623) Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre2 Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3879680 .. Extending file size to 3174464 .. done Extending file size to 6016224 .. Extending file size to 7928768 .. Extending file size to 7189920 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7383680 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7942880 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8344192 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8694176 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8012672 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8444256 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8895744 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9045408 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8643968 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9293472 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9236864 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 1056 1057 1059 1061 1063 done 192.168.202.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg216-client.virtnet /mnt/lustre2 (opts:) 192.168.202.116@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg216-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 19:14:47 (1743203687) Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 5376 2202880 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 5376 2202880 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0293387 s, 71.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0853988 s, 24.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg216-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 19:14:51 (1743203691) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0248708 s, 84.3 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 19:14:53 (1743203693) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x88d:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb92:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb92:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb93:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0xb8d:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 19:14:55 (1743203695) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0264708 s, 79.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb94:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb8e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb8f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 19:14:57 (1743203697) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0278474 s, 75.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb95:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb96:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb90:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb95:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb96:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb90:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb91:0x0] } PASS 204b (1s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 19:14:58 (1743203698) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0275878 s, 76.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb97:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb92:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb97:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb92:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb93:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 19:15:21 (1743203721) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg216-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.01786 s, 58.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb98:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb95:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb94:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb98:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb95:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb94:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb99:0x0] } PASS 204d (41s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 19:16:02 (1743203762) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg216-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb96:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb97:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0271556 s, 77.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb96:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb97:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb9a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb96:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb97:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb9a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (43s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 19:16:45 (1743203805) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg216-server mds-ost sync done. Creating new pool test_204f oleg216-server: Pool lustre.test_204f created Adding targets to pool oleg216-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg216-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0248858 s, 84.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb98:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb9b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb98:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb9b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg216-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg216-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg216-server: Pool lustre.test_204f destroyed PASS 204f (52s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 19:17:37 (1743203857) PASS 205a (1s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 19:17:38 (1743203858) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 19:17:40 (1743203860) oleg216-server: Pool lustre.test_206 created Adding targets to pool oleg216-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg216-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg216-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg216-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg216-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (14s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 19:17:54 (1743203874) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0209508 s, 50.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00971911 s, 108 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x7641824 Mirror 3: 0x7641824 PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 19:17:56 (1743203876) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 19:17:57 (1743203877) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (0s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 19:17:57 (1743203877) Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.017208 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x18e0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1740000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1740000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1740000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1700000 Read this (4 bytes): BBBB 192.168.202.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg216-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 19:18:10 (1743203890) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00280887 s, 2.9 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 19:18:12 (1743203892) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00949911 s, 110 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x8b2 lmm_fid: [0x200000404:0x8b2:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x8b2 lmm_fid: [0x200000404:0x8b2:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xba6:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322836146 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-03-28 19:18:12.000000000 -0400 Modify: 2025-03-28 19:18:12.000000000 -0400 Change: 2025-03-28 19:18:13.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 19:18:14 (1743203894) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (0s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 19:18:15 (1743203895) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0970957 s, 422 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0137869 s, 297 kB/s size after second write -rw-r--r-- 1 root root 4096 Mar 28 19:18 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x8b4:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Mar 28 19:18 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 698 sec ============ 19:18:16 (1743203896) === sanity-flr: start cleanup 19:18:17 (1743203897) === === sanity-flr: finish cleanup 19:18:18 (1743203898) ===