-----============= acceptance-small: sanity-flr ============----- Tue Jun 10 21:31:11 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg243-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 21:31:16 (1749605476) === oleg243-client.virtnet: executing check_config_client /mnt/lustre oleg243-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg243-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012aaf5000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012aaf5000.idle_timeout=debug disable quota as required oleg243-server: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 21:31:23 (1749605483) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7264] == sanity-flr test 0a: lfs mirror create with -N option == 21:31:24 (1749605484) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 21:31:27 (1749605487) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 21:31:28 (1749605488) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (0s) == sanity-flr test 0d: lfs mirror extend with -N option == 21:31:28 (1749605488) PASS 0d (5s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 21:31:33 (1749605493) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (0s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 21:31:33 (1749605493) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 21:31:34 (1749605494) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88012aaf5000.stats=clear osc.lustre-OST0001-osc-ffff88012aaf5000.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 21:31:36 (1749605496) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 21:31:38 (1749605498) PASS 0j (1s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 21:31:39 (1749605499) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 21:31:40 (1749605500) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 21:31:42 (1749605502) PASS 2 (1s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 21:31:43 (1749605503) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 21:31:44 (1749605504) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (1s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 21:31:45 (1749605505) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 21:31:48 (1749605508) PASS 7 (1s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 21:31:49 (1749605509) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0606269 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0321368 s, 32.6 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 21:31:51 (1749605511) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0136027 s, 154 MB/s PASS 22 (1s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 21:31:52 (1749605512) Stopping /mnt/lustre-ost1 (opts:) on oleg243-server oleg243-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012aaf5000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012aaf5000.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg243-server: oleg243-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg243-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg243-server oleg243-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012aaf5000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012aaf5000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg243-server: oleg243-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg243-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (20s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 21:32:12 (1749605532) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.0800761 s, 65.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg243-server oleg243-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012aaf5000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012aaf5000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg243-server: oleg243-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg243-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (9s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 21:32:21 (1749605541) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg243-server oleg243-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012aaf5000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012aaf5000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg243-server: oleg243-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg243-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg243-server oleg243-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012aaf5000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012aaf5000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg243-server: oleg243-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg243-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (16s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 21:32:37 (1749605557) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.71133 s, 73.7 MB/s -rw-r--r-- 1 root root 52428800 Jun 10 21:32 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.214s user 0m0.001s sys 0m0.115s Stopping /mnt/lustre-ost1 (opts:) on oleg243-server oleg243-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012aaf5000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012aaf5000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.200s user 0m0.002s sys 0m0.106s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg243-server: oleg243-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg243-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg243-server oleg243-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012aaf5000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012aaf5000.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost1)... real 0m0.222s user 0m0.001s sys 0m0.107s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg243-server: oleg243-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg243-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg243-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012aaf5000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012aaf5000.ost_server_uuid in FULL state after 0 sec PASS 33b (21s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 21:32:58 (1749605578) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (0s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 21:32:58 (1749605578) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 21:32:59 (1749605579) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 21:33:00 (1749605580) PASS 35 (1s) == sanity-flr test 36a: write to mirrored files ========== 21:33:01 (1749605581) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.116319 s, 108 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 21:33:04 (1749605584) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00728332 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00752071 s, 139 MB/s PASS 36b (1s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 21:33:05 (1749605585) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 21:33:08 (1749605588) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000526707 s, 1.9 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000548756 s, 1.9 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000693747 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000573352 s, 1.8 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 21:33:10 (1749605590) create test files with size 15732223 .. e87e1e1143e7ad58586f096f6b761b56 - d7ef3a58d6d8ea236cb407f05c3b086a - 5d9808c1fdd280b50dfed559d98a2347 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg243-client.virtnet /mnt/lustre (opts:) Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre PASS 37 (4s) == sanity-flr test 38: resync ============================ 21:33:14 (1749605594) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 21:33:17 (1749605597) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 21:33:19 (1749605599) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0211431 s, 49.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00719081 s, 146 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (1s) == sanity-flr test 41: lfs mirror resync check =========== 21:33:20 (1749605600) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0282452 s, 74.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0907192 s, 46.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0349997 s, 59.9 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 21:33:23 (1749605603) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 21:33:24 (1749605604) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (0s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 21:33:24 (1749605604) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00753087 s, 139 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 21:33:26 (1749605606) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 21:33:27 (1749605607) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (1s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 21:33:29 (1749605609) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0610389 s, 172 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg243-client.virtnet /mnt/lustre (opts:) Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (24s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 21:33:53 (1749605633) llite.lustre-ffff8800aaadd000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0661964 s, 158 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg243-client.virtnet /mnt/lustre (opts:) Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff88012aad8800.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 21:33:56 (1749605636) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (6s) == sanity-flr test 46: Verify setstripe --copy option ==== 21:34:02 (1749605642) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (6s) == sanity-flr test 47: Verify mirror obj alloc =========== 21:34:08 (1749605648) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 21:34:09 (1749605649) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0587454 s, 53.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0428824 s, 73.4 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: b95aa23dcc1c9431301c21e3c36d132a - mirror 1 checksum: d1d49b874f1c1f3e8d78a5ee967f4a16 - mirror 2 checksum: b95aa23dcc1c9431301c21e3c36d132a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Jun 10 21:34:10 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Jun 10 21:34:10 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: b95aa23dcc1c9431301c21e3c36d132a - mirror 1 checksum: d1d49b874f1c1f3e8d78a5ee967f4a16 - mirror 2 checksum: d1d49b874f1c1f3e8d78a5ee967f4a16 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Jun 10 21:34:10 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Jun 10 21:34:10 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 48 (2s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 21:34:11 (1749605651) Failing mds1 on oleg243-server Stopping /mnt/lustre-mds1 (opts:) on oleg243-server 21:34:13 (1749605653) shut down facet: mds1 facet_host: oleg243-server facet_failover_host: oleg243-server Failover mds1 to oleg243-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 21:34:25 (1749605665) targets are mounted 21:34:25 (1749605665) facet_failover done oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00710744 s, 148 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b:0x0] } PASS 50A (19s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 21:34:30 (1749605670) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 21:34:31 (1749605671) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 21:34:32 (1749605672) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00228932 s, 7.2 MB/s PASS 50c (1s) == sanity-flr test 50d: mirror rsync keep holes ========== 21:34:33 (1749605673) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 21:34:34 (1749605674) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 21:34:35 (1749605675) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (3s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 21:34:38 (1749605678) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 21:34:56 (1749605696) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (16s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 21:35:13 (1749605713) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (16s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 21:35:29 (1749605729) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 21:35:31 (1749605731) mirror create pid 22783 mirror split pid 22784 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4948: 22783 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4948: 22784 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (61s) == sanity-flr test 70a: flr mode fsx test ================ 21:36:32 (1749605792) Seed set to 5793 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x2d7f6 000001 1749605793.160825 trunc from 00000000 to 0x02d7f5 (0x2d7f6 bytes) 000004 1749605793.164471 mapread 0x02d3ab thru 0x02d7f5 (0x44b bytes) 000005 1749605793.166164 trunc from 0x02d7f6 to 0x0263dc (0x7419 bytes) 000008 1749605793.170888 MIRROR_EXTEND to 2 mirrors 000009 1749605793.261450 mapwrite 0x01327f thru 0x01bec0 (0x8c42 bytes) 000010 1749605793.303310 trunc from 0x0263dd to 0x01eb8e (0x784e bytes) 000011 1749605793.311189 mapread 0x012575 thru 0x0134b4 (0xf40 bytes) 000012 1749605793.313578 trunc from 0x01eb8f to 0x011828 (0xd366 bytes) 000014 1749605793.317393 read 0x00f304 thru 0x011828 (0x2525 bytes) 000015 1749605793.317784 write 0x02dba4 thru 0x035601 (0x7a5e bytes) 000016 1749605793.318541 mapread 0x02fe30 thru 0x035601 (0x57d2 bytes) 000018 1749605793.320337 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) truncating to largest ever: 0x39002 000020 1749605793.353609 trunc from 0x035602 to 0x039001 (0x3a00 bytes) 000024 1749605793.358488 MIRROR_EXTEND to 2 mirrors 000025 1749605793.461468 write 0x03b2ac thru 0x03c50d (0x1262 bytes) 000026 1749605793.474395 trunc from 0x03c50e to 0x033eba (0x8653 bytes) 000027 1749605793.482191 mapwrite 0x0284f9 thru 0x033c7d (0xb785 bytes) 000028 1749605793.502761 read 0x009ea0 thru 0x00b3bf (0x1520 bytes) 000031 1749605793.505156 read 0x001b03 thru 0x0046ca (0x2bc8 bytes) 000032 1749605793.508147 trunc from 0x033ebb to 0x0380a7 (0x41ed bytes) 000034 1749605793.512053 mapwrite 0x019ebd thru 0x01cbf9 (0x2d3d bytes) 000035 1749605793.525716 mapwrite 0x014021 thru 0x0159fb (0x19db bytes) 000037 1749605793.536690 write 0x033ede thru 0x038692 (0x47b5 bytes) 000038 1749605793.538763 mapread 0x00c03f thru 0x010196 (0x4158 bytes) 000039 1749605793.539422 read 0x033aa8 thru 0x038692 (0x4beb bytes) 000040 1749605793.539782 write 0x035fb2 thru 0x03f050 (0x909f bytes) 000042 1749605793.540668 mapwrite 0x038e8d thru 0x03ffff (0x7173 bytes) 000043 1749605793.557486 read 0x00b408 thru 0x00db4e (0x2747 bytes) 000044 1749605793.557831 trunc from 0x040000 to 0x01bd4c (0x242b3 bytes) 000045 1749605793.561494 read 0x015c73 thru 0x019ce2 (0x4070 bytes) 000046 1749605793.561885 write 0x00c8ad thru 0x014605 (0x7d59 bytes) 000047 1749605793.562573 trunc from 0x01bd4d to 0x026310 (0xa5c4 bytes) 000048 1749605793.567686 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000049 1749605793.602494 write 0x0156c4 thru 0x017c37 (0x2574 bytes) 000050 1749605793.605186 write 0x03d863 thru 0x03ffff (0x279d bytes) 000053 1749605793.606955 trunc from 0x040000 to 0x033e3a (0xc1c5 bytes) 000056 1749605793.611332 write 0x008344 thru 0x014708 (0xc3c5 bytes) 000057 1749605793.615150 read 0x00ce87 thru 0x0122cf (0x5449 bytes) 000059 1749605793.616140 mapread 0x010e61 thru 0x01a384 (0x9524 bytes) 000060 1749605793.618536 write 0x017f5d thru 0x01f89e (0x7942 bytes) 000062 1749605793.619286 read 0x02671d thru 0x02faa0 (0x9384 bytes) 000066 1749605793.624329 MIRROR_EXTEND to 2 mirrors 000067 1749605793.723937 read 0x00413a thru 0x007907 (0x37ce bytes) 000068 1749605793.725573 trunc from 0x033e3b to 0x024f04 (0xef36 bytes) 000069 1749605793.744627 mapwrite 0x030c14 thru 0x03aad9 (0x9ec6 bytes) 000070 1749605793.768218 read 0x016d5f thru 0x02484d (0xdaef bytes) 000072 1749605793.772706 write 0x0161d1 thru 0x021aa5 (0xb8d5 bytes) 000074 1749605793.775505 MIRROR_RESYNC 2 mirrors 000075 1749605793.867985 trunc from 0x03aada to 0x022eb1 (0x17c28 bytes) 000076 1749605793.884152 mapread 0x002358 thru 0x00deba (0xbb63 bytes) 000077 1749605793.889491 trunc from 0x022eb2 to 0x0120aa (0x10e07 bytes) 000079 1749605793.893558 read 0x0082b4 thru 0x0120aa (0x9df7 bytes) 000080 1749605793.895117 MIRROR_EXTEND to 3 mirrors 000082 1749605793.985361 trunc from 0x0120ab to 0x0380f5 (0x2604b bytes) 000083 1749605794.005147 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000084 1749605794.051612 mapread 0x027d59 thru 0x02b97e (0x3c26 bytes) 000085 1749605794.056059 trunc from 0x0380f6 to 0x038886 (0x791 bytes) 000087 1749605794.059755 trunc from 0x038887 to 0x0217ed (0x17099 bytes) 000088 1749605794.063039 read 0x017163 thru 0x021345 (0xa1e3 bytes) truncating to largest ever: 0x3d08e 000089 1749605794.065392 trunc from 0x0217ee to 0x03d08d (0x1b8a0 bytes) 000091 1749605794.069403 read 0x037b6d thru 0x03d08d (0x5521 bytes) 000093 1749605794.074182 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000094 1749605794.103893 mapread 0x0041ba thru 0x0100f6 (0xbf3d bytes) 000095 1749605794.110181 read 0x003560 thru 0x003797 (0x238 bytes) 000097 1749605794.110739 mapwrite 0x03e842 thru 0x03ffff (0x17be bytes) 000101 1749605794.128526 write 0x03ddd4 thru 0x03ffff (0x222c bytes) 000103 1749605794.130325 MIRROR_EXTEND to 2 mirrors 000105 1749605794.220842 mapread 0x0081e3 thru 0x010ce8 (0x8b06 bytes) 000107 1749605794.225078 trunc from 0x040000 to 0x039ef8 (0x6107 bytes) 000110 1749605794.243342 trunc from 0x039ef9 to 0x00aa1e (0x2f4da bytes) 000111 1749605794.247607 mapread 0x001e29 thru 0x001e49 (0x21 bytes) 000112 1749605794.250141 mapwrite 0x03088c thru 0x031afb (0x1270 bytes) 000114 1749605794.267931 trunc from 0x031afc to 0x008db2 (0x28d49 bytes) 000117 1749605794.272775 mapwrite 0x03c712 thru 0x03feb2 (0x37a1 bytes) 000119 1749605794.291083 read 0x01dcbe thru 0x027f85 (0xa2c8 bytes) 000120 1749605794.293750 mapread 0x0358c0 thru 0x03feb2 (0xa5f3 bytes) 000122 1749605794.295884 MIRROR_RESYNC 2 mirrors 000123 1749605794.382745 mapread 0x03f0b4 thru 0x03feb2 (0xdff bytes) 000124 1749605794.383116 write 0x03c35c thru 0x03ffff (0x3ca4 bytes) 000125 1749605794.396282 read 0x00cd5f thru 0x01950d (0xc7af bytes) 000126 1749605794.398670 trunc from 0x040000 to 0x00846b (0x37b94 bytes) 000127 1749605794.407099 trunc from 0x00846c to 0x039ada (0x3166f bytes) 000128 1749605794.410438 read 0x02d8ea thru 0x02ec23 (0x133a bytes) 000129 1749605794.414002 mapwrite 0x011ea9 thru 0x01708b (0x51e3 bytes) 000130 1749605794.427457 mapwrite 0x022038 thru 0x027a5b (0x5a24 bytes) 000131 1749605794.440602 mapread 0x010856 thru 0x013730 (0x2edb bytes) 000133 1749605794.442788 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000134 1749605794.471962 read 0x0050d3 thru 0x008851 (0x377f bytes) 000136 1749605794.473842 mapwrite 0x0280a4 thru 0x035500 (0xd45d bytes) 000137 1749605794.493270 read 0x00fff2 thru 0x012e63 (0x2e72 bytes) 000138 1749605794.496208 mapread 0x0100fc thru 0x019e39 (0x9d3e bytes) 000139 1749605794.496753 mapwrite 0x03769c thru 0x039739 (0x209e bytes) 000141 1749605794.508521 trunc from 0x039adb to 0x011a6d (0x2806d bytes) 000142 1749605794.512753 mapwrite 0x02afcb thru 0x02c26c (0x12a2 bytes) 000146 1749605794.533366 write 0x038c15 thru 0x03ffff (0x73eb bytes) 000147 1749605794.534151 write 0x017b53 thru 0x026248 (0xe6f6 bytes) 000154 1749605794.539235 write 0x0063e7 thru 0x0114dd (0xb0f7 bytes) 000156 1749605794.540273 read 0x002fa6 thru 0x00f9f4 (0xca4f bytes) 000158 1749605794.541160 read 0x003160 thru 0x008908 (0x57a9 bytes) 000159 1749605794.541517 trunc from 0x040000 to 0x02e267 (0x11d98 bytes) 000165 1749605794.546486 mapread 0x005b25 thru 0x01110e (0xb5ea bytes) 000166 1749605794.547032 mapread 0x00c97f thru 0x014df3 (0x8475 bytes) 000167 1749605794.549775 mapwrite 0x0319c3 thru 0x03ec12 (0xd250 bytes) 000169 1749605794.571636 read 0x011540 thru 0x0147a5 (0x3266 bytes) 000170 1749605794.572134 mapread 0x01a09e thru 0x0220a0 (0x8003 bytes) 000172 1749605794.573199 mapwrite 0x03045d thru 0x0374d8 (0x707c bytes) 000174 1749605794.587625 MIRROR_EXTEND to 2 mirrors 000177 1749605794.675518 mapwrite 0x0035b4 thru 0x006f22 (0x396f bytes) 000178 1749605794.717778 MIRROR_RESYNC 2 mirrors 000179 1749605794.801959 write 0x01753d thru 0x01e074 (0x6b38 bytes) 000180 1749605794.816070 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000182 1749605794.846744 trunc from 0x03ec13 to 0x030d93 (0xde7f bytes) 000184 1749605794.853880 write 0x0229f5 thru 0x02965f (0x6c6b bytes) 000186 1749605794.857044 write 0x0369e9 thru 0x03757e (0xb96 bytes) 000187 1749605794.859852 mapwrite 0x034a5f thru 0x03b836 (0x6dd8 bytes) 000195 1749605794.880205 mapread 0x03630f thru 0x03b836 (0x5528 bytes) 000196 1749605794.880796 mapwrite 0x032a2a thru 0x039103 (0x66da bytes) 000197 1749605794.892337 trunc from 0x03b837 to 0x00bbc5 (0x2fc71 bytes) 000198 1749605794.896602 mapwrite 0x00cf41 thru 0x017902 (0xa9c2 bytes) 000200 1749605794.919043 trunc from 0x017903 to 0x0145d8 (0x332a bytes) 000201 1749605794.922468 trunc from 0x0145d9 to 0x0247b1 (0x101d9 bytes) 000202 1749605794.925843 trunc from 0x0247b2 to 0x02c25f (0x7aae bytes) 000203 1749605794.928941 mapread 0x005d1b thru 0x00ab5e (0x4e44 bytes) 000206 1749605794.930949 MIRROR_EXTEND to 2 mirrors 000207 1749605795.013604 mapread 0x00c060 thru 0x0104f9 (0x449a bytes) 000208 1749605795.018207 write 0x02d633 thru 0x03092a (0x32f8 bytes) 000209 1749605795.031659 read 0x00a845 thru 0x0133da (0x8b96 bytes) 000210 1749605795.033762 mapread 0x00f71d thru 0x01306a (0x394e bytes) 000211 1749605795.034245 read 0x014b0a thru 0x01e514 (0x9a0b bytes) 000212 1749605795.037125 read 0x0079a0 thru 0x00cdd0 (0x5431 bytes) 000213 1749605795.037441 mapread 0x029655 thru 0x03092a (0x72d6 bytes) 000215 1749605795.038058 write 0x03f424 thru 0x03ffff (0xbdc bytes) 000217 1749605795.038726 trunc from 0x040000 to 0x023f60 (0x1c09f bytes) 000218 1749605795.046288 mapread 0x01ac56 thru 0x023f60 (0x930b bytes) 000219 1749605795.046857 mapwrite 0x0007d1 thru 0x00a675 (0x9ea5 bytes) 000223 1749605795.060923 mapwrite 0x014918 thru 0x01efcb (0xa6b4 bytes) 000224 1749605795.074503 read 0x00d602 thru 0x0174b4 (0x9eb3 bytes) 000226 1749605795.076306 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000228 1749605795.105097 mapread 0x02325c thru 0x023f60 (0xd05 bytes) 000229 1749605795.107180 mapwrite 0x01214b thru 0x01ccba (0xab70 bytes) 000230 1749605795.122997 mapread 0x004ba2 thru 0x00f632 (0xaa91 bytes) 000231 1749605795.125604 mapread 0x01c18b thru 0x023f60 (0x7dd6 bytes) 000232 1749605795.126011 trunc from 0x023f61 to 0x02de5c (0x9efc bytes) 000233 1749605795.129311 mapwrite 0x008c1b thru 0x0144e6 (0xb8cc bytes) 000236 1749605795.141158 trunc from 0x02de5d to 0x031acb (0x3c6f bytes) 000240 1749605795.145823 write 0x031615 thru 0x03ca81 (0xb46d bytes) 000241 1749605795.148403 trunc from 0x03ca82 to 0x0156f3 (0x2738e bytes) 000242 1749605795.152142 write 0x009d96 thru 0x0153e5 (0xb650 bytes) 000244 1749605795.153121 mapwrite 0x038a35 thru 0x03ffff (0x75cb bytes) 000245 1749605795.172241 write 0x027077 thru 0x032952 (0xb8dc bytes) 000246 1749605795.174827 read 0x027745 thru 0x035287 (0xdb43 bytes) 000248 1749605795.175334 read 0x0096c3 thru 0x01494f (0xb28d bytes) 000250 1749605795.175996 mapwrite 0x01624b thru 0x016f5f (0xd15 bytes) 000255 1749605795.190121 read 0x0240db thru 0x033fae (0xfed4 bytes) 000259 1749605795.191551 trunc from 0x040000 to 0x00e1a0 (0x31e5f bytes) 000262 1749605795.196801 read 0x001d3f thru 0x00e1a0 (0xc462 bytes) 000263 1749605795.197264 trunc from 0x00e1a1 to 0x0326c2 (0x24522 bytes) 000264 1749605795.200641 read 0x02e8d5 thru 0x02fa7e (0x11aa bytes) 000265 1749605795.203832 mapwrite 0x00e880 thru 0x01720d (0x898e bytes) 000266 1749605795.216697 trunc from 0x0326c3 to 0x01ccb7 (0x15a0b bytes) 000268 1749605795.220485 read 0x007ebf thru 0x00e3a6 (0x64e8 bytes) 000269 1749605795.220792 read 0x01b43f thru 0x01ccb7 (0x1879 bytes) 000270 1749605795.221108 read 0x0046a4 thru 0x010c08 (0xc565 bytes) 000273 1749605795.221819 read 0x0173a3 thru 0x01ccb7 (0x5915 bytes) 000274 1749605795.222174 mapread 0x010ef3 thru 0x0113f6 (0x504 bytes) 000275 1749605795.222677 read 0x0011cf thru 0x007542 (0x6374 bytes) 000276 1749605795.222983 trunc from 0x01ccb8 to 0x004c6d (0x1804a bytes) 000280 1749605795.227261 mapwrite 0x00b154 thru 0x013702 (0x85af bytes) 000283 1749605795.245982 read 0x004f62 thru 0x00879c (0x383b bytes) 000284 1749605795.247451 MIRROR_EXTEND to 2 mirrors 000285 1749605795.329277 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000290 1749605795.358619 MIRROR_EXTEND to 2 mirrors 000293 1749605795.441364 MIRROR_EXTEND to 3 mirrors 000294 1749605795.525144 write 0x013cc8 thru 0x01da4f (0x9d88 bytes) 000296 1749605795.539921 MIRROR_RESYNC 3 mirrors 000298 1749605795.637125 mapwrite 0x01550a thru 0x0195e8 (0x40df bytes) 000300 1749605795.676966 read 0x01bd4c thru 0x01da4f (0x1d04 bytes) 000302 1749605795.677512 mapread 0x01b775 thru 0x01da4f (0x22db bytes) 000303 1749605795.677901 mapwrite 0x023ee5 thru 0x02b7e2 (0x78fe bytes) 000304 1749605795.700676 mapwrite 0x023372 thru 0x0313bb (0xe04a bytes) 000305 1749605795.718866 mapwrite 0x015655 thru 0x01e70e (0x90ba bytes) 000306 1749605795.731820 mapread 0x003946 thru 0x00463e (0xcf9 bytes) 000307 1749605795.734134 read 0x02557c thru 0x03097b (0xb400 bytes) 000308 1749605795.734485 mapwrite 0x031dbd thru 0x03cda5 (0xafe9 bytes) 000309 1749605795.754492 MIRROR_EXTEND to 4 mirrors 000310 1749605795.839075 write 0x02b0d3 thru 0x036e0a (0xbd38 bytes) 000311 1749605795.854388 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000312 1749605795.899580 write 0x016d12 thru 0x01ac11 (0x3f00 bytes) 000313 1749605795.902078 trunc from 0x03cda6 to 0x03b0d7 (0x1cce bytes) 000314 1749605795.908898 read 0x022a3e thru 0x0320f5 (0xf6b8 bytes) 000315 1749605795.911066 mapread 0x0142d3 thru 0x01a79c (0x64ca bytes) 000316 1749605795.913366 write 0x00826d thru 0x00833e (0xd2 bytes) 000321 1749605795.916986 mapread 0x02aebe thru 0x0343f3 (0x9536 bytes) 000325 1749605795.920568 write 0x0067b6 thru 0x0075e5 (0xe30 bytes) 000326 1749605795.921262 mapread 0x033c57 thru 0x038c33 (0x4fdd bytes) 000327 1749605795.921946 trunc from 0x03b0d8 to 0x03a637 (0xaa0 bytes) 000329 1749605795.925802 write 0x019e42 thru 0x021588 (0x7747 bytes) 000331 1749605795.927745 write 0x01016c thru 0x017e38 (0x7ccd bytes) 000332 1749605795.929544 MIRROR_EXTEND to 4 mirrors 000333 1749605796.033599 mapwrite 0x0168c1 thru 0x021a0e (0xb14e bytes) 000334 1749605796.078236 mapwrite 0x0038c1 thru 0x006537 (0x2c77 bytes) 000335 1749605796.092223 mapread 0x02cc30 thru 0x03a637 (0xda08 bytes) 000336 1749605796.096619 read 0x0300c1 thru 0x0357ad (0x56ed bytes) 000339 1749605796.097560 trunc from 0x03a638 to 0x032e9d (0x779a bytes) 000340 1749605796.106752 MIRROR_EXTEND to 5 mirrors 000341 1749605796.207255 MIRROR_RESYNC 5 mirrors 000342 1749605796.341901 read 0x01996f thru 0x0204e0 (0x6b72 bytes) 000343 1749605796.343927 read 0x02af28 thru 0x032e9d (0x7f76 bytes) 000345 1749605796.346676 trunc from 0x032e9e to 0x01f8a5 (0x135f8 bytes) 000346 1749605796.360385 trunc from 0x01f8a6 to 0x0296af (0x9e0a bytes) 000347 1749605796.364140 mapread 0x00b5ba thru 0x00e335 (0x2d7c bytes) 000349 1749605796.368882 write 0x014d81 thru 0x018e2f (0x40af bytes) 000350 1749605796.371139 trunc from 0x0296b0 to 0x010306 (0x193a9 bytes) 000352 1749605796.374884 mapwrite 0x03944a thru 0x03ffff (0x6bb6 bytes) 000353 1749605796.392190 mapread 0x033785 thru 0x03b7b6 (0x8032 bytes) 000354 1749605796.392811 write 0x03620c thru 0x03bf4d (0x5d42 bytes) 000356 1749605796.393588 trunc from 0x040000 to 0x026075 (0x19f8a bytes) 000357 1749605796.397051 mapread 0x01287f thru 0x01f787 (0xcf09 bytes) 000363 1749605796.412227 mapread 0x00bd3f thru 0x015fef (0xa2b1 bytes) 000367 1749605796.416255 mapwrite 0x01cfe6 thru 0x021cf0 (0x4d0b bytes) 000371 1749605796.431930 write 0x007f4e thru 0x00a942 (0x29f5 bytes) 000372 1749605796.434551 trunc from 0x026076 to 0x026800 (0x78b bytes) 000373 1749605796.438299 read 0x00a1dc thru 0x015eba (0xbcdf bytes) 000374 1749605796.439805 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000375 1749605796.486815 trunc from 0x026801 to 0x033da1 (0xd5a1 bytes) 000380 1749605796.491727 trunc from 0x033da2 to 0x006fcc (0x2cdd5 bytes) 000382 1749605796.495941 trunc from 0x006fcd to 0x03a750 (0x33784 bytes) 000385 1749605796.500143 write 0x0388b5 thru 0x03eff6 (0x6742 bytes) 000386 1749605796.502171 mapread 0x032486 thru 0x039f60 (0x7adb bytes) 000387 1749605796.505539 write 0x007876 thru 0x014f5d (0xd6e8 bytes) 000390 1749605796.509892 read 0x0128db thru 0x01ac23 (0x8349 bytes) 000392 1749605796.512461 read 0x0331e9 thru 0x034fb3 (0x1dcb bytes) 000394 1749605796.513041 trunc from 0x03eff7 to 0x032b17 (0xc4df bytes) 000399 1749605796.517838 read 0x01c96d thru 0x024f73 (0x8607 bytes) 000402 1749605796.522261 MIRROR_EXTEND to 5 mirrors 000405 1749605796.626876 mapread 0x007f79 thru 0x010e48 (0x8ed0 bytes) 000406 1749605796.633764 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000414 1749605796.705430 MIRROR_EXTEND to 5 mirrors 000416 1749605796.805884 write 0x039f46 thru 0x03e833 (0x48ee bytes) 000417 1749605796.819828 write 0x035308 thru 0x036e71 (0x1b6a bytes) 000418 1749605796.822503 read 0x0233fa thru 0x0236dd (0x2e4 bytes) 000419 1749605796.824102 read 0x03e449 thru 0x03e833 (0x3eb bytes) 000420 1749605796.824378 read 0x0326e6 thru 0x034f14 (0x282f bytes) 000427 1749605796.829968 mapwrite 0x0208d0 thru 0x0240b6 (0x37e7 bytes) 000428 1749605796.844142 read 0x00263a thru 0x00553a (0x2f01 bytes) 000429 1749605796.846254 MIRROR_EXTEND to 6 mirrors 000430 1749605796.937558 mapread 0x0170f5 thru 0x01e8d2 (0x77de bytes) 000432 1749605796.945314 MIRROR_EXTEND to 7 mirrors 000436 1749605797.043850 write 0x00d0ad thru 0x0124cf (0x5423 bytes) 000437 1749605797.059739 write 0x004798 thru 0x00d5c4 (0x8e2d bytes) truncating to largest ever: 0x3e792 000438 1749605797.062136 trunc from 0x03e834 to 0x03e791 (0xa2 bytes) 000440 1749605797.069608 mapread 0x025946 thru 0x02cc60 (0x731b bytes) 000441 1749605797.073387 MIRROR_RESYNC 7 mirrors 000442 1749605797.272789 read 0x0261ce thru 0x02f46e (0x92a1 bytes) 000443 1749605797.274717 mapread 0x037040 thru 0x03e791 (0x7752 bytes) 000444 1749605797.278452 mapwrite 0x02ce74 thru 0x03c319 (0xf4a6 bytes) 000445 1749605797.320609 MIRROR_EXTEND to 8 mirrors 000447 1749605797.413836 trunc from 0x03e792 to 0x003a63 (0x3ad2e bytes) 000448 1749605797.431153 read 0x000806 thru 0x00182b (0x1026 bytes) 000450 1749605797.432989 mapread 0x0006e1 thru 0x000ab3 (0x3d3 bytes) 000452 1749605797.433710 write 0x0046f2 thru 0x00d9f2 (0x9301 bytes) 000453 1749605797.434377 trunc from 0x00d9f3 to 0x01b4ae (0xdabc bytes) 000454 1749605797.438405 write 0x00ee85 thru 0x01a623 (0xb79f bytes) 000455 1749605797.440983 mapread 0x017915 thru 0x01b4ae (0x3b9a bytes) 000456 1749605797.442561 read 0x004516 thru 0x00d935 (0x9420 bytes) 000457 1749605797.442880 mapwrite 0x00887f thru 0x00a179 (0x18fb bytes) 000458 1749605797.457795 MIRROR_RESYNC 8 mirrors 000459 1749605797.645779 MIRROR_RESYNC 8 mirrors 000460 1749605797.689416 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000461 1749605797.749941 read 0x0031fe thru 0x005823 (0x2626 bytes) 000462 1749605797.752359 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000463 1749605797.814187 mapread 0x004f7f thru 0x00d381 (0x8403 bytes) 000465 1749605797.819531 trunc from 0x01b4af to 0x01cbc7 (0x1719 bytes) 000466 1749605797.839181 read 0x01a468 thru 0x01cbc7 (0x2760 bytes) 000467 1749605797.841082 trunc from 0x01cbc8 to 0x0317bf (0x14bf8 bytes) 000469 1749605797.847373 MIRROR_EXTEND to 7 mirrors 000470 1749605797.942124 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000474 1749605798.010921 write 0x0250e2 thru 0x02c22c (0x714b bytes) 000475 1749605798.028121 mapwrite 0x00839e thru 0x00d3de (0x5041 bytes) 000476 1749605798.043992 read 0x013d5a thru 0x019b44 (0x5deb bytes) 000477 1749605798.046250 mapwrite 0x0362b5 thru 0x03ffff (0x9d4b bytes) 000479 1749605798.072099 trunc from 0x040000 to 0x006baf (0x39450 bytes) 000484 1749605798.077045 mapwrite 0x01191b thru 0x01ddc0 (0xc4a6 bytes) 000489 1749605798.102112 read 0x001d13 thru 0x00b1cf (0x94bd bytes) 000490 1749605798.106046 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000491 1749605798.155736 read 0x00013c thru 0x004849 (0x470e bytes) 000493 1749605798.158440 mapwrite 0x029297 thru 0x02acb3 (0x1a1d bytes) 000494 1749605798.176526 write 0x039cf3 thru 0x03d0dc (0x33ea bytes) 000497 1749605798.177566 trunc from 0x03d0dd to 0x0299db (0x13701 bytes) 000498 1749605798.181590 read 0x02200c thru 0x0297bc (0x77b1 bytes) 000500 1749605798.185290 write 0x039f6d thru 0x03ffff (0x6093 bytes) 000501 1749605798.185965 write 0x02ca36 thru 0x03837f (0xb94a bytes) 000502 1749605798.188883 mapwrite 0x02242f thru 0x02a3ea (0x7fbc bytes) 000506 1749605798.205048 read 0x003851 thru 0x00cd6a (0x951a bytes) 000508 1749605798.205822 mapread 0x01f65d thru 0x022c7e (0x3622 bytes) 000510 1749605798.208083 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000511 1749605798.256086 mapwrite 0x0000d2 thru 0x005305 (0x5234 bytes) 000512 1749605798.272859 write 0x01f585 thru 0x02962f (0xa0ab bytes) 000513 1749605798.277588 MIRROR_EXTEND to 5 mirrors 000514 1749605798.382231 mapwrite 0x038c5d thru 0x03b2d9 (0x267d bytes) 000515 1749605798.425082 mapread 0x016986 thru 0x024c0a (0xe285 bytes) 000517 1749605798.429881 trunc from 0x040000 to 0x02352a (0x1cad5 bytes) 000520 1749605798.438279 read 0x00128e thru 0x00a6ab (0x941e bytes) 000521 1749605798.440643 write 0x00a502 thru 0x0118b9 (0x73b8 bytes) 000525 1749605798.442962 mapwrite 0x027ce9 thru 0x033e51 (0xc169 bytes) 000526 1749605798.462988 mapwrite 0x0222b0 thru 0x02c6f9 (0xa44a bytes) 000527 1749605798.474325 read 0x00403b thru 0x00a0d3 (0x6099 bytes) 000528 1749605798.474679 write 0x02b1b1 thru 0x02db91 (0x29e1 bytes) 000531 1749605798.475536 mapwrite 0x002824 thru 0x004559 (0x1d36 bytes) 000532 1749605798.486908 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000533 1749605798.531814 write 0x0060dc thru 0x00d154 (0x7079 bytes) 000535 1749605798.534807 read 0x029c7a thru 0x030550 (0x68d7 bytes) 000539 1749605798.537348 trunc from 0x033e52 to 0x024a48 (0xf409 bytes) 000543 1749605798.541693 mapread 0x021e74 thru 0x024a48 (0x2bd5 bytes) 000544 1749605798.545363 MIRROR_RESYNC 4 mirrors 000546 1749605798.675930 mapwrite 0x03dafc thru 0x03ffff (0x2504 bytes) 000547 1749605798.708546 mapread 0x032a18 thru 0x0374e5 (0x4ace bytes) 000548 1749605798.710792 read 0x01dfb3 thru 0x027539 (0x9587 bytes) 000551 1749605798.713285 trunc from 0x040000 to 0x02c6f9 (0x13906 bytes) 000552 1749605798.717694 read 0x02aa54 thru 0x02c6f9 (0x1ca6 bytes) 000553 1749605798.720099 mapwrite 0x01d191 thru 0x0237b4 (0x6624 bytes) 000555 1749605798.732096 write 0x01c633 thru 0x02c277 (0xfc45 bytes) 000557 1749605798.732988 mapwrite 0x010a57 thru 0x018f22 (0x84cc bytes) 000558 1749605798.744776 read 0x01d53d thru 0x0228d8 (0x539c bytes) 000559 1749605798.745062 mapread 0x00012f thru 0x00ae8d (0xad5f bytes) 000565 1749605798.746608 read 0x027d2e thru 0x02c6f9 (0x49cc bytes) 000566 1749605798.746889 mapread 0x006f9e thru 0x016756 (0xf7b9 bytes) 000567 1749605798.747300 trunc from 0x02c6fa to 0x013d0e (0x189eb bytes) 000569 1749605798.751536 read 0x00b4af thru 0x00d508 (0x205a bytes) 000570 1749605798.751821 write 0x008d9e thru 0x00a34a (0x15ad bytes) 000572 1749605798.752458 read 0x000060 thru 0x008e35 (0x8dd6 bytes) 000574 1749605798.752943 read 0x0084ef thru 0x009a77 (0x1589 bytes) 000576 1749605798.753486 read 0x000838 thru 0x005a0e (0x51d7 bytes) 000584 1749605798.756581 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000585 1749605798.802132 write 0x02f66f thru 0x037288 (0x7c1a bytes) 000586 1749605798.803001 read 0x0298dc thru 0x02deb3 (0x45d8 bytes) 000587 1749605798.806003 read 0x027021 thru 0x02d2c0 (0x62a0 bytes) 000588 1749605798.809814 trunc from 0x037289 to 0x009b76 (0x2d712 bytes) 000590 1749605798.815614 mapread 0x002645 thru 0x009b76 (0x7532 bytes) 000591 1749605798.816435 mapwrite 0x029d19 thru 0x039628 (0xf910 bytes) 000593 1749605798.856353 read 0x02acdc thru 0x02ea74 (0x3d99 bytes) 000594 1749605798.856776 trunc from 0x039629 to 0x03c6cb (0x30a3 bytes) 000595 1749605798.860931 trunc from 0x03c6cc to 0x010866 (0x2be65 bytes) 000597 1749605798.865894 write 0x00e3b6 thru 0x018e85 (0xaad0 bytes) 000598 1749605798.870060 MIRROR_EXTEND to 4 mirrors 000599 1749605798.987970 write 0x039c7a thru 0x03a2c1 (0x648 bytes) 000600 1749605799.006946 trunc from 0x03a2c2 to 0x01ae2f (0x1f492 bytes) 000602 1749605799.020779 mapread 0x015d5e thru 0x01ae2f (0x50d2 bytes) 000606 1749605799.033016 MIRROR_RESYNC 4 mirrors 000607 1749605799.236318 write 0x027351 thru 0x032ba6 (0xb856 bytes) 000608 1749605799.253104 mapread 0x019879 thru 0x01b9c6 (0x214e bytes) 000610 1749605799.259039 mapwrite 0x00022c thru 0x009717 (0x94ec bytes) 000611 1749605799.277461 read 0x001bac thru 0x005538 (0x398d bytes) 000614 1749605799.278579 write 0x02b027 thru 0x03a310 (0xf2ea bytes) 000616 1749605799.279968 read 0x016618 thru 0x01db7c (0x7565 bytes) 000619 1749605799.280877 mapwrite 0x023752 thru 0x029eb0 (0x675f bytes) 000620 1749605799.296027 read 0x0247c9 thru 0x02cd85 (0x85bd bytes) 000621 1749605799.296496 write 0x024a37 thru 0x02ee9b (0xa465 bytes) 000622 1749605799.297293 read 0x027a00 thru 0x02dec1 (0x64c2 bytes) 000623 1749605799.299216 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000624 1749605799.352131 write 0x037b2b thru 0x03872c (0xc02 bytes) 000625 1749605799.354919 write 0x035e5a thru 0x03ffff (0xa1a6 bytes) 000626 1749605799.357804 write 0x010a5e thru 0x01861f (0x7bc2 bytes) 000627 1749605799.360320 trunc from 0x040000 to 0x020016 (0x1ffe9 bytes) 000628 1749605799.367836 mapread 0x01610b thru 0x0175e7 (0x14dd bytes) 000629 1749605799.368292 write 0x007821 thru 0x016a83 (0xf263 bytes) 000631 1749605799.370759 mapread 0x0110c6 thru 0x020016 (0xef51 bytes) 000633 1749605799.373519 mapwrite 0x02e80d thru 0x031c10 (0x3404 bytes) 000634 1749605799.393074 write 0x0249cf thru 0x025444 (0xa76 bytes) 000635 1749605799.395305 mapread 0x0144a2 thru 0x022b95 (0xe6f4 bytes) 000637 1749605799.396052 mapwrite 0x03c5f7 thru 0x03ffff (0x3a09 bytes) 000638 1749605799.412197 trunc from 0x040000 to 0x020268 (0x1fd97 bytes) 000640 1749605799.415773 mapread 0x009c4b thru 0x010878 (0x6c2e bytes) 000643 1749605799.416894 read 0x00718d thru 0x011215 (0xa089 bytes) 000644 1749605799.417255 write 0x033d4e thru 0x03cf9c (0x924f bytes) 000645 1749605799.419104 MIRROR_RESYNC 3 mirrors 000646 1749605799.526810 mapread 0x001031 thru 0x00fd1b (0xeceb bytes) 000649 1749605799.533538 trunc from 0x03cf9d to 0x021e4e (0x1b14e bytes) 000652 1749605799.552560 read 0x01796f thru 0x021e4e (0xa4e0 bytes) 000653 1749605799.556344 MIRROR_EXTEND to 4 mirrors 000654 1749605799.643874 write 0x016a02 thru 0x017560 (0xb5f bytes) 000656 1749605799.657975 mapread 0x000702 thru 0x002e4f (0x274e bytes) 000657 1749605799.662356 write 0x015a63 thru 0x0250c0 (0xf65e bytes) 000659 1749605799.665784 trunc from 0x0250c1 to 0x039578 (0x144b8 bytes) 000662 1749605799.675472 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000663 1749605799.727890 read 0x0330b4 thru 0x033f79 (0xec6 bytes) 000666 1749605799.730593 mapwrite 0x03aae5 thru 0x03ffff (0x551b bytes) 000668 1749605799.751446 mapwrite 0x00db5d thru 0x00f60b (0x1aaf bytes) 000669 1749605799.764527 mapread 0x003f0c thru 0x005773 (0x1868 bytes) 000670 1749605799.768225 MIRROR_RESYNC 3 mirrors 000671 1749605799.876445 read 0x033f6d thru 0x03f9b1 (0xba45 bytes) 000672 1749605799.880081 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000674 1749605799.959212 read 0x0023a2 thru 0x00ee9c (0xcafb bytes) 000675 1749605799.963101 mapwrite 0x0162ee thru 0x01d330 (0x7043 bytes) 000676 1749605800.017467 mapread 0x032675 thru 0x0343ff (0x1d8b bytes) 000680 1749605800.022085 mapwrite 0x00bb5a thru 0x00e8e4 (0x2d8b bytes) 000681 1749605800.040176 read 0x001151 thru 0x00851e (0x73ce bytes) 000683 1749605800.046095 MIRROR_RESYNC 2 mirrors 000684 1749605800.160017 trunc from 0x040000 to 0x02db9a (0x12465 bytes) 000685 1749605800.180363 mapread 0x028072 thru 0x02db9a (0x5b29 bytes) 000686 1749605800.186300 read 0x00e40f thru 0x00f75e (0x1350 bytes) 000687 1749605800.188330 write 0x03ab20 thru 0x03dbcc (0x30ad bytes) 000691 1749605800.190774 mapwrite 0x034692 thru 0x039f0b (0x587a bytes) 000695 1749605800.211137 read 0x01b8c7 thru 0x02706b (0xb7a5 bytes) 000698 1749605800.214890 read 0x02b7b5 thru 0x039c1a (0xe466 bytes) 000699 1749605800.215350 trunc from 0x03dbcd to 0x027af9 (0x160d3 bytes) 000700 1749605800.220039 read 0x00f41e thru 0x00feb5 (0xa98 bytes) 000701 1749605800.221431 MIRROR_EXTEND to 3 mirrors 000704 1749605800.313469 write 0x032083 thru 0x0336d2 (0x1650 bytes) 000705 1749605800.328404 write 0x03e5b1 thru 0x03ffff (0x1a4f bytes) 000706 1749605800.329068 mapwrite 0x0328bf thru 0x039bae (0x72f0 bytes) 000707 1749605800.347509 mapread 0x039299 thru 0x03ffff (0x6d67 bytes) 000708 1749605800.347994 mapread 0x018caf thru 0x01e791 (0x5ae3 bytes) 000710 1749605800.350692 mapread 0x0014d5 thru 0x00b345 (0x9e71 bytes) 000711 1749605800.354302 mapread 0x01cd81 thru 0x0295f3 (0xc873 bytes) 000712 1749605800.356791 trunc from 0x040000 to 0x024260 (0x1bd9f bytes) 000713 1749605800.365077 read 0x00b292 thru 0x015047 (0x9db6 bytes) 000716 1749605800.367196 mapwrite 0x0381c4 thru 0x03ffff (0x7e3c bytes) 000717 1749605800.388323 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000718 1749605800.434900 mapwrite 0x0337c6 thru 0x0364e1 (0x2d1c bytes) 000722 1749605800.450430 mapwrite 0x022c75 thru 0x031db7 (0xf143 bytes) 000723 1749605800.468808 mapread 0x03d4b0 thru 0x03ffff (0x2b50 bytes) 000725 1749605800.470907 mapread 0x01b832 thru 0x02a20d (0xe9dc bytes) 000728 1749605800.473545 mapread 0x00d5e3 thru 0x00ebb8 (0x15d6 bytes) 000729 1749605800.475892 read 0x00d8c9 thru 0x0190f1 (0xb829 bytes) 000731 1749605800.477809 MIRROR_EXTEND to 3 mirrors 000732 1749605800.572891 mapwrite 0x01bf49 thru 0x029123 (0xd1db bytes) 000734 1749605800.620893 MIRROR_RESYNC 3 mirrors 000735 1749605800.721843 mapwrite 0x02252a thru 0x024dea (0x28c1 bytes) 000736 1749605800.763284 mapread 0x03d0ec thru 0x03ffff (0x2f14 bytes) 000737 1749605800.765708 mapread 0x026c5f thru 0x02edac (0x814e bytes) 000738 1749605800.768056 trunc from 0x040000 to 0x038132 (0x7ecd bytes) 000739 1749605800.775930 mapread 0x01f6af thru 0x02071c (0x106e bytes) 000740 1749605800.776463 write 0x018262 thru 0x02260f (0xa3ae bytes) 000741 1749605800.778951 mapread 0x02cf26 thru 0x0316a1 (0x477c bytes) 000742 1749605800.779473 mapwrite 0x00280a thru 0x004fed (0x27e4 bytes) 000743 1749605800.795184 MIRROR_EXTEND to 4 mirrors 000744 1749605800.886104 write 0x00dc13 thru 0x01105f (0x344d bytes) 000745 1749605800.900999 mapwrite 0x006065 thru 0x008663 (0x25ff bytes) 000746 1749605800.916492 write 0x0356e5 thru 0x03fa4f (0xa36b bytes) 000747 1749605800.918990 mapwrite 0x03d7d6 thru 0x03e3ec (0xc17 bytes) 000751 1749605800.932651 trunc from 0x03fa50 to 0x034187 (0xb8c8 bytes) 000753 1749605800.942777 mapread 0x031bf8 thru 0x034187 (0x2590 bytes) 000756 1749605800.943729 read 0x00445a thru 0x00cc37 (0x87de bytes) 000757 1749605800.944037 trunc from 0x034188 to 0x024331 (0xfe56 bytes) 000758 1749605800.948228 trunc from 0x024332 to 0x004978 (0x1f9b9 bytes) 000759 1749605800.952528 read 0x00242e thru 0x004978 (0x254b bytes) 000761 1749605800.953537 read 0x0033cb thru 0x004978 (0x15ae bytes) 000765 1749605800.954672 mapread 0x003b23 thru 0x004978 (0xe56 bytes) 000766 1749605800.955285 read 0x002812 thru 0x004978 (0x2167 bytes) 000767 1749605800.957889 MIRROR_RESYNC 4 mirrors 000769 1749605801.104423 read 0x004629 thru 0x004978 (0x350 bytes) 000770 1749605801.106511 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000772 1749605801.192243 trunc from 0x004979 to 0x02be6e (0x274f6 bytes) 000773 1749605801.220649 mapwrite 0x004c27 thru 0x0123f2 (0xd7cc bytes) 000774 1749605801.258027 mapread 0x011e60 thru 0x017df9 (0x5f9a bytes) 000775 1749605801.261573 MIRROR_RESYNC 3 mirrors 000778 1749605801.382826 MIRROR_EXTEND to 4 mirrors 000781 1749605801.523380 mapread 0x009b36 thru 0x00ed61 (0x522c bytes) 000782 1749605801.528351 trunc from 0x02be6f to 0x00d9c9 (0x1e4a5 bytes) 000783 1749605801.549802 trunc from 0x00d9ca to 0x00a99a (0x302f bytes) 000785 1749605801.555781 MIRROR_EXTEND to 5 mirrors 000786 1749605801.673465 mapread 0x001993 thru 0x00a99a (0x9008 bytes) 000788 1749605801.683402 write 0x03fd97 thru 0x03ffff (0x269 bytes) 000789 1749605801.703901 mapread 0x024afd thru 0x0292ff (0x4803 bytes) 000790 1749605801.708628 mapread 0x000a1b thru 0x005fc8 (0x55ae bytes) 000792 1749605801.714656 write 0x025ff1 thru 0x03090a (0xa91a bytes) 000795 1749605801.717155 trunc from 0x040000 to 0x03cb73 (0x348c bytes) 000796 1749605801.729862 mapread 0x030e18 thru 0x03cb73 (0xbd5c bytes) 000799 1749605801.736680 write 0x02639c thru 0x02bd9e (0x5a03 bytes) 000801 1749605801.739349 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000802 1749605801.799951 mapwrite 0x01cb9b thru 0x0244a0 (0x7906 bytes) 000805 1749605801.820265 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000807 1749605801.863887 mapread 0x02d5c9 thru 0x0372bf (0x9cf7 bytes) 000809 1749605801.870018 read 0x02e8cb thru 0x036012 (0x7748 bytes) 000814 1749605801.872308 MIRROR_EXTEND to 4 mirrors 000816 1749605801.980387 trunc from 0x03cb74 to 0x03a01f (0x2b54 bytes) 000817 1749605802.001670 write 0x00748f thru 0x016304 (0xee76 bytes) 000818 1749605802.004716 write 0x0142e3 thru 0x01dd1b (0x9a39 bytes) 000819 1749605802.006448 mapread 0x001d84 thru 0x007a1a (0x5c97 bytes) 000821 1749605802.009158 read 0x01a235 thru 0x025c13 (0xb9df bytes) 000822 1749605802.011287 trunc from 0x03a020 to 0x02467f (0x159a0 bytes) 000824 1749605802.016193 trunc from 0x024680 to 0x0017fa (0x22e85 bytes) 000825 1749605802.019868 read 0x00081a thru 0x0017fa (0xfe1 bytes) 000829 1749605802.022777 read 0x000ea6 thru 0x0017fa (0x955 bytes) 000830 1749605802.024759 MIRROR_EXTEND to 5 mirrors 000831 1749605802.103748 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000833 1749605802.160342 trunc from 0x0017fb to 0x021642 (0x1fe48 bytes) 000835 1749605802.178293 MIRROR_RESYNC 4 mirrors 000836 1749605802.306603 read 0x00ba93 thru 0x00c1e6 (0x754 bytes) 000837 1749605802.308147 mapread 0x00e193 thru 0x01aa29 (0xc897 bytes) 000839 1749605802.313815 trunc from 0x021643 to 0x032d27 (0x116e5 bytes) 000840 1749605802.332183 MIRROR_RESYNC 4 mirrors 000841 1749605802.458848 write 0x022458 thru 0x031c23 (0xf7cc bytes) 000844 1749605802.476155 trunc from 0x032d28 to 0x02a7c9 (0x855e bytes) 000845 1749605802.483440 read 0x012e79 thru 0x01cdd2 (0x9f5a bytes) 000846 1749605802.485664 read 0x004a3f thru 0x011144 (0xc706 bytes) 000848 1749605802.488863 write 0x013634 thru 0x02349b (0xfe68 bytes) 000849 1749605802.489968 write 0x022a34 thru 0x02faf8 (0xd0c5 bytes) 000850 1749605802.491195 trunc from 0x02faf9 to 0x01ee9d (0x10c5b bytes) 000851 1749605802.495621 trunc from 0x01ee9e to 0x031e9c (0x12fff bytes) 000852 1749605802.501407 MIRROR_RESYNC 4 mirrors 000854 1749605802.626340 read 0x007b66 thru 0x007d06 (0x1a1 bytes) 000855 1749605802.628323 mapwrite 0x021f29 thru 0x023537 (0x160f bytes) 000857 1749605802.680922 MIRROR_RESYNC 4 mirrors 000860 1749605802.816251 write 0x02d523 thru 0x0393d5 (0xbeb3 bytes) 000862 1749605802.829686 trunc from 0x0393d6 to 0x0254cf (0x13f06 bytes) 000863 1749605802.837380 trunc from 0x0254d0 to 0x00581c (0x1fcb3 bytes) 000864 1749605802.841233 mapwrite 0x024e0e thru 0x025777 (0x96a bytes) 000865 1749605802.859126 mapwrite 0x018fb0 thru 0x020973 (0x79c4 bytes) 000866 1749605802.872964 mapwrite 0x01042a thru 0x01ff54 (0xfb2b bytes) 000867 1749605802.889344 mapwrite 0x02d046 thru 0x02e32f (0x12ea bytes) 000868 1749605802.910902 mapread 0x025020 thru 0x02736b (0x234c bytes) 000869 1749605802.911527 write 0x00788c thru 0x008fde (0x1753 bytes) 000870 1749605802.914751 trunc from 0x02e330 to 0x033e3e (0x5b0f bytes) 000875 1749605802.921857 MIRROR_RESYNC 4 mirrors 000876 1749605803.048531 trunc from 0x033e3f to 0x00226f (0x31bcf bytes) 000877 1749605803.067331 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000879 1749605803.114539 write 0x01ffe9 thru 0x029eb6 (0x9ece bytes) 000880 1749605803.115307 trunc from 0x029eb7 to 0x033266 (0x93b0 bytes) 000881 1749605803.119088 mapread 0x000469 thru 0x001933 (0x14cb bytes) 000885 1749605803.122424 write 0x02069b thru 0x025499 (0x4dff bytes) 000886 1749605803.122943 write 0x02c4a5 thru 0x032d29 (0x6885 bytes) 000887 1749605803.126026 mapread 0x015a20 thru 0x019880 (0x3e61 bytes) 000888 1749605803.129549 read 0x01a0c2 thru 0x02135b (0x729a bytes) 000892 1749605803.130924 trunc from 0x033267 to 0x023c3a (0xf62c bytes) 000893 1749605803.135644 trunc from 0x023c3b to 0x00a1b4 (0x19a86 bytes) 000895 1749605803.140597 mapread 0x00347c thru 0x00a1b4 (0x6d39 bytes) 000897 1749605803.144719 mapwrite 0x0230c5 thru 0x03131d (0xe259 bytes) 000898 1749605803.176303 MIRROR_EXTEND to 4 mirrors 000899 1749605803.265837 write 0x03c3ea thru 0x03ffff (0x3c16 bytes) 000901 1749605803.279281 mapread 0x03fbb2 thru 0x03ffff (0x44e bytes) 000902 1749605803.280005 read 0x01fab9 thru 0x023d0f (0x4257 bytes) 000903 1749605803.282406 mapread 0x035d74 thru 0x03a778 (0x4a05 bytes) 000904 1749605803.286634 read 0x023983 thru 0x0307ee (0xce6c bytes) 000905 1749605803.289705 mapread 0x00e381 thru 0x010198 (0x1e18 bytes) 000907 1749605803.291548 MIRROR_RESYNC 4 mirrors 000908 1749605803.417324 trunc from 0x040000 to 0x030da2 (0xf25d bytes) 000909 1749605803.434324 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000913 1749605803.476673 MIRROR_RESYNC 3 mirrors 000914 1749605803.574276 read 0x028b58 thru 0x030da2 (0x824b bytes) 000915 1749605803.576371 mapread 0x024885 thru 0x02b686 (0x6e02 bytes) 000916 1749605803.580451 trunc from 0x030da3 to 0x02329c (0xdb06 bytes) 000918 1749605803.597724 mapwrite 0x02848d thru 0x035e07 (0xd97b bytes) 000920 1749605803.619823 read 0x00f13b thru 0x01ddc9 (0xec8f bytes) 000921 1749605803.622746 read 0x00c1ad thru 0x019d93 (0xdbe7 bytes) 000922 1749605803.624173 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000923 1749605803.675166 mapwrite 0x008ddf thru 0x00d45d (0x467f bytes) 000924 1749605803.693881 mapwrite 0x03e15d thru 0x03ffff (0x1ea3 bytes) 000925 1749605803.714481 MIRROR_RESYNC 2 mirrors 000927 1749605803.797110 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa08:0x0]: Invalid argument (22) 000930 1749605803.826304 write 0x0392b7 thru 0x03ffff (0x6d49 bytes) 000932 1749605803.832106 MIRROR_EXTEND to 2 mirrors 000933 1749605803.945040 write 0x008fd7 thru 0x00ee31 (0x5e5b bytes) 000934 1749605803.961265 MIRROR_EXTEND to 3 mirrors 000935 1749605804.091910 trunc from 0x040000 to 0x02f634 (0x109cb bytes) 000936 1749605804.109130 read 0x0078a7 thru 0x009b06 (0x2260 bytes) 000937 1749605804.111401 mapwrite 0x01eb27 thru 0x026935 (0x7e0f bytes) 000938 1749605804.136701 MIRROR_EXTEND to 4 mirrors 000943 1749605804.256309 trunc from 0x02f635 to 0x003b52 (0x2bae2 bytes) 000945 1749605804.278208 write 0x018935 thru 0x027349 (0xea15 bytes) 000947 1749605804.281272 write 0x0123ae thru 0x019177 (0x6dca bytes) 000948 1749605804.283927 read 0x0199ff thru 0x01c6c1 (0x2cc3 bytes) 000949 1749605804.284427 mapwrite 0x03a2a7 thru 0x03d041 (0x2d9b bytes) truncating to largest ever: 0x3fa2b 000953 1749605804.312067 trunc from 0x03d042 to 0x03fa2a (0x29e9 bytes) 000954 1749605804.316598 read 0x03069d thru 0x036822 (0x6186 bytes) 000956 1749605804.320700 write 0x01b82e thru 0x02a05a (0xe82d bytes) 000957 1749605804.321516 mapread 0x0224d4 thru 0x02f422 (0xcf4f bytes) 000960 1749605804.322616 mapread 0x012be5 thru 0x01d5dd (0xa9f9 bytes) 000961 1749605804.323189 trunc from 0x03fa2b to 0x01917f (0x268ab bytes) 000963 1749605804.327589 read 0x00355f thru 0x00cd83 (0x9825 bytes) 000965 1749605804.328207 mapread 0x008b9b thru 0x013022 (0xa488 bytes) 000966 1749605804.328760 mapwrite 0x034a1c thru 0x03f556 (0xab3b bytes) 000967 1749605804.350961 mapwrite 0x03ce82 thru 0x03ffff (0x317e bytes) 000968 1749605804.367518 mapread 0x0249d6 thru 0x024eb7 (0x4e2 bytes) 000969 1749605804.368265 mapread 0x02b5e7 thru 0x03087e (0x5298 bytes) 000970 1749605804.368755 write 0x00b537 thru 0x012741 (0x720b bytes) 000973 1749605804.369903 write 0x039a51 thru 0x03aac7 (0x1077 bytes) 000976 1749605804.370878 mapwrite 0x00ca80 thru 0x015af3 (0x9074 bytes) 000977 1749605804.383951 mapwrite 0x007799 thru 0x011a0b (0xa273 bytes) 000978 1749605804.396772 write 0x006528 thru 0x012862 (0xc33b bytes) 000979 1749605804.397499 trunc from 0x040000 to 0x03b6f1 (0x490e bytes) 000980 1749605804.402835 MIRROR_EXTEND to 5 mirrors 000983 1749605804.515698 trunc from 0x03b6f2 to 0x037870 (0x3e81 bytes) 000985 1749605804.534356 mapread 0x015269 thru 0x0165a0 (0x1338 bytes) 000986 1749605804.537787 mapwrite 0x02c53b thru 0x0314d1 (0x4f97 bytes) 000987 1749605804.555233 mapread 0x00e911 thru 0x0112c4 (0x29b4 bytes) 000990 1749605804.559768 mapwrite 0x030296 thru 0x037bcb (0x7936 bytes) 000992 1749605804.582189 mapread 0x01717b thru 0x0180a6 (0xf2c bytes) 000993 1749605804.583077 mapwrite 0x027b68 thru 0x031d94 (0xa22d bytes) 000994 1749605804.597096 MIRROR_RESYNC 5 mirrors 000997 1749605804.799906 MIRROR_RESYNC 5 mirrors 000998 1749605804.855448 MIRROR_RESYNC 5 mirrors 000999 1749605804.913169 mapwrite 0x00afad thru 0x018126 (0xd17a bytes) 001000 1749605804.972626 trunc from 0x037bcc to 0x00fdef (0x27ddc bytes) All operations completed A-OK! PASS 70a (14s) == sanity-flr test 71: check mirror extend parent fid ==== 21:36:46 (1749605806) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (0s) == sanity-flr test 200a: stress test ===================== 21:36:46 (1749605806) Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre2 Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3142272 .. Extending file size to 8273760 .. Extending file size to 6098368 .. failed Extending file size to 8205568 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8209440 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8382176 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8400672 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8819104 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9383552 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9119648 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 31273 31274 31276 31277 31280 192.168.202.143@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg243-client.virtnet /mnt/lustre2 (opts:) 192.168.202.143@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg243-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 21:37:50 (1749605870) Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre2 Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8655904 .. Extending file size to 6858112 .. done Extending file size to 7804224 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8341472 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9041472 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8585600 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9249632 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9054400 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9116320 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9296576 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9261024 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 2967 2968 2970 2972 2974 192.168.202.143@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg243-client.virtnet /mnt/lustre2 (opts:) 192.168.202.143@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg243-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 21:38:53 (1749605933) Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 5504 2202624 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 5504 2202624 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.034399 s, 61.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0794322 s, 26.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.143@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg243-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 21:38:58 (1749605938) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0279274 s, 75.1 MB/s PASS 202 (1s) == sanity-flr test 203: mirror file preserve mirror ID === 21:38:59 (1749605939) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa56:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe2e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe2e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe2f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0xe29:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 21:39:01 (1749605941) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0249088 s, 84.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe30:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe2a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe2b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 21:39:03 (1749605943) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0278353 s, 75.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe31:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe32:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe2c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe31:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe32:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe2c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe2d:0x0] } PASS 204b (1s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 21:39:04 (1749605944) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0300311 s, 69.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe33:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe2e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe33:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe2e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe2f:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 21:39:27 (1749605967) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg243-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0249284 s, 42.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe34:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe31:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe30:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe34:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe31:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe30:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe35:0x0] } PASS 204d (44s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 21:40:11 (1749606011) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg243-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe32:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe33:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0450362 s, 46.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe32:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe33:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe36:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe32:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe33:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe36:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 21:40:57 (1749606057) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg243-server mds-ost sync done. Creating new pool test_204f oleg243-server: Pool lustre.test_204f created Adding targets to pool oleg243-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg243-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.041538 s, 50.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe34:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe37:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe34:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe37:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg243-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg243-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg243-server: Pool lustre.test_204f destroyed PASS 204f (56s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 21:41:53 (1749606113) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 21:41:55 (1749606115) PASS 205b (1s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 21:41:56 (1749606116) oleg243-server: Pool lustre.test_206 created Adding targets to pool oleg243-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg243-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg243-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg243-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg243-server: Pool lustre.test_206 destroyed PASS 206 (15s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 21:42:11 (1749606131) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0175369 s, 59.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00704997 s, 149 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x18d88750 Mirror 3: 0x18d88750 PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 21:42:13 (1749606133) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 21:42:14 (1749606134) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (0s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 21:42:15 (1749606135) Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0213745 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1b30000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1ca0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1ca0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1ca0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xf90000 Read this (4 bytes): BBBB 192.168.202.143@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg243-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 21:42:28 (1749606148) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00248024 s, 3.3 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 21:42:30 (1749606150) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0135181 s, 77.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0xa7b lmm_fid: [0x200000404:0xa7b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0xa7b lmm_fid: [0x200000404:0xa7b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe42:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322836603 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-06-10 21:42:31.000000000 -0400 Modify: 2025-06-10 21:42:31.000000000 -0400 Change: 2025-06-10 21:42:31.000000000 -0400 Birth: - PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 21:42:33 (1749606153) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 21:42:34 (1749606154) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.116266 s, 352 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0160448 s, 255 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 10 21:42 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa7d:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 10 21:42 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 684 sec ============ 21:42:36 (1749606156) === sanity-flr: start cleanup 21:42:37 (1749606157) === === sanity-flr: finish cleanup 21:42:39 (1749606159) ===