-----============= acceptance-small: sanity-flr ============----- Tue Jun 10 03:32:57 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg119-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 03:33:02 (1749540782) === oleg119-client.virtnet: executing check_config_client /mnt/lustre oleg119-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg119-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b5967000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b5967000.idle_timeout=debug disable quota as required oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 03:33:10 (1749540790) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7926] == sanity-flr test 0a: lfs mirror create with -N option == 03:33:11 (1749540791) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 03:33:14 (1749540794) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 03:33:15 (1749540795) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 03:33:16 (1749540796) PASS 0d (5s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 03:33:21 (1749540801) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 03:33:22 (1749540802) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (0s) == sanity-flr test 0g: lfs mirror create flags support === 03:33:22 (1749540802) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b5967000.stats=clear osc.lustre-OST0001-osc-ffff8800b5967000.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 03:33:24 (1749540804) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 03:33:26 (1749540806) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 03:33:28 (1749540808) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 03:33:29 (1749540809) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (1s) == sanity-flr test 2: create components from existing files ========================================================== 03:33:30 (1749540810) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 03:33:32 (1749540812) PASS 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 03:33:34 (1749540814) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (1s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 03:33:35 (1749540815) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 03:33:38 (1749540818) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 03:33:40 (1749540820) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB) copied, 0.180622 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0354545 s, 29.6 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 03:33:42 (1749540822) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00903164 s, 116 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 03:33:44 (1749540824) Stopping /mnt/lustre-ost1 (opts:) on oleg119-server oleg119-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5967000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5967000.ost_server_uuid in DISCONN state after 3 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg119-server: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg119-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg119-server oleg119-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5967000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5967000.ost_server_uuid in DISCONN state after 2 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg119-server: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg119-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (25s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 03:34:09 (1749540849) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB) copied, 0.0918344 s, 68.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg119-server oleg119-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5967000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5967000.ost_server_uuid in DISCONN state after 3 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg119-server: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg119-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (12s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 03:34:22 (1749540862) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg119-server oleg119-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5967000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5967000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg119-server: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg119-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg119-server oleg119-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5967000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5967000.ost_server_uuid in DISCONN state after 3 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg119-server: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg119-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (19s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 03:34:41 (1749540881) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.788121 s, 66.5 MB/s -rw-r--r-- 1 root root 52428800 Jun 10 03:34 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.292s user 0m0.000s sys 0m0.120s Stopping /mnt/lustre-ost1 (opts:) on oleg119-server oleg119-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5967000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5967000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.255s user 0m0.000s sys 0m0.110s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg119-server: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg119-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg119-server oleg119-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5967000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5967000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost1)... real 0m0.282s user 0m0.000s sys 0m0.104s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg119-server: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg119-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg119-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b5967000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5967000.ost_server_uuid in FULL state after 0 sec PASS 33b (22s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 03:35:03 (1749540903) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (0s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 03:35:03 (1749540903) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 03:35:04 (1749540904) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 03:35:05 (1749540905) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 03:35:07 (1749540907) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.143079 s, 87.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 03:35:10 (1749540910) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0120401 s, 87.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0115022 s, 91.2 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 03:35:12 (1749540912) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 03:35:15 (1749540915) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000739387 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000613644 s, 1.7 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000635134 s, 1.6 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000622876 s, 1.6 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 03:35:17 (1749540917) create test files with size 15737687 .. 1e0be4261c7b78cb9e3fe30eea93684a - 5ef169929cb07959077dd463913b0b56 - cafeb986a35f87ef0ede1d8027344890 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg119-client.virtnet /mnt/lustre (opts:) Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre PASS 37 (5s) == sanity-flr test 38: resync ============================ 03:35:22 (1749540922) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 03:35:25 (1749540925) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 03:35:27 (1749540927) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0179053 s, 58.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00866186 s, 121 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x41:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 03:35:29 (1749540929) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0186219 s, 113 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0823229 s, 50.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0343629 s, 61.0 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) == sanity-flr test 42: lfs mirror verify ================= 03:35:31 (1749540931) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (0s) == sanity-flr test 43a: mirror pick on write ============= 03:35:31 (1749540931) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 03:35:32 (1749540932) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00799505 s, 131 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 03:35:34 (1749540934) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0740531 s, 42.5 MB/s PASS 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 03:35:36 (1749540936) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4a:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x48:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x48:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (1s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 03:35:38 (1749540938) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0719709 s, 146 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg119-client.virtnet /mnt/lustre (opts:) Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (23s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 03:36:01 (1749540961) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800aae70000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0692841 s, 151 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg119-client.virtnet /mnt/lustre (opts:) Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff88012c970800.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 03:36:04 (1749540964) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (6s) == sanity-flr test 46: Verify setstripe --copy option ==== 03:36:10 (1749540970) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (6s) == sanity-flr test 47: Verify mirror obj alloc =========== 03:36:16 (1749540976) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (0s) == sanity-flr test 48: Verify snapshot mirror ============ 03:36:16 (1749540976) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0750032 s, 41.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0481998 s, 65.3 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: c6af349b7a81797695d1d8084143a574 - mirror 1 checksum: 3d103369820a329ff75f6d5bd9a82558 - mirror 2 checksum: c6af349b7a81797695d1d8084143a574 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Jun 10 03:36:17 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Jun 10 03:36:17 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: c6af349b7a81797695d1d8084143a574 - mirror 1 checksum: 3d103369820a329ff75f6d5bd9a82558 - mirror 2 checksum: 3d103369820a329ff75f6d5bd9a82558 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Jun 10 03:36:17 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Jun 10 03:36:17 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } PASS 48 (2s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 03:36:18 (1749540978) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00955861 s, 110 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 529408.. 530431: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 03:36:20 (1749540980) Failing mds1 on oleg119-server Stopping /mnt/lustre-mds1 (opts:) on oleg119-server 03:36:27 (1749540987) shut down facet: mds1 facet_host: oleg119-server facet_failover_host: oleg119-server Failover mds1 to oleg119-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0000 03:36:39 (1749540999) targets are mounted 03:36:39 (1749540999) facet_failover done oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0094218 s, 111 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } PASS 50A (27s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 03:36:47 (1749541007) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00813748 s, 2.5 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00391024 s, 5.2 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 03:36:49 (1749541009) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00911911 s, 2.2 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0025102 s, 8.2 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 03:36:51 (1749541011) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00216821 s, 7.6 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 03:36:53 (1749541013) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00873374 s, 2.3 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 03:36:54 (1749541014) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00159706 s, 2.6 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00807223 s, 507 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 03:36:56 (1749541016) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (3s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 03:36:59 (1749541019) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 03:37:17 (1749541037) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 03:37:34 (1749541054) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 03:37:51 (1749541071) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (1s) == sanity-flr test 70: mirror create and split race ====== 03:37:52 (1749541072) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 26685 mirror split pid 26686 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4948: 26685 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4948: 26686 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 03:38:54 (1749541134) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 1134 skipping zero size read skipping zero length zero range fallocating to largest ever: 0x7e05 000003 1749541134.937698 fallocate 0x003f09 thru 0x007e04 (0x3efc bytes) skipping zero size read skipping zero length punch hole 000006 1749541134.940022 write 0x01b7af thru 0x01fe66 (0x46b8 bytes) 000007 1749541134.940649 write 0x038e85 thru 0x03c9fe (0x3b7a bytes) zero_range to largest ever: 0x1e781 000008 1749541134.941310 trunc from 0x013a74 to 0x01e780 (0xad0d bytes) 000009 1749541134.945128 fallocate 0x006bd7 thru 0x0141ad (0xd5d7 bytes) 000010 1749541134.946656 trunc from 0x01feb7 to 0x027969 (0x7ab3 bytes) 000011 1749541134.947952 mapwrite 0x03e5fb thru 0x03ffff (0x1a05 bytes) 000012 1749541134.959182 trunc from 0x040000 to 0x004cd1 (0x3b32e bytes) 000014 1749541134.963089 mapwrite 0x020c9e thru 0x030035 (0xf398 bytes) 000015 1749541134.977658 write 0x036297 thru 0x0372cc (0x1036 bytes) 000016 1749541134.978260 mapread 0x01d2be thru 0x022a84 (0x57c7 bytes) truncating to largest ever: 0x36550 000017 1749541134.978857 trunc from 0x0372cd to 0x03654f (0xd7d bytes) 000018 1749541134.983239 MIRROR_EXTEND to 2 mirrors 000019 1749541135.088072 write 0x01fb3f thru 0x021c47 (0x2109 bytes) 000020 1749541135.100304 write 0x02284e thru 0x02c8fe (0xa0b1 bytes) 000021 1749541135.104291 read 0x0080c9 thru 0x0173c4 (0xf2fc bytes) 000022 1749541135.106972 write 0x0119b5 thru 0x01f8f6 (0xdf42 bytes) 000023 1749541135.108006 read 0x000b3c thru 0x004d91 (0x4256 bytes) 000024 1749541135.110619 trunc from 0x036550 to 0x0019cd (0x34b82 bytes) 000025 1749541135.116652 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x9b1:0x0]: Invalid argument (22) truncating to largest ever: 0x3a71e 000026 1749541135.146355 trunc from 0x0019ce to 0x03a71d (0x38d50 bytes) 000027 1749541135.151262 mapwrite 0x0357f6 thru 0x03ffff (0xa80a bytes) 000028 1749541135.168069 mapread 0x0300b1 thru 0x032281 (0x21d1 bytes) 000029 1749541135.168654 fallocate 0x00e858 thru 0x015122 (0x68cb bytes) 000030 1749541135.170368 fallocate 0x030917 thru 0x035223 (0x490d bytes) 000031 1749541135.171943 fallocate 0x02e73b thru 0x02fe41 (0x1707 bytes) 000032 1749541135.175227 MIRROR_EXTEND to 2 mirrors 000033 1749541135.266942 mapread 0x0076f4 thru 0x007d37 (0x644 bytes) 000034 1749541135.268669 mapwrite 0x03a4d1 thru 0x03ffff (0x5b2f bytes) 000035 1749541135.296348 trunc from 0x040000 to 0x02fe8d (0x10172 bytes) 000036 1749541135.306949 MIRROR_RESYNC 2 mirrors 000038 1749541135.391144 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x9b1:0x0]: Invalid argument (22) 000039 1749541135.420166 mapread 0x0231ed thru 0x028b09 (0x591d bytes) 000041 1749541135.424658 fallocate 0x00ebdf thru 0x01e682 (0xfaa4 bytes) 000043 1749541135.428870 read 0x0004d5 thru 0x004f9f (0x4acb bytes) 000045 1749541135.431065 mapwrite 0x0188c9 thru 0x0215a7 (0x8cdf bytes) 000046 1749541135.441414 fallocate 0x00ed0c thru 0x012700 (0x39f5 bytes) 000047 1749541135.442677 fallocate 0x029dac thru 0x02f7d4 (0x5a29 bytes) 000048 1749541135.445408 MIRROR_EXTEND to 2 mirrors 000049 1749541135.549998 read 0x00d3b7 thru 0x01862b (0xb275 bytes) 000050 1749541135.552828 trunc from 0x01d02b to 0x029cfe (0xccd4 bytes) 000051 1749541135.560993 mapwrite 0x01adeb thru 0x026324 (0xb53a bytes) 000052 1749541135.573835 mapread 0x0100b7 thru 0x0123f9 (0x2343 bytes) 000053 1749541135.576008 trunc from 0x022e4a to 0x02a34a (0x7501 bytes) 000054 1749541135.577472 punch from 0x00da58 to 0x019454 (0xb9fd bytes) 000055 1749541135.578898 mapread 0x01d9c2 thru 0x01eb59 (0x1198 bytes) 000057 1749541135.579511 mapwrite 0x011163 thru 0x020a17 (0xf8b5 bytes) 000058 1749541135.587948 read 0x01a219 thru 0x01fb2c (0x5914 bytes) 000059 1749541135.588383 mapwrite 0x035091 thru 0x0376d1 (0x2641 bytes) 000060 1749541135.605491 mapread 0x007071 thru 0x00db62 (0x6af2 bytes) 000061 1749541135.606184 mapwrite 0x03f0f1 thru 0x03ffff (0xf0f bytes) 000062 1749541135.615853 mapwrite 0x02b6bc thru 0x0386e3 (0xd028 bytes) 000063 1749541135.624458 trunc from 0x040000 to 0x00959f (0x36a60 bytes) 000064 1749541135.630226 write 0x004f3c thru 0x011ba6 (0xcc6b bytes) 000065 1749541135.631442 write 0x0201cd thru 0x021154 (0xf88 bytes) 000067 1749541135.632098 mapread 0x017233 thru 0x021154 (0x9f22 bytes) 000068 1749541135.643301 trunc from 0x021155 to 0x0019b6 (0x1f79e bytes) 000069 1749541135.647478 punch from 0x00018c to 0x0019b6 (0x182b bytes) 000070 1749541135.648750 trunc from 0x000c4d to 0x0019b6 (0xd6a bytes) 000071 1749541135.649884 fallocate 0x015454 thru 0x01aa8b (0x5638 bytes) 000072 1749541135.651495 fallocate 0x00e10b thru 0x016db2 (0x8ca8 bytes) 000073 1749541135.653017 mapwrite 0x03a4c0 thru 0x03cdc5 (0x2906 bytes) 000074 1749541135.666857 MIRROR_EXTEND to 3 mirrors 000075 1749541135.758479 write 0x01374d thru 0x015e4c (0x2700 bytes) 000076 1749541135.768823 trunc from 0x03cdc6 to 0x02bb22 (0x112a3 bytes) truncating to largest ever: 0x3dc0e 000077 1749541135.778522 trunc from 0x02bb23 to 0x03dc0d (0x120eb bytes) 000078 1749541135.782207 trunc from 0x03dc0e to 0x01e655 (0x1f5b8 bytes) 000079 1749541135.785806 punch from 0x000a5b to 0x00da44 (0xcfea bytes) 000080 1749541135.787318 fallocate 0x007299 thru 0x00ab0c (0x3874 bytes) 000081 1749541135.788710 punch from 0x014be6 to 0x017c0d (0x3028 bytes) 000082 1749541135.794024 MIRROR_RESYNC 3 mirrors 000083 1749541135.924305 mapread 0x00c941 thru 0x013efb (0x75bb bytes) 000084 1749541135.929440 MIRROR_RESYNC 3 mirrors 000085 1749541135.964239 fallocate 0x0331c8 thru 0x034492 (0x12cb bytes) 000086 1749541135.971102 mapread 0x008054 thru 0x014f9c (0xcf49 bytes) 000088 1749541135.976983 fallocate 0x01a8f1 thru 0x029abe (0xf1ce bytes) 000089 1749541135.978569 write 0x033e4c thru 0x0341ef (0x3a4 bytes) 000091 1749541135.979551 fallocate 0x01eceb thru 0x02890e (0x9c24 bytes) 000092 1749541135.980879 trunc from 0x0341f0 to 0x019955 (0x1a89a bytes) 000093 1749541135.989885 MIRROR_EXTEND to 4 mirrors 000094 1749541136.087371 fallocate 0x013bfd thru 0x01e26c (0xa670 bytes) 000095 1749541136.098988 fallocate 0x00f369 thru 0x019285 (0x9f1d bytes) 000096 1749541136.100705 trunc from 0x019956 to 0x00281f (0x17136 bytes) 000097 1749541136.106928 read 0x00199b thru 0x00281f (0xe85 bytes) 000098 1749541136.108936 trunc from 0x001cc4 to 0x00281f (0xb5c bytes) 000099 1749541136.110890 trunc from 0x00184f to 0x00281f (0xfd1 bytes) 000100 1749541136.112450 fallocate 0x02c407 thru 0x03895d (0xc557 bytes) 000101 1749541136.114258 read 0x0034db thru 0x00f6ce (0xc1f4 bytes) truncating to largest ever: 0x3ec6e 000102 1749541136.118475 trunc from 0x03895e to 0x03ec6d (0x6310 bytes) 000103 1749541136.122191 trunc from 0x018ab1 to 0x027b26 (0xf076 bytes) 000104 1749541136.123752 trunc from 0x03ec6e to 0x01341c (0x2b851 bytes) 000105 1749541136.127741 mapread 0x010ec4 thru 0x01341c (0x2559 bytes) 000106 1749541136.128437 trunc from 0x01341d to 0x028e0d (0x159f1 bytes) 000107 1749541136.133530 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x9b1:0x0]: Invalid argument (22) 000108 1749541136.178799 mapread 0x012916 thru 0x01fe14 (0xd4ff bytes) 000109 1749541136.184162 read 0x00733e thru 0x0122fb (0xafbe bytes) 000110 1749541136.185841 punch from 0x00b745 to 0x00e813 (0x30cf bytes) 000111 1749541136.187114 trunc from 0x028e0e to 0x00d42b (0x1b9e2 bytes) 000112 1749541136.191194 fallocate 0x02def1 thru 0x03badb (0xdbeb bytes) 000113 1749541136.193769 MIRROR_EXTEND to 4 mirrors 000114 1749541136.290631 mapread 0x007b7a thru 0x00d42b (0x58b2 bytes) 000116 1749541136.297119 read 0x009398 thru 0x00d42b (0x4094 bytes) 000117 1749541136.297383 punch from 0x00b7fc to 0x00d42b (0x1c30 bytes) 000119 1749541136.307786 fallocate 0x033a9c thru 0x03d9e5 (0x9f4a bytes) 000120 1749541136.309406 mapwrite 0x0253cf thru 0x034bf2 (0xf824 bytes) 000121 1749541136.335665 mapread 0x0124c2 thru 0x01728f (0x4dce bytes) 000122 1749541136.338637 mapwrite 0x02d7af thru 0x031a53 (0x42a5 bytes) 000123 1749541136.344057 trunc from 0x00f002 to 0x01a84e (0xb84d bytes) 000124 1749541136.347297 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x9b1:0x0]: Invalid argument (22) 000126 1749541136.405763 punch from 0x00d47f to 0x00d92c (0x4ae bytes) 000127 1749541136.410081 trunc from 0x00939d to 0x00be2d (0x2a91 bytes) 000128 1749541136.411676 read 0x014b78 thru 0x019810 (0x4c99 bytes) 000129 1749541136.414498 punch from 0x015833 to 0x025351 (0xfb1f bytes) 000130 1749541136.416819 trunc from 0x00c10c to 0x011397 (0x528c bytes) 000131 1749541136.418617 trunc from 0x034180 to 0x034bf2 (0xa73 bytes) 000132 1749541136.419940 trunc from 0x034bf3 to 0x0316c4 (0x352e bytes) 000133 1749541136.425835 read 0x025c7c thru 0x02c092 (0x6417 bytes) 000134 1749541136.429574 write 0x030804 thru 0x03b31a (0xab17 bytes) 000135 1749541136.432612 MIRROR_RESYNC 3 mirrors 000136 1749541136.591421 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x9b1:0x0]: Invalid argument (22) 000137 1749541136.654880 trunc from 0x01198f to 0x019669 (0x7cdb bytes) 000138 1749541136.664002 fallocate 0x005085 thru 0x006e77 (0x1df3 bytes) 000139 1749541136.665568 mapread 0x0129b8 thru 0x01d3f1 (0xaa3a bytes) 000140 1749541136.672013 mapwrite 0x034b57 thru 0x03a89b (0x5d45 bytes) 000141 1749541136.681313 punch from 0x01d360 to 0x022789 (0x542a bytes) 000142 1749541136.683844 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x9b1:0x0]: Invalid argument (22) 000143 1749541136.717782 mapread 0x01b2cf thru 0x01c12f (0xe61 bytes) 000144 1749541136.721114 trunc from 0x011564 to 0x0149cf (0x346c bytes) 000145 1749541136.722555 trunc from 0x03b31b to 0x03661b (0x4cff bytes) 000147 1749541136.731802 mapwrite 0x00d168 thru 0x019291 (0xc12a bytes) 000150 1749541136.744774 punch from 0x01f795 to 0x022a7e (0x32ea bytes) 000151 1749541136.746356 read 0x01b093 thru 0x0257e7 (0xa755 bytes) 000152 1749541136.749839 MIRROR_EXTEND to 2 mirrors 000154 1749541136.861940 read 0x000d4f thru 0x00705b (0x630d bytes) 000155 1749541136.864019 punch from 0x00d74d to 0x014c5e (0x7512 bytes) 000159 1749541136.873826 mapread 0x0058ae thru 0x014bad (0xf300 bytes) 000160 1749541136.881379 mapread 0x00b85a thru 0x010e4f (0x55f6 bytes) 000161 1749541136.881788 read 0x0265d6 thru 0x034926 (0xe351 bytes) 000162 1749541136.884866 read 0x01e5bd thru 0x0260ab (0x7aef bytes) 000163 1749541136.885339 mapread 0x02d5c5 thru 0x03661b (0x9057 bytes) 000164 1749541136.885952 mapread 0x01174e thru 0x01307a (0x192d bytes) 000165 1749541136.886510 mapwrite 0x02feb0 thru 0x03392a (0x3a7b bytes) 000167 1749541136.893623 mapread 0x022bde thru 0x02c38f (0x97b2 bytes) 000168 1749541136.894115 fallocate 0x027952 thru 0x02b398 (0x3a47 bytes) 000170 1749541136.896525 mapwrite 0x032240 thru 0x039322 (0x70e3 bytes) 000171 1749541136.917865 write 0x035b13 thru 0x03b139 (0x5627 bytes) 000172 1749541136.918590 trunc from 0x03b13a to 0x019db0 (0x21389 bytes) 000173 1749541136.923571 punch from 0x004688 to 0x0088ad (0x4226 bytes) 000174 1749541136.926376 punch from 0x019830 to 0x019db0 (0x581 bytes) 000175 1749541136.928579 fallocate 0x002f83 thru 0x004774 (0x17f2 bytes) 000178 1749541136.931383 mapread 0x008575 thru 0x00d4e5 (0x4f71 bytes) 000180 1749541136.933816 read 0x0196b5 thru 0x019db0 (0x6fc bytes) 000181 1749541136.936992 mapwrite 0x01315f thru 0x022441 (0xf2e3 bytes) 000182 1749541136.951186 trunc from 0x005050 to 0x010a15 (0xb9c6 bytes) 000183 1749541136.953013 mapread 0x0110fd thru 0x01501a (0x3f1e bytes) 000184 1749541136.953626 punch from 0x01db7f to 0x022441 (0x48c3 bytes) fallocating to largest ever: 0x40000 000185 1749541136.954998 fallocate 0x036206 thru 0x03ffff (0x9dfa bytes) 000186 1749541136.956472 trunc from 0x040000 to 0x030990 (0xf66f bytes) 000187 1749541136.960004 trunc from 0x030991 to 0x01bbd3 (0x14dbd bytes) 000188 1749541136.963867 trunc from 0x01bbd4 to 0x01b2d6 (0x8fd bytes) 000189 1749541136.967505 trunc from 0x003e0e to 0x00cd22 (0x8f15 bytes) 000190 1749541136.969017 mapwrite 0x01f599 thru 0x02e302 (0xed6a bytes) 000191 1749541136.984595 trunc from 0x02e303 to 0x00562e (0x28cd4 bytes) 000192 1749541136.989047 MIRROR_RESYNC 2 mirrors 000193 1749541137.075003 fallocate 0x035ae5 thru 0x03aa70 (0x4f8c bytes) 000194 1749541137.084339 write 0x016964 thru 0x023554 (0xcbf1 bytes) 000195 1749541137.085487 write 0x03754b thru 0x03ffff (0x8ab5 bytes) 000196 1749541137.087647 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x9b1:0x0]: Invalid argument (22) 000197 1749541137.121938 fallocate 0x011b5a thru 0x019c95 (0x813c bytes) 000198 1749541137.123620 write 0x02921a thru 0x02fdc8 (0x6baf bytes) 000199 1749541137.127233 mapread 0x006ab8 thru 0x009ff0 (0x3539 bytes) 000200 1749541137.130379 mapwrite 0x00fb01 thru 0x0147d9 (0x4cd9 bytes) 000201 1749541137.139103 trunc from 0x040000 to 0x0349e9 (0xb616 bytes) 000202 1749541137.143065 mapwrite 0x03a929 thru 0x03ffff (0x56d7 bytes) 000203 1749541137.154648 mapwrite 0x0135ff thru 0x02140c (0xde0e bytes) 000204 1749541137.162746 write 0x002505 thru 0x00f83b (0xd337 bytes) 000205 1749541137.163709 read 0x03e71c thru 0x03ffff (0x18e4 bytes) 000206 1749541137.163998 mapread 0x007f42 thru 0x011e75 (0x9f34 bytes) 000207 1749541137.164443 fallocate 0x015051 thru 0x01badd (0x6a8d bytes) 000208 1749541137.166061 trunc from 0x00e26e to 0x01637a (0x810d bytes) 000210 1749541137.170102 read 0x024b01 thru 0x032094 (0xd594 bytes) 000211 1749541137.172724 mapwrite 0x010699 thru 0x01bd2c (0xb694 bytes) 000213 1749541137.178699 fallocate 0x026360 thru 0x028155 (0x1df6 bytes) 000214 1749541137.179917 mapread 0x03de6d thru 0x03ffff (0x2193 bytes) 000216 1749541137.180474 write 0x039db2 thru 0x03ffff (0x624e bytes) 000217 1749541137.180978 write 0x011cfb thru 0x021b4f (0xfe55 bytes) 000218 1749541137.181641 fallocate 0x03d5f2 thru 0x03ffff (0x2a0e bytes) 000219 1749541137.182828 mapwrite 0x018a24 thru 0x01aa4f (0x202c bytes) 000220 1749541137.187930 trunc from 0x035a45 to 0x03ffff (0xa5bb bytes) 000221 1749541137.189221 write 0x03d0ce thru 0x03ffff (0x2f32 bytes) 000222 1749541137.190771 mapread 0x03a83b thru 0x03ffff (0x57c5 bytes) 000223 1749541137.194292 write 0x010b43 thru 0x01cb26 (0xbfe4 bytes) 000224 1749541137.195240 trunc from 0x036fb4 to 0x03b1e8 (0x4235 bytes) 000225 1749541137.196483 trunc from 0x040000 to 0x01bab3 (0x2454c bytes) 000227 1749541137.200550 trunc from 0x00a533 to 0x0122d1 (0x7d9f bytes) 000228 1749541137.203928 punch from 0x012669 to 0x01bab3 (0x944b bytes) 000229 1749541137.205111 read 0x018e8e thru 0x01bab3 (0x2c26 bytes) 000230 1749541137.207406 punch from 0x01349f to 0x01bab3 (0x8615 bytes) 000231 1749541137.208698 fallocate 0x03ab9f thru 0x03ffff (0x5461 bytes) 000232 1749541137.210183 punch from 0x019a42 to 0x01bab3 (0x2072 bytes) 000234 1749541137.211954 read 0x00fca0 thru 0x0185fd (0x895e bytes) 000235 1749541137.214045 mapwrite 0x0033e4 thru 0x005cd6 (0x28f3 bytes) 000236 1749541137.218529 fallocate 0x023f87 thru 0x027307 (0x3381 bytes) 000237 1749541137.220119 read 0x01c14e thru 0x0261ff (0xa0b2 bytes) 000238 1749541137.222310 punch from 0x00f964 to 0x01efab (0xf648 bytes) 000239 1749541137.223648 fallocate 0x029209 thru 0x0350e0 (0xbed8 bytes) 000240 1749541137.225018 punch from 0x00f049 to 0x01cef2 (0xdeaa bytes) 000241 1749541137.226286 punch from 0x00f92e to 0x012448 (0x2b1b bytes) 000242 1749541137.227644 write 0x03d93f thru 0x03ffff (0x26c1 bytes) 000243 1749541137.228161 write 0x011e6b thru 0x01f788 (0xd91e bytes) 000244 1749541137.231259 fallocate 0x02871a thru 0x033c32 (0xb519 bytes) 000245 1749541137.232726 trunc from 0x006a30 to 0x00fa47 (0x9018 bytes) 000246 1749541137.234205 trunc from 0x040000 to 0x01b592 (0x24a6d bytes) 000247 1749541137.238579 mapwrite 0x01d6f2 thru 0x02d642 (0xff51 bytes) 000249 1749541137.267415 punch from 0x01ffff to 0x0201cd (0x1cf bytes) 000251 1749541137.269427 mapread 0x000359 thru 0x00663d (0x62e5 bytes) 000252 1749541137.271286 trunc from 0x01f2dd to 0x02b840 (0xc564 bytes) 000253 1749541137.272852 mapwrite 0x01f0fa thru 0x020742 (0x1649 bytes) 000254 1749541137.278009 punch from 0x028364 to 0x02d642 (0x52df bytes) 000255 1749541137.279133 trunc from 0x02d643 to 0x02d2ee (0x354 bytes) 000256 1749541137.282218 read 0x005b81 thru 0x00e55b (0x89db bytes) 000257 1749541137.286234 MIRROR_EXTEND to 2 mirrors 000258 1749541137.391300 trunc from 0x001215 to 0x00768f (0x647b bytes) 000259 1749541137.400317 read 0x00954a thru 0x01542d (0xbee4 bytes) 000260 1749541137.403023 fallocate 0x006e6b thru 0x0146e5 (0xd87b bytes) 000261 1749541137.404468 punch from 0x00e802 to 0x0130e0 (0x48df bytes) 000262 1749541137.406297 read 0x0027f6 thru 0x00b30e (0x8b19 bytes) 000263 1749541137.410527 trunc from 0x018af0 to 0x01b35f (0x2870 bytes) 000264 1749541137.412276 read 0x02c5f7 thru 0x02d2ee (0xcf8 bytes) 000265 1749541137.412542 trunc from 0x00de26 to 0x01b8fd (0xdad8 bytes) 000266 1749541137.414282 read 0x01dcfb thru 0x0267da (0x8ae0 bytes) 000268 1749541137.414846 trunc from 0x013b9d to 0x01922d (0x5691 bytes) 000269 1749541137.416253 trunc from 0x01ada9 to 0x02268e (0x78e6 bytes) 000270 1749541137.419311 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x9b1:0x0]: Invalid argument (22) 000271 1749541137.453059 mapwrite 0x0226e3 thru 0x02bde8 (0x9706 bytes) 000272 1749541137.464792 trunc from 0x02d2ef to 0x03eca0 (0x119b2 bytes) 000274 1749541137.474111 punch from 0x034307 to 0x03eca0 (0xa99a bytes) 000276 1749541137.475620 trunc from 0x033b1b to 0x03bb08 (0x7fee bytes) 000277 1749541137.476904 write 0x026ca3 thru 0x02f002 (0x8360 bytes) 000278 1749541137.478804 fallocate 0x00c840 thru 0x00cf61 (0x722 bytes) 000281 1749541137.480995 mapwrite 0x034311 thru 0x03f5e1 (0xb2d1 bytes) 000282 1749541137.494888 trunc from 0x03f5e2 to 0x02a1f5 (0x153ec bytes) 000283 1749541137.499361 mapwrite 0x0294ee thru 0x02d9e5 (0x44f8 bytes) 000284 1749541137.511457 read 0x012106 thru 0x01b8a6 (0x97a1 bytes) 000285 1749541137.514102 mapwrite 0x028a0e thru 0x033bc1 (0xb1b4 bytes) 000286 1749541137.529497 punch from 0x00231d to 0x005457 (0x313b bytes) 000287 1749541137.531292 mapwrite 0x009bb6 thru 0x00c001 (0x244c bytes) 000288 1749541137.538430 trunc from 0x002fa4 to 0x0054b8 (0x2515 bytes) 000289 1749541137.540170 fallocate 0x01e7a7 thru 0x02428c (0x5ae6 bytes) 000292 1749541137.544771 MIRROR_EXTEND to 2 mirrors 000293 1749541137.647384 fallocate 0x02ec9c thru 0x039093 (0xa3f8 bytes) 000294 1749541137.657011 fallocate 0x01b91d thru 0x02472b (0x8e0f bytes) 000296 1749541137.659148 mapread 0x008ac1 thru 0x013f3f (0xb47f bytes) 000297 1749541137.669025 trunc from 0x033bc2 to 0x02abb4 (0x900d bytes) 000298 1749541137.681416 mapread 0x0066d5 thru 0x00af35 (0x4861 bytes) 000299 1749541137.682016 trunc from 0x01d004 to 0x02abb4 (0xdbb1 bytes) 000300 1749541137.683564 trunc from 0x02abb5 to 0x025205 (0x59af bytes) 000301 1749541137.687659 trunc from 0x020d73 to 0x025205 (0x4493 bytes) 000302 1749541137.688862 mapread 0x0098c2 thru 0x016ac3 (0xd202 bytes) 000303 1749541137.689318 mapread 0x01ab0c thru 0x025205 (0xa6fa bytes) 000304 1749541137.692313 punch from 0x00c8ac to 0x00de42 (0x1597 bytes) 000305 1749541137.693554 mapwrite 0x006cac thru 0x00d20a (0x655f bytes) 000306 1749541137.700723 read 0x01e1de thru 0x025205 (0x7028 bytes) 000307 1749541137.701072 mapwrite 0x03f369 thru 0x03ffff (0xc97 bytes) 000308 1749541137.712438 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x9b1:0x0]: Invalid argument (22) 000309 1749541137.744880 read 0x02c54d thru 0x038db1 (0xc865 bytes) 000310 1749541137.747434 mapwrite 0x009dca thru 0x010137 (0x636e bytes) 000311 1749541137.759692 read 0x034233 thru 0x03ffff (0xbdcd bytes) 000312 1749541137.762997 mapread 0x0209a6 thru 0x02b9b3 (0xb00e bytes) 000313 1749541137.763627 punch from 0x03b7be to 0x03ffff (0x4842 bytes) 000314 1749541137.765047 mapwrite 0x0302e3 thru 0x03e594 (0xe2b2 bytes) 000315 1749541137.774581 mapwrite 0x01915c thru 0x01f69c (0x6541 bytes) 000317 1749541137.780247 read 0x0217f0 thru 0x02f354 (0xdb65 bytes) 000318 1749541137.780748 write 0x0080c4 thru 0x00f9ad (0x78ea bytes) 000319 1749541137.783163 mapwrite 0x007185 thru 0x010808 (0x9684 bytes) 000320 1749541137.788563 mapread 0x0162a2 thru 0x01f1ee (0x8f4d bytes) 000321 1749541137.789162 trunc from 0x040000 to 0x01594d (0x2a6b2 bytes) 000322 1749541137.793188 punch from 0x00e503 to 0x01594d (0x744b bytes) 000323 1749541137.795422 read 0x000450 thru 0x00b1c2 (0xad73 bytes) 000324 1749541137.795729 fallocate 0x00be37 thru 0x01a360 (0xe52a bytes) 000325 1749541137.797054 punch from 0x00b6c1 to 0x013317 (0x7c57 bytes) 000327 1749541137.798902 mapwrite 0x00e274 thru 0x00f0b4 (0xe41 bytes) 000328 1749541137.805722 write 0x00bd6d thru 0x016a91 (0xad25 bytes) 000329 1749541137.807982 MIRROR_EXTEND to 2 mirrors 000330 1749541137.886881 write 0x034693 thru 0x03ffff (0xb96d bytes) 000331 1749541137.897856 MIRROR_RESYNC 2 mirrors 000332 1749541137.985360 trunc from 0x00108b to 0x0098f2 (0x8868 bytes) 000333 1749541137.997664 fallocate 0x0137ef thru 0x0219c1 (0xe1d3 bytes) 000336 1749541137.999854 trunc from 0x040000 to 0x019808 (0x267f7 bytes) 000337 1749541138.014147 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x9b1:0x0]: Invalid argument (22) 000338 1749541138.053214 read 0x002e9b thru 0x0092ea (0x6450 bytes) 000339 1749541138.056417 trunc from 0x019809 to 0x018a8e (0xd7a bytes) 000341 1749541138.062357 read 0x015170 thru 0x017907 (0x2798 bytes) 000342 1749541138.065578 mapread 0x00b15d thru 0x013394 (0x8238 bytes) 000343 1749541138.066149 trunc from 0x00b8b3 to 0x00dd48 (0x2496 bytes) 000344 1749541138.067821 read 0x017031 thru 0x018a8e (0x1a5e bytes) 000345 1749541138.068193 write 0x025c7a thru 0x029782 (0x3b09 bytes) 000347 1749541138.068931 fallocate 0x0114fd thru 0x013626 (0x212a bytes) 000348 1749541138.070850 write 0x0124d5 thru 0x0204f3 (0xe01f bytes) 000349 1749541138.073609 fallocate 0x007bf6 thru 0x00b71b (0x3b26 bytes) 000350 1749541138.075502 punch from 0x01603b to 0x01d372 (0x7338 bytes) 000351 1749541138.079588 mapwrite 0x0023d6 thru 0x0043ad (0x1fd8 bytes) 000352 1749541138.084560 trunc from 0x029783 to 0x01a2cf (0xf4b3 bytes) 000353 1749541138.088833 trunc from 0x01a2d0 to 0x00cb2e (0xd7a1 bytes) 000354 1749541138.092230 write 0x031743 thru 0x03ee32 (0xd6f0 bytes) 000355 1749541138.093219 punch from 0x01a76c to 0x02463e (0x9ed3 bytes) 000356 1749541138.094709 fallocate 0x03b5c6 thru 0x03ffff (0x4a3a bytes) 000358 1749541138.096620 write 0x02cca0 thru 0x03bf20 (0xf281 bytes) 000359 1749541138.100288 trunc from 0x040000 to 0x02f429 (0x10bd6 bytes) 000362 1749541138.105498 mapread 0x0038c6 thru 0x0050f0 (0x182b bytes) 000363 1749541138.106145 mapwrite 0x034d89 thru 0x03d7e3 (0x8a5b bytes) 000364 1749541138.127209 MIRROR_EXTEND to 2 mirrors 000365 1749541138.226182 fallocate 0x015b50 thru 0x01b66e (0x5b1f bytes) 000366 1749541138.235307 mapwrite 0x00bd8a thru 0x018637 (0xc8ae bytes) 000367 1749541138.252338 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x9b1:0x0]: Invalid argument (22) 000368 1749541138.287052 mapread 0x00ec46 thru 0x01ca69 (0xde24 bytes) 000369 1749541138.294797 punch from 0x00c034 to 0x010877 (0x4844 bytes) 000370 1749541138.297589 mapread 0x018339 thru 0x01a42c (0x20f4 bytes) 000371 1749541138.298111 trunc from 0x03d7e4 to 0x014e6f (0x28974 bytes) 000372 1749541138.310116 trunc from 0x014e70 to 0x0057cb (0xf6a4 bytes) 000373 1749541138.314143 punch from 0x000185 to 0x0057cb (0x5647 bytes) 000374 1749541138.315623 write 0x030443 thru 0x03804b (0x7c09 bytes) 000375 1749541138.316587 trunc from 0x02dced to 0x0366de (0x89f2 bytes) 000376 1749541138.322344 MIRROR_EXTEND to 2 mirrors 000377 1749541138.422660 write 0x03da25 thru 0x03ffff (0x25db bytes) 000378 1749541138.430569 mapwrite 0x01af7d thru 0x01f1a6 (0x422a bytes) 000379 1749541138.440403 mapwrite 0x024c03 thru 0x032e2a (0xe228 bytes) 000381 1749541138.451106 fallocate 0x03fd08 thru 0x03ffff (0x2f8 bytes) 000382 1749541138.452980 read 0x00f8de thru 0x016ff1 (0x7714 bytes) 000383 1749541138.455590 trunc from 0x040000 to 0x031f4a (0xe0b5 bytes) 000384 1749541138.465692 read 0x002fa2 thru 0x007a33 (0x4a92 bytes) 000385 1749541138.468931 fallocate 0x0031eb thru 0x00c375 (0x918b bytes) 000386 1749541138.470763 trunc from 0x031f4b to 0x01a006 (0x17f44 bytes) 000388 1749541138.475977 mapread 0x0104da thru 0x019fb4 (0x9adb bytes) 000389 1749541138.476681 trunc from 0x0165ee to 0x01a006 (0x3a19 bytes) 000390 1749541138.478985 mapwrite 0x015727 thru 0x01693d (0x1217 bytes) 000391 1749541138.485640 read 0x00a316 thru 0x01882b (0xe516 bytes) 000392 1749541138.488397 mapwrite 0x00edab thru 0x01c2b3 (0xd509 bytes) 000393 1749541138.502766 MIRROR_RESYNC 2 mirrors 000394 1749541138.612842 fallocate 0x027286 thru 0x0310ec (0x9e67 bytes) 000395 1749541138.623037 MIRROR_EXTEND to 3 mirrors 000397 1749541138.734707 write 0x03815a thru 0x03ffff (0x7ea6 bytes) 000398 1749541138.744026 fallocate 0x00eb03 thru 0x0185bb (0x9ab9 bytes) 000399 1749541138.745650 trunc from 0x040000 to 0x0282c6 (0x17d39 bytes) 000400 1749541138.754250 mapwrite 0x02b1d0 thru 0x02ca8c (0x18bd bytes) 000401 1749541138.764568 trunc from 0x02ca8d to 0x010e48 (0x1bc44 bytes) 000402 1749541138.768407 fallocate 0x017fcb thru 0x022172 (0xa1a8 bytes) 000403 1749541138.769996 fallocate 0x001bfb thru 0x00ca62 (0xae68 bytes) 000404 1749541138.771470 punch from 0x016d65 to 0x01a7b8 (0x3a54 bytes) 000405 1749541138.772793 punch from 0x0147de to 0x021e7c (0xd69f bytes) 000406 1749541138.774232 write 0x01cc0b thru 0x028912 (0xbd08 bytes) 000407 1749541138.777105 mapwrite 0x015760 thru 0x023c79 (0xe51a bytes) 000408 1749541138.784548 fallocate 0x022559 thru 0x023a15 (0x14bd bytes) 000409 1749541138.786029 mapwrite 0x0302b1 thru 0x039a1c (0x976c bytes) 000410 1749541138.799000 write 0x03ff42 thru 0x03ffff (0xbe bytes) 000411 1749541138.799566 mapwrite 0x03772d thru 0x039549 (0x1e1d bytes) 000412 1749541138.804972 mapread 0x031323 thru 0x032d56 (0x1a34 bytes) 000413 1749541138.806890 MIRROR_EXTEND to 4 mirrors 000414 1749541138.919274 mapread 0x0148ae thru 0x019fb9 (0x570c bytes) 000415 1749541138.923531 write 0x01bd03 thru 0x022dcf (0x70cd bytes) 000416 1749541138.934193 read 0x02f4f5 thru 0x037ad9 (0x85e5 bytes) 000417 1749541138.936184 trunc from 0x0150c5 to 0x01711b (0x2057 bytes) 000418 1749541138.937762 trunc from 0x040000 to 0x01981e (0x267e1 bytes) 000419 1749541138.947253 MIRROR_RESYNC 4 mirrors 000421 1749541139.128868 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x9b1:0x0]: Invalid argument (22) 000422 1749541139.181839 mapwrite 0x03a32c thru 0x03ffff (0x5cd4 bytes) 000423 1749541139.201677 trunc from 0x005ff7 to 0x00b195 (0x519f bytes) 000425 1749541139.204577 trunc from 0x040000 to 0x03aef1 (0x510e bytes) 000426 1749541139.211677 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x9b1:0x0]: Invalid argument (22) 000427 1749541139.262898 fallocate 0x0144b7 thru 0x01674c (0x2296 bytes) 000429 1749541139.264725 mapread 0x00b778 thru 0x013fe8 (0x8871 bytes) 000430 1749541139.270017 fallocate 0x0061c4 thru 0x008f59 (0x2d96 bytes) 000431 1749541139.271990 mapread 0x01402c thru 0x017793 (0x3768 bytes) 000432 1749541139.274559 trunc from 0x03aef2 to 0x02df54 (0xcf9d bytes) 000433 1749541139.280148 fallocate 0x02fbe3 thru 0x03513f (0x555d bytes) 000435 1749541139.283010 trunc from 0x00f710 to 0x019cd6 (0xa5c7 bytes) 000437 1749541139.287760 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x9b1:0x0]: Invalid argument (22) 000438 1749541139.318909 mapread 0x0118b1 thru 0x020d62 (0xf4b2 bytes) 000439 1749541139.325853 trunc from 0x02df55 to 0x0308af (0x295b bytes) 000440 1749541139.331116 write 0x0165e9 thru 0x01ce47 (0x685f bytes) 000441 1749541139.331914 trunc from 0x0308b0 to 0x000a08 (0x2fea7 bytes) 000442 1749541139.336706 trunc from 0x0000c0 to 0x000a08 (0x949 bytes) 000443 1749541139.337975 fallocate 0x035616 thru 0x03c945 (0x7330 bytes) 000444 1749541139.340017 mapwrite 0x013047 thru 0x016e06 (0x3dc0 bytes) 000445 1749541139.353773 punch from 0x001fc3 to 0x009d85 (0x7dc3 bytes) 000446 1749541139.355303 fallocate 0x0342a1 thru 0x039a0f (0x576f bytes) 000447 1749541139.356914 punch from 0x015cad to 0x016e06 (0x115a bytes) 000448 1749541139.358561 read 0x0002be thru 0x004a5f (0x47a2 bytes) 000449 1749541139.361605 punch from 0x0135ea to 0x016e06 (0x381d bytes) 000450 1749541139.364497 mapwrite 0x023396 thru 0x02caa4 (0x970f bytes) 000451 1749541139.380361 trunc from 0x02caa5 to 0x03fca3 (0x131ff bytes) 000453 1749541139.384588 write 0x03c2cb thru 0x03ffff (0x3d35 bytes) 000454 1749541139.387148 write 0x028918 thru 0x033889 (0xaf72 bytes) 000456 1749541139.389402 punch from 0x038d5e to 0x03ffff (0x72a2 bytes) 000457 1749541139.393976 write 0x014248 thru 0x0157d3 (0x158c bytes) 000458 1749541139.397085 trunc from 0x040000 to 0x0313e2 (0xec1d bytes) 000459 1749541139.401999 trunc from 0x0313e3 to 0x03ade9 (0x9a07 bytes) 000460 1749541139.406932 read 0x01a2c5 thru 0x01e3a0 (0x40dc bytes) 000462 1749541139.407764 mapread 0x02c734 thru 0x02fa8f (0x335c bytes) 000463 1749541139.408439 fallocate 0x03da62 thru 0x03ffff (0x259e bytes) 000464 1749541139.410272 fallocate 0x022eab thru 0x0245f8 (0x174e bytes) 000465 1749541139.411903 punch from 0x012b57 to 0x02284d (0xfcf7 bytes) 000466 1749541139.415050 fallocate 0x009ab1 thru 0x014117 (0xa667 bytes) 000467 1749541139.417069 read 0x00edc3 thru 0x01475c (0x599a bytes) 000468 1749541139.420372 trunc from 0x03adea to 0x03d90f (0x2b26 bytes) 000469 1749541139.425669 mapread 0x01b7c1 thru 0x01cb66 (0x13a6 bytes) 000470 1749541139.426296 mapwrite 0x010cce thru 0x01abbf (0x9ef2 bytes) 000472 1749541139.434995 read 0x033c3d thru 0x034596 (0x95a bytes) 000473 1749541139.435269 write 0x0337b9 thru 0x035adb (0x2323 bytes) 000474 1749541139.435682 punch from 0x015459 to 0x024ed6 (0xfa7e bytes) 000475 1749541139.437562 trunc from 0x01296e to 0x01a181 (0x7814 bytes) 000479 1749541139.441122 trunc from 0x03d910 to 0x018bcf (0x24d40 bytes) 000480 1749541139.446041 mapread 0x01794d thru 0x018bcf (0x1283 bytes) 000481 1749541139.448991 mapwrite 0x018409 thru 0x01c9ef (0x45e7 bytes) 000482 1749541139.459635 fallocate 0x035d0f thru 0x03ffff (0xa2f1 bytes) 000483 1749541139.461587 trunc from 0x01c9f0 to 0x00ee5f (0xdb90 bytes) 000484 1749541139.465863 write 0x0151d5 thru 0x01a0f9 (0x4f25 bytes) 000485 1749541139.466623 write 0x013380 thru 0x01d116 (0x9d97 bytes) 000488 1749541139.469734 punch from 0x0063d2 to 0x009f64 (0x3b93 bytes) 000489 1749541139.473039 MIRROR_EXTEND to 2 mirrors 000490 1749541139.582961 mapwrite 0x00d1b5 thru 0x00f686 (0x24d2 bytes) 000494 1749541139.614192 trunc from 0x01bb82 to 0x01d116 (0x1595 bytes) 000496 1749541139.616501 fallocate 0x0111a1 thru 0x012711 (0x1571 bytes) 000497 1749541139.619106 MIRROR_RESYNC 2 mirrors 000499 1749541139.746077 mapwrite 0x03dcdb thru 0x03ffff (0x2325 bytes) 000500 1749541139.767114 trunc from 0x040000 to 0x02f3ad (0x10c52 bytes) 000501 1749541139.771572 read 0x02ba83 thru 0x02f3ad (0x392b bytes) 000503 1749541139.774348 write 0x0357fe thru 0x03f9f5 (0xa1f8 bytes) 000504 1749541139.775269 punch from 0x01f848 to 0x01f9dd (0x196 bytes) 000505 1749541139.778220 MIRROR_RESYNC 2 mirrors 000506 1749541139.889652 punch from 0x01e6b7 to 0x02c7e2 (0xe12c bytes) 000507 1749541139.899718 mapread 0x0342e8 thru 0x03a000 (0x5d19 bytes) 000508 1749541139.906479 fallocate 0x023823 thru 0x023d36 (0x514 bytes) 000509 1749541139.908500 mapread 0x013b99 thru 0x01cda5 (0x920d bytes) 000510 1749541139.914968 fallocate 0x01056d thru 0x01465e (0x40f2 bytes) 000511 1749541139.917271 fallocate 0x001ed8 thru 0x0106ae (0xe7d7 bytes) 000512 1749541139.919256 mapread 0x034295 thru 0x03afc2 (0x6d2e bytes) 000513 1749541139.919920 fallocate 0x016cfc thru 0x018763 (0x1a68 bytes) 000514 1749541139.921484 fallocate 0x02dd7e thru 0x036401 (0x8684 bytes) 000515 1749541139.923430 write 0x01aa48 thru 0x02a1ed (0xf7a6 bytes) 000516 1749541139.926070 mapwrite 0x01e673 thru 0x028bd8 (0xa566 bytes) 000517 1749541139.934654 trunc from 0x01d7c1 to 0x02a864 (0xd0a4 bytes) 000518 1749541139.936653 punch from 0x038a18 to 0x03ee88 (0x6471 bytes) 000519 1749541139.937970 read 0x03e13b thru 0x03f9f5 (0x18bb bytes) 000520 1749541139.939126 mapwrite 0x02236e thru 0x030d57 (0xe9ea bytes) 000521 1749541139.947680 MIRROR_EXTEND to 3 mirrors 000522 1749541140.062278 read 0x00ccd0 thru 0x0192e8 (0xc619 bytes) 000523 1749541140.066426 mapread 0x01c19d thru 0x0204f8 (0x435c bytes) 000525 1749541140.075780 write 0x007c33 thru 0x0129d7 (0xada5 bytes) 000526 1749541140.087843 write 0x025a95 thru 0x0286c7 (0x2c33 bytes) 000527 1749541140.091124 mapread 0x02fb58 thru 0x03752b (0x79d4 bytes) 000528 1749541140.095146 mapwrite 0x03afda thru 0x03ddd8 (0x2dff bytes) 000529 1749541140.100901 mapwrite 0x03b4a6 thru 0x03b79a (0x2f5 bytes) 000530 1749541140.103996 fallocate 0x022128 thru 0x02e242 (0xc11b bytes) 000531 1749541140.105526 mapwrite 0x02785f thru 0x02c08a (0x482c bytes) 000532 1749541140.110349 read 0x0165e9 thru 0x023029 (0xca41 bytes) 000533 1749541140.112638 trunc from 0x03f9f6 to 0x003830 (0x3c1c5 bytes) 000534 1749541140.118084 mapread 0x00357c thru 0x003830 (0x2b5 bytes) 000538 1749541140.121244 MIRROR_EXTEND to 4 mirrors 000540 1749541140.222299 trunc from 0x000e75 to 0x001c06 (0xd92 bytes) 000541 1749541140.232548 mapwrite 0x01c70b thru 0x021503 (0x4df9 bytes) 000543 1749541140.249878 write 0x023c44 thru 0x02a743 (0x6b00 bytes) 000544 1749541140.251798 MIRROR_RESYNC 4 mirrors 000545 1749541140.417304 mapread 0x0132c7 thru 0x01a278 (0x6fb2 bytes) 000546 1749541140.423204 fallocate 0x006e05 thru 0x00cd75 (0x5f71 bytes) 000547 1749541140.436060 mapwrite 0x003c9b thru 0x008e89 (0x51ef bytes) 000548 1749541140.445772 read 0x01cbe6 thru 0x01efb1 (0x23cc bytes) 000549 1749541140.447447 trunc from 0x018d15 to 0x01dbfc (0x4ee8 bytes) 000551 1749541140.449547 write 0x01f93b thru 0x02c934 (0xcffa bytes) 000552 1749541140.452926 mapwrite 0x017549 thru 0x01f1da (0x7c92 bytes) 000553 1749541140.463540 write 0x032f7e thru 0x038f55 (0x5fd8 bytes) 000554 1749541140.464251 punch from 0x024f86 to 0x02666c (0x16e7 bytes) 000555 1749541140.466496 punch from 0x0341dc to 0x038f55 (0x4d7a bytes) 000556 1749541140.470178 mapread 0x0297d8 thru 0x033ad6 (0xa2ff bytes) 000557 1749541140.472777 punch from 0x015362 to 0x018b7e (0x381d bytes) 000558 1749541140.474496 trunc from 0x01d3e8 to 0x021465 (0x407e bytes) 000559 1749541140.476032 punch from 0x0320af to 0x038f55 (0x6ea7 bytes) 000560 1749541140.477597 mapread 0x0043e6 thru 0x00ea06 (0xa621 bytes) 000561 1749541140.478235 trunc from 0x014051 to 0x01c024 (0x7fd4 bytes) 000562 1749541140.480473 mapwrite 0x02b5bc thru 0x032a28 (0x746d bytes) 000563 1749541140.490682 punch from 0x015a79 to 0x0227a0 (0xcd28 bytes) 000564 1749541140.492630 trunc from 0x038f56 to 0x03ac35 (0x1ce0 bytes) 000565 1749541140.505221 punch from 0x030227 to 0x0366ef (0x64c9 bytes) 000566 1749541140.506964 write 0x004425 thru 0x00e99e (0xa57a bytes) 000567 1749541140.508045 punch from 0x016c09 to 0x026635 (0xfa2d bytes) 000568 1749541140.510216 trunc from 0x0201fd to 0x022807 (0x260b bytes) 000569 1749541140.512021 mapwrite 0x01e608 thru 0x02b109 (0xcb02 bytes) 000570 1749541140.523620 fallocate 0x03082a thru 0x037de8 (0x75bf bytes) 000571 1749541140.527245 MIRROR_EXTEND to 5 mirrors 000572 1749541140.648181 trunc from 0x03ac36 to 0x035a8a (0x51ab bytes) 000573 1749541140.662735 mapwrite 0x023045 thru 0x025e58 (0x2e14 bytes) 000574 1749541140.671065 mapread 0x0098ff thru 0x00e733 (0x4e35 bytes) 000575 1749541140.673636 mapwrite 0x00ca18 thru 0x0124b6 (0x5a9f bytes) 000576 1749541140.679253 mapwrite 0x03de4b thru 0x03ffff (0x21b5 bytes) 000578 1749541140.691998 mapread 0x019eb4 thru 0x01bcf8 (0x1e45 bytes) 000579 1749541140.694790 mapread 0x032ed6 thru 0x03ffff (0xd12a bytes) 000581 1749541140.697785 mapread 0x012d7d thru 0x01c752 (0x99d6 bytes) 000582 1749541140.698243 trunc from 0x040000 to 0x015598 (0x2aa67 bytes) 000583 1749541140.702563 punch from 0x0116a5 to 0x015598 (0x3ef4 bytes) 000585 1749541140.704814 mapwrite 0x03d3ac thru 0x03ffff (0x2c54 bytes) 000586 1749541140.717147 mapread 0x01f63a thru 0x024575 (0x4f3c bytes) 000587 1749541140.720166 mapread 0x00a064 thru 0x01439f (0xa33c bytes) 000588 1749541140.722755 write 0x0334c0 thru 0x03b49c (0x7fdd bytes) 000589 1749541140.725771 punch from 0x03d5f7 to 0x03ffff (0x2a09 bytes) 000590 1749541140.727695 mapread 0x02284c thru 0x02a231 (0x79e6 bytes) 000591 1749541140.728218 trunc from 0x040000 to 0x00c073 (0x33f8c bytes) 000592 1749541140.733117 trunc from 0x004af3 to 0x00c073 (0x7581 bytes) 000593 1749541140.734842 mapread 0x0098cc thru 0x00c073 (0x27a8 bytes) 000594 1749541140.737220 fallocate 0x01090b thru 0x018bca (0x82c0 bytes) 000595 1749541140.738734 trunc from 0x006960 to 0x0072d4 (0x975 bytes) 000596 1749541140.740177 trunc from 0x00c074 to 0x02bab2 (0x1fa3f bytes) 000597 1749541140.744236 write 0x02598f thru 0x029330 (0x39a2 bytes) 000598 1749541140.746985 mapwrite 0x02474f thru 0x025e7b (0x172d bytes) 000599 1749541140.754026 MIRROR_EXTEND to 6 mirrors 000600 1749541140.878889 trunc from 0x01ab84 to 0x01ef0f (0x438c bytes) 000601 1749541140.891276 write 0x010216 thru 0x011eb0 (0x1c9b bytes) 000602 1749541140.895352 mapread 0x01d1b9 thru 0x01eccd (0x1b15 bytes) 000603 1749541140.898777 mapread 0x007a23 thru 0x010bdf (0x91bd bytes) 000604 1749541140.905418 punch from 0x00f64b to 0x015f09 (0x68bf bytes) 000605 1749541140.910482 write 0x000c83 thru 0x0041c9 (0x3547 bytes) 000606 1749541140.914130 fallocate 0x0113a2 thru 0x0119d4 (0x633 bytes) 000607 1749541140.916444 mapread 0x001d7a thru 0x00e08d (0xc314 bytes) 000608 1749541140.917078 mapread 0x02009c thru 0x02bab2 (0xba17 bytes) 000609 1749541140.922389 fallocate 0x011709 thru 0x019802 (0x80fa bytes) 000610 1749541140.925492 mapread 0x02346b thru 0x02bab2 (0x8648 bytes) 000611 1749541140.926205 mapwrite 0x02d9ff thru 0x038fff (0xb601 bytes) 000612 1749541140.956175 MIRROR_EXTEND to 7 mirrors 000613 1749541141.086836 mapread 0x02e479 thru 0x0367ca (0x8352 bytes) 000614 1749541141.095662 fallocate 0x012dfe thru 0x0185cb (0x57ce bytes) 000615 1749541141.106054 trunc from 0x039000 to 0x018f42 (0x200bd bytes) 000616 1749541141.115458 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x9b1:0x0]: Invalid argument (22) 000618 1749541141.179744 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x9b1:0x0]: Invalid argument (22) 000619 1749541141.234083 mapread 0x016799 thru 0x016e6c (0x6d4 bytes) 000620 1749541141.236382 trunc from 0x0064f6 to 0x00c5d9 (0x60e4 bytes) 000621 1749541141.239279 MIRROR_RESYNC 5 mirrors 000622 1749541141.446372 fallocate 0x01a997 thru 0x02235b (0x79c5 bytes) 000623 1749541141.455898 trunc from 0x012c68 to 0x013845 (0xbde bytes) 000624 1749541141.459053 MIRROR_RESYNC 5 mirrors 000625 1749541141.640811 MIRROR_RESYNC 5 mirrors 000626 1749541141.685944 fallocate 0x02cc79 thru 0x03a375 (0xd6fd bytes) 000627 1749541141.694774 trunc from 0x018f43 to 0x01d79f (0x485d bytes) 000628 1749541141.704699 mapread 0x009f6f thru 0x017be1 (0xdc73 bytes) 000629 1749541141.711585 write 0x03ca8d thru 0x03ffff (0x3573 bytes) 000630 1749541141.713767 MIRROR_RESYNC 5 mirrors 000631 1749541141.934850 fallocate 0x00bf31 thru 0x012eb0 (0x6f80 bytes) 000632 1749541141.943055 punch from 0x012565 to 0x01b511 (0x8fad bytes) 000633 1749541141.944603 mapwrite 0x0266ff thru 0x02b2bc (0x4bbe bytes) 000634 1749541141.955180 fallocate 0x01ac56 thru 0x02234d (0x76f8 bytes) 000635 1749541141.956848 punch from 0x0252d7 to 0x02d968 (0x8692 bytes) 000636 1749541141.958088 trunc from 0x036451 to 0x039403 (0x2fb3 bytes) 000638 1749541141.959947 write 0x013da5 thru 0x022dc0 (0xf01c bytes) 000639 1749541141.963558 mapread 0x0121c5 thru 0x0215e7 (0xf423 bytes) 000640 1749541141.964269 fallocate 0x01c229 thru 0x026d00 (0xaad8 bytes) 000641 1749541141.965955 mapwrite 0x03b174 thru 0x03c5be (0x144b bytes) 000642 1749541141.973787 mapread 0x039b5c thru 0x03ffff (0x64a4 bytes) 000643 1749541141.975826 MIRROR_RESYNC 5 mirrors 000644 1749541142.235457 write 0x007a26 thru 0x015763 (0xdd3e bytes) 000645 1749541142.246316 punch from 0x03ccd7 to 0x03ffff (0x3329 bytes) 000646 1749541142.248408 punch from 0x00debf to 0x01814b (0xa28d bytes) 000647 1749541142.252592 mapwrite 0x021e90 thru 0x023548 (0x16b9 bytes) 000648 1749541142.261148 trunc from 0x020d85 to 0x0227f6 (0x1a72 bytes) 000649 1749541142.263236 punch from 0x009dd9 to 0x00d2a9 (0x34d1 bytes) 000650 1749541142.266688 MIRROR_RESYNC 5 mirrors 000651 1749541142.650050 punch from 0x02505d to 0x034eb4 (0xfe58 bytes) 000652 1749541142.663233 punch from 0x02e3f7 to 0x034750 (0x635a bytes) 000654 1749541142.665587 punch from 0x02549d to 0x02b174 (0x5cd8 bytes) 000655 1749541142.667364 trunc from 0x040000 to 0x008748 (0x378b7 bytes) 000656 1749541142.680374 punch from 0x00256d to 0x008748 (0x61dc bytes) 000657 1749541142.682293 trunc from 0x002a3f to 0x008748 (0x5d0a bytes) 000658 1749541142.684303 mapread 0x002a19 thru 0x008748 (0x5d30 bytes) 000659 1749541142.690144 mapwrite 0x0292dd thru 0x02e06a (0x4d8e bytes) 000660 1749541142.704182 read 0x0158cc thru 0x021849 (0xbf7e bytes) 000661 1749541142.707178 write 0x018dc4 thru 0x023ed3 (0xb110 bytes) 000662 1749541142.708135 trunc from 0x0280b6 to 0x02a57f (0x24ca bytes) 000663 1749541142.710667 trunc from 0x02e06b to 0x011488 (0x1cbe2 bytes) 000664 1749541142.714762 trunc from 0x00d6c5 to 0x011488 (0x3dc4 bytes) 000665 1749541142.717031 punch from 0x004dfc to 0x011488 (0xc68d bytes) 000666 1749541142.718547 write 0x034d43 thru 0x038664 (0x3922 bytes) 000668 1749541142.719389 trunc from 0x038665 to 0x037d32 (0x932 bytes) 000669 1749541142.723287 read 0x00dd87 thru 0x017de0 (0xa05a bytes) 000670 1749541142.728245 mapread 0x0322ce thru 0x036f3b (0x4c6e bytes) 000671 1749541142.728902 fallocate 0x00ff5a thru 0x0119bd (0x1a64 bytes) 000672 1749541142.731834 trunc from 0x0074e7 to 0x00db4a (0x6664 bytes) 000673 1749541142.733741 fallocate 0x0321cd thru 0x036a0b (0x483f bytes) 000675 1749541142.736318 trunc from 0x037d33 to 0x03c790 (0x4a5e bytes) 000676 1749541142.740500 trunc from 0x028909 to 0x028c7d (0x375 bytes) 000677 1749541142.742443 trunc from 0x03c791 to 0x03c336 (0x45a bytes) 000678 1749541142.747700 fallocate 0x01bdb3 thru 0x028c00 (0xce4e bytes) 000679 1749541142.750115 punch from 0x02f045 to 0x03317a (0x4136 bytes) 000680 1749541142.752919 trunc from 0x03c337 to 0x00bd9e (0x30598 bytes) 000681 1749541142.759459 mapwrite 0x03a012 thru 0x03ffff (0x5fee bytes) 000682 1749541142.775178 fallocate 0x00f9f4 thru 0x01277d (0x2d8a bytes) 000683 1749541142.777815 read 0x00c66d thru 0x018e1d (0xc7b1 bytes) 000684 1749541142.782014 mapread 0x01b7fb thru 0x01df34 (0x273a bytes) 000685 1749541142.782742 fallocate 0x018825 thru 0x026b78 (0xe354 bytes) 000686 1749541142.784958 trunc from 0x01954e to 0x01bc32 (0x26e5 bytes) 000687 1749541142.787484 trunc from 0x02f5eb to 0x035515 (0x5f2b bytes) 000688 1749541142.789302 fallocate 0x003f4b thru 0x00dbbd (0x9c73 bytes) 000689 1749541142.790903 write 0x03db48 thru 0x03ffff (0x24b8 bytes) 000690 1749541142.791587 mapwrite 0x019470 thru 0x01d2c5 (0x3e56 bytes) 000691 1749541142.802663 trunc from 0x0303d4 to 0x03b4d4 (0xb101 bytes) 000692 1749541142.804814 mapread 0x01617c thru 0x01f948 (0x97cd bytes) 000693 1749541142.805381 punch from 0x005eb1 to 0x013a80 (0xdbd0 bytes) 000694 1749541142.806953 mapread 0x015e61 thru 0x01b8b0 (0x5a50 bytes) 000695 1749541142.807356 read 0x02e4fe thru 0x03c00f (0xdb12 bytes) 000696 1749541142.811003 fallocate 0x02c6b8 thru 0x02d7fe (0x1147 bytes) 000697 1749541142.813373 read 0x0384c2 thru 0x03ffff (0x7b3e bytes) 000698 1749541142.813774 trunc from 0x01d892 to 0x02bbce (0xe33d bytes) 000699 1749541142.815733 trunc from 0x040000 to 0x00c811 (0x337ee bytes) 000700 1749541142.824116 MIRROR_RESYNC 5 mirrors 000701 1749541142.973802 fallocate 0x021cb7 thru 0x02cc8c (0xafd6 bytes) 000703 1749541142.984313 fallocate 0x03dbf1 thru 0x03ffff (0x240f bytes) 000704 1749541142.986029 trunc from 0x00e1f8 to 0x0153e2 (0x71eb bytes) 000705 1749541142.987746 write 0x001b41 thru 0x005aaa (0x3f6a bytes) 000706 1749541142.990995 punch from 0x00d8c8 to 0x0101b9 (0x28f2 bytes) 000707 1749541142.992588 trunc from 0x00aeb2 to 0x00b12e (0x27d bytes) 000708 1749541142.994128 write 0x023a83 thru 0x02bf89 (0x8507 bytes) 000709 1749541142.997515 write 0x03592f thru 0x035d5a (0x42c bytes) 000710 1749541142.998134 punch from 0x01762b to 0x01cb08 (0x54de bytes) 000711 1749541143.000186 mapwrite 0x03b6a6 thru 0x03ffff (0x495a bytes) 000712 1749541143.025095 read 0x022fd8 thru 0x025579 (0x25a2 bytes) 000713 1749541143.025575 trunc from 0x006e07 to 0x00b314 (0x450e bytes) 000714 1749541143.027794 mapread 0x021480 thru 0x023860 (0x23e1 bytes) 000715 1749541143.029569 punch from 0x03eb34 to 0x03ffff (0x14cc bytes) 000716 1749541143.032295 trunc from 0x00fb43 to 0x01b575 (0xba33 bytes) 000717 1749541143.034727 punch from 0x0332eb to 0x03ffff (0xcd15 bytes) 000718 1749541143.037012 mapread 0x018a42 thru 0x020d50 (0x830f bytes) 000719 1749541143.039853 punch from 0x00ef66 to 0x0147d7 (0x5872 bytes) 000720 1749541143.041858 trunc from 0x02e7c2 to 0x037d66 (0x95a5 bytes) 000722 1749541143.043704 punch from 0x01df01 to 0x02c631 (0xe731 bytes) 000723 1749541143.045858 read 0x01040d thru 0x014aec (0x46e0 bytes) 000724 1749541143.052120 fallocate 0x00d45a thru 0x00e55d (0x1104 bytes) 000725 1749541143.054722 trunc from 0x040000 to 0x029178 (0x16e87 bytes) 000726 1749541143.059959 fallocate 0x034969 thru 0x0350ff (0x797 bytes) 000727 1749541143.061682 write 0x01379c thru 0x014dcd (0x1632 bytes) 000728 1749541143.062355 trunc from 0x031fe8 to 0x0350ff (0x3118 bytes) 000729 1749541143.064047 mapread 0x010790 thru 0x020676 (0xfee7 bytes) 000731 1749541143.064938 mapwrite 0x01166a thru 0x020f64 (0xf8fb bytes) 000732 1749541143.075749 MIRROR_RESYNC 5 mirrors 000733 1749541143.371986 MIRROR_EXTEND to 6 mirrors 000736 1749541143.514305 MIRROR_EXTEND to 7 mirrors 000738 1749541143.674504 trunc from 0x035100 to 0x029da6 (0xb359 bytes) 000739 1749541143.695580 punch from 0x00c970 to 0x0127e4 (0x5e75 bytes) 000740 1749541143.698137 fallocate 0x03b33b thru 0x03dc75 (0x293b bytes) 000741 1749541143.700173 write 0x01cbee thru 0x02b60a (0xea1d bytes) 000743 1749541143.702698 trunc from 0x02b60b to 0x011bcd (0x19a3d bytes) 000744 1749541143.709289 mapwrite 0x009f8e thru 0x010b4a (0x6bbd bytes) 000746 1749541143.722817 trunc from 0x011bce to 0x006d86 (0xae47 bytes) 000747 1749541143.727922 punch from 0x0049dc to 0x006d86 (0x23ab bytes) 000748 1749541143.729778 mapread 0x001f27 thru 0x0037d1 (0x18ab bytes) 000750 1749541143.733060 fallocate 0x0062b9 thru 0x00dda3 (0x7aeb bytes) 000751 1749541143.735982 punch from 0x00b31b to 0x00b6dd (0x3c3 bytes) 000752 1749541143.737718 write 0x010350 thru 0x01e7bb (0xe46c bytes) 000753 1749541143.738796 trunc from 0x01e7bc to 0x037484 (0x18cc9 bytes) 000754 1749541143.744700 MIRROR_EXTEND to 8 mirrors 000755 1749541143.890473 MIRROR_RESYNC 8 mirrors 000756 1749541144.241906 mapwrite 0x03615c thru 0x038fbd (0x2e62 bytes) 000757 1749541144.266720 fallocate 0x0094a4 thru 0x01641e (0xcf7b bytes) 000758 1749541144.269553 MIRROR_RESYNC 8 mirrors 000759 1749541144.587074 read 0x0277f7 thru 0x029d07 (0x2511 bytes) 000760 1749541144.589640 mapread 0x020612 thru 0x02b769 (0xb158 bytes) 000761 1749541144.596722 read 0x019172 thru 0x01a18b (0x101a bytes) 000762 1749541144.600344 mapwrite 0x035695 thru 0x03ffff (0xa96b bytes) 000763 1749541144.632373 write 0x038c63 thru 0x03ffff (0x739d bytes) 000764 1749541144.633220 read 0x0365ee thru 0x0375d8 (0xfeb bytes) 000765 1749541144.633603 read 0x0005fb thru 0x010561 (0xff67 bytes) 000766 1749541144.637608 mapwrite 0x03c3ab thru 0x03d480 (0x10d6 bytes) 000767 1749541144.642454 mapwrite 0x00be64 thru 0x0137d6 (0x7973 bytes) 000768 1749541144.648866 mapread 0x03ce8e thru 0x03ffff (0x3172 bytes) 000769 1749541144.649318 read 0x01a35f thru 0x01fc46 (0x58e8 bytes) 000770 1749541144.649604 trunc from 0x040000 to 0x00a13d (0x35ec2 bytes) 000772 1749541144.656525 write 0x00aa2d thru 0x01801b (0xd5ef bytes) 000773 1749541144.657934 read 0x015b8f thru 0x01801b (0x248d bytes) 000774 1749541144.658391 trunc from 0x01801c to 0x00f2fa (0x8d21 bytes) 000775 1749541144.663682 mapwrite 0x01c7e8 thru 0x029b57 (0xd370 bytes) 000776 1749541144.679482 trunc from 0x00a047 to 0x00e4ac (0x4466 bytes) 000777 1749541144.682096 fallocate 0x011382 thru 0x01fc31 (0xe8b0 bytes) 000778 1749541144.684219 mapwrite 0x03670b thru 0x03ffff (0x98f5 bytes) 000779 1749541144.705114 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x9b1:0x0]: Invalid argument (22) 000780 1749541144.768179 read 0x00a5ff thru 0x010323 (0x5d25 bytes) 000781 1749541144.771263 write 0x0359ab thru 0x03ad08 (0x535e bytes) 000782 1749541144.777539 MIRROR_RESYNC 7 mirrors 000783 1749541145.146070 write 0x01e7ed thru 0x01f0ec (0x900 bytes) 000784 1749541145.156306 mapread 0x01d0c7 thru 0x01d11d (0x57 bytes) 000785 1749541145.158622 fallocate 0x01ed6e thru 0x023281 (0x4514 bytes) 000786 1749541145.160394 mapread 0x036c3e thru 0x0377f6 (0xbb9 bytes) 000787 1749541145.163486 mapwrite 0x0277d8 thru 0x028d3e (0x1567 bytes) 000788 1749541145.172561 fallocate 0x020f23 thru 0x025e4e (0x4f2c bytes) 000789 1749541145.174236 write 0x01bac1 thru 0x025876 (0x9db6 bytes) 000791 1749541145.175476 trunc from 0x0266de to 0x02e472 (0x7d95 bytes) 000792 1749541145.177487 trunc from 0x040000 to 0x021ecc (0x1e133 bytes) 000793 1749541145.184659 mapwrite 0x00f049 thru 0x018718 (0x96d0 bytes) 000794 1749541145.195675 punch from 0x01cb56 to 0x021ecc (0x5377 bytes) 000795 1749541145.197519 mapwrite 0x02a6f6 thru 0x031e5a (0x7765 bytes) 000796 1749541145.209903 punch from 0x00bfbc to 0x01985d (0xd8a2 bytes) 000797 1749541145.211593 trunc from 0x031e5b to 0x037fc3 (0x6169 bytes) 000798 1749541145.216537 MIRROR_RESYNC 7 mirrors 000799 1749541145.605380 fallocate 0x0298cf thru 0x02e115 (0x4847 bytes) 000800 1749541145.614668 trunc from 0x037fc4 to 0x025fe8 (0x11fdb bytes) 000801 1749541145.621126 MIRROR_RESYNC 7 mirrors 000802 1749541145.971532 read 0x000e3e thru 0x002b4f (0x1d12 bytes) 000803 1749541145.973583 mapread 0x0105cf thru 0x01ee8e (0xe8c0 bytes) 000804 1749541145.981803 fallocate 0x02a82f thru 0x03a21a (0xf9ec bytes) 000805 1749541145.993866 trunc from 0x01d825 to 0x01dc72 (0x44e bytes) 000806 1749541145.995520 fallocate 0x0263f0 thru 0x0331b4 (0xcdc5 bytes) 000808 1749541145.997206 mapwrite 0x010203 thru 0x01cf79 (0xcd77 bytes) 000809 1749541146.013607 MIRROR_RESYNC 7 mirrors 000810 1749541146.381138 fallocate 0x019011 thru 0x01de40 (0x4e30 bytes) 000811 1749541146.390124 MIRROR_EXTEND to 8 mirrors 000812 1749541146.526512 trunc from 0x01f2e2 to 0x02086a (0x1589 bytes) 000813 1749541146.537960 trunc from 0x03a21b to 0x008d6f (0x314ab bytes) 000814 1749541146.544692 mapread 0x001b61 thru 0x008141 (0x65e1 bytes) 000815 1749541146.550519 mapread 0x0077c8 thru 0x008d6f (0x15a8 bytes) 000817 1749541146.551471 punch from 0x002724 to 0x008d6f (0x664c bytes) 000818 1749541146.553578 trunc from 0x008d70 to 0x030676 (0x27907 bytes) 000819 1749541146.558405 write 0x012839 thru 0x01ac23 (0x83eb bytes) 000820 1749541146.562532 fallocate 0x03c7ee thru 0x03ffff (0x3812 bytes) 000821 1749541146.564554 trunc from 0x00a96d to 0x01181d (0x6eb1 bytes) 000823 1749541146.566508 fallocate 0x01247c thru 0x01ddf4 (0xb979 bytes) 000824 1749541146.568443 mapwrite 0x014f19 thru 0x02250e (0xd5f6 bytes) 000825 1749541146.579442 mapwrite 0x0084db thru 0x00a6b1 (0x21d7 bytes) 000826 1749541146.587938 read 0x020982 thru 0x022dfa (0x2479 bytes) 000827 1749541146.588279 mapread 0x02b8d2 thru 0x030676 (0x4da5 bytes) 000829 1749541146.591262 trunc from 0x030677 to 0x03671f (0x60a9 bytes) 000830 1749541146.595583 mapread 0x019c56 thru 0x025796 (0xbb41 bytes) 000831 1749541146.598073 trunc from 0x013daa to 0x01444d (0x6a4 bytes) 000832 1749541146.599409 trunc from 0x036720 to 0x02f73b (0x6fe4 bytes) 000833 1749541146.602901 mapwrite 0x009362 thru 0x00aae1 (0x1780 bytes) 000835 1749541146.607408 fallocate 0x007dfa thru 0x011a32 (0x9c39 bytes) 000837 1749541146.609271 punch from 0x00c5f1 to 0x00fadd (0x34ed bytes) 000838 1749541146.611018 punch from 0x0273d3 to 0x0293d1 (0x1fff bytes) 000840 1749541146.612439 write 0x00906e thru 0x012494 (0x9427 bytes) 000841 1749541146.613049 trunc from 0x02f73c to 0x02f7ba (0x7f bytes) 000842 1749541146.616410 punch from 0x006bce to 0x0165c2 (0xf9f5 bytes) 000843 1749541146.621170 MIRROR_RESYNC 8 mirrors 000844 1749541146.895017 read 0x02e29a thru 0x02f7ba (0x1521 bytes) 000845 1749541146.896628 read 0x0054c6 thru 0x00b82f (0x636a bytes) 000846 1749541146.900261 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x9b1:0x0]: Invalid argument (22) 000847 1749541146.969931 trunc from 0x01ac44 to 0x0246e9 (0x9aa6 bytes) 000849 1749541146.978226 mapwrite 0x03e958 thru 0x03ffff (0x16a8 bytes) 000850 1749541146.989815 trunc from 0x010e49 to 0x01fc51 (0xee09 bytes) 000851 1749541146.991265 trunc from 0x040000 to 0x0313d4 (0xec2b bytes) 000852 1749541146.995347 trunc from 0x002b96 to 0x00fd20 (0xd18b bytes) 000853 1749541146.997179 trunc from 0x0313d5 to 0x02ecbc (0x2718 bytes) 000854 1749541147.002433 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x9b1:0x0]: Invalid argument (22) 000855 1749541147.063601 trunc from 0x02e95e to 0x02ecbc (0x35f bytes) 000856 1749541147.065733 punch from 0x026372 to 0x026429 (0xb8 bytes) 000857 1749541147.070103 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x9b1:0x0]: Invalid argument (22) 000858 1749541147.123904 write 0x00beb2 thru 0x014a8d (0x8bdc bytes) 000859 1749541147.128646 write 0x01fba6 thru 0x0291a8 (0x9603 bytes) 000861 1749541147.136337 MIRROR_EXTEND to 6 mirrors 000862 1749541147.260684 mapwrite 0x00286a thru 0x0038ef (0x1086 bytes) 000863 1749541147.292540 mapread 0x006205 thru 0x00ca87 (0x6883 bytes) 000864 1749541147.293238 write 0x02c716 thru 0x0341cf (0x7aba bytes) 000866 1749541147.299038 trunc from 0x00b406 to 0x01a71e (0xf319 bytes) 000867 1749541147.300914 read 0x031f6f thru 0x0341cf (0x2261 bytes) 000868 1749541147.301350 mapread 0x022c12 thru 0x030887 (0xdc76 bytes) 000869 1749541147.304217 read 0x032645 thru 0x0341cf (0x1b8b bytes) 000870 1749541147.304581 punch from 0x02d226 to 0x02db0d (0x8e8 bytes) 000871 1749541147.308969 mapread 0x01ac16 thru 0x01f973 (0x4d5e bytes) 000872 1749541147.312671 mapwrite 0x01a339 thru 0x028b4e (0xe816 bytes) 000873 1749541147.320902 read 0x026b54 thru 0x0341cf (0xd67c bytes) 000875 1749541147.321673 read 0x01af4f thru 0x01fc9a (0x4d4c bytes) 000876 1749541147.322060 write 0x01bbd1 thru 0x02029a (0x46ca bytes) 000877 1749541147.322961 read 0x02f3fb thru 0x0341cf (0x4dd5 bytes) 000878 1749541147.324993 MIRROR_RESYNC 6 mirrors 000879 1749541147.522844 trunc from 0x006277 to 0x015e8a (0xfc14 bytes) 000880 1749541147.533002 punch from 0x0044b1 to 0x0108b2 (0xc402 bytes) 000881 1749541147.535028 punch from 0x00151d to 0x00d2d6 (0xbdba bytes) 000882 1749541147.536970 write 0x01de75 thru 0x02d446 (0xf5d2 bytes) 000883 1749541147.540658 trunc from 0x0179d4 to 0x017afd (0x12a bytes) 000884 1749541147.543339 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x9b1:0x0]: Invalid argument (22) 000885 1749541147.604382 punch from 0x0082e9 to 0x008d3a (0xa52 bytes) 000886 1749541147.606286 read 0x01aff0 thru 0x029713 (0xe724 bytes) 000887 1749541147.609581 write 0x03d94e thru 0x03ffff (0x26b2 bytes) 000888 1749541147.613145 write 0x008d1f thru 0x013c6f (0xaf51 bytes) 000889 1749541147.617117 trunc from 0x02ec67 to 0x031614 (0x29ae bytes) 000891 1749541147.619724 write 0x02b000 thru 0x033387 (0x8388 bytes) 000892 1749541147.621873 fallocate 0x02e897 thru 0x030377 (0x1ae1 bytes) 000893 1749541147.624600 mapwrite 0x0343de thru 0x03a638 (0x625b bytes) 000894 1749541147.635395 punch from 0x000c9f to 0x005139 (0x449b bytes) 000895 1749541147.637636 fallocate 0x01bade thru 0x01f94c (0x3e6f bytes) 000896 1749541147.640575 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x9b1:0x0]: Invalid argument (22) 000897 1749541147.686957 fallocate 0x025388 thru 0x02d2ec (0x7f65 bytes) 000898 1749541147.690504 MIRROR_RESYNC 4 mirrors 000899 1749541147.844140 read 0x01a14a thru 0x01efb0 (0x4e67 bytes) 000900 1749541147.847540 mapwrite 0x009c74 thru 0x013fae (0xa33b bytes) 000901 1749541147.879605 write 0x008aa0 thru 0x00d565 (0x4ac6 bytes) 000902 1749541147.880190 write 0x02bbaa thru 0x03a427 (0xe87e bytes) 000903 1749541147.886174 MIRROR_EXTEND to 5 mirrors 000904 1749541147.992245 trunc from 0x040000 to 0x01e3c4 (0x21c3b bytes) 000905 1749541148.008597 MIRROR_RESYNC 5 mirrors 000906 1749541148.172458 mapread 0x008f23 thru 0x018d17 (0xfdf5 bytes) 000908 1749541148.180710 write 0x00e80e thru 0x01bee5 (0xd6d8 bytes) 000909 1749541148.195075 MIRROR_RESYNC 5 mirrors 000910 1749541148.342388 trunc from 0x019bed to 0x01e3c4 (0x47d8 bytes) 000913 1749541148.353451 punch from 0x01ad6b to 0x01e3c4 (0x365a bytes) 000914 1749541148.355375 read 0x01c626 thru 0x01e3c4 (0x1d9f bytes) 000915 1749541148.358236 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x9b1:0x0]: Invalid argument (22) 000916 1749541148.412150 write 0x02ea97 thru 0x02fc34 (0x119e bytes) 000917 1749541148.416182 trunc from 0x02fc35 to 0x02f8fc (0x338 bytes) 000918 1749541148.420856 trunc from 0x0014eb to 0x00becd (0xa9e3 bytes) 000919 1749541148.422863 trunc from 0x02f8fd to 0x032ffa (0x36fe bytes) 000920 1749541148.428672 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x9b1:0x0]: Invalid argument (22) 000922 1749541148.474772 mapwrite 0x0219b7 thru 0x02bdc8 (0xa412 bytes) 000923 1749541148.484253 fallocate 0x03638a thru 0x038ce2 (0x2959 bytes) 000924 1749541148.485902 read 0x026c57 thru 0x030451 (0x97fb bytes) 000925 1749541148.487897 trunc from 0x032ffb to 0x01b17a (0x17e80 bytes) 000926 1749541148.492192 trunc from 0x00622d to 0x007e10 (0x1be4 bytes) 000927 1749541148.493262 write 0x001080 thru 0x00ff23 (0xeea4 bytes) 000928 1749541148.495858 fallocate 0x006d87 thru 0x0070b6 (0x330 bytes) 000929 1749541148.497227 mapread 0x0137db thru 0x01831b (0x4b41 bytes) 000930 1749541148.499310 read 0x006a3d thru 0x006cdf (0x2a3 bytes) 000931 1749541148.499557 punch from 0x01142e to 0x017b9a (0x676d bytes) 000932 1749541148.500962 mapread 0x00387f thru 0x00ab0b (0x728d bytes) 000933 1749541148.501466 mapread 0x005b4f thru 0x00dbce (0x8080 bytes) 000935 1749541148.502022 fallocate 0x0301cd thru 0x03fc11 (0xfa45 bytes) 000936 1749541148.503621 punch from 0x000c4a to 0x00fbc2 (0xef79 bytes) 000938 1749541148.507827 mapread 0x00eb5c thru 0x0172fd (0x87a2 bytes) 000939 1749541148.511229 punch from 0x011f1d to 0x01b17a (0x925e bytes) 000940 1749541148.513054 read 0x015f3f thru 0x01612c (0x1ee bytes) 000941 1749541148.515751 write 0x02f637 thru 0x03ad38 (0xb702 bytes) 000943 1749541148.516782 mapread 0x01a900 thru 0x020a04 (0x6105 bytes) 000945 1749541148.519408 trunc from 0x03ad39 to 0x031c2e (0x910a bytes) 000946 1749541148.523842 fallocate 0x008d3e thru 0x00f725 (0x69e8 bytes) 000947 1749541148.525122 trunc from 0x010477 to 0x0155b7 (0x5141 bytes) 000948 1749541148.526202 mapwrite 0x00aca2 thru 0x016e6d (0xc1cc bytes) 000949 1749541148.538822 MIRROR_RESYNC 3 mirrors 000950 1749541148.660835 punch from 0x0150c8 to 0x02136e (0xc2a7 bytes) 000951 1749541148.670769 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x9b1:0x0]: Invalid argument (22) 000952 1749541148.716876 trunc from 0x031c2f to 0x02ca22 (0x520c bytes) 000953 1749541148.726760 mapread 0x028eea thru 0x02ca22 (0x3b39 bytes) 000954 1749541148.732588 MIRROR_EXTEND to 3 mirrors 000955 1749541148.853184 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x9b1:0x0]: Invalid argument (22) 000956 1749541148.908308 fallocate 0x026081 thru 0x02753d (0x14bd bytes) 000958 1749541148.917395 fallocate 0x01b494 thru 0x023446 (0x7fb3 bytes) 000959 1749541148.918827 mapwrite 0x00af7c thru 0x00c204 (0x1289 bytes) 000960 1749541148.927624 punch from 0x018b78 to 0x019ba5 (0x102e bytes) 000961 1749541148.928930 trunc from 0x01fc36 to 0x028878 (0x8c43 bytes) 000962 1749541148.931567 MIRROR_EXTEND to 3 mirrors 000963 1749541149.053348 mapread 0x00f678 thru 0x0173d4 (0x7d5d bytes) 000964 1749541149.062412 mapread 0x003e36 thru 0x00b705 (0x78d0 bytes) 000965 1749541149.069811 read 0x0227f8 thru 0x02c498 (0x9ca1 bytes) 000966 1749541149.074562 mapread 0x00926a thru 0x00caf5 (0x388c bytes) 000967 1749541149.078160 read 0x0027e5 thru 0x00f80b (0xd027 bytes) 000968 1749541149.080226 read 0x01a231 thru 0x02404d (0x9e1d bytes) 000969 1749541149.084604 trunc from 0x0004ee to 0x004897 (0x43aa bytes) 000970 1749541149.094499 trunc from 0x024eea to 0x02938a (0x44a1 bytes) 000971 1749541149.096079 mapread 0x0015eb thru 0x00f853 (0xe269 bytes) 000973 1749541149.112268 read 0x02952f thru 0x02ca22 (0x34f4 bytes) 000974 1749541149.116547 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x9b1:0x0]: Invalid argument (22) 000975 1749541149.164854 mapwrite 0x02326a thru 0x02dc73 (0xaa0a bytes) 000976 1749541149.182610 mapwrite 0x02ee37 thru 0x038f9b (0xa165 bytes) 000977 1749541149.196935 trunc from 0x03184e to 0x038f9b (0x774e bytes) 000978 1749541149.198543 punch from 0x02ce44 to 0x02ec6e (0x1e2b bytes) 000979 1749541149.199878 mapread 0x0241d8 thru 0x02b843 (0x766c bytes) 000980 1749541149.200521 mapwrite 0x02a874 thru 0x02c21d (0x19aa bytes) 000981 1749541149.206581 trunc from 0x038f9c to 0x03dc83 (0x4ce8 bytes) 000982 1749541149.210834 trunc from 0x03dc84 to 0x00c8d9 (0x313aa bytes) 000983 1749541149.215348 read 0x00a069 thru 0x00c8d9 (0x2871 bytes) 000984 1749541149.217178 read 0x0039ce thru 0x0087d7 (0x4e0a bytes) 000985 1749541149.219839 trunc from 0x00c8da to 0x03b539 (0x2ec60 bytes) 000986 1749541149.223732 trunc from 0x03b53a to 0x03f9bd (0x4484 bytes) 000987 1749541149.228105 mapwrite 0x012a3f thru 0x0193a2 (0x6964 bytes) 000988 1749541149.237599 punch from 0x03913d to 0x03f9bd (0x6881 bytes) 000989 1749541149.239172 trunc from 0x03f9be to 0x029adf (0x15ede bytes) 000990 1749541149.243264 fallocate 0x00afad thru 0x00bb0d (0xb61 bytes) 000991 1749541149.244913 read 0x0077aa thru 0x007b3b (0x392 bytes) 000992 1749541149.245288 trunc from 0x01ceef to 0x01d2b4 (0x3c6 bytes) 000993 1749541149.246702 mapwrite 0x00676b thru 0x00c779 (0x600f bytes) 000994 1749541149.253618 mapwrite 0x0202ec thru 0x028cfb (0x8a10 bytes) 000995 1749541149.258862 read 0x00aa25 thru 0x013f62 (0x953e bytes) 000996 1749541149.259237 read 0x0100c7 thru 0x011269 (0x11a3 bytes) 000997 1749541149.259607 write 0x0354e6 thru 0x03784c (0x2367 bytes) 000998 1749541149.260158 trunc from 0x03784d to 0x03df6c (0x6720 bytes) 001000 1749541149.264542 trunc from 0x015f9e to 0x016e2d (0xe90 bytes) All operations completed A-OK! PASS 70a (16s) == sanity-flr test 71: check mirror extend parent fid ==== 03:39:10 (1749541150) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000404:0x9ec:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (2s) == sanity-flr test 200a: stress test ===================== 03:39:12 (1749541152) Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre2 Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7421920 .. Extending file size to 7714272 .. done Extending file size to 7818944 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8474400 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8183616 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9055008 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9298880 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9058112 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9262080 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 3150 3151 3152 3154 3156 192.168.201.119@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg119-client.virtnet /mnt/lustre2 (opts:) 192.168.201.119@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg119-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 03:40:16 (1749541216) Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre2 Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2063328 .. Extending file size to 2123584 .. Extending file size to 6805600 .. done Extending file size to 7172320 .. Extending file size to 8916480 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8121728 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9337920 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9093312 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9214688 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9247264 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 7761 7762 7764 7766 7768 192.168.201.119@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg119-client.virtnet /mnt/lustre2 (opts:) 192.168.201.119@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg119-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (62s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 03:41:18 (1749541278) Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2716 1284972 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3272 1284416 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18960 3588060 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18908 3588112 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37868 7176172 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2716 1284972 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 3272 1284416 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18960 3588060 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18908 3588112 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37868 7176172 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0336259 s, 62.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0488949 s, 42.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.119@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg119-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 03:41:23 (1749541283) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0255243 s, 82.2 MB/s PASS 202 (1s) == sanity-flr test 203: mirror file preserve mirror ID === 03:41:24 (1749541284) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x61:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 03:41:26 (1749541286) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0230049 s, 91.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xc98:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xc97:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xc98:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 03:41:28 (1749541288) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0226829 s, 92.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xc99:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xc9a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xc99:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xc99:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xc9a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xc99:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xc9a:0x0] } PASS 204b (1s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 03:41:29 (1749541289) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0219926 s, 95.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 03:41:52 (1749541312) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg119-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0142196 s, 73.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xc9b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xc9c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xc9b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xc9b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xc9c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xc9b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xc9c:0x0] } PASS 204d (32s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 03:42:24 (1749541344) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg119-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xc9d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xc9e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0208277 s, 101 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xc9d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xc9e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xc9d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xc9d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xc9e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xc9d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (31s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 03:42:55 (1749541375) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg119-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg119-server: Pool lustre.test_204f created Adding targets to pool oleg119-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg119-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0232461 s, 90.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg119-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg119-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg119-server: Pool lustre.test_204f destroyed PASS 204f (44s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 03:43:39 (1749541419) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 03:43:41 (1749541421) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 03:43:43 (1749541423) oleg119-server: Pool lustre.test_206 created Adding targets to pool oleg119-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg119-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg119-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg119-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg119-server: Pool lustre.test_206 destroyed PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 03:43:56 (1749541436) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0186516 s, 56.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00914408 s, 115 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xee40865a Mirror 3: 0xee40865a PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 03:43:58 (1749541438) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 03:43:59 (1749541439) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 03:44:00 (1749541440) Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.00898668 s, 0.4 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2560000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x15a0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x15a0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x15a0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x10d0000 Read this (4 bytes): BBBB 192.168.201.119@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg119-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 03:44:13 (1749541453) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00274023 s, 3.0 MB/s the very first read read with warm cache PASS 209b (1s) == sanity-flr test 210a: handle broken mirrored lovea ==== 03:44:14 (1749541454) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102981 s, 102 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7e lmm_fid: [0x200000405:0x7e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7e lmm_fid: [0x200000405:0x7e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb7:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611262 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-06-10 03:44:15.000000000 -0400 Modify: 2025-06-10 03:44:15.000000000 -0400 Change: 2025-06-10 03:44:15.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 03:44:16 (1749541456) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0090551 s, 116 MB/s oleg119-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg119-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg119-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg119-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg119-server: O/280000401/d24/b8: File not found by ext2_lookup PASS 210b (10s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 03:44:26 (1749541466) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0195404 s, 2.1 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00511214 s, 801 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 10 03:44 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x83:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 10 03:44 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 690 sec ============ 03:44:28 (1749541468) === sanity-flr: start cleanup 03:44:28 (1749541468) === === sanity-flr: finish cleanup 03:44:30 (1749541470) ===