-----============= acceptance-small: sanity-flr ============----- Mon Jun 9 10:12:57 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg432-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 10:13:02 (1749478382) === oleg432-client.virtnet: executing check_config_client /mnt/lustre oleg432-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg432-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012a48c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012a48c800.idle_timeout=debug disable quota as required oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 10:13:10 (1749478390) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7925] == sanity-flr test 0a: lfs mirror create with -N option == 10:13:11 (1749478391) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:13:14 (1749478394) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (0s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:13:15 (1749478395) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (0s) == sanity-flr test 0d: lfs mirror extend with -N option == 10:13:15 (1749478395) PASS 0d (4s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:13:19 (1749478399) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:13:20 (1749478400) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 10:13:21 (1749478401) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88012a48c800.stats=clear osc.lustre-OST0001-osc-ffff88012a48c800.stats=clear PASS 0g (1s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:13:22 (1749478402) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:13:25 (1749478405) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:13:26 (1749478406) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:13:27 (1749478407) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 10:13:29 (1749478409) PASS 2 (1s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:13:30 (1749478410) PASS 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:13:32 (1749478412) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (1s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:13:33 (1749478413) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:13:36 (1749478416) PASS 7 (1s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:13:37 (1749478417) 15+0 records in 15+0 records out 15728640 bytes (16 MB) copied, 0.24846 s, 63.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0178603 s, 58.7 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:13:39 (1749478419) 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.119108 s, 141 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:13:41 (1749478421) Stopping /mnt/lustre-ost1 (opts:) on oleg432-server oleg432-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a48c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a48c800.ost_server_uuid in DISCONN state after 2 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg432-server: oleg432-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg432-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg432-server oleg432-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a48c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a48c800.ost_server_uuid in DISCONN state after 4 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg432-server: oleg432-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg432-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (26s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:14:07 (1749478447) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB) copied, 0.109563 s, 67.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg432-server oleg432-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a48c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a48c800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg432-server: oleg432-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg432-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (9s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:14:16 (1749478456) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg432-server oleg432-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a48c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a48c800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg432-server: oleg432-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg432-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg432-server oleg432-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a48c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a48c800.ost_server_uuid in DISCONN state after 4 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg432-server: oleg432-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg432-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (20s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:14:37 (1749478477) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.747447 s, 70.1 MB/s -rw-r--r-- 1 root root 52428800 Jun 9 10:14 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.232s user 0m0.001s sys 0m0.104s Stopping /mnt/lustre-ost1 (opts:) on oleg432-server oleg432-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a48c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a48c800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.254s user 0m0.001s sys 0m0.117s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg432-server: oleg432-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg432-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg432-server oleg432-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a48c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a48c800.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost1)... real 0m0.258s user 0m0.000s sys 0m0.111s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg432-server: oleg432-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg432-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg432-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012a48c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a48c800.ost_server_uuid in FULL state after 0 sec PASS 33b (21s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:14:57 (1749478497) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:14:58 (1749478498) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:14:59 (1749478499) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (0s) == sanity-flr test 35: allow to write to mirrored files == 10:14:59 (1749478499) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 10:15:01 (1749478501) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.127947 s, 98.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (2s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:15:03 (1749478503) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0094852 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100367 s, 104 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:15:05 (1749478505) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:15:08 (1749478508) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000546523 s, 1.9 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000492774 s, 2.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000572943 s, 1.8 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000660999 s, 1.5 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 10:15:10 (1749478510) create test files with size 15751027 .. f5fa5605edf728fbb080fcf6e63f1865 - da2205f04f7d34a995d1a2a1d438a03f - cfb4b90198cb3ab4d8625f53faeeff96 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg432-client.virtnet /mnt/lustre (opts:) Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre PASS 37 (4s) == sanity-flr test 38: resync ============================ 10:15:14 (1749478514) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (2s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:15:17 (1749478517) PASS 39 (1s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:15:18 (1749478518) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0180428 s, 58.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x37:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x38:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x37:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101328 s, 103 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x38:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x39:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 10:15:20 (1749478520) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0203967 s, 103 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0799696 s, 52.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0356499 s, 58.8 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) == sanity-flr test 42: lfs mirror verify ================= 10:15:22 (1749478522) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 10:15:23 (1749478523) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:15:24 (1749478524) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00910102 s, 115 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (1s) == sanity-flr test 44a: lfs mirror split check =========== 10:15:25 (1749478525) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0705617 s, 44.6 MB/s PASS 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:15:28 (1749478528) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x44:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x44:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x44:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (1s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:15:30 (1749478530) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0781379 s, 134 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg432-client.virtnet /mnt/lustre (opts:) Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (22s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:15:52 (1749478552) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800a237c800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0643843 s, 163 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg432-client.virtnet /mnt/lustre (opts:) Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800a9c0c000.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:15:55 (1749478555) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (7s) == sanity-flr test 46: Verify setstripe --copy option ==== 10:16:02 (1749478562) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (6s) == sanity-flr test 47: Verify mirror obj alloc =========== 10:16:08 (1749478568) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 10:16:09 (1749478569) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.06926 s, 45.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0497038 s, 63.3 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 79c8a65d0770ee36ebca0ea994333d52 - mirror 1 checksum: 656bfb16c4d0a295631c946f36493c8d - mirror 2 checksum: 79c8a65d0770ee36ebca0ea994333d52 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Jun 9 10:16:09 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Jun 9 10:16:09 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 79c8a65d0770ee36ebca0ea994333d52 - mirror 1 checksum: 656bfb16c4d0a295631c946f36493c8d - mirror 2 checksum: 656bfb16c4d0a295631c946f36493c8d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Jun 9 10:16:09 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Jun 9 10:16:09 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } PASS 48 (2s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 10:16:11 (1749478571) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00924704 s, 113 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (1s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:16:12 (1749478572) Failing mds1 on oleg432-server Stopping /mnt/lustre-mds1 (opts:) on oleg432-server 10:16:20 (1749478580) shut down facet: mds1 facet_host: oleg432-server facet_failover_host: oleg432-server Failover mds1 to oleg432-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-MDT0000 10:16:33 (1749478593) targets are mounted 10:16:33 (1749478593) facet_failover done oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00806198 s, 130 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } PASS 50A (28s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 10:16:40 (1749478600) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00703369 s, 2.9 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00407784 s, 5.0 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 10:16:42 (1749478602) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00832274 s, 2.5 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00342553 s, 6.0 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:16:44 (1749478604) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.0019384 s, 8.5 MB/s PASS 50c (1s) == sanity-flr test 50d: mirror rsync keep holes ========== 10:16:45 (1749478605) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00854729 s, 2.4 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:16:47 (1749478607) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00147573 s, 2.8 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00666307 s, 615 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:16:49 (1749478609) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x11:0x0]: Invalid argument (22) PASS 60b (3s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:16:52 (1749478612) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (17s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 10:17:09 (1749478629) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 10:17:26 (1749478646) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 10:17:43 (1749478663) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (1s) == sanity-flr test 70: mirror create and split race ====== 10:17:44 (1749478664) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 26680 mirror split pid 26681 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4948: 26680 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4948: 26681 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 10:18:46 (1749478726) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 8726 truncating to largest ever: 0x3ec39 000001 1749478726.867855 trunc from 00000000 to 0x03ec38 (0x3ec39 bytes) 000003 1749478726.871450 write 0x0267f4 thru 0x02f5ad (0x8dba bytes) 000004 1749478726.874132 mapread 0x025070 thru 0x029389 (0x431a bytes) 000005 1749478726.876043 fallocate 0x01fe0b thru 0x0277f2 (0x79e8 bytes) 000008 1749478726.878073 read 0x00b486 thru 0x011295 (0x5e10 bytes) 000009 1749478726.879917 write 0x00fa51 thru 0x01d618 (0xdbc8 bytes) 000011 1749478726.881829 punch from 0x0386e3 to 0x03ec38 (0x6556 bytes) 000012 1749478726.882909 read 0x02fe4f thru 0x039c8b (0x9e3d bytes) 000013 1749478726.885513 trunc from 0x03ec39 to 0x03b7b2 (0x3486 bytes) 000014 1749478726.889495 mapwrite 0x031d84 thru 0x0355ad (0x382a bytes) 000015 1749478726.897377 read 0x01181c thru 0x01e48f (0xcc74 bytes) 000016 1749478726.897808 write 0x02a309 thru 0x038b21 (0xe819 bytes) 000017 1749478726.898815 trunc from 0x03b7b3 to 0x0241a5 (0x1760d bytes) 000018 1749478726.902653 fallocate 0x02c3c1 thru 0x032768 (0x63a8 bytes) 000019 1749478726.904316 mapread 0x01882b thru 0x01fb9f (0x7375 bytes) 000020 1749478726.904945 mapwrite 0x00abbd thru 0x00de45 (0x3289 bytes) 000022 1749478726.911089 read 0x00ac98 thru 0x012ed2 (0x823b bytes) 000023 1749478726.911571 write 0x0345b8 thru 0x03dc37 (0x9680 bytes) 000024 1749478726.912471 trunc from 0x03dc38 to 0x021412 (0x1c825 bytes) 000025 1749478726.917482 read 0x008b63 thru 0x009116 (0x5b4 bytes) 000026 1749478726.917811 mapread 0x00d360 thru 0x0120ed (0x4d8e bytes) 000027 1749478726.918379 trunc from 0x021413 to 0x03a79e (0x1938c bytes) 000028 1749478726.922332 trunc from 0x00cdea to 0x00dc1a (0xe31 bytes) 000029 1749478726.923943 punch from 0x037c85 to 0x03a79e (0x2b1a bytes) 000030 1749478726.925404 read 0x0377fe thru 0x03a79e (0x2fa1 bytes) 000032 1749478726.928561 trunc from 0x032aa9 to 0x03a79e (0x7cf6 bytes) 000033 1749478726.930063 trunc from 0x03a79f to 0x023ba8 (0x16bf6 bytes) 000034 1749478726.933659 punch from 0x0020b1 to 0x00906f (0x6fbf bytes) 000035 1749478726.935059 trunc from 0x00521a to 0x0078ab (0x2692 bytes) 000036 1749478726.936398 punch from 0x0179fc to 0x019369 (0x196e bytes) 000039 1749478726.939746 MIRROR_EXTEND to 2 mirrors 000040 1749478727.072470 mapwrite 0x01f4c6 thru 0x025f43 (0x6a7e bytes) 000041 1749478727.093043 write 0x0336ee thru 0x039049 (0x595c bytes) 000042 1749478727.093797 mapwrite 0x025c1a thru 0x02ade7 (0x51ce bytes) 000043 1749478727.100308 punch from 0x009d23 to 0x00de3a (0x4118 bytes) 000044 1749478727.101867 read 0x01e1e3 thru 0x022a2f (0x484d bytes) 000045 1749478727.103704 MIRROR_EXTEND to 3 mirrors 000047 1749478727.217337 mapwrite 0x0202c3 thru 0x02f252 (0xef90 bytes) 000048 1749478727.249799 read 0x033752 thru 0x036ef3 (0x37a2 bytes) 000049 1749478727.251734 punch from 0x0177bb to 0x01dc55 (0x649b bytes) 000050 1749478727.253558 fallocate 0x015b02 thru 0x01642e (0x92d bytes) 000052 1749478727.255245 punch from 0x02fedc to 0x03528d (0x53b2 bytes) 000053 1749478727.257044 mapread 0x02d14c thru 0x02fcae (0x2b63 bytes) 000054 1749478727.259784 read 0x006a19 thru 0x00a284 (0x386c bytes) 000055 1749478727.262190 write 0x016940 thru 0x0195f6 (0x2cb7 bytes) 000056 1749478727.263782 trunc from 0x03904a to 0x011966 (0x276e3 bytes) 000057 1749478727.273731 MIRROR_RESYNC 3 mirrors 000059 1749478727.418791 punch from 0x00c8cf to 0x00f713 (0x2e45 bytes) 000060 1749478727.426650 punch from 0x00b2de to 0x011966 (0x6689 bytes) 000061 1749478727.427906 write 0x0030f2 thru 0x007b96 (0x4aa5 bytes) 000062 1749478727.430102 write 0x01434b thru 0x017d40 (0x39f6 bytes) 000063 1749478727.430747 trunc from 0x00cb1f to 0x015f5d (0x943f bytes) 000064 1749478727.433849 fallocate 0x00590a thru 0x007274 (0x196b bytes) 000066 1749478727.435614 read 0x00ee63 thru 0x017d40 (0x8ede bytes) 000067 1749478727.437978 mapwrite 0x01a35c thru 0x024803 (0xa4a8 bytes) 000068 1749478727.455021 fallocate 0x02ca33 thru 0x031f5f (0x552d bytes) 000070 1749478727.456589 mapread 0x0169e0 thru 0x024803 (0xde24 bytes) 000071 1749478727.458199 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000072 1749478727.503986 mapread 0x00e09a thru 0x0180f4 (0xa05b bytes) 000073 1749478727.511209 punch from 0x005e23 to 0x0088de (0x2abc bytes) 000074 1749478727.513087 trunc from 0x024804 to 0x009e9c (0x1a967 bytes) 000075 1749478727.519382 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000076 1749478727.548600 trunc from 0x008ef6 to 0x009e9c (0xfa7 bytes) 000077 1749478727.550056 fallocate 0x0175c8 thru 0x01f1db (0x7c14 bytes) 000078 1749478727.551374 trunc from 0x01f1dc to 0x005b68 (0x19673 bytes) fallocating to largest ever: 0x3f648 000079 1749478727.556413 fallocate 0x039352 thru 0x03f647 (0x62f6 bytes) 000080 1749478727.559358 MIRROR_EXTEND to 2 mirrors 000081 1749478727.651686 trunc from 0x03f648 to 0x03daf4 (0x1b53 bytes) 000082 1749478727.664775 mapwrite 0x03e1f4 thru 0x03ffff (0x1e0c bytes) 000083 1749478727.676047 mapread 0x02197a thru 0x030e3a (0xf4c1 bytes) 000084 1749478727.680974 trunc from 0x040000 to 0x0175f7 (0x28a08 bytes) 000085 1749478727.685022 write 0x02bbbe thru 0x038435 (0xc878 bytes) 000086 1749478727.686010 read 0x001e6b thru 0x002a10 (0xba6 bytes) 000087 1749478727.687717 fallocate 0x0131bc thru 0x015650 (0x2495 bytes) 000088 1749478727.689418 write 0x02d448 thru 0x034ad1 (0x768a bytes) 000089 1749478727.690121 mapwrite 0x01b047 thru 0x01b8fd (0x8b7 bytes) 000090 1749478727.697322 trunc from 0x015a4f to 0x01626f (0x821 bytes) 000091 1749478727.698889 write 0x011d4a thru 0x015ed7 (0x418e bytes) 000092 1749478727.701459 write 0x00af0b thru 0x00f6df (0x47d5 bytes) 000093 1749478727.702206 trunc from 0x038436 to 0x02488a (0x13bab bytes) 000094 1749478727.706047 mapwrite 0x018647 thru 0x027b62 (0xf51c bytes) 000095 1749478727.719910 mapwrite 0x026fb1 thru 0x0356eb (0xe73b bytes) 000096 1749478727.734274 mapwrite 0x01100e thru 0x017138 (0x612b bytes) 000097 1749478727.739813 write 0x000c1b thru 0x007164 (0x654a bytes) 000098 1749478727.742200 mapread 0x02b290 thru 0x0348ff (0x9670 bytes) 000099 1749478727.742816 punch from 0x00e51d to 0x01074b (0x222f bytes) 000101 1749478727.744682 punch from 0x028643 to 0x0356eb (0xd0a9 bytes) 000102 1749478727.745997 write 0x036949 thru 0x03ffff (0x96b7 bytes) 000103 1749478727.746764 trunc from 0x03b87a to 0x03ffff (0x4786 bytes) 000105 1749478727.750392 trunc from 0x040000 to 0x03df55 (0x20aa bytes) fallocating to largest ever: 0x40000 000106 1749478727.754196 fallocate 0x03b506 thru 0x03ffff (0x4afa bytes) 000107 1749478727.755759 fallocate 0x00e29c thru 0x01989d (0xb602 bytes) 000108 1749478727.758236 MIRROR_RESYNC 2 mirrors 000109 1749478727.871923 mapread 0x02f5cb thru 0x02f8a3 (0x2d9 bytes) 000110 1749478727.873512 trunc from 0x00b2c6 to 0x012709 (0x7444 bytes) 000111 1749478727.881361 read 0x009a00 thru 0x00c9eb (0x2fec bytes) 000112 1749478727.883086 read 0x03b22a thru 0x03df55 (0x2d2c bytes) 000113 1749478727.886692 fallocate 0x03ddea thru 0x03ffff (0x2216 bytes) 000114 1749478727.888040 trunc from 0x03b6b3 to 0x03ffff (0x494d bytes) 000115 1749478727.889584 mapread 0x0114ec thru 0x0202b2 (0xedc7 bytes) 000116 1749478727.890234 fallocate 0x025f31 thru 0x02c255 (0x6325 bytes) 000118 1749478727.891759 mapread 0x0003fc thru 0x000fa1 (0xba6 bytes) 000119 1749478727.892296 read 0x0244d5 thru 0x028e45 (0x4971 bytes) 000120 1749478727.892706 mapwrite 0x0067f4 thru 0x0100bb (0x98c8 bytes) 000121 1749478727.898514 fallocate 0x02ac80 thru 0x02b2d6 (0x657 bytes) 000122 1749478727.899992 fallocate 0x031f2b thru 0x03bd57 (0x9e2d bytes) 000123 1749478727.901487 punch from 0x023de3 to 0x025a9d (0x1cbb bytes) 000124 1749478727.902947 mapread 0x00c753 thru 0x012d0e (0x65bc bytes) 000125 1749478727.903368 fallocate 0x023ed0 thru 0x033e7e (0xffaf bytes) 000127 1749478727.905385 punch from 0x0057ab to 0x011f7b (0xc7d1 bytes) 000129 1749478727.907460 trunc from 0x040000 to 0x033475 (0xcb8a bytes) 000130 1749478727.915791 mapread 0x03083b thru 0x033475 (0x2c3b bytes) 000131 1749478727.918786 read 0x0247d0 thru 0x02cb70 (0x83a1 bytes) 000132 1749478727.921544 read 0x00d9da thru 0x013ec0 (0x64e7 bytes) 000133 1749478727.922025 read 0x02933f thru 0x0320af (0x8d71 bytes) 000134 1749478727.922461 mapread 0x01b740 thru 0x01d6ea (0x1fab bytes) 000135 1749478727.923021 write 0x0257fe thru 0x02a21d (0x4a20 bytes) 000136 1749478727.923777 mapread 0x0222fe thru 0x027ce1 (0x59e4 bytes) 000137 1749478727.924387 punch from 0x02fd50 to 0x033475 (0x3726 bytes) 000138 1749478727.926060 mapread 0x017ce6 thru 0x01b5b6 (0x38d1 bytes) 000139 1749478727.926574 fallocate 0x01460a thru 0x021649 (0xd040 bytes) 000140 1749478727.928033 mapread 0x00008b thru 0x00ec69 (0xebdf bytes) 000141 1749478727.928622 trunc from 0x033476 to 0x024986 (0xeaef bytes) 000142 1749478727.933609 MIRROR_RESYNC 2 mirrors 000144 1749478728.011475 trunc from 0x01b5d9 to 0x01f792 (0x41ba bytes) 000145 1749478728.019644 mapread 0x00ea7b thru 0x017c37 (0x91bd bytes) 000146 1749478728.025095 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000148 1749478728.055305 read 0x0192fb thru 0x021f73 (0x8c79 bytes) 000149 1749478728.057601 trunc from 0x024987 to 0x03fdb4 (0x1b42e bytes) 000150 1749478728.065920 trunc from 0x03abf9 to 0x03fdb4 (0x51bc bytes) 000151 1749478728.067369 write 0x019051 thru 0x02157f (0x852f bytes) 000152 1749478728.068246 trunc from 0x03fdb5 to 0x02c894 (0x13520 bytes) 000153 1749478728.071551 punch from 0x02a512 to 0x02c894 (0x2383 bytes) 000154 1749478728.072918 fallocate 0x0147a7 thru 0x021ed0 (0xd72a bytes) 000155 1749478728.074275 mapwrite 0x030c1d thru 0x032eb6 (0x229a bytes) 000156 1749478728.086716 write 0x01f24f thru 0x0222fc (0x30ae bytes) 000157 1749478728.088414 trunc from 0x02ebeb to 0x032eb6 (0x42cc bytes) 000158 1749478728.090054 read 0x030b05 thru 0x032eb6 (0x23b2 bytes) 000159 1749478728.093235 trunc from 0x01df6d to 0x021bdc (0x3c70 bytes) 000160 1749478728.096009 read 0x02328c thru 0x02d839 (0xa5ae bytes) 000162 1749478728.097063 read 0x00610d thru 0x00d798 (0x768c bytes) 000163 1749478728.097566 write 0x008a10 thru 0x0162c0 (0xd8b1 bytes) 000164 1749478728.098708 trunc from 0x006873 to 0x0095a0 (0x2d2e bytes) 000166 1749478728.103268 mapwrite 0x039510 thru 0x03bf21 (0x2a12 bytes) 000167 1749478728.114321 trunc from 0x03bf22 to 0x00e478 (0x2daa9 bytes) 000168 1749478728.118534 read 0x000708 thru 0x00e478 (0xdd71 bytes) 000170 1749478728.119513 trunc from 0x00e479 to 0x039719 (0x2b2a1 bytes) 000171 1749478728.123320 trunc from 0x0030cc to 0x00e53f (0xb474 bytes) 000175 1749478728.125914 mapread 0x02ec8d thru 0x039719 (0xaa8d bytes) 000176 1749478728.129778 mapread 0x00a946 thru 0x0143a5 (0x9a60 bytes) 000177 1749478728.130474 mapread 0x02eeec thru 0x039719 (0xa82e bytes) 000178 1749478728.130915 read 0x0345ac thru 0x039719 (0x516e bytes) 000180 1749478728.131768 trunc from 0x018b63 to 0x0271b7 (0xe655 bytes) 000181 1749478728.133567 punch from 0x005254 to 0x012d8a (0xdb37 bytes) 000182 1749478728.135055 fallocate 0x008b49 thru 0x00f3c5 (0x687d bytes) 000183 1749478728.136597 mapwrite 0x020c60 thru 0x0240be (0x345f bytes) 000184 1749478728.144897 punch from 0x001026 to 0x0065fe (0x55d9 bytes) 000185 1749478728.146475 write 0x03cda3 thru 0x03ffff (0x325d bytes) 000186 1749478728.147112 fallocate 0x0336cf thru 0x03c503 (0x8e35 bytes) 000187 1749478728.148631 punch from 0x03b0bf to 0x03ffff (0x4f41 bytes) 000188 1749478728.151344 fallocate 0x032010 thru 0x033fbe (0x1faf bytes) 000189 1749478728.152867 mapwrite 0x036381 thru 0x03ffff (0x9c7f bytes) 000190 1749478728.159694 fallocate 0x000187 thru 0x00d079 (0xcef3 bytes) 000191 1749478728.161697 write 0x01d111 thru 0x0213a2 (0x4292 bytes) 000192 1749478728.162340 write 0x03b26a thru 0x03ec8a (0x3a21 bytes) 000193 1749478728.162860 trunc from 0x040000 to 0x020633 (0x1f9cc bytes) 000194 1749478728.167145 read 0x018c7d thru 0x01d73e (0x4ac2 bytes) 000198 1749478728.169161 mapread 0x0112c3 thru 0x01848e (0x71cc bytes) 000199 1749478728.169842 punch from 0x006484 to 0x012845 (0xc3c2 bytes) 000200 1749478728.171477 trunc from 0x002d50 to 0x00f9cb (0xcc7c bytes) 000201 1749478728.172698 punch from 0x017011 to 0x020633 (0x9623 bytes) 000202 1749478728.175628 read 0x01c1c2 thru 0x020633 (0x4472 bytes) 000203 1749478728.177982 write 0x002244 thru 0x005616 (0x33d3 bytes) 000204 1749478728.178506 read 0x005033 thru 0x0074cc (0x249a bytes) 000205 1749478728.178797 punch from 0x0075e5 to 0x0130f1 (0xbb0d bytes) 000206 1749478728.180229 trunc from 0x020634 to 0x03c987 (0x1c354 bytes) 000207 1749478728.183933 punch from 0x0145a2 to 0x01da03 (0x9462 bytes) 000209 1749478728.185666 trunc from 0x03c988 to 0x025391 (0x175f6 bytes) 000210 1749478728.189131 trunc from 0x013518 to 0x021eb7 (0xe9a0 bytes) 000211 1749478728.190703 punch from 0x000301 to 0x00209e (0x1d9e bytes) 000212 1749478728.193569 fallocate 0x02b667 thru 0x033e15 (0x87af bytes) 000213 1749478728.195224 fallocate 0x02600a thru 0x031dce (0xbdc5 bytes) 000214 1749478728.196715 trunc from 0x00130f to 0x00f421 (0xe113 bytes) 000217 1749478728.199033 mapread 0x01a5dc thru 0x024772 (0xa197 bytes) 000218 1749478728.202511 punch from 0x02254d to 0x02b6b2 (0x9166 bytes) 000219 1749478728.203926 write 0x0110dd thru 0x018fc9 (0x7eed bytes) 000220 1749478728.204683 mapread 0x00741b thru 0x008b3b (0x1721 bytes) 000221 1749478728.205071 trunc from 0x033e16 to 0x03d9c5 (0x9bb0 bytes) 000224 1749478728.209429 punch from 0x00b376 to 0x00b81a (0x4a5 bytes) 000225 1749478728.212689 MIRROR_EXTEND to 2 mirrors 000226 1749478728.325993 trunc from 0x03d9c6 to 0x024713 (0x192b2 bytes) 000227 1749478728.338987 write 0x028c02 thru 0x0352a5 (0xc6a4 bytes) 000229 1749478728.340227 fallocate 0x0304af thru 0x03f903 (0xf455 bytes) 000230 1749478728.341758 fallocate 0x00264a thru 0x008ebf (0x6876 bytes) 000231 1749478728.342990 mapwrite 0x01aad2 thru 0x01ab2e (0x5d bytes) 000233 1749478728.349452 trunc from 0x02e301 to 0x03b6e9 (0xd3e9 bytes) 000234 1749478728.350832 write 0x0010eb thru 0x00d910 (0xc826 bytes) 000236 1749478728.353660 punch from 0x00d293 to 0x014d07 (0x7a75 bytes) 000237 1749478728.357063 mapread 0x027fe5 thru 0x02f762 (0x777e bytes) 000238 1749478728.361265 read 0x015dfb thru 0x01b743 (0x5949 bytes) 000239 1749478728.363759 mapwrite 0x007644 thru 0x01494e (0xd30b bytes) 000240 1749478728.371610 fallocate 0x03d562 thru 0x03ffff (0x2a9e bytes) 000241 1749478728.373012 read 0x00cf4d thru 0x014aee (0x7ba2 bytes) 000242 1749478728.373354 trunc from 0x00ad95 to 0x016adc (0xbd48 bytes) 000243 1749478728.374706 punch from 0x00ae15 to 0x01535e (0xa54a bytes) 000244 1749478728.377343 MIRROR_EXTEND to 3 mirrors 000246 1749478728.483871 fallocate 0x02803f thru 0x0366f8 (0xe6ba bytes) 000247 1749478728.493566 punch from 0x029885 to 0x034958 (0xb0d4 bytes) 000248 1749478728.495440 write 0x020606 thru 0x024a2c (0x4427 bytes) 000249 1749478728.500284 trunc from 0x00dfdc to 0x0171f4 (0x9219 bytes) 000250 1749478728.503747 MIRROR_RESYNC 3 mirrors 000251 1749478728.668037 mapwrite 0x01728f thru 0x020ddc (0x9b4e bytes) 000252 1749478728.697159 mapread 0x03503f thru 0x039d56 (0x4d18 bytes) 000253 1749478728.700969 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000254 1749478728.744901 read 0x026e0e thru 0x02fdd2 (0x8fc5 bytes) 000256 1749478728.747669 punch from 0x035254 to 0x03d818 (0x85c5 bytes) 000259 1749478728.749920 trunc from 0x040000 to 0x003e56 (0x3c1a9 bytes) 000260 1749478728.758461 trunc from 0x0011b9 to 0x003e56 (0x2c9e bytes) 000261 1749478728.759982 punch from 0x0028c0 to 0x003e56 (0x1597 bytes) 000262 1749478728.761197 write 0x035094 thru 0x03d6b2 (0x861f bytes) 000263 1749478728.762224 write 0x02229b thru 0x024b00 (0x2866 bytes) 000264 1749478728.764679 punch from 0x03a94d to 0x03d6b2 (0x2d66 bytes) 000265 1749478728.768104 punch from 0x029453 to 0x031eeb (0x8a99 bytes) 000266 1749478728.769457 trunc from 0x016106 to 0x01f224 (0x911f bytes) 000267 1749478728.771086 mapread 0x035509 thru 0x03d6b2 (0x81aa bytes) 000268 1749478728.773077 mapread 0x0010eb thru 0x007bb5 (0x6acb bytes) 000269 1749478728.775500 write 0x00e757 thru 0x018420 (0x9cca bytes) 000270 1749478728.777231 trunc from 0x02e648 to 0x03362c (0x4fe5 bytes) 000271 1749478728.779709 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000272 1749478728.809464 mapwrite 0x003040 thru 0x00a7fe (0x77bf bytes) 000273 1749478728.819029 trunc from 0x03d6b3 to 0x023047 (0x1a66b bytes) 000274 1749478728.823822 mapwrite 0x0329ab thru 0x039f39 (0x758f bytes) 000277 1749478728.836237 mapwrite 0x009da7 thru 0x00fba2 (0x5dfc bytes) 000279 1749478728.841661 mapwrite 0x03ce21 thru 0x03ffff (0x31df bytes) 000280 1749478728.851897 fallocate 0x0132e5 thru 0x01440a (0x1126 bytes) 000281 1749478728.853729 punch from 0x00f5dd to 0x010445 (0xe69 bytes) 000282 1749478728.855283 trunc from 0x0165f4 to 0x01cfed (0x69fa bytes) 000283 1749478728.856854 fallocate 0x022a2e thru 0x02ea5e (0xc031 bytes) 000284 1749478728.858785 trunc from 0x040000 to 0x038a43 (0x75bc bytes) 000285 1749478728.863685 trunc from 0x038a44 to 0x01539b (0x236a8 bytes) 000287 1749478728.867881 mapwrite 0x0004ba thru 0x00c1f3 (0xbd3a bytes) 000288 1749478728.876618 write 0x00b60c thru 0x015c5c (0xa651 bytes) 000289 1749478728.878895 trunc from 0x015c5d to 0x00dd02 (0x7f5a bytes) 000290 1749478728.883017 read 0x0021d6 thru 0x006208 (0x4033 bytes) 000291 1749478728.883444 punch from 0x001474 to 0x00d65e (0xc1eb bytes) 000292 1749478728.886229 trunc from 0x006380 to 0x00b0ea (0x4d6b bytes) 000293 1749478728.887699 mapwrite 0x03198d thru 0x03275b (0xdcf bytes) 000294 1749478728.898505 trunc from 0x03275c to 0x02a6fd (0x805e bytes) 000295 1749478728.901960 mapwrite 0x0097e6 thru 0x00c356 (0x2b71 bytes) 000297 1749478728.908974 mapread 0x017f74 thru 0x022703 (0xa790 bytes) 000298 1749478728.913239 write 0x03d9b8 thru 0x03ffff (0x2648 bytes) 000299 1749478728.913888 mapread 0x0140d7 thru 0x01cdf3 (0x8d1d bytes) 000301 1749478728.914903 write 0x03f192 thru 0x03ffff (0xe6e bytes) 000302 1749478728.915249 punch from 0x0377cc to 0x03ffff (0x8834 bytes) 000303 1749478728.917989 read 0x024aeb thru 0x030cae (0xc1c4 bytes) 000304 1749478728.920597 read 0x003365 thru 0x00417b (0xe17 bytes) 000305 1749478728.921077 read 0x01c8ce thru 0x02ab30 (0xe263 bytes) 000306 1749478728.921549 write 0x0122b8 thru 0x016ddf (0x4b28 bytes) 000307 1749478728.922088 trunc from 0x035a6a to 0x03911a (0x36b1 bytes) 000308 1749478728.923402 fallocate 0x015cdf thru 0x019a11 (0x3d33 bytes) 000309 1749478728.924663 mapwrite 0x0305e1 thru 0x03aecf (0xa8ef bytes) 000310 1749478728.931940 trunc from 0x03f209 to 0x03ffff (0xdf7 bytes) 000311 1749478728.933129 write 0x039fef thru 0x03ffff (0x6011 bytes) 000312 1749478728.933690 trunc from 0x040000 to 0x021c54 (0x1e3ab bytes) 000314 1749478728.937978 mapread 0x0201de thru 0x021c54 (0x1a77 bytes) 000315 1749478728.938534 mapwrite 0x010e06 thru 0x017d70 (0x6f6b bytes) 000316 1749478728.944140 fallocate 0x036033 thru 0x03ffff (0x9fcd bytes) 000318 1749478728.945907 mapread 0x01c118 thru 0x025adf (0x99c8 bytes) 000319 1749478728.950600 mapread 0x0329c5 thru 0x03ffff (0xd63b bytes) 000321 1749478728.966067 MIRROR_EXTEND to 2 mirrors 000322 1749478729.073807 trunc from 0x040000 to 0x027c77 (0x18388 bytes) 000323 1749478729.084270 mapread 0x00eaca thru 0x01e4e1 (0xfa18 bytes) 000324 1749478729.090782 write 0x01bd4e thru 0x0234d9 (0x778c bytes) 000325 1749478729.091798 fallocate 0x01b303 thru 0x028ba3 (0xd8a1 bytes) 000326 1749478729.093421 mapread 0x020cf1 thru 0x027c77 (0x6f87 bytes) 000327 1749478729.095108 mapread 0x008d65 thru 0x017146 (0xe3e2 bytes) 000328 1749478729.097404 read 0x01a57a thru 0x01fbe5 (0x566c bytes) 000329 1749478729.097773 write 0x019781 thru 0x0219b1 (0x8231 bytes) 000330 1749478729.098465 trunc from 0x027c78 to 0x02490d (0x336a bytes) 000331 1749478729.104267 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000333 1749478729.138317 trunc from 0x02490e to 0x0000db (0x24832 bytes) 000334 1749478729.142575 trunc from 0x000050 to 0x0000db (0x8c bytes) 000335 1749478729.143870 mapwrite 0x03047d thru 0x0348bf (0x4443 bytes) 000336 1749478729.156597 MIRROR_EXTEND to 2 mirrors 000337 1749478729.245603 write 0x02a1f8 thru 0x03586c (0xb675 bytes) 000338 1749478729.255248 trunc from 0x007227 to 0x00a16e (0x2f48 bytes) 000340 1749478729.257745 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000341 1749478729.290826 punch from 0x021e55 to 0x02263f (0x7eb bytes) 000342 1749478729.292150 read 0x012bd4 thru 0x021d0b (0xf138 bytes) 000343 1749478729.294420 read 0x02046c thru 0x029be6 (0x977b bytes) 000344 1749478729.297014 write 0x00675c thru 0x00f7a2 (0x9047 bytes) 000345 1749478729.298874 trunc from 0x020064 to 0x02adfa (0xad97 bytes) 000346 1749478729.300507 punch from 0x00d718 to 0x01a67a (0xcf63 bytes) 000347 1749478729.303894 mapwrite 0x03cf77 thru 0x03ffff (0x3089 bytes) 000348 1749478729.318051 trunc from 0x036064 to 0x03c0b6 (0x6053 bytes) 000349 1749478729.319708 write 0x025d91 thru 0x0300b1 (0xa321 bytes) 000350 1749478729.321866 mapwrite 0x023cdc thru 0x027855 (0x3b7a bytes) 000351 1749478729.328892 trunc from 0x01edff to 0x0204ec (0x16ee bytes) 000352 1749478729.330226 trunc from 0x040000 to 0x01f797 (0x20868 bytes) 000353 1749478729.334087 punch from 0x00062d to 0x00c7a7 (0xc17b bytes) 000354 1749478729.336940 MIRROR_EXTEND to 2 mirrors 000355 1749478729.432154 mapwrite 0x02cc9e thru 0x02e16a (0x14cd bytes) 000356 1749478729.454609 punch from 0x01f08d to 0x01f490 (0x404 bytes) 000357 1749478729.456441 trunc from 0x02e16b to 0x02c36c (0x1dfe bytes) 000359 1749478729.461954 punch from 0x0268a3 to 0x02c36c (0x5aca bytes) 000360 1749478729.463538 mapwrite 0x01898c thru 0x01e7bd (0x5e32 bytes) 000361 1749478729.472664 trunc from 0x0054ad to 0x00e5b5 (0x9109 bytes) 000362 1749478729.474306 punch from 0x00d681 to 0x01cd1f (0xf69f bytes) 000363 1749478729.475744 trunc from 0x00e4a4 to 0x016ff9 (0x8b56 bytes) 000364 1749478729.478310 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000365 1749478729.512348 mapread 0x0082f9 thru 0x00c7ff (0x4507 bytes) 000367 1749478729.517387 mapwrite 0x013e58 thru 0x017d85 (0x3f2e bytes) 000368 1749478729.523700 write 0x0019cb thru 0x006ae7 (0x511d bytes) 000372 1749478729.528394 punch from 0x00a609 to 0x014575 (0x9f6d bytes) 000373 1749478729.530006 mapread 0x020ec4 thru 0x025dfa (0x4f37 bytes) 000375 1749478729.533365 punch from 0x011025 to 0x01f97f (0xe95b bytes) 000376 1749478729.534982 mapread 0x00d902 thru 0x0198c6 (0xbfc5 bytes) 000377 1749478729.539701 trunc from 0x012a45 to 0x0209c6 (0xdf82 bytes) 000379 1749478729.541779 mapread 0x01b1b6 thru 0x026065 (0xaeb0 bytes) 000380 1749478729.544393 read 0x0258ab thru 0x02c36c (0x6ac2 bytes) 000381 1749478729.544817 trunc from 0x00aaf1 to 0x00b167 (0x677 bytes) 000382 1749478729.546230 punch from 0x02bbb3 to 0x02c36c (0x7ba bytes) 000383 1749478729.547816 mapread 0x024ad9 thru 0x02c36c (0x7894 bytes) 000385 1749478729.550153 fallocate 0x009efd thru 0x017564 (0xd668 bytes) 000386 1749478729.551440 trunc from 0x020120 to 0x02bdab (0xbc8c bytes) 000387 1749478729.552983 punch from 0x019121 to 0x01b06e (0x1f4e bytes) 000388 1749478729.554290 write 0x01388d thru 0x014313 (0xa87 bytes) 000389 1749478729.557537 MIRROR_EXTEND to 2 mirrors 000390 1749478729.658061 mapread 0x00cc3d thru 0x01260a (0x59ce bytes) 000391 1749478729.662432 read 0x00fb8c thru 0x01a60c (0xaa81 bytes) 000392 1749478729.663918 mapwrite 0x026ddc thru 0x029c3b (0x2e60 bytes) 000393 1749478729.691423 read 0x029f1d thru 0x02c36c (0x2450 bytes) 000394 1749478729.691908 trunc from 0x025cd7 to 0x02c36c (0x6696 bytes) 000395 1749478729.693931 fallocate 0x022de1 thru 0x0231c5 (0x3e5 bytes) 000396 1749478729.695512 read 0x00ff66 thru 0x01da6d (0xdb08 bytes) 000397 1749478729.698857 punch from 0x01bf29 to 0x021b27 (0x5bff bytes) 000398 1749478729.700189 punch from 0x01818d to 0x025ad2 (0xd946 bytes) 000399 1749478729.701825 trunc from 0x02c36d to 0x0264cd (0x5e9f bytes) 000400 1749478729.710732 mapread 0x00b266 thru 0x014a78 (0x9813 bytes) 000401 1749478729.711509 trunc from 0x0264ce to 0x0388d1 (0x12404 bytes) 000402 1749478729.715287 mapread 0x0141bd thru 0x01dcc4 (0x9b08 bytes) 000403 1749478729.722651 read 0x0343b0 thru 0x0366e2 (0x2333 bytes) 000404 1749478729.723013 write 0x00e690 thru 0x0121d7 (0x3b48 bytes) 000405 1749478729.723616 mapread 0x02d4e6 thru 0x0388d1 (0xb3ec bytes) 000406 1749478729.724158 fallocate 0x036959 thru 0x03ffff (0x96a7 bytes) 000407 1749478729.725756 punch from 0x0360d7 to 0x03eca5 (0x8bcf bytes) 000408 1749478729.728241 MIRROR_EXTEND to 3 mirrors 000409 1749478729.832680 trunc from 0x01cd53 to 0x01ede5 (0x2093 bytes) 000410 1749478729.841983 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000411 1749478729.884316 trunc from 0x040000 to 0x03eaca (0x1535 bytes) 000412 1749478729.892129 write 0x031df9 thru 0x03daa2 (0xbcaa bytes) 000413 1749478729.895237 mapwrite 0x00fc90 thru 0x010608 (0x979 bytes) 000414 1749478729.902429 fallocate 0x028920 thru 0x02f9b4 (0x7095 bytes) 000415 1749478729.904237 punch from 0x034f10 to 0x03eaca (0x9bbb bytes) 000416 1749478729.905814 fallocate 0x017627 thru 0x01fdfd (0x87d7 bytes) 000417 1749478729.907319 trunc from 0x00e27e to 0x016aad (0x8830 bytes) 000418 1749478729.908694 read 0x031d4c thru 0x035619 (0x38ce bytes) 000419 1749478729.910376 fallocate 0x00896e thru 0x012bd2 (0xa265 bytes) 000420 1749478729.911785 trunc from 0x01b722 to 0x01c61d (0xefc bytes) 000421 1749478729.913239 fallocate 0x015fda thru 0x02481f (0xe846 bytes) 000423 1749478729.915422 trunc from 0x03eacb to 0x01755d (0x2756d bytes) 000424 1749478729.919726 mapread 0x0019b7 thru 0x00cb41 (0xb18b bytes) 000425 1749478729.922328 mapread 0x00110f thru 0x0015ca (0x4bc bytes) 000426 1749478729.922856 trunc from 0x01755e to 0x03b7f6 (0x24299 bytes) 000427 1749478729.926379 trunc from 0x0121d6 to 0x01adce (0x8bf9 bytes) 000428 1749478729.927841 mapwrite 0x004291 thru 0x00fb13 (0xb883 bytes) 000429 1749478729.935830 punch from 0x033055 to 0x03b7f6 (0x87a2 bytes) 000430 1749478729.937302 trunc from 0x0067ad to 0x0103f8 (0x9c4c bytes) 000431 1749478729.939010 fallocate 0x03e028 thru 0x03ffff (0x1fd8 bytes) 000432 1749478729.940596 trunc from 0x00cec7 to 0x01c733 (0xf86d bytes) 000433 1749478729.941980 trunc from 0x03b7f7 to 0x011770 (0x2a086 bytes) 000434 1749478729.946967 MIRROR_EXTEND to 3 mirrors 000435 1749478730.050399 write 0x011df5 thru 0x01c3bc (0xa5c8 bytes) 000436 1749478730.059269 fallocate 0x0236fd thru 0x02eb69 (0xb46d bytes) 000437 1749478730.060741 write 0x00fe34 thru 0x01f473 (0xf640 bytes) 000438 1749478730.063640 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000439 1749478730.106104 trunc from 0x01f474 to 0x03169e (0x1222b bytes) 000440 1749478730.110446 punch from 0x020c32 to 0x02fd50 (0xf11f bytes) 000441 1749478730.111953 trunc from 0x0171e8 to 0x025b63 (0xe97c bytes) 000442 1749478730.113427 mapread 0x01dfd8 thru 0x01dff9 (0x22 bytes) 000443 1749478730.115010 trunc from 0x016697 to 0x024174 (0xdade bytes) 000444 1749478730.116252 fallocate 0x025ba1 thru 0x03348c (0xd8ec bytes) 000445 1749478730.118894 MIRROR_RESYNC 2 mirrors 000446 1749478730.205358 write 0x00f2e1 thru 0x01680c (0x752c bytes) 000447 1749478730.214009 mapwrite 0x016b2e thru 0x0239ce (0xcea1 bytes) 000448 1749478730.224496 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000449 1749478730.254001 mapread 0x01bff6 thru 0x01c89e (0x8a9 bytes) 000450 1749478730.256790 write 0x0253b0 thru 0x03072f (0xb380 bytes) 000451 1749478730.262119 trunc from 0x03169f to 0x030bc2 (0xadc bytes) 000452 1749478730.269678 trunc from 0x011b31 to 0x02092f (0xedff bytes) 000453 1749478730.271368 mapread 0x003cd1 thru 0x004de2 (0x1112 bytes) 000454 1749478730.274247 write 0x01cf76 thru 0x023349 (0x63d4 bytes) 000456 1749478730.276967 write 0x024e47 thru 0x025817 (0x9d1 bytes) 000458 1749478730.277982 trunc from 0x0069a8 to 0x00875b (0x1db4 bytes) 000459 1749478730.279421 mapread 0x016581 thru 0x0182c9 (0x1d49 bytes) 000460 1749478730.279824 fallocate 0x039af4 thru 0x03c7cd (0x2cda bytes) 000461 1749478730.281050 trunc from 0x02401a to 0x030701 (0xc6e8 bytes) 000462 1749478730.284522 mapwrite 0x035839 thru 0x03b9bd (0x6185 bytes) 000464 1749478730.295872 mapwrite 0x028329 thru 0x02b86e (0x3546 bytes) 000465 1749478730.301563 trunc from 0x03b9be to 0x01def1 (0x1dacc bytes) 000467 1749478730.305225 fallocate 0x03fb11 thru 0x03ffff (0x4ef bytes) 000469 1749478730.307079 punch from 0x00f79e to 0x01def1 (0xe754 bytes) 000470 1749478730.309528 MIRROR_EXTEND to 2 mirrors 000471 1749478730.404347 read 0x013dab thru 0x01def1 (0xa147 bytes) 000472 1749478730.406936 read 0x00e61a thru 0x0118dc (0x32c3 bytes) 000473 1749478730.409255 write 0x02f9da thru 0x0319bf (0x1fe6 bytes) 000474 1749478730.418433 trunc from 0x009022 to 0x012352 (0x9331 bytes) 000475 1749478730.421305 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000477 1749478730.455151 trunc from 0x0319c0 to 0x01f65d (0x12362 bytes) 000479 1749478730.464149 fallocate 0x019293 thru 0x01d314 (0x4082 bytes) 000480 1749478730.465864 mapwrite 0x015518 thru 0x01e157 (0x8c40 bytes) 000481 1749478730.476425 mapwrite 0x0389c2 thru 0x03ffff (0x763e bytes) 000482 1749478730.487316 write 0x03323e thru 0x0385d7 (0x539a bytes) 000483 1749478730.488267 mapread 0x01ae73 thru 0x025119 (0xa2a7 bytes) 000484 1749478730.490976 punch from 0x032b72 to 0x03cfa0 (0xa42f bytes) 000485 1749478730.494138 write 0x013f48 thru 0x021c4d (0xdd06 bytes) 000486 1749478730.495180 punch from 0x02a8d3 to 0x03178d (0x6ebb bytes) 000488 1749478730.496788 fallocate 0x01c441 thru 0x0268c7 (0xa487 bytes) 000489 1749478730.498206 trunc from 0x002157 to 0x00a82a (0x86d4 bytes) 000490 1749478730.499618 mapread 0x00dd2d thru 0x012390 (0x4664 bytes) 000492 1749478730.502299 read 0x028121 thru 0x031ad8 (0x99b8 bytes) 000493 1749478730.504249 write 0x0101d6 thru 0x02014c (0xff77 bytes) 000494 1749478730.505017 trunc from 0x040000 to 0x02428c (0x1bd73 bytes) 000495 1749478730.508875 punch from 0x0023e3 to 0x00c75b (0xa379 bytes) 000498 1749478730.510795 write 0x01d88c thru 0x025cd7 (0x844c bytes) 000499 1749478730.511408 trunc from 0x004485 to 0x010909 (0xc485 bytes) 000500 1749478730.514706 trunc from 0x01eb39 to 0x02335a (0x4822 bytes) 000501 1749478730.515856 write 0x03f44b thru 0x03ffff (0xbb5 bytes) 000502 1749478730.516282 mapread 0x02fdc6 thru 0x038d68 (0x8fa3 bytes) 000503 1749478730.519969 mapwrite 0x00b531 thru 0x011c8a (0x675a bytes) 000504 1749478730.527946 read 0x00e366 thru 0x01a3e1 (0xc07c bytes) 000505 1749478730.528322 read 0x01473c thru 0x01d311 (0x8bd6 bytes) 000506 1749478730.528708 trunc from 0x01fcd2 to 0x02a063 (0xa392 bytes) 000507 1749478730.531268 MIRROR_EXTEND to 2 mirrors 000508 1749478730.632019 fallocate 0x025af8 thru 0x031d3c (0xc245 bytes) 000510 1749478730.640778 trunc from 0x040000 to 0x039515 (0x6aea bytes) 000511 1749478730.649573 mapread 0x034fa0 thru 0x039515 (0x4576 bytes) 000512 1749478730.654379 write 0x008888 thru 0x00912b (0x8a4 bytes) 000513 1749478730.656759 trunc from 0x039516 to 0x00604b (0x334ca bytes) 000514 1749478730.660919 trunc from 0x00604c to 0x030ab7 (0x2aa6c bytes) 000515 1749478730.665622 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000517 1749478730.696584 fallocate 0x01dde9 thru 0x02a5e8 (0xc800 bytes) 000518 1749478730.698240 mapread 0x01be14 thru 0x0261ce (0xa3bb bytes) 000521 1749478730.704843 punch from 0x030879 to 0x030ab7 (0x23f bytes) 000522 1749478730.706098 punch from 0x00b9b0 to 0x016b2a (0xb17b bytes) 000523 1749478730.707322 mapwrite 0x0004b4 thru 0x00d1c4 (0xcd11 bytes) 000524 1749478730.715750 mapwrite 0x034d6d thru 0x03936e (0x4602 bytes) 000525 1749478730.726574 punch from 0x015ce6 to 0x01fd29 (0xa044 bytes) 000526 1749478730.727961 mapread 0x002e5e thru 0x00a026 (0x71c9 bytes) 000527 1749478730.729464 MIRROR_EXTEND to 2 mirrors 000528 1749478730.821460 fallocate 0x00096a thru 0x00bc64 (0xb2fb bytes) 000529 1749478730.830852 read 0x0074ca thru 0x011e1c (0xa953 bytes) 000530 1749478730.833367 trunc from 0x027d6a to 0x02d8d8 (0x5b6f bytes) 000531 1749478730.834882 fallocate 0x01384b thru 0x01cea8 (0x965e bytes) 000532 1749478730.836468 punch from 0x01b72a to 0x0285df (0xceb6 bytes) 000533 1749478730.837869 punch from 0x003c69 to 0x005689 (0x1a21 bytes) 000534 1749478730.839319 mapread 0x00c542 thru 0x019c44 (0xd703 bytes) 000536 1749478730.845054 trunc from 0x009c6a to 0x00f318 (0x56af bytes) 000537 1749478730.846552 fallocate 0x0161ed thru 0x0164a7 (0x2bb bytes) 000538 1749478730.848250 trunc from 0x020b11 to 0x02eebe (0xe3ae bytes) 000539 1749478730.849720 punch from 0x009e47 to 0x009f5e (0x118 bytes) 000541 1749478730.851150 trunc from 0x03936f to 0x027f6e (0x11400 bytes) 000542 1749478730.861332 mapwrite 0x00464e thru 0x0081ea (0x3b9d bytes) 000543 1749478730.869096 write 0x02f9ab thru 0x0380c7 (0x871d bytes) 000544 1749478730.869896 read 0x02b1e5 thru 0x036950 (0xb76c bytes) 000545 1749478730.872228 punch from 0x030503 to 0x0380c7 (0x7bc5 bytes) 000546 1749478730.875661 trunc from 0x012a3a to 0x0200b4 (0xd67b bytes) 000547 1749478730.877285 trunc from 0x0380c8 to 0x0253b6 (0x12d11 bytes) 000548 1749478730.881865 mapwrite 0x02e0fd thru 0x02e988 (0x88c bytes) 000550 1749478730.892393 mapwrite 0x03f3d6 thru 0x03ffff (0xc2a bytes) 000552 1749478730.903031 MIRROR_EXTEND to 3 mirrors 000553 1749478731.009732 read 0x033e29 thru 0x03ffff (0xc1d7 bytes) 000554 1749478731.012223 trunc from 0x040000 to 0x0174c4 (0x28b3b bytes) 000555 1749478731.025229 write 0x006f4e thru 0x0095d2 (0x2685 bytes) 000556 1749478731.028007 mapread 0x0124e0 thru 0x0174c4 (0x4fe5 bytes) 000557 1749478731.030375 read 0x015711 thru 0x0174c4 (0x1db4 bytes) 000558 1749478731.030790 read 0x00bbea thru 0x0141ae (0x85c5 bytes) 000559 1749478731.032682 trunc from 0x01533e to 0x0174c4 (0x2187 bytes) 000561 1749478731.035323 MIRROR_RESYNC 3 mirrors 000562 1749478731.153742 mapread 0x00ab90 thru 0x010784 (0x5bf5 bytes) 000563 1749478731.159583 MIRROR_RESYNC 3 mirrors 000564 1749478731.195627 mapwrite 0x01192a thru 0x0189c8 (0x709f bytes) 000567 1749478731.216724 read 0x002269 thru 0x0119c2 (0xf75a bytes) 000568 1749478731.220086 MIRROR_EXTEND to 4 mirrors 000569 1749478731.306802 mapwrite 0x00d5ad thru 0x018d31 (0xb785 bytes) 000570 1749478731.332051 fallocate 0x03881a thru 0x03d8e5 (0x50cc bytes) 000571 1749478731.333543 mapwrite 0x01cbed thru 0x02b04d (0xe461 bytes) 000572 1749478731.347622 mapwrite 0x013cac thru 0x017f31 (0x4286 bytes) 000573 1749478731.351619 mapread 0x014ba2 thru 0x0242d1 (0xf730 bytes) 000574 1749478731.352279 trunc from 0x02b04e to 0x020a93 (0xa5ba bytes) 000575 1749478731.356200 punch from 0x009b22 to 0x009c1b (0xfa bytes) 000576 1749478731.357410 trunc from 0x018110 to 0x01efae (0x6e9f bytes) 000577 1749478731.358894 write 0x0339c7 thru 0x0348be (0xef8 bytes) 000578 1749478731.359732 punch from 0x02480b to 0x0254bf (0xcb5 bytes) 000580 1749478731.361265 mapread 0x016f39 thru 0x025b0d (0xebd5 bytes) 000581 1749478731.365135 punch from 0x02343b to 0x02dc68 (0xa82e bytes) 000582 1749478731.366448 mapread 0x023dbe thru 0x02506e (0x12b1 bytes) 000584 1749478731.368854 mapread 0x015dec thru 0x018747 (0x295c bytes) 000585 1749478731.369313 mapwrite 0x027eb3 thru 0x0365f2 (0xe740 bytes) 000587 1749478731.381641 read 0x022992 thru 0x027f56 (0x55c5 bytes) 000588 1749478731.382056 mapread 0x014d9d thru 0x01f922 (0xab86 bytes) 000589 1749478731.383810 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000590 1749478731.427971 punch from 0x02e880 to 0x0365f2 (0x7d73 bytes) 000591 1749478731.429623 read 0x001415 thru 0x008a37 (0x7623 bytes) 000592 1749478731.431965 mapread 0x00ddfc thru 0x01ddb5 (0xffba bytes) 000593 1749478731.438269 mapread 0x025d8a thru 0x03517e (0xf3f5 bytes) 000595 1749478731.443037 mapwrite 0x03c747 thru 0x03ffff (0x38b9 bytes) 000596 1749478731.454724 trunc from 0x040000 to 0x00092c (0x3f6d3 bytes) 000597 1749478731.458574 mapread 0x000197 thru 0x00092c (0x796 bytes) 000598 1749478731.460136 read 0x0005c5 thru 0x00092c (0x368 bytes) 000599 1749478731.460452 punch from 0x0001f9 to 0x00092c (0x734 bytes) 000600 1749478731.461685 trunc from 0x00092d to 0x01bfae (0x1b682 bytes) 000602 1749478731.465253 trunc from 0x007b05 to 0x008b5b (0x1057 bytes) 000603 1749478731.466695 read 0x012509 thru 0x01bfae (0x9aa6 bytes) 000604 1749478731.469007 punch from 0x014534 to 0x01bfae (0x7a7b bytes) 000605 1749478731.470192 trunc from 0x0030d9 to 0x00f901 (0xc829 bytes) 000606 1749478731.471505 fallocate 0x03e17c thru 0x03ffff (0x1e84 bytes) 000608 1749478731.472932 trunc from 0x000e67 to 0x00f3b6 (0xe550 bytes) 000609 1749478731.474272 fallocate 0x038894 thru 0x03ffff (0x776c bytes) 000610 1749478731.475657 mapwrite 0x03b13b thru 0x03ffff (0x4ec5 bytes) 000611 1749478731.486491 fallocate 0x01c47f thru 0x025b97 (0x9719 bytes) 000612 1749478731.489284 MIRROR_EXTEND to 4 mirrors 000613 1749478731.584876 read 0x0081f4 thru 0x00cb54 (0x4961 bytes) 000614 1749478731.587002 read 0x03704e thru 0x03ffff (0x8fb2 bytes) 000615 1749478731.589576 trunc from 0x03ea88 to 0x03ffff (0x1578 bytes) 000616 1749478731.597764 mapread 0x03bfb7 thru 0x03ffff (0x4049 bytes) 000617 1749478731.602808 mapread 0x01aa05 thru 0x021fce (0x75ca bytes) 000618 1749478731.609861 mapread 0x0036b8 thru 0x00e7ac (0xb0f5 bytes) 000619 1749478731.620878 punch from 0x007641 to 0x0159ad (0xe36d bytes) 000620 1749478731.622340 trunc from 0x040000 to 0x03ab4e (0x54b1 bytes) 000621 1749478731.637872 punch from 0x02ad73 to 0x02fa2b (0x4cb9 bytes) 000622 1749478731.639357 fallocate 0x032974 thru 0x038298 (0x5925 bytes) 000623 1749478731.640876 write 0x03d27b thru 0x03ffff (0x2d85 bytes) 000624 1749478731.641483 trunc from 0x040000 to 0x0176d0 (0x2892f bytes) 000625 1749478731.645056 mapwrite 0x02d78b thru 0x03767f (0x9ef5 bytes) 000626 1749478731.669781 read 0x0080ba thru 0x008382 (0x2c9 bytes) 000627 1749478731.673671 mapread 0x01c54f thru 0x02b901 (0xf3b3 bytes) 000629 1749478731.674675 punch from 0x00d8f7 to 0x014190 (0x689a bytes) 000630 1749478731.676521 write 0x010cb4 thru 0x01c0f7 (0xb444 bytes) 000631 1749478731.678635 write 0x036dce thru 0x03febe (0x90f1 bytes) 000632 1749478731.679399 fallocate 0x0346a8 thru 0x03ffff (0xb958 bytes) 000633 1749478731.680945 write 0x02d6ab thru 0x03be6f (0xe7c5 bytes) 000634 1749478731.681982 mapread 0x03081f thru 0x03f5ff (0xede1 bytes) 000635 1749478731.682912 trunc from 0x0363a6 to 0x03e46d (0x80c8 bytes) 000636 1749478731.686462 punch from 0x015872 to 0x020acf (0xb25e bytes) 000637 1749478731.687773 write 0x010c3f thru 0x01fad1 (0xee93 bytes) 000638 1749478731.690425 mapread 0x01282d thru 0x016091 (0x3865 bytes) 000639 1749478731.690933 mapread 0x01a3dc thru 0x01f087 (0x4cac bytes) 000640 1749478731.692642 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000641 1749478731.744403 trunc from 0x01963b to 0x027da6 (0xe76c bytes) 000642 1749478731.745998 read 0x0323c4 thru 0x03eed1 (0xcb0e bytes) 000643 1749478731.748563 trunc from 0x03c53b to 0x03ffff (0x3ac5 bytes) 000644 1749478731.749983 trunc from 0x040000 to 0x0161ff (0x29e00 bytes) 000645 1749478731.754471 mapread 0x007991 thru 0x01394a (0xbfba bytes) 000646 1749478731.760433 punch from 0x0085dc to 0x010885 (0x82aa bytes) 000647 1749478731.761924 mapread 0x014f5d thru 0x0161ff (0x12a3 bytes) 000648 1749478731.762460 mapread 0x00d904 thru 0x0161ff (0x88fc bytes) 000649 1749478731.764689 write 0x01b156 thru 0x02551f (0xa3ca bytes) 000650 1749478731.766846 MIRROR_RESYNC 3 mirrors 000651 1749478731.890894 fallocate 0x023fc2 thru 0x024ce2 (0xd21 bytes) 000652 1749478731.898856 mapread 0x009acc thru 0x011333 (0x7868 bytes) 000653 1749478731.903329 trunc from 0x006a88 to 0x00f2a4 (0x881d bytes) 000654 1749478731.904822 mapwrite 0x026b67 thru 0x02ee4c (0x82e6 bytes) 000655 1749478731.919759 fallocate 0x015e37 thru 0x017bae (0x1d78 bytes) 000656 1749478731.921095 mapread 0x00355f thru 0x008b03 (0x55a5 bytes) 000657 1749478731.923505 mapread 0x01778c thru 0x01d5c4 (0x5e39 bytes) 000658 1749478731.925552 mapwrite 0x035204 thru 0x038647 (0x3444 bytes) 000659 1749478731.935987 mapwrite 0x0347bb thru 0x039688 (0x4ece bytes) 000660 1749478731.946323 trunc from 0x039689 to 0x036c02 (0x2a86 bytes) 000661 1749478731.950024 fallocate 0x00a3b6 thru 0x019339 (0xef84 bytes) 000662 1749478731.951414 trunc from 0x018589 to 0x028355 (0xfdcd bytes) 000663 1749478731.952874 trunc from 0x036c03 to 0x01eaa0 (0x18162 bytes) 000664 1749478731.957777 MIRROR_EXTEND to 4 mirrors 000665 1749478732.050382 MIRROR_EXTEND to 5 mirrors 000666 1749478732.134210 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000667 1749478732.184272 fallocate 0x01d98b thru 0x02249a (0x4b10 bytes) 000668 1749478732.193090 mapwrite 0x020569 thru 0x0300bd (0xfb55 bytes) 000669 1749478732.210685 read 0x015a5d thru 0x01e686 (0x8c2a bytes) 000670 1749478732.212825 trunc from 0x001e79 to 0x00f86e (0xd9f6 bytes) 000671 1749478732.214312 mapread 0x010f2f thru 0x0130df (0x21b1 bytes) 000672 1749478732.216503 fallocate 0x03e3ed thru 0x03ffff (0x1c13 bytes) 000673 1749478732.217974 punch from 0x035d31 to 0x0386c6 (0x2996 bytes) 000674 1749478732.219128 trunc from 0x040000 to 0x03621b (0x9de4 bytes) 000676 1749478732.222938 punch from 0x019ce7 to 0x02922f (0xf549 bytes) 000677 1749478732.224924 mapwrite 0x03260c thru 0x03eac4 (0xc4b9 bytes) 000678 1749478732.239479 mapread 0x02f868 thru 0x038241 (0x89da bytes) 000679 1749478732.240073 read 0x03e5cf thru 0x03eac4 (0x4f6 bytes) 000680 1749478732.240404 fallocate 0x023382 thru 0x02525a (0x1ed9 bytes) 000681 1749478732.241911 punch from 0x022c03 to 0x02e949 (0xbd47 bytes) 000682 1749478732.243302 fallocate 0x010c9e thru 0x01fb44 (0xeea7 bytes) 000683 1749478732.244818 mapwrite 0x02f5e1 thru 0x03a08d (0xaaad bytes) 000684 1749478732.250724 trunc from 0x03eac5 to 0x00a5dd (0x344e7 bytes) 000685 1749478732.254920 trunc from 0x00827c to 0x00a5dd (0x2362 bytes) 000686 1749478732.256238 mapwrite 0x01a915 thru 0x01b783 (0xe6f bytes) 000687 1749478732.267599 mapwrite 0x018c1f thru 0x01f451 (0x6833 bytes) 000688 1749478732.279591 mapread 0x012c3c thru 0x017213 (0x45d8 bytes) 000689 1749478732.280366 read 0x01a4e2 thru 0x01f451 (0x4f70 bytes) 000690 1749478732.280879 punch from 0x00d401 to 0x00db21 (0x721 bytes) 000691 1749478732.282456 trunc from 0x01cb8e to 0x01f451 (0x28c4 bytes) 000692 1749478732.285019 MIRROR_RESYNC 4 mirrors 000693 1749478732.422937 read 0x01764f thru 0x01f12a (0x7adc bytes) 000694 1749478732.425032 fallocate 0x00e5f1 thru 0x016b24 (0x8534 bytes) 000695 1749478732.432457 trunc from 0x006be0 to 0x00fd18 (0x9139 bytes) 000696 1749478732.433901 trunc from 0x01f452 to 0x012b0d (0xc944 bytes) 000697 1749478732.442143 fallocate 0x00d84e thru 0x00ded0 (0x683 bytes) 000699 1749478732.443892 trunc from 0x012b0e to 0x01a4b3 (0x79a6 bytes) 000701 1749478732.448077 write 0x0204b7 thru 0x02cf9c (0xcae6 bytes) 000702 1749478732.449408 mapread 0x017ef8 thru 0x018e33 (0xf3c bytes) 000703 1749478732.452139 write 0x014414 thru 0x01f0fb (0xace8 bytes) 000704 1749478732.455107 mapread 0x01d844 thru 0x024bb8 (0x7375 bytes) 000705 1749478732.455610 trunc from 0x0162b9 to 0x021a99 (0xb7e1 bytes) 000706 1749478732.458845 trunc from 0x027d9b to 0x02cf9c (0x5202 bytes) 000707 1749478732.460093 trunc from 0x023ea9 to 0x02cf9c (0x90f4 bytes) 000708 1749478732.461415 punch from 0x01dcd4 to 0x02491c (0x6c49 bytes) 000709 1749478732.462746 trunc from 0x02cf9d to 0x00cae3 (0x204b9 bytes) 000711 1749478732.467995 MIRROR_EXTEND to 5 mirrors 000712 1749478732.549975 write 0x02e707 thru 0x03cad9 (0xe3d3 bytes) 000713 1749478732.558646 read 0x00630a thru 0x016085 (0xfd7c bytes) 000714 1749478732.560746 fallocate 0x03bf35 thru 0x03ffff (0x40cb bytes) 000715 1749478732.562324 mapread 0x001775 thru 0x003303 (0x1b8f bytes) 000716 1749478732.566143 punch from 0x032b20 to 0x03937e (0x685f bytes) 000717 1749478732.569710 trunc from 0x0368e2 to 0x03cad9 (0x61f8 bytes) 000719 1749478732.571495 mapread 0x000158 thru 0x009594 (0x943d bytes) 000720 1749478732.574537 MIRROR_RESYNC 5 mirrors 000721 1749478732.768479 write 0x02edde thru 0x03a254 (0xb477 bytes) 000722 1749478732.777520 fallocate 0x03d197 thru 0x03ffff (0x2e69 bytes) 000724 1749478732.779542 fallocate 0x00ac5c thru 0x00efc6 (0x436b bytes) 000725 1749478732.781125 mapread 0x0015cc thru 0x00d07b (0xbab0 bytes) 000726 1749478732.785128 write 0x02a498 thru 0x02e909 (0x4472 bytes) 000727 1749478732.788948 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000729 1749478732.839697 mapwrite 0x00fa5f thru 0x01d73b (0xdcdd bytes) 000730 1749478732.853090 write 0x021cd1 thru 0x02ffb9 (0xe2e9 bytes) 000731 1749478732.855932 mapwrite 0x003c6a thru 0x007141 (0x34d8 bytes) 000732 1749478732.861867 fallocate 0x03b58a thru 0x03ffff (0x4a76 bytes) 000733 1749478732.863473 mapwrite 0x02b828 thru 0x037341 (0xbb1a bytes) 000734 1749478732.870868 trunc from 0x008b37 to 0x009192 (0x65c bytes) 000735 1749478732.872241 trunc from 0x040000 to 0x02b483 (0x14b7c bytes) 000736 1749478732.886135 read 0x00b53c thru 0x015e13 (0xa8d8 bytes) 000737 1749478732.888141 punch from 0x023d4a to 0x02454b (0x802 bytes) 000738 1749478732.889789 trunc from 0x02b484 to 0x02e68f (0x320c bytes) 000739 1749478732.893814 trunc from 0x02e690 to 0x02c5d0 (0x20bf bytes) 000740 1749478732.897093 punch from 0x0256b1 to 0x02ae44 (0x5794 bytes) 000742 1749478732.901917 MIRROR_EXTEND to 5 mirrors 000743 1749478733.009053 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000745 1749478733.066146 read 0x019dce thru 0x01cb63 (0x2d96 bytes) 000746 1749478733.068375 write 0x03ba41 thru 0x03ffff (0x45bf bytes) 000747 1749478733.076999 mapread 0x03f6f3 thru 0x03ffff (0x90d bytes) 000748 1749478733.077461 trunc from 0x040000 to 0x035c81 (0xa37e bytes) 000749 1749478733.085768 mapwrite 0x00be39 thru 0x019382 (0xd54a bytes) 000750 1749478733.097310 fallocate 0x002207 thru 0x00fb26 (0xd920 bytes) 000751 1749478733.100058 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000752 1749478733.142486 trunc from 0x035c82 to 0x0100eb (0x25b96 bytes) 000753 1749478733.147064 trunc from 0x0100ec to 0x01fa18 (0xf92d bytes) 000754 1749478733.150341 read 0x01c76e thru 0x01e915 (0x21a8 bytes) 000755 1749478733.151840 trunc from 0x0074f1 to 0x008b8d (0x169d bytes) 000756 1749478733.153063 fallocate 0x00dbbf thru 0x01545a (0x789c bytes) 000757 1749478733.154637 fallocate 0x01e6dd thru 0x026b9c (0x84c0 bytes) 000758 1749478733.155913 read 0x016836 thru 0x0263b5 (0xfb80 bytes) 000759 1749478733.158398 write 0x036d96 thru 0x03ee78 (0x80e3 bytes) 000760 1749478733.160312 MIRROR_RESYNC 3 mirrors 000762 1749478733.261508 trunc from 0x03ee79 to 0x036ae9 (0x838f bytes) 000763 1749478733.273484 mapread 0x016bcd thru 0x020e28 (0xa25c bytes) 000764 1749478733.279415 mapread 0x03361c thru 0x036ae9 (0x34ce bytes) 000766 1749478733.281696 read 0x02a5dd thru 0x031419 (0x6e3d bytes) 000769 1749478733.283987 punch from 0x025cbe to 0x02ad12 (0x5055 bytes) 000770 1749478733.287061 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000771 1749478733.331691 fallocate 0x020ff5 thru 0x0212cb (0x2d7 bytes) 000772 1749478733.333530 write 0x03d98b thru 0x03ffff (0x2675 bytes) 000774 1749478733.335626 mapread 0x0023d5 thru 0x01138e (0xefba bytes) 000775 1749478733.341648 mapread 0x03b98f thru 0x03ffff (0x4671 bytes) 000776 1749478733.344841 MIRROR_RESYNC 2 mirrors 000777 1749478733.445496 write 0x02c76b thru 0x033cfe (0x7594 bytes) 000778 1749478733.454529 fallocate 0x013dc7 thru 0x01dc1f (0x9e59 bytes) 000779 1749478733.456086 mapwrite 0x01a313 thru 0x01fac7 (0x57b5 bytes) 000780 1749478733.463927 trunc from 0x00866e to 0x00a816 (0x21a9 bytes) 000781 1749478733.465474 trunc from 0x040000 to 0x024e75 (0x1b18a bytes) 000782 1749478733.475527 read 0x01ee73 thru 0x024e75 (0x6003 bytes) 000783 1749478733.477451 punch from 0x0003fc to 0x00c06f (0xbc74 bytes) 000784 1749478733.478995 fallocate 0x01804b thru 0x01cc56 (0x4c0c bytes) 000785 1749478733.480375 trunc from 0x00af97 to 0x00b1c1 (0x22b bytes) 000786 1749478733.481631 trunc from 0x024e76 to 0x02d355 (0x84e0 bytes) 000788 1749478733.485328 mapwrite 0x02c083 thru 0x03b1ec (0xf16a bytes) 000789 1749478733.498300 trunc from 0x03b1ed to 0x03d31a (0x212e bytes) 000791 1749478733.501894 fallocate 0x01b365 thru 0x01f51a (0x41b6 bytes) 000793 1749478733.503512 write 0x03e1ac thru 0x03ffff (0x1e54 bytes) 000795 1749478733.505869 MIRROR_RESYNC 2 mirrors 000796 1749478733.623081 read 0x03439a thru 0x03ffff (0xbc66 bytes) 000797 1749478733.625232 read 0x023d7e thru 0x02a02b (0x62ae bytes) 000798 1749478733.627938 read 0x0185ba thru 0x0248bc (0xc303 bytes) 000799 1749478733.628198 read 0x029d5c thru 0x038c2a (0xeecf bytes) 000800 1749478733.628413 mapread 0x01b843 thru 0x026088 (0xa846 bytes) 000801 1749478733.628865 mapwrite 0x027e6e thru 0x028969 (0xafc bytes) 000803 1749478733.654505 fallocate 0x03391c thru 0x03cc05 (0x92ea bytes) 000804 1749478733.656018 write 0x02c2a0 thru 0x03a6b8 (0xe419 bytes) 000805 1749478733.656971 mapwrite 0x03f63e thru 0x03ffff (0x9c2 bytes) 000806 1749478733.661297 trunc from 0x02507b to 0x02ad0c (0x5c92 bytes) 000807 1749478733.663847 MIRROR_EXTEND to 3 mirrors 000808 1749478733.768417 write 0x00a996 thru 0x019c70 (0xf2db bytes) 000809 1749478733.778266 trunc from 0x040000 to 0x035e2e (0xa1d1 bytes) 000811 1749478733.786885 mapread 0x028456 thru 0x02fce8 (0x7893 bytes) 000812 1749478733.789563 punch from 0x008275 to 0x0113bd (0x9149 bytes) 000814 1749478733.793239 mapread 0x0196ba thru 0x028c56 (0xf59d bytes) 000815 1749478733.797063 read 0x02105c thru 0x02a4eb (0x9490 bytes) 000816 1749478733.797374 trunc from 0x035e2f to 0x00b0d5 (0x2ad59 bytes) 000817 1749478733.801227 mapwrite 0x0105ce thru 0x01726b (0x6c9e bytes) 000818 1749478733.812264 mapread 0x00f6b7 thru 0x01667f (0x6fc9 bytes) 000819 1749478733.812758 trunc from 0x01726c to 0x011938 (0x5933 bytes) 000820 1749478733.816362 mapread 0x00be40 thru 0x011938 (0x5af9 bytes) 000821 1749478733.816954 mapwrite 0x012b51 thru 0x01fa36 (0xcee6 bytes) 000822 1749478733.830704 write 0x01207f thru 0x01a3da (0x835c bytes) 000823 1749478733.831355 read 0x011301 thru 0x017037 (0x5d37 bytes) 000824 1749478733.831709 write 0x02685f thru 0x0276c5 (0xe67 bytes) 000825 1749478733.832198 write 0x0272c6 thru 0x02f3e3 (0x811e bytes) 000826 1749478733.832931 fallocate 0x006e7f thru 0x00c6ec (0x586e bytes) 000827 1749478733.834527 trunc from 0x02f3e4 to 0x01ad09 (0x146da bytes) 000828 1749478733.838297 fallocate 0x0019e7 thru 0x001f0e (0x528 bytes) 000829 1749478733.839659 read 0x00ad3a thru 0x013c20 (0x8ee7 bytes) 000830 1749478733.840020 read 0x000147 thru 0x008139 (0x7ff3 bytes) 000831 1749478733.842278 fallocate 0x0117ae thru 0x0192f3 (0x7b46 bytes) 000832 1749478733.844094 fallocate 0x00c0cf thru 0x01114d (0x507f bytes) 000833 1749478733.845770 mapwrite 0x03a26b thru 0x03d053 (0x2de9 bytes) 000834 1749478733.858381 write 0x007d6f thru 0x01067b (0x890d bytes) 000835 1749478733.859123 read 0x01e574 thru 0x01f933 (0x13c0 bytes) 000836 1749478733.859463 read 0x026418 thru 0x029ae5 (0x36ce bytes) 000837 1749478733.859834 trunc from 0x03d054 to 0x032f32 (0xa121 bytes) 000838 1749478733.863322 trunc from 0x011c2b to 0x019006 (0x73dc bytes) 000839 1749478733.866239 MIRROR_RESYNC 3 mirrors 000840 1749478734.003500 punch from 0x03126d to 0x032f32 (0x1cc6 bytes) 000841 1749478734.012712 mapread 0x014415 thru 0x015b66 (0x1752 bytes) 000842 1749478734.016803 MIRROR_EXTEND to 4 mirrors 000843 1749478734.124073 punch from 0x03162f to 0x032f32 (0x1904 bytes) 000844 1749478734.131923 mapread 0x002c83 thru 0x0077db (0x4b59 bytes) 000845 1749478734.136833 mapwrite 0x0112ea thru 0x01aed8 (0x9bef bytes) 000846 1749478734.147244 write 0x00b099 thru 0x01480b (0x9773 bytes) 000847 1749478734.148966 trunc from 0x02da2c to 0x03200c (0x45e1 bytes) 000848 1749478734.150798 fallocate 0x014cd9 thru 0x01fb3d (0xae65 bytes) 000849 1749478734.152265 punch from 0x00b830 to 0x01995e (0xe12f bytes) 000850 1749478734.155691 mapwrite 0x027848 thru 0x0283e8 (0xba1 bytes) 000851 1749478734.161101 mapread 0x002eb5 thru 0x00c61c (0x9768 bytes) 000852 1749478734.163674 trunc from 0x032f33 to 0x03bea3 (0x8f71 bytes) 000853 1749478734.168909 trunc from 0x029c2e to 0x03783e (0xdc11 bytes) 000854 1749478734.170448 write 0x00ee67 thru 0x01b295 (0xc42f bytes) 000855 1749478734.172838 MIRROR_RESYNC 4 mirrors 000856 1749478734.385251 trunc from 0x000479 to 0x00785c (0x73e4 bytes) 000858 1749478734.393801 read 0x030aee thru 0x032d16 (0x2229 bytes) 000859 1749478734.395448 trunc from 0x01656c to 0x01bdda (0x586f bytes) 000860 1749478734.396746 fallocate 0x00feb3 thru 0x01b4dc (0xb62a bytes) 000861 1749478734.398113 write 0x03f11f thru 0x03ffff (0xee1 bytes) 000862 1749478734.398698 trunc from 0x040000 to 0x003a1d (0x3c5e2 bytes) 000863 1749478734.406357 trunc from 0x003445 to 0x003a1d (0x5d9 bytes) 000864 1749478734.407704 mapread 0x001d39 thru 0x003a1d (0x1ce5 bytes) 000865 1749478734.411222 write 0x02562e thru 0x0329a5 (0xd378 bytes) 000867 1749478734.412279 fallocate 0x02fe09 thru 0x03c0e0 (0xc2d8 bytes) 000868 1749478734.413808 mapwrite 0x001e37 thru 0x009353 (0x751d bytes) 000869 1749478734.421765 trunc from 0x02c5e0 to 0x030ab0 (0x44d1 bytes) 000870 1749478734.423014 mapread 0x020387 thru 0x025464 (0x50de bytes) 000872 1749478734.425688 trunc from 0x02becf to 0x0329a5 (0x6ad7 bytes) 000873 1749478734.427250 mapwrite 0x002291 thru 0x009564 (0x72d4 bytes) 000874 1749478734.431549 fallocate 0x001249 thru 0x006a1d (0x57d5 bytes) 000875 1749478734.432807 mapread 0x01421c thru 0x01c9e0 (0x87c5 bytes) 000876 1749478734.436058 fallocate 0x0216a9 thru 0x025a09 (0x4361 bytes) 000877 1749478734.437474 trunc from 0x021e9d to 0x02635e (0x44c2 bytes) 000878 1749478734.440211 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000879 1749478734.487060 punch from 0x006527 to 0x007547 (0x1021 bytes) 000880 1749478734.489829 MIRROR_RESYNC 3 mirrors 000881 1749478734.624291 write 0x0370d1 thru 0x03ffff (0x8f2f bytes) 000883 1749478734.630894 fallocate 0x002a73 thru 0x00dc91 (0xb21f bytes) 000884 1749478734.632448 trunc from 0x02f981 to 0x0367e5 (0x6e65 bytes) 000885 1749478734.635017 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000886 1749478734.682387 write 0x02d305 thru 0x030e20 (0x3b1c bytes) 000887 1749478734.685953 trunc from 0x040000 to 0x030911 (0xf6ee bytes) 000888 1749478734.692564 punch from 0x000322 to 0x0053dc (0x50bb bytes) 000889 1749478734.694183 trunc from 0x030912 to 0x00e673 (0x2229e bytes) 000890 1749478734.697561 fallocate 0x02e833 thru 0x033826 (0x4ff4 bytes) 000891 1749478734.698955 mapread 0x00883e thru 0x00e673 (0x5e36 bytes) 000892 1749478734.701106 punch from 0x002725 to 0x003450 (0xd2c bytes) 000893 1749478734.702318 punch from 0x008c95 to 0x00cadf (0x3e4b bytes) 000894 1749478734.703657 fallocate 0x024336 thru 0x024bbd (0x888 bytes) 000895 1749478734.704924 write 0x03327e thru 0x03c35a (0x90dd bytes) 000896 1749478734.705774 trunc from 0x03c35b to 0x00e278 (0x2e0e2 bytes) 000897 1749478734.709498 read 0x000df1 thru 0x004914 (0x3b24 bytes) 000899 1749478734.711666 punch from 0x001c08 to 0x00b4bd (0x98b6 bytes) 000900 1749478734.713050 write 0x02b16d thru 0x0357c4 (0xa658 bytes) 000901 1749478734.714048 mapread 0x02a1b8 thru 0x032400 (0x8249 bytes) 000902 1749478734.716236 trunc from 0x032ba1 to 0x03376b (0xbcb bytes) 000903 1749478734.719342 trunc from 0x02c5d2 to 0x0357c4 (0x91f3 bytes) 000904 1749478734.720916 write 0x013e33 thru 0x01c4a9 (0x8677 bytes) 000905 1749478734.724413 punch from 0x021b16 to 0x02ae23 (0x930e bytes) 000906 1749478734.725910 write 0x01c776 thru 0x024a4e (0x82d9 bytes) 000907 1749478734.728844 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000908 1749478734.758549 write 0x035dfb thru 0x036494 (0x69a bytes) 000909 1749478734.762286 MIRROR_EXTEND to 2 mirrors 000910 1749478734.866841 punch from 0x0311fa to 0x036494 (0x529b bytes) 000911 1749478734.874465 punch from 0x02a012 to 0x036494 (0xc483 bytes) 000912 1749478734.875825 trunc from 0x010437 to 0x016053 (0x5c1d bytes) 000913 1749478734.877139 trunc from 0x036495 to 0x01b915 (0x1ab7f bytes) 000914 1749478734.885976 mapread 0x014078 thru 0x0173a0 (0x3329 bytes) 000915 1749478734.890193 fallocate 0x000b0c thru 0x00c815 (0xbd0a bytes) 000916 1749478734.891659 trunc from 0x01b916 to 0x00da80 (0xde95 bytes) 000917 1749478734.895567 trunc from 0x004f03 to 0x005902 (0xa00 bytes) 000918 1749478734.896817 trunc from 0x00da81 to 0x007490 (0x65f0 bytes) 000919 1749478734.900086 mapread 0x000986 thru 0x007490 (0x6b0b bytes) 000920 1749478734.903226 mapwrite 0x02ffc9 thru 0x030fa9 (0xfe1 bytes) 000921 1749478734.912446 punch from 0x006100 to 0x01466e (0xe56f bytes) 000922 1749478734.913841 trunc from 0x0277e2 to 0x027f15 (0x734 bytes) 000923 1749478734.915011 fallocate 0x035bf7 thru 0x03cde5 (0x71ef bytes) 000924 1749478734.916328 mapread 0x019317 thru 0x01a534 (0x121e bytes) 000926 1749478734.918915 read 0x02c345 thru 0x030fa9 (0x4c65 bytes) 000927 1749478734.920424 MIRROR_RESYNC 2 mirrors 000928 1749478735.016978 fallocate 0x01d499 thru 0x02675d (0x92c5 bytes) 000930 1749478735.023951 trunc from 0x030faa to 0x002c04 (0x2e3a5 bytes) 000931 1749478735.031353 write 0x01a69b thru 0x01ceb9 (0x281f bytes) 000932 1749478735.031996 trunc from 0x01ceba to 0x007fe4 (0x14ed5 bytes) 000933 1749478735.035191 write 0x02c7cc thru 0x031f2b (0x5760 bytes) 000934 1749478735.035840 punch from 0x009a18 to 0x00ce0d (0x33f6 bytes) 000935 1749478735.037143 mapwrite 0x00d9fe thru 0x01988f (0xbe92 bytes) 000936 1749478735.049145 mapread 0x008cd3 thru 0x018257 (0xf585 bytes) 000938 1749478735.049929 trunc from 0x003c47 to 0x012f26 (0xf2e0 bytes) 000939 1749478735.051299 trunc from 0x012db5 to 0x01ae8e (0x80da bytes) 000940 1749478735.052510 mapread 0x0221f6 thru 0x023e98 (0x1ca3 bytes) 000941 1749478735.054950 fallocate 0x00abe0 thru 0x018f7e (0xe39f bytes) 000942 1749478735.056319 write 0x03d058 thru 0x03ffff (0x2fa8 bytes) 000943 1749478735.056867 punch from 0x0177ef to 0x025f30 (0xe742 bytes) 000944 1749478735.058144 mapread 0x03cc4b thru 0x03ffff (0x33b5 bytes) 000945 1749478735.060300 read 0x028885 thru 0x033ffa (0xb776 bytes) 000946 1749478735.061637 write 0x02b1f8 thru 0x02f853 (0x465c bytes) 000947 1749478735.062199 mapread 0x02ec1e thru 0x0386b5 (0x9a98 bytes) 000948 1749478735.064828 MIRROR_RESYNC 2 mirrors 000950 1749478735.182632 MIRROR_EXTEND to 3 mirrors 000951 1749478735.287737 mapread 0x017c6b thru 0x01fef4 (0x828a bytes) 000952 1749478735.292294 write 0x03de93 thru 0x03ffff (0x216d bytes) 000953 1749478735.302130 write 0x03ef47 thru 0x03ffff (0x10b9 bytes) 000954 1749478735.302635 write 0x038bb4 thru 0x03bccd (0x311a bytes) 000955 1749478735.303121 fallocate 0x01eda2 thru 0x027085 (0x82e4 bytes) 000956 1749478735.304460 fallocate 0x02dabd thru 0x033f9b (0x64df bytes) 000957 1749478735.305741 write 0x03af2a thru 0x03ffff (0x50d6 bytes) 000958 1749478735.306288 mapread 0x02d962 thru 0x034542 (0x6be1 bytes) 000959 1749478735.308431 write 0x0283ac thru 0x035784 (0xd3d9 bytes) 000960 1749478735.310607 read 0x002616 thru 0x00e233 (0xbc1e bytes) 000961 1749478735.312660 mapread 0x02ba79 thru 0x03b120 (0xf6a8 bytes) 000962 1749478735.313360 trunc from 0x02fc0b to 0x03c3fb (0xc7f1 bytes) 000963 1749478735.317033 write 0x03de49 thru 0x03ffff (0x21b7 bytes) 000964 1749478735.318747 MIRROR_RESYNC 3 mirrors 000965 1749478735.453012 mapread 0x01e90e thru 0x020942 (0x2035 bytes) 000966 1749478735.465105 read 0x037f2e thru 0x0390c0 (0x1193 bytes) 000968 1749478735.470991 mapread 0x010eaf thru 0x014df1 (0x3f43 bytes) 000969 1749478735.474450 trunc from 0x040000 to 0x004930 (0x3b6cf bytes) 000970 1749478735.487502 mapread 0x0046a7 thru 0x004930 (0x28a bytes) 000971 1749478735.489604 punch from 0x003c66 to 0x004930 (0xccb bytes) 000972 1749478735.490846 punch from 0x002cce to 0x004930 (0x1c63 bytes) 000973 1749478735.492001 trunc from 0x001317 to 0x004930 (0x361a bytes) 000974 1749478735.493241 mapread 0x000737 thru 0x004930 (0x41fa bytes) 000975 1749478735.495724 write 0x00ba52 thru 0x010b0a (0x50b9 bytes) 000976 1749478735.496508 trunc from 0x008031 to 0x00df90 (0x5f60 bytes) 000978 1749478735.499866 read 0x00212a thru 0x005a96 (0x396d bytes) 000979 1749478735.501956 trunc from 0x010b0b to 0x03ab7a (0x2a070 bytes) 000980 1749478735.506641 write 0x027e30 thru 0x02b4fc (0x36cd bytes) 000981 1749478735.509735 write 0x036aa5 thru 0x03b89e (0x4dfa bytes) 000982 1749478735.510451 fallocate 0x02949e thru 0x02d38a (0x3eed bytes) 000984 1749478735.512373 write 0x00543f thru 0x0066e1 (0x12a3 bytes) 000985 1749478735.512997 mapwrite 0x024bea thru 0x02e4e4 (0x98fb bytes) 000986 1749478735.519253 fallocate 0x025fb3 thru 0x0331ea (0xd238 bytes) 000987 1749478735.520683 fallocate 0x0396b0 thru 0x03ffff (0x6950 bytes) 000988 1749478735.522023 mapwrite 0x031a94 thru 0x03ffff (0xe56c bytes) 000991 1749478735.532663 read 0x01d35b thru 0x02858f (0xb235 bytes) 000992 1749478735.533018 trunc from 0x040000 to 0x009239 (0x36dc6 bytes) 000993 1749478735.536837 fallocate 0x022191 thru 0x02af66 (0x8dd6 bytes) 000994 1749478735.538393 mapwrite 0x0122b3 thru 0x02080c (0xe55a bytes) 000995 1749478735.547303 fallocate 0x0144d4 thru 0x01bb29 (0x7656 bytes) 000996 1749478735.548861 trunc from 0x0007e6 to 0x0082bc (0x7ad7 bytes) 000997 1749478735.550464 fallocate 0x002ad3 thru 0x00eb95 (0xc0c3 bytes) 000998 1749478735.552059 write 0x0126f8 thru 0x01760c (0x4f15 bytes) 000999 1749478735.552757 trunc from 0x02af67 to 0x0388f0 (0xd98a bytes) 001000 1749478735.556273 fallocate 0x01237f thru 0x014cba (0x293c bytes) All operations completed A-OK! PASS 70a (10s) == sanity-flr test 71: check mirror extend parent fid ==== 10:18:56 (1749478736) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x200000405:0x60:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (2s) == sanity-flr test 200a: stress test ===================== 10:18:58 (1749478738) Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre2 Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 2502816 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8675328 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3829344 .. Extending file size to 7714272 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7730176 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7895808 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 8081408 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8588064 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8704128 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8717728 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8871776 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8902816 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8927936 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9165248 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8947136 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 5289 5290 5292 5294 5296 done 192.168.204.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg432-client.virtnet /mnt/lustre2 (opts:) 192.168.204.132@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg432-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 10:20:02 (1749478802) Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre2 Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2633952 .. Extending file size to 1035360 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8321792 .. Extending file size to 1860672 .. Extending file size to 8011264 .. Extending file size to 8363424 .. failed Extending file size to 8495552 .. Extending file size to 8944704 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8542592 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9152576 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8643168 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8926976 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8973728 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8999584 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 9592 9593 9595 9597 9599 done 192.168.204.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg432-client.virtnet /mnt/lustre2 (opts:) 192.168.204.132@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg432-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 10:21:05 (1749478865) Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2824 1284864 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3380 1284308 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 14708 3587352 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 22676 3572168 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37384 7159520 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2824 1284864 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 3380 1284308 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 14708 3587352 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 22676 3572168 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37384 7159520 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0328936 s, 63.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0472917 s, 44.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg432-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 10:21:10 (1749478870) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0247419 s, 84.8 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 10:21:11 (1749478871) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x6c:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbc:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbc:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbd:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x9c:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 10:21:13 (1749478873) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0202199 s, 104 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xfdb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xfdc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xfdd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (1s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 10:21:14 (1749478874) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0284015 s, 73.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x9d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x9d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x9e:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 10:21:16 (1749478876) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0195497 s, 107 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x9f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x9f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa0:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 10:21:39 (1749478899) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg432-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0125736 s, 83.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xfdc:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xfdf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xfde:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xfdc:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xfdf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xfde:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xfdd:0x0] } PASS 204d (34s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 10:22:13 (1749478933) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg432-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0367713 s, 57.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (33s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 10:22:46 (1749478966) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg432-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg432-server: Pool lustre.test_204f created Adding targets to pool oleg432-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg432-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0191758 s, 109 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg432-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg432-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg432-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (44s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 10:23:30 (1749479010) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 10:23:32 (1749479012) PASS 205b (1s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 10:23:33 (1749479013) oleg432-server: Pool lustre.test_206 created Adding targets to pool oleg432-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg432-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg432-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg432-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg432-server: Pool lustre.test_206 destroyed PASS 206 (15s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 10:23:48 (1749479028) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0183117 s, 57.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106777 s, 98.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x71ce2cae Mirror 3: 0x71ce2cae PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 10:23:50 (1749479030) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 10:23:51 (1749479031) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 10:23:52 (1749479032) Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.00803698 s, 0.5 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1310000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xda0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xda0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xda0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xd60000 Read this (4 bytes): BBBB 192.168.204.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg432-client.virtnet /mnt/lustre2 (opts:) PASS 209a (14s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 10:24:06 (1749479046) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.0022438 s, 3.7 MB/s the very first read read with warm cache PASS 209b (1s) == sanity-flr test 210a: handle broken mirrored lovea ==== 10:24:07 (1749479047) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104441 s, 100 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x8d lmm_fid: [0x200000405:0x8d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x8d lmm_fid: [0x200000405:0x8d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611277 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-06-09 10:24:08.000000000 -0400 Modify: 2025-06-09 10:24:08.000000000 -0400 Change: 2025-06-09 10:24:08.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 10:24:09 (1749479049) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00899386 s, 117 MB/s oleg432-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg432-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg432-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg432-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg432-server: O/280000401/d14/ce: File not found by ext2_lookup PASS 210b (10s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 10:24:19 (1749479059) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0483153 s, 848 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00962398 s, 426 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 9 10:24 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x92:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 9 10:24 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test complete, duration 684 sec ============ 10:24:22 (1749479062) === sanity-flr: start cleanup 10:24:22 (1749479062) === === sanity-flr: finish cleanup 10:24:24 (1749479064) ===