-----============= acceptance-small: sanity-flr ============----- Tue Mar 11 07:17:58 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg316-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 07:18:03 (1741691883) === oleg316-client.virtnet: executing check_config_client /mnt/lustre oleg316-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg316-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012a60b800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012a60b800.idle_timeout=debug disable quota as required oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 07:18:11 (1741691891) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7593] == sanity-flr test 0a: lfs mirror create with -N option == 07:18:12 (1741691892) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 07:18:15 (1741691895) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 07:18:16 (1741691896) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (0s) == sanity-flr test 0d: lfs mirror extend with -N option == 07:18:16 (1741691896) PASS 0d (5s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 07:18:21 (1741691901) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 07:18:22 (1741691902) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (0s) == sanity-flr test 0g: lfs mirror create flags support === 07:18:22 (1741691902) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88012a60b800.stats=clear osc.lustre-OST0001-osc-ffff88012a60b800.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 07:18:24 (1741691904) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 07:18:26 (1741691906) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 07:18:28 (1741691908) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (0s) == sanity-flr test 1: create components with setstripe options ========================================================== 07:18:28 (1741691908) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 07:18:30 (1741691910) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 07:18:32 (1741691912) PASS 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 07:18:33 (1741691913) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td lfs extend: cannot get the last init comp error: lfs extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 07:18:35 (1741691915) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 07:18:37 (1741691917) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 07:18:39 (1741691919) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0213673 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0187237 s, 56.0 MB/s PASS 21 (1s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 07:18:40 (1741691920) 14+0 records in 14+0 records out 14680064 bytes (15 MB) copied, 0.0882097 s, 166 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 07:18:42 (1741691922) Stopping /mnt/lustre-ost1 (opts:) on oleg316-server oleg316-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a60b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a60b800.ost_server_uuid in DISCONN state after 0 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg316-server: oleg316-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg316-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg316-server oleg316-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a60b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a60b800.ost_server_uuid in DISCONN state after 3 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg316-server: oleg316-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg316-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (22s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 07:19:04 (1741691944) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.0880592 s, 59.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg316-server oleg316-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a60b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a60b800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg316-server: oleg316-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg316-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (11s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 07:19:15 (1741691955) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg316-server oleg316-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a60b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a60b800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg316-server: oleg316-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg316-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg316-server oleg316-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a60b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a60b800.ost_server_uuid in DISCONN state after 4 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg316-server: oleg316-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg316-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (20s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 07:19:35 (1741691975) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.782102 s, 67.0 MB/s -rw-r--r-- 1 root root 52428800 Mar 11 07:19 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.218s user 0m0.001s sys 0m0.133s Stopping /mnt/lustre-ost1 (opts:) on oleg316-server oleg316-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a60b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a60b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.200s user 0m0.000s sys 0m0.129s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg316-server: oleg316-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg316-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg316-server oleg316-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a60b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a60b800.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost1)... real 0m0.250s user 0m0.000s sys 0m0.128s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg316-server: oleg316-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg316-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg316-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012a60b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a60b800.ost_server_uuid in FULL state after 0 sec PASS 33b (22s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 07:19:57 (1741691997) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 07:19:58 (1741691998) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (0s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 07:19:58 (1741691998) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 07:19:59 (1741691999) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 07:20:01 (1741692001) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.118258 s, 106 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (2s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 07:20:03 (1741692003) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00941729 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100383 s, 104 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 07:20:05 (1741692005) PASS 36c (2s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 07:20:07 (1741692007) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000692922 s, 1.5 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000541807 s, 1.9 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00047974 s, 2.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000390213 s, 2.6 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 07:20:09 (1741692009) create test files with size 15758192 .. c2d830f0da807bc89dcd8087fb3f59f8 - 6c709d3d1048a2b62d812b5deb76cb0a - 3143f379bae51498e274bf84d25fd055 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg316-client.virtnet /mnt/lustre (opts:) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre PASS 37 (4s) == sanity-flr test 38: resync ============================ 07:20:14 (1741692014) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (2s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 07:20:16 (1741692016) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 07:20:18 (1741692018) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.019013 s, 55.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00917111 s, 114 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (1s) == sanity-flr test 41: lfs mirror resync check =========== 07:20:19 (1741692019) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0184084 s, 114 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0780659 s, 53.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0346308 s, 60.6 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 07:20:22 (1741692022) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (0s) == sanity-flr test 43a: mirror pick on write ============= 07:20:22 (1741692022) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 07:20:23 (1741692023) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00982559 s, 107 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 07:20:25 (1741692025) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0736025 s, 42.7 MB/s PASS 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 07:20:27 (1741692027) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x48:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x46:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x46:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 07:20:29 (1741692029) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0729987 s, 144 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg316-client.virtnet /mnt/lustre (opts:) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (23s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 07:20:52 (1741692052) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff88012aa93800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0692276 s, 151 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg316-client.virtnet /mnt/lustre (opts:) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800a951f000.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 07:20:55 (1741692055) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (8s) == sanity-flr test 46: Verify setstripe --copy option ==== 07:21:03 (1741692063) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (6s) == sanity-flr test 47: Verify mirror obj alloc =========== 07:21:09 (1741692069) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 07:21:10 (1741692070) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0593097 s, 53.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0424214 s, 74.2 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 2692b6ae865166abae7ff7295e1e2922 - mirror 1 checksum: 8be7ccce283946999e502c1ac6acb281 - mirror 2 checksum: 2692b6ae865166abae7ff7295e1e2922 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Mar 11 07:21:11 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Mar 11 07:21:11 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 2692b6ae865166abae7ff7295e1e2922 - mirror 1 checksum: 8be7ccce283946999e502c1ac6acb281 - mirror 2 checksum: 8be7ccce283946999e502c1ac6acb281 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Mar 11 07:21:11 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Mar 11 07:21:11 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } PASS 48 (1s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 07:21:11 (1741692071) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00767163 s, 137 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 529408.. 530431: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 07:21:13 (1741692073) Failing mds1 on oleg316-server Stopping /mnt/lustre-mds1 (opts:) on oleg316-server 07:21:20 (1741692080) shut down facet: mds1 facet_host: oleg316-server facet_failover_host: oleg316-server Failover mds1 to oleg316-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-MDT0000 07:21:33 (1741692093) targets are mounted 07:21:33 (1741692093) facet_failover done oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.008357 s, 125 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } PASS 50A (27s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 07:21:40 (1741692100) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00728876 s, 2.8 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00399261 s, 5.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 07:21:42 (1741692102) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00777833 s, 2.6 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00309303 s, 6.6 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 07:21:44 (1741692104) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00217901 s, 7.5 MB/s PASS 50c (1s) == sanity-flr test 50d: mirror rsync keep holes ========== 07:21:46 (1741692106) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00893481 s, 2.3 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 07:21:47 (1741692107) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00185454 s, 2.2 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00662252 s, 618 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 07:21:49 (1741692109) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (3s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 07:21:52 (1741692112) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (17s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 07:22:09 (1741692129) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 07:22:26 (1741692146) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 07:22:43 (1741692163) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (1s) == sanity-flr test 70: mirror create and split race ====== 07:22:44 (1741692164) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 26251 mirror split pid 26252 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4866: 26251 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4866: 26252 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 07:23:46 (1741692226) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 2226 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x2dabd 000001 1741692226.592913 trunc from 00000000 to 0x02dabc (0x2dabd bytes) 000003 1741692226.597549 read 0x0130d3 thru 0x01557b (0x24a9 bytes) 000004 1741692226.599211 write 0x02f118 thru 0x03b2d0 (0xc1b9 bytes) 000005 1741692226.600071 write 0x000d5c thru 0x00e145 (0xd3ea bytes) 000006 1741692226.604142 write 0x00d11b thru 0x01cac0 (0xf9a6 bytes) 000007 1741692226.607107 mapwrite 0x02fb09 thru 0x030975 (0xe6d bytes) 000008 1741692226.615114 write 0x020161 thru 0x0272b6 (0x7156 bytes) 000009 1741692226.617493 read 0x016e22 thru 0x0207d0 (0x99af bytes) 000010 1741692226.617824 fallocate 0x01d018 thru 0x01e847 (0x1830 bytes) fallocating to largest ever: 0x3d44f 000011 1741692226.619359 fallocate 0x0337ef thru 0x03d44e (0x9c60 bytes) 000013 1741692226.620754 mapread 0x028d89 thru 0x03899a (0xfc12 bytes) 000014 1741692226.624022 fallocate 0x02f3b1 thru 0x03ab93 (0xb7e3 bytes) 000015 1741692226.625364 write 0x0278d8 thru 0x028ba5 (0x12ce bytes) 000016 1741692226.625740 mapwrite 0x00e83a thru 0x0108a3 (0x206a bytes) 000017 1741692226.630252 mapwrite 0x012312 thru 0x015375 (0x3064 bytes) 000020 1741692226.634141 fallocate 0x0056d6 thru 0x012ba0 (0xd4cb bytes) 000021 1741692226.635495 punch from 0x016b3c to 0x01f41f (0x88e4 bytes) 000022 1741692226.636906 read 0x0197e2 thru 0x01e427 (0x4c46 bytes) 000023 1741692226.639162 punch from 0x0286c8 to 0x033cf6 (0xb62f bytes) 000024 1741692226.640497 punch from 0x028e0b to 0x0360e1 (0xd2d7 bytes) 000025 1741692226.641856 mapwrite 0x038ecf thru 0x03ffff (0x7131 bytes) 000026 1741692226.651987 fallocate 0x039ee4 thru 0x03c290 (0x23ad bytes) fallocating to largest ever: 0x40000 000027 1741692226.653500 fallocate 0x03c8bd thru 0x03ffff (0x3743 bytes) 000029 1741692226.655187 trunc from 0x040000 to 0x037d5e (0x82a1 bytes) 000030 1741692226.661026 MIRROR_EXTEND to 2 mirrors 000031 1741692226.761443 fallocate 0x02eb9d thru 0x02efbe (0x422 bytes) 000032 1741692226.769521 trunc from 0x037d5f to 0x029f0f (0xde4f bytes) 000033 1741692226.779405 MIRROR_EXTEND to 3 mirrors 000034 1741692226.889318 read 0x00bc71 thru 0x01a90b (0xec9b bytes) 000035 1741692226.893822 trunc from 0x029f10 to 0x00ef83 (0x1af8c bytes) 000036 1741692226.909693 MIRROR_RESYNC 3 mirrors 000037 1741692226.991786 mapread 0x00974e thru 0x00ef83 (0x5836 bytes) 000038 1741692226.996927 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000039 1741692227.043767 read 0x00ebae thru 0x00ef83 (0x3d6 bytes) 000041 1741692227.045448 mapwrite 0x03a6e7 thru 0x03c802 (0x211c bytes) 000042 1741692227.067619 fallocate 0x01fc16 thru 0x0240fd (0x44e8 bytes) 000044 1741692227.069293 trunc from 0x03c803 to 0x003c21 (0x38be1 bytes) 000045 1741692227.074821 fallocate 0x005c5c thru 0x012206 (0xc5ab bytes) 000046 1741692227.076270 trunc from 0x003c22 to 0x034884 (0x30c63 bytes) 000047 1741692227.080060 trunc from 0x034885 to 0x034ea2 (0x61e bytes) 000049 1741692227.083771 read 0x010966 thru 0x012884 (0x1f1f bytes) 000050 1741692227.087030 trunc from 0x034ea3 to 0x0243f7 (0x10aab bytes) 000051 1741692227.090758 mapread 0x0070f3 thru 0x008abc (0x19ca bytes) 000052 1741692227.091228 read 0x009cb6 thru 0x00d15f (0x34aa bytes) 000053 1741692227.091528 write 0x0292ab thru 0x02fa3a (0x6790 bytes) 000054 1741692227.093559 MIRROR_EXTEND to 3 mirrors 000055 1741692227.188784 mapwrite 0x015c01 thru 0x018e21 (0x3221 bytes) 000056 1741692227.220136 mapread 0x01bb0c thru 0x026850 (0xad45 bytes) 000057 1741692227.223000 read 0x02cd77 thru 0x02fa3a (0x2cc4 bytes) 000058 1741692227.224353 read 0x007e98 thru 0x00def4 (0x605d bytes) 000059 1741692227.226428 write 0x0225f0 thru 0x02e12a (0xbb3b bytes) 000060 1741692227.228145 fallocate 0x021538 thru 0x027c4f (0x6718 bytes) 000061 1741692227.229712 write 0x036c49 thru 0x03ffff (0x93b7 bytes) 000062 1741692227.230613 punch from 0x02da29 to 0x033dfc (0x63d4 bytes) 000063 1741692227.234162 mapwrite 0x00d417 thru 0x01ad80 (0xd96a bytes) 000064 1741692227.241313 MIRROR_EXTEND to 4 mirrors 000065 1741692227.350590 mapwrite 0x0012de thru 0x008f62 (0x7c85 bytes) 000066 1741692227.382172 fallocate 0x02f750 thru 0x0375dc (0x7e8d bytes) 000067 1741692227.383509 mapread 0x039081 thru 0x03ffff (0x6f7f bytes) 000068 1741692227.385540 read 0x000089 thru 0x006bf7 (0x6b6f bytes) 000070 1741692227.387181 read 0x0115f7 thru 0x01f537 (0xdf41 bytes) 000071 1741692227.389552 read 0x013b32 thru 0x022431 (0xe900 bytes) 000072 1741692227.389827 punch from 0x02b504 to 0x03152e (0x602b bytes) 000074 1741692227.391415 mapwrite 0x014e71 thru 0x01cfdb (0x816b bytes) 000076 1741692227.396604 read 0x0069cd thru 0x00bdc1 (0x53f5 bytes) 000077 1741692227.396921 fallocate 0x01f75b thru 0x02ca93 (0xd339 bytes) 000078 1741692227.398350 mapread 0x0237cc thru 0x02ec0a (0xb43f bytes) 000079 1741692227.400962 mapread 0x026f99 thru 0x031e1a (0xae82 bytes) 000081 1741692227.401896 write 0x015b6a thru 0x019219 (0x36b0 bytes) 000082 1741692227.403486 read 0x018a54 thru 0x018abd (0x6a bytes) 000083 1741692227.403749 read 0x00794a thru 0x00b064 (0x371b bytes) 000084 1741692227.404093 fallocate 0x017a52 thru 0x02202d (0xa5dc bytes) 000085 1741692227.405500 mapwrite 0x032b07 thru 0x03d9c7 (0xaec1 bytes) 000086 1741692227.410630 write 0x0126e5 thru 0x01c665 (0x9f81 bytes) 000087 1741692227.411291 write 0x009879 thru 0x00a45b (0xbe3 bytes) 000088 1741692227.411659 trunc from 0x040000 to 0x006daa (0x39255 bytes) 000089 1741692227.421194 fallocate 0x0256ad thru 0x02af80 (0x58d4 bytes) 000091 1741692227.422823 trunc from 0x02af81 to 0x01cae7 (0xe499 bytes) 000092 1741692227.426564 trunc from 0x01cae8 to 0x0188b4 (0x4233 bytes) 000093 1741692227.430382 punch from 0x006a2c to 0x014d42 (0xe317 bytes) 000094 1741692227.431586 mapwrite 0x01950c thru 0x024ac9 (0xb5be bytes) 000096 1741692227.443639 fallocate 0x024fec thru 0x02868d (0x36a2 bytes) 000097 1741692227.445040 fallocate 0x01126b thru 0x01efe0 (0xdd76 bytes) 000098 1741692227.446277 punch from 0x003655 to 0x0061a5 (0x2b51 bytes) 000099 1741692227.448515 MIRROR_EXTEND to 5 mirrors 000100 1741692227.551112 fallocate 0x0288a6 thru 0x036b44 (0xe29f bytes) 000101 1741692227.559430 punch from 0x020ffe to 0x024ac9 (0x3acc bytes) 000102 1741692227.560654 mapwrite 0x01cc29 thru 0x0206bb (0x3a93 bytes) 000103 1741692227.569680 mapwrite 0x00e520 thru 0x019d40 (0xb821 bytes) 000104 1741692227.578244 MIRROR_RESYNC 5 mirrors 000105 1741692227.784996 punch from 0x0226ae to 0x022e05 (0x758 bytes) 000106 1741692227.794298 MIRROR_EXTEND to 6 mirrors 000107 1741692227.894273 mapwrite 0x001806 thru 0x005d6b (0x4566 bytes) 000108 1741692227.925566 write 0x01bd80 thru 0x01eb2b (0x2dac bytes) 000109 1741692227.928826 mapwrite 0x017061 thru 0x01c2a0 (0x5240 bytes) 000110 1741692227.933483 mapread 0x004f34 thru 0x00e2af (0x937c bytes) 000111 1741692227.933951 write 0x025ee2 thru 0x0330dd (0xd1fc bytes) 000112 1741692227.934724 read 0x000f25 thru 0x0090c3 (0x819f bytes) 000113 1741692227.936269 trunc from 0x0330de to 0x01ae3d (0x182a0 bytes) 000114 1741692227.945103 mapread 0x00f8b8 thru 0x00fd30 (0x479 bytes) 000116 1741692227.945706 read 0x00a0e8 thru 0x015af9 (0xba12 bytes) 000117 1741692227.947549 mapread 0x01143e thru 0x01ae3d (0x9a00 bytes) 000118 1741692227.947972 mapwrite 0x00c28e thru 0x0155e3 (0x9356 bytes) 000119 1741692227.956938 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000120 1741692228.007894 MIRROR_RESYNC 5 mirrors 000121 1741692228.160466 read 0x009cd7 thru 0x00bcef (0x2019 bytes) 000122 1741692228.162389 mapwrite 0x001f09 thru 0x010a0e (0xeb06 bytes) 000123 1741692228.193828 mapwrite 0x00912d thru 0x015a2d (0xc901 bytes) 000125 1741692228.202868 punch from 0x018a0b to 0x01ae3d (0x2433 bytes) 000126 1741692228.204546 trunc from 0x01ae3e to 0x020fe4 (0x61a7 bytes) 000127 1741692228.210612 MIRROR_EXTEND to 6 mirrors 000128 1741692228.314502 mapread 0x020040 thru 0x020fe4 (0xfa5 bytes) 000129 1741692228.317885 punch from 0x01c031 to 0x020fe4 (0x4fb4 bytes) 000130 1741692228.327610 mapwrite 0x026680 thru 0x02a39c (0x3d1d bytes) 000132 1741692228.342612 punch from 0x0282b5 to 0x02a39c (0x20e8 bytes) 000133 1741692228.343984 punch from 0x0215d8 to 0x02a39c (0x8dc5 bytes) 000134 1741692228.345357 mapwrite 0x025179 thru 0x03163d (0xc4c5 bytes) 000135 1741692228.358706 mapread 0x00cefa thru 0x018d3d (0xbe44 bytes) 000136 1741692228.362973 fallocate 0x022f51 thru 0x027453 (0x4503 bytes) 000137 1741692228.364568 trunc from 0x03163e to 0x00849f (0x2919e bytes) 000138 1741692228.368624 punch from 0x007548 to 0x007f29 (0x9e2 bytes) 000140 1741692228.369995 fallocate 0x00c18a thru 0x0129f2 (0x6869 bytes) 000142 1741692228.371606 punch from 0x0042aa to 0x004a7b (0x7d2 bytes) 000143 1741692228.372714 trunc from 0x0129f3 to 0x021e39 (0xf447 bytes) 000144 1741692228.376452 mapread 0x016f28 thru 0x020e2f (0x9f08 bytes) 000145 1741692228.380485 trunc from 0x021e3a to 0x026df3 (0x4fba bytes) 000147 1741692228.384085 mapread 0x005004 thru 0x014b0d (0xfb0a bytes) 000148 1741692228.387718 write 0x03e86f thru 0x03ffff (0x1791 bytes) 000149 1741692228.388256 trunc from 0x040000 to 0x026ff8 (0x19007 bytes) 000150 1741692228.392472 MIRROR_RESYNC 6 mirrors 000153 1741692228.537705 mapread 0x00c457 thru 0x00c7c0 (0x36a bytes) 000154 1741692228.540684 MIRROR_RESYNC 6 mirrors 000156 1741692228.583377 trunc from 0x026ff9 to 0x01d8a4 (0x9754 bytes) 000158 1741692228.592728 mapwrite 0x01c5f4 thru 0x029728 (0xd135 bytes) 000159 1741692228.607949 mapread 0x00b620 thru 0x015544 (0x9f25 bytes) 000160 1741692228.611902 trunc from 0x029729 to 0x00c484 (0x1d2a4 bytes) 000163 1741692228.615858 read 0x007701 thru 0x00c484 (0x4d84 bytes) 000165 1741692228.616576 read 0x004555 thru 0x00796c (0x3418 bytes) 000166 1741692228.618438 mapwrite 0x017c7f thru 0x017f5d (0x2df bytes) 000167 1741692228.627536 read 0x013ec7 thru 0x017f5d (0x4097 bytes) 000168 1741692228.627976 fallocate 0x0162c9 thru 0x022b7d (0xc8b5 bytes) 000169 1741692228.629547 write 0x008372 thru 0x009e68 (0x1af7 bytes) 000170 1741692228.630031 punch from 0x011a0c to 0x012b37 (0x112c bytes) 000171 1741692228.631350 trunc from 0x017f5e to 0x029d3c (0x11ddf bytes) 000172 1741692228.634827 mapread 0x02222b thru 0x029d3c (0x7b12 bytes) 000174 1741692228.639016 MIRROR_EXTEND to 7 mirrors 000175 1741692228.742666 read 0x004d39 thru 0x00c385 (0x764d bytes) 000176 1741692228.746885 punch from 0x007b1e to 0x00b495 (0x3978 bytes) 000177 1741692228.755163 read 0x0242ff thru 0x024c1d (0x91f bytes) 000178 1741692228.757830 write 0x01c7b8 thru 0x022d18 (0x6561 bytes) 000179 1741692228.759439 write 0x00a4d1 thru 0x0107b8 (0x62e8 bytes) 000181 1741692228.760491 punch from 0x008b5c to 0x0149e8 (0xbe8d bytes) 000184 1741692228.764286 write 0x017fab thru 0x0221d7 (0xa22d bytes) 000185 1741692228.764876 trunc from 0x029d3d to 0x02462d (0x570f bytes) 000186 1741692228.773675 read 0x01cd1e thru 0x02462d (0x7910 bytes) 000187 1741692228.774098 punch from 0x01aeb9 to 0x02462d (0x9775 bytes) 000188 1741692228.777275 fallocate 0x00c4c3 thru 0x015123 (0x8c61 bytes) 000190 1741692228.778979 mapwrite 0x026a65 thru 0x02732e (0x8ca bytes) 000192 1741692228.789657 punch from 0x00cfd4 to 0x0143f2 (0x741f bytes) 000193 1741692228.790894 mapwrite 0x01b893 thru 0x0253b4 (0x9b22 bytes) 000194 1741692228.795693 mapwrite 0x027725 thru 0x036346 (0xec22 bytes) 000195 1741692228.807022 mapread 0x02b581 thru 0x02c7ed (0x126d bytes) 000196 1741692228.807401 trunc from 0x036347 to 0x022944 (0x13a02 bytes) 000198 1741692228.811439 mapread 0x019a0e thru 0x022944 (0x8f37 bytes) 000199 1741692228.811941 write 0x033d46 thru 0x037f0c (0x41c7 bytes) 000200 1741692228.812713 write 0x02ef64 thru 0x03edd2 (0xfe6f bytes) 000202 1741692228.815032 trunc from 0x03edd3 to 0x032868 (0xc56a bytes) 000203 1741692228.818635 write 0x015f30 thru 0x01fce4 (0x9db5 bytes) 000204 1741692228.820715 MIRROR_EXTEND to 8 mirrors 000205 1741692228.924088 read 0x00a012 thru 0x016e5f (0xce4e bytes) 000207 1741692228.929683 read 0x00ac9f thru 0x013194 (0x84f6 bytes) 000209 1741692228.933153 read 0x01d8d8 thru 0x02a570 (0xcc99 bytes) 000210 1741692228.937223 fallocate 0x03d152 thru 0x03dfbe (0xe6d bytes) 000211 1741692228.945500 write 0x027c7e thru 0x031eca (0xa24d bytes) 000212 1741692228.950056 mapwrite 0x030828 thru 0x030a3a (0x213 bytes) 000213 1741692228.957426 MIRROR_RESYNC 8 mirrors 000217 1741692229.331700 punch from 0x038e17 to 0x03dfbe (0x51a8 bytes) 000218 1741692229.340038 fallocate 0x021fd3 thru 0x028446 (0x6474 bytes) 000219 1741692229.342519 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000221 1741692229.385548 write 0x030c89 thru 0x03f26d (0xe5e5 bytes) 000222 1741692229.388531 punch from 0x030c34 to 0x03e883 (0xdc50 bytes) 000223 1741692229.392635 mapread 0x020337 thru 0x025046 (0x4d10 bytes) 000225 1741692229.396677 fallocate 0x010ee2 thru 0x01da7d (0xcb9c bytes) 000226 1741692229.398257 fallocate 0x00213e thru 0x00d799 (0xb65c bytes) 000228 1741692229.399839 trunc from 0x03f26e to 0x03de3a (0x1433 bytes) 000229 1741692229.403187 mapwrite 0x019221 thru 0x0211ae (0x7f8e bytes) 000230 1741692229.410459 trunc from 0x03de3b to 0x0244ff (0x1993b bytes) 000232 1741692229.414165 mapread 0x016456 thru 0x0186e6 (0x2291 bytes) 000233 1741692229.414534 trunc from 0x024500 to 0x0398c8 (0x153c9 bytes) 000234 1741692229.417671 fallocate 0x03bbd4 thru 0x03ffff (0x442c bytes) 000235 1741692229.419082 read 0x035fca thru 0x036b59 (0xb90 bytes) 000236 1741692229.420463 punch from 0x00a6cf to 0x00fae8 (0x541a bytes) 000238 1741692229.421781 punch from 0x00dde3 to 0x01122e (0x344c bytes) 000239 1741692229.423122 write 0x037ea7 thru 0x03d68c (0x57e6 bytes) 000240 1741692229.425600 trunc from 0x040000 to 0x022f85 (0x1d07a bytes) 000241 1741692229.429206 mapread 0x01bab9 thru 0x02286d (0x6db5 bytes) 000243 1741692229.430081 trunc from 0x022f86 to 0x0109d2 (0x125b3 bytes) 000244 1741692229.433571 trunc from 0x0109d3 to 0x007aea (0x8ee8 bytes) 000245 1741692229.437875 MIRROR_EXTEND to 8 mirrors 000246 1741692229.513268 trunc from 0x007aeb to 0x015793 (0xdca9 bytes) 000249 1741692229.527346 mapwrite 0x024707 thru 0x02a3f1 (0x5ceb bytes) 000250 1741692229.544658 fallocate 0x031f6e thru 0x03839a (0x642d bytes) 000252 1741692229.546687 write 0x035573 thru 0x03ffff (0xaa8d bytes) 000253 1741692229.548542 mapwrite 0x00b235 thru 0x01a0ba (0xee86 bytes) 000254 1741692229.562429 read 0x00ebfa thru 0x019aef (0xaef6 bytes) 000255 1741692229.563069 read 0x01560f thru 0x01dd1b (0x870d bytes) 000256 1741692229.565923 trunc from 0x040000 to 0x0381db (0x7e24 bytes) 000257 1741692229.570338 write 0x02ff25 thru 0x0368bf (0x699b bytes) 000259 1741692229.571211 punch from 0x01fff0 to 0x025f0c (0x5f1d bytes) 000260 1741692229.572787 mapread 0x02c6cf thru 0x034a54 (0x8386 bytes) 000261 1741692229.573310 fallocate 0x018b0d thru 0x02874e (0xfc42 bytes) 000262 1741692229.574715 trunc from 0x0381dc to 0x014eb0 (0x2332b bytes) 000263 1741692229.578794 punch from 0x0085f3 to 0x00c077 (0x3a85 bytes) 000266 1741692229.582268 MIRROR_EXTEND to 9 mirrors 000267 1741692229.680895 fallocate 0x02f8cf thru 0x038cbf (0x93f1 bytes) 000268 1741692229.689807 mapwrite 0x03e11e thru 0x03ffff (0x1ee2 bytes) 000269 1741692229.706573 trunc from 0x040000 to 0x032737 (0xd8c8 bytes) 000270 1741692229.711207 trunc from 0x032738 to 0x004a6f (0x2dcc8 bytes) 000271 1741692229.715076 fallocate 0x02c759 thru 0x02f563 (0x2e0b bytes) 000272 1741692229.716756 write 0x024b71 thru 0x024e3d (0x2cd bytes) 000274 1741692229.720192 mapread 0x020cda thru 0x02155a (0x881 bytes) 000276 1741692229.721179 mapread 0x01cedb thru 0x02c9ee (0xfb14 bytes) 000277 1741692229.723780 punch from 0x02c86a to 0x02f563 (0x2cfa bytes) 000278 1741692229.725413 mapread 0x02cfda thru 0x02f563 (0x258a bytes) 000279 1741692229.727416 trunc from 0x02f564 to 0x00f4fb (0x20068 bytes) 000280 1741692229.731777 mapread 0x0096cb thru 0x00f4fb (0x5e31 bytes) 000281 1741692229.737033 MIRROR_EXTEND to 10 mirrors 000283 1741692229.833609 read 0x00ee88 thru 0x00f4fb (0x674 bytes) 000285 1741692229.835225 mapwrite 0x03d47b thru 0x03ffff (0x2b85 bytes) 000286 1741692229.856258 write 0x01bab6 thru 0x01e7a2 (0x2ced bytes) 000287 1741692229.858515 mapwrite 0x01bfbe thru 0x023421 (0x7464 bytes) 000288 1741692229.864340 MIRROR_RESYNC 10 mirrors 000289 1741692230.172932 mapwrite 0x025ee7 thru 0x02ef23 (0x903d bytes) 000290 1741692230.202347 mapwrite 0x03b8c1 thru 0x03d41d (0x1b5d bytes) 000291 1741692230.208152 read 0x03caa2 thru 0x03ffff (0x355e bytes) 000292 1741692230.208455 mapread 0x01322f thru 0x019b41 (0x6913 bytes) 000293 1741692230.212124 MIRROR_RESYNC 10 mirrors 000295 1741692230.595567 read 0x003a51 thru 0x004b34 (0x10e4 bytes) 000296 1741692230.597170 mapwrite 0x03edcf thru 0x03ffff (0x1231 bytes) 000297 1741692230.623814 read 0x01299a thru 0x0187d1 (0x5e38 bytes) 000298 1741692230.628047 MIRROR_EXTEND to 11 mirrors 000299 1741692230.741569 mapread 0x004f24 thru 0x00dc27 (0x8d04 bytes) 000300 1741692230.749109 write 0x02f769 thru 0x03c2d4 (0xcb6c bytes) 000301 1741692230.763273 MIRROR_EXTEND to 12 mirrors 000303 1741692230.883546 mapwrite 0x01febf thru 0x02d2ac (0xd3ee bytes) 000304 1741692230.919004 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000305 1741692230.964702 punch from 0x01a0cd to 0x01c600 (0x2534 bytes) 000306 1741692230.966151 mapread 0x0213ca thru 0x026c85 (0x58bc bytes) 000307 1741692230.971502 fallocate 0x02fc50 thru 0x035f16 (0x62c7 bytes) 000308 1741692230.973182 punch from 0x014c94 to 0x023de7 (0xf154 bytes) 000309 1741692230.974546 mapread 0x035432 thru 0x03b909 (0x64d8 bytes) 000310 1741692230.977848 punch from 0x009777 to 0x00e423 (0x4cad bytes) 000311 1741692230.979061 write 0x02d7e1 thru 0x02fac5 (0x22e5 bytes) 000312 1741692230.983257 MIRROR_EXTEND to 12 mirrors 000313 1741692231.093545 mapwrite 0x029bc6 thru 0x02e889 (0x4cc4 bytes) 000314 1741692231.128324 read 0x00d7a7 thru 0x01b681 (0xdedb bytes) 000315 1741692231.132535 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000316 1741692231.186271 mapwrite 0x034c8b thru 0x03ffff (0xb375 bytes) 000317 1741692231.198309 mapwrite 0x00aa16 thru 0x019668 (0xec53 bytes) 000318 1741692231.209371 trunc from 0x040000 to 0x021962 (0x1e69d bytes) 000319 1741692231.214418 fallocate 0x024c89 thru 0x02f0b1 (0xa429 bytes) 000320 1741692231.216213 mapwrite 0x02cb84 thru 0x0340b2 (0x752f bytes) 000321 1741692231.228694 read 0x019975 thru 0x024edb (0xb567 bytes) 000323 1741692231.231445 read 0x027aee thru 0x029c5c (0x216f bytes) 000324 1741692231.231702 punch from 0x02d2e4 to 0x02f118 (0x1e35 bytes) 000325 1741692231.233288 trunc from 0x0340b3 to 0x001d1c (0x32396 bytes) 000327 1741692231.237785 read 0x00144f thru 0x001d1c (0x8ce bytes) 000328 1741692231.239808 write 0x016d8b thru 0x020ca9 (0x9f1f bytes) 000331 1741692231.243439 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000332 1741692231.303164 mapwrite 0x0064f9 thru 0x01358f (0xd097 bytes) 000333 1741692231.319558 mapread 0x002f78 thru 0x00eee6 (0xbf6f bytes) 000334 1741692231.320041 fallocate 0x03215d thru 0x03b0dc (0x8f80 bytes) 000335 1741692231.321892 write 0x00efa8 thru 0x010c15 (0x1c6e bytes) 000336 1741692231.323721 mapread 0x00b3a5 thru 0x00f7bb (0x4417 bytes) 000337 1741692231.324434 trunc from 0x03b0dd to 0x03166a (0x9a72 bytes) 000338 1741692231.328321 punch from 0x006d3c to 0x00d84a (0x6b0f bytes) 000339 1741692231.330025 trunc from 0x03166b to 0x0352be (0x3c54 bytes) 000341 1741692231.334349 write 0x01992b thru 0x0269a4 (0xd07a bytes) 000342 1741692231.336833 mapwrite 0x0238be thru 0x02c5e8 (0x8d2b bytes) 000343 1741692231.345450 write 0x00254c thru 0x004a64 (0x2519 bytes) 000344 1741692231.345954 punch from 0x0002e0 to 0x005c87 (0x59a8 bytes) 000345 1741692231.348556 trunc from 0x0352bf to 0x019f54 (0x1b36a bytes) 000346 1741692231.353273 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000347 1741692231.399172 fallocate 0x03fabb thru 0x03ffff (0x545 bytes) 000348 1741692231.400790 read 0x005590 thru 0x014980 (0xf3f1 bytes) 000349 1741692231.403127 write 0x034431 thru 0x03ffff (0xbbcf bytes) 000350 1741692231.405706 mapread 0x014673 thru 0x01783c (0x31ca bytes) 000353 1741692231.409394 write 0x00f7f2 thru 0x01e331 (0xeb40 bytes) 000354 1741692231.410094 mapread 0x019678 thru 0x0250b1 (0xba3a bytes) 000355 1741692231.412266 mapread 0x033643 thru 0x03db8e (0xa54c bytes) 000356 1741692231.415731 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000357 1741692231.460914 read 0x02c933 thru 0x02f7cb (0x2e99 bytes) 000358 1741692231.464283 MIRROR_EXTEND to 9 mirrors 000359 1741692231.573631 fallocate 0x01d24b thru 0x028f77 (0xbd2d bytes) 000360 1741692231.581737 fallocate 0x01e06c thru 0x02d9d7 (0xf96c bytes) 000361 1741692231.583160 trunc from 0x040000 to 0x0353af (0xac50 bytes) 000362 1741692231.591548 mapwrite 0x01a688 thru 0x01c1ae (0x1b27 bytes) 000363 1741692231.602219 MIRROR_RESYNC 9 mirrors 000364 1741692231.998710 punch from 0x02b938 to 0x02f118 (0x37e1 bytes) 000365 1741692232.010085 punch from 0x015387 to 0x01a855 (0x54cf bytes) 000366 1741692232.011902 trunc from 0x0353b0 to 0x038441 (0x3092 bytes) 000367 1741692232.021259 read 0x02c9c4 thru 0x02f4a6 (0x2ae3 bytes) 000368 1741692232.024286 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000369 1741692232.086140 mapread 0x021cab thru 0x02b36c (0x96c2 bytes) 000370 1741692232.095226 trunc from 0x038442 to 0x01b784 (0x1ccbd bytes) 000371 1741692232.101347 trunc from 0x01b785 to 0x01ba4b (0x2c7 bytes) 000372 1741692232.105934 write 0x01978c thru 0x01f359 (0x5bce bytes) 000373 1741692232.108966 punch from 0x00e026 to 0x01a572 (0xc54d bytes) 000374 1741692232.113230 punch from 0x01d11d to 0x01f359 (0x223d bytes) 000375 1741692232.114834 fallocate 0x01f820 thru 0x02f1a4 (0xf985 bytes) 000376 1741692232.116558 punch from 0x013377 to 0x016cb9 (0x3943 bytes) 000377 1741692232.118076 punch from 0x0002b1 to 0x0021ee (0x1f3e bytes) 000378 1741692232.119491 fallocate 0x002ddc thru 0x012507 (0xf72c bytes) 000380 1741692232.122666 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000382 1741692232.169069 trunc from 0x01f35a to 0x03cc23 (0x1d8ca bytes) 000383 1741692232.173555 fallocate 0x00992e thru 0x00afa1 (0x1674 bytes) 000384 1741692232.175962 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000385 1741692232.221485 mapread 0x024e1d thru 0x029679 (0x485d bytes) 000386 1741692232.228604 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000388 1741692232.273123 write 0x002c51 thru 0x00aefc (0x82ac bytes) 000389 1741692232.276853 write 0x024126 thru 0x0276a4 (0x357f bytes) 000390 1741692232.279107 fallocate 0x019b51 thru 0x01cee6 (0x3396 bytes) 000392 1741692232.281919 trunc from 0x03cc24 to 0x0338c0 (0x9363 bytes) 000393 1741692232.285663 write 0x00ee1f thru 0x01607d (0x725f bytes) 000394 1741692232.288435 mapwrite 0x00a6a2 thru 0x00d2df (0x2c3e bytes) 000395 1741692232.295596 MIRROR_RESYNC 5 mirrors 000396 1741692232.539385 trunc from 0x0338c1 to 0x020d42 (0x12b7e bytes) 000397 1741692232.552674 read 0x006e9a thru 0x0138c2 (0xca29 bytes) 000398 1741692232.555471 read 0x00da64 thru 0x00feb9 (0x2456 bytes) 000399 1741692232.555778 mapread 0x008e18 thru 0x017fbc (0xf1a5 bytes) 000400 1741692232.558943 punch from 0x01a3cb to 0x01bee3 (0x1b19 bytes) 000401 1741692232.560599 write 0x0120ff thru 0x014a29 (0x292b bytes) 000402 1741692232.561188 trunc from 0x020d43 to 0x002dda (0x1df68 bytes) 000403 1741692232.567958 mapwrite 0x002b4a thru 0x00f4ae (0xc965 bytes) 000405 1741692232.597086 MIRROR_EXTEND to 6 mirrors 000408 1741692232.697017 mapwrite 0x011eff thru 0x016b07 (0x4c09 bytes) 000409 1741692232.724000 trunc from 0x016b08 to 0x038f2d (0x22426 bytes) 000410 1741692232.729962 punch from 0x00cd43 to 0x017834 (0xaaf2 bytes) 000411 1741692232.733282 MIRROR_RESYNC 6 mirrors 000414 1741692232.947750 fallocate 0x0107a8 thru 0x01c998 (0xc1f1 bytes) 000415 1741692232.956239 trunc from 0x038f2e to 0x007e41 (0x310ec bytes) 000416 1741692232.963547 trunc from 0x007e42 to 0x030f70 (0x2912f bytes) 000417 1741692232.967156 mapwrite 0x01da92 thru 0x026d1d (0x928c bytes) 000419 1741692232.977486 fallocate 0x02284f thru 0x02cdfc (0xa5ae bytes) 000420 1741692232.978812 fallocate 0x010ae4 thru 0x01cad1 (0xbfee bytes) 000421 1741692232.981465 MIRROR_RESYNC 6 mirrors 000422 1741692233.225806 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000423 1741692233.282702 write 0x03bd2c thru 0x03ffff (0x42d4 bytes) 000427 1741692233.292938 write 0x012237 thru 0x017194 (0x4f5e bytes) 000429 1741692233.296064 write 0x021cd9 thru 0x02bc2e (0x9f56 bytes) 000431 1741692233.301138 punch from 0x037639 to 0x03ffff (0x89c7 bytes) 000432 1741692233.305150 read 0x010a20 thru 0x010b25 (0x106 bytes) 000433 1741692233.306474 read 0x004c6f thru 0x01226e (0xd600 bytes) 000435 1741692233.309624 punch from 0x02439a to 0x028787 (0x43ee bytes) 000437 1741692233.311554 mapwrite 0x00c4eb thru 0x01845a (0xbf70 bytes) 000438 1741692233.318760 write 0x0329a4 thru 0x03762d (0x4c8a bytes) 000439 1741692233.319293 write 0x03d333 thru 0x03df5b (0xc29 bytes) 000441 1741692233.319885 punch from 0x00fc42 to 0x019a68 (0x9e27 bytes) 000442 1741692233.321322 mapwrite 0x0166f3 thru 0x0194f1 (0x2dff bytes) 000444 1741692233.329636 mapread 0x004820 thru 0x006cba (0x249b bytes) 000446 1741692233.330248 mapwrite 0x0127b1 thru 0x01af01 (0x8751 bytes) 000447 1741692233.339064 fallocate 0x0169d5 thru 0x020626 (0x9c52 bytes) 000448 1741692233.340591 write 0x016301 thru 0x021b35 (0xb835 bytes) 000453 1741692233.342368 read 0x03ed5e thru 0x03ffff (0x12a2 bytes) 000454 1741692233.342666 punch from 0x012945 to 0x01de5b (0xb517 bytes) 000455 1741692233.346132 read 0x00fee6 thru 0x01fe42 (0xff5d bytes) 000456 1741692233.348510 fallocate 0x00cd8e thru 0x01b5ef (0xe862 bytes) 000457 1741692233.350098 mapread 0x009663 thru 0x00bd27 (0x26c5 bytes) 000458 1741692233.350710 fallocate 0x02b39c thru 0x032276 (0x6edb bytes) 000459 1741692233.353903 MIRROR_RESYNC 5 mirrors 000461 1741692233.691521 trunc from 0x040000 to 0x03a45b (0x5ba4 bytes) 000462 1741692233.704188 read 0x000258 thru 0x0039cd (0x3776 bytes) 000463 1741692233.705735 mapread 0x02f5fe thru 0x02f925 (0x328 bytes) 000464 1741692233.708390 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000465 1741692233.757509 mapread 0x003e14 thru 0x008413 (0x4600 bytes) 000466 1741692233.762023 mapread 0x01a619 thru 0x01c0e5 (0x1acd bytes) 000467 1741692233.764411 read 0x01c9ca thru 0x01f735 (0x2d6c bytes) 000469 1741692233.764942 read 0x02a73e thru 0x02f013 (0x48d6 bytes) 000470 1741692233.770651 punch from 0x0118c5 to 0x01ffd3 (0xe70f bytes) 000471 1741692233.773618 MIRROR_EXTEND to 5 mirrors 000473 1741692233.904248 write 0x025c4c thru 0x02cd3e (0x70f3 bytes) 000476 1741692233.914459 mapread 0x017ebd thru 0x01bb0d (0x3c51 bytes) 000477 1741692233.916749 trunc from 0x03a45c to 0x03a887 (0x42c bytes) 000478 1741692233.924470 read 0x025662 thru 0x0268d6 (0x1275 bytes) 000480 1741692233.924955 punch from 0x023fc9 to 0x02c131 (0x8169 bytes) 000482 1741692233.928323 punch from 0x01d681 to 0x02357f (0x5eff bytes) 000484 1741692233.929983 mapwrite 0x0238f6 thru 0x02549d (0x1ba8 bytes) 000486 1741692233.936227 write 0x004c17 thru 0x013a3e (0xee28 bytes) 000487 1741692233.939808 fallocate 0x024f99 thru 0x02ee17 (0x9e7f bytes) 000488 1741692233.942847 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000489 1741692233.990188 write 0x02d13b thru 0x030679 (0x353f bytes) 000490 1741692233.993810 write 0x033d1c thru 0x037509 (0x37ee bytes) 000491 1741692233.996254 trunc from 0x03a888 to 0x029bc6 (0x10cc1 bytes) 000492 1741692234.002279 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000493 1741692234.045596 punch from 0x00fa96 to 0x01a9ca (0xaf35 bytes) 000494 1741692234.047156 write 0x00852b thru 0x0136af (0xb185 bytes) 000495 1741692234.050504 mapread 0x00bb3c thru 0x016120 (0xa5e5 bytes) 000496 1741692234.053948 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000497 1741692234.096840 read 0x022328 thru 0x022fec (0xcc5 bytes) 000499 1741692234.098776 trunc from 0x029bc7 to 0x016b00 (0x130c6 bytes) 000500 1741692234.103156 fallocate 0x02e404 thru 0x036e90 (0x8a8d bytes) 000502 1741692234.104831 mapwrite 0x01e8ef thru 0x02d993 (0xf0a5 bytes) 000503 1741692234.121411 fallocate 0x01d6cf thru 0x01dd93 (0x6c5 bytes) 000504 1741692234.122852 fallocate 0x01bf38 thru 0x0270ed (0xb1b6 bytes) 000505 1741692234.124442 punch from 0x00d655 to 0x0138cd (0x6279 bytes) 000506 1741692234.127262 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000507 1741692234.154958 trunc from 0x02d994 to 0x006978 (0x2701b bytes) 000510 1741692234.160059 fallocate 0x00f06a thru 0x019c3e (0xabd5 bytes) 000513 1741692234.162125 read 0x00f798 thru 0x0138a2 (0x410b bytes) 000514 1741692234.164085 trunc from 0x019c3f to 0x01c274 (0x2636 bytes) 000515 1741692234.168438 punch from 0x007dd6 to 0x00857d (0x7a8 bytes) 000517 1741692234.169806 mapwrite 0x0214a2 thru 0x0224d5 (0x1034 bytes) 000518 1741692234.179476 trunc from 0x0224d6 to 0x03e893 (0x1c3be bytes) 000520 1741692234.183008 write 0x01d10a thru 0x02c22b (0xf122 bytes) 000521 1741692234.186845 trunc from 0x03e894 to 0x00c6f1 (0x321a2 bytes) 000523 1741692234.190820 write 0x00b0c1 thru 0x00bf3d (0xe7d bytes) 000525 1741692234.193261 read 0x002439 thru 0x007a0c (0x55d4 bytes) 000526 1741692234.194732 punch from 0x0064ae to 0x008836 (0x2389 bytes) 000527 1741692234.196681 trunc from 0x00c6f2 to 0x03f2f7 (0x32c06 bytes) 000528 1741692234.200951 trunc from 0x03f2f8 to 0x03dadd (0x181a bytes) 000529 1741692234.204667 punch from 0x032d56 to 0x03a917 (0x7bc2 bytes) 000531 1741692234.206352 mapwrite 0x0113f1 thru 0x01a104 (0x8d14 bytes) 000532 1741692234.216908 trunc from 0x03dade to 0x030663 (0xd47a bytes) 000533 1741692234.221067 trunc from 0x030664 to 0x003585 (0x2d0de bytes) 000534 1741692234.225052 read 0x002c75 thru 0x003585 (0x911 bytes) 000536 1741692234.225727 mapread 0x002c56 thru 0x003585 (0x930 bytes) 000537 1741692234.226169 mapwrite 0x03f62d thru 0x03ffff (0x9d3 bytes) 000539 1741692234.237856 trunc from 0x040000 to 0x02a9db (0x15624 bytes) 000540 1741692234.242092 mapread 0x00204a thru 0x005e78 (0x3e2f bytes) 000542 1741692234.243382 fallocate 0x0378a4 thru 0x03ffb0 (0x870d bytes) 000543 1741692234.245123 write 0x004485 thru 0x00ae85 (0x6a01 bytes) 000544 1741692234.245700 mapwrite 0x005c0b thru 0x00c662 (0x6a58 bytes) 000545 1741692234.251757 read 0x003cec thru 0x00ae0f (0x7124 bytes) 000546 1741692234.252117 trunc from 0x02a9dc to 0x02dbf6 (0x321b bytes) 000547 1741692234.255688 mapwrite 0x004695 thru 0x0137e0 (0xf14c bytes) 000548 1741692234.263204 mapread 0x020046 thru 0x02a4a4 (0xa45f bytes) 000549 1741692234.263912 write 0x023246 thru 0x026811 (0x35cc bytes) 000552 1741692234.265384 mapread 0x019f81 thru 0x01cef3 (0x2f73 bytes) 000556 1741692234.266417 read 0x010831 thru 0x01ac2a (0xa3fa bytes) 000557 1741692234.266717 read 0x00a829 thru 0x00b27b (0xa53 bytes) 000558 1741692234.266996 write 0x01ba4a thru 0x020300 (0x48b7 bytes) 000559 1741692234.267705 trunc from 0x02dbf7 to 0x00c1c3 (0x21a33 bytes) 000560 1741692234.272337 fallocate 0x02decc thru 0x0377ad (0x98e2 bytes) 000561 1741692234.274067 punch from 0x009ba7 to 0x00c1c3 (0x261d bytes) 000563 1741692234.275925 write 0x01b612 thru 0x02b3bf (0xfdae bytes) 000564 1741692234.277227 write 0x0042f6 thru 0x00cb1f (0x882a bytes) 000566 1741692234.280696 MIRROR_EXTEND to 2 mirrors 000568 1741692234.385661 MIRROR_EXTEND to 3 mirrors 000569 1741692234.492695 write 0x011367 thru 0x01fa6f (0xe709 bytes) 000570 1741692234.504525 mapwrite 0x012253 thru 0x019610 (0x73be bytes) 000571 1741692234.512546 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000572 1741692234.560417 mapread 0x0013ca thru 0x00181a (0x451 bytes) 000573 1741692234.563109 MIRROR_EXTEND to 3 mirrors 000574 1741692234.669814 MIRROR_RESYNC 3 mirrors 000575 1741692234.773142 write 0x00dbf6 thru 0x00f0b5 (0x14c0 bytes) 000576 1741692234.782515 read 0x01541e thru 0x019f68 (0x4b4b bytes) 000577 1741692234.784716 write 0x00c63f thru 0x0108c3 (0x4285 bytes) 000578 1741692234.787087 read 0x006a3f thru 0x00c3ef (0x59b1 bytes) 000579 1741692234.787498 read 0x016e7f thru 0x01d6a4 (0x6826 bytes) 000580 1741692234.790755 mapread 0x01237b thru 0x015c99 (0x391f bytes) 000581 1741692234.791455 write 0x01452f thru 0x01dad0 (0x95a2 bytes) 000582 1741692234.792197 trunc from 0x02b3c0 to 0x01eb71 (0xc84e bytes) 000583 1741692234.800431 mapwrite 0x0176de thru 0x0247e1 (0xd104 bytes) 000584 1741692234.818283 write 0x03a94e thru 0x03ffff (0x56b2 bytes) 000586 1741692234.819359 trunc from 0x040000 to 0x03f2a2 (0xd5d bytes) 000588 1741692234.823405 trunc from 0x03f2a3 to 0x00fd79 (0x2f529 bytes) 000589 1741692234.828640 MIRROR_EXTEND to 4 mirrors 000590 1741692234.919122 MIRROR_EXTEND to 5 mirrors 000591 1741692235.009269 MIRROR_RESYNC 5 mirrors 000592 1741692235.130008 mapread 0x006216 thru 0x00b92c (0x5717 bytes) 000593 1741692235.134437 read 0x0031cc thru 0x0062bb (0x30f0 bytes) 000594 1741692235.134742 mapread 0x00086c thru 0x006420 (0x5bb5 bytes) 000597 1741692235.138270 punch from 0x00ca65 to 0x00d5eb (0xb87 bytes) 000599 1741692235.148408 MIRROR_RESYNC 5 mirrors 000600 1741692235.300143 mapread 0x00fcb8 thru 0x00fd79 (0xc2 bytes) 000601 1741692235.300707 trunc from 0x00fd7a to 0x02419b (0x14422 bytes) 000602 1741692235.310944 mapread 0x003e1b thru 0x00fec8 (0xc0ae bytes) 000603 1741692235.317925 trunc from 0x02419c to 0x00d1ed (0x16fae bytes) 000604 1741692235.322918 write 0x000deb thru 0x0102af (0xf4c5 bytes) 000605 1741692235.325264 write 0x023040 thru 0x030fcb (0xdf8c bytes) 000606 1741692235.326104 punch from 0x01139a to 0x01d2f5 (0xbf5c bytes) 000608 1741692235.327897 mapwrite 0x01041d thru 0x019700 (0x92e4 bytes) 000609 1741692235.339487 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000610 1741692235.389111 trunc from 0x030fcc to 0x038a7a (0x7aaf bytes) 000611 1741692235.393748 mapwrite 0x03c693 thru 0x03e6ac (0x201a bytes) 000613 1741692235.411735 read 0x00ab36 thru 0x013581 (0x8a4c bytes) 000614 1741692235.414022 trunc from 0x03e6ad to 0x003ae6 (0x3abc6 bytes) 000615 1741692235.418449 mapread 0x000797 thru 0x001de1 (0x164b bytes) 000617 1741692235.421835 punch from 0x003556 to 0x003ae6 (0x591 bytes) 000619 1741692235.423484 mapread 0x000721 thru 0x003ae6 (0x33c6 bytes) 000620 1741692235.425167 punch from 0x00349f to 0x003ae6 (0x648 bytes) 000621 1741692235.426550 trunc from 0x003ae7 to 0x024d14 (0x2122e bytes) 000622 1741692235.430379 punch from 0x015e49 to 0x021b2c (0xbce4 bytes) 000623 1741692235.431783 mapwrite 0x003d72 thru 0x0045db (0x86a bytes) 000624 1741692235.439323 read 0x00491c thru 0x00d0e3 (0x87c8 bytes) 000625 1741692235.439704 fallocate 0x000905 thru 0x003566 (0x2c62 bytes) 000626 1741692235.441341 punch from 0x0194bf to 0x01c3e7 (0x2f29 bytes) 000627 1741692235.442714 mapwrite 0x0350f9 thru 0x03ffff (0xaf07 bytes) 000628 1741692235.457012 fallocate 0x01a88e thru 0x027945 (0xd0b8 bytes) 000629 1741692235.458900 mapwrite 0x013121 thru 0x01c827 (0x9707 bytes) 000630 1741692235.466138 MIRROR_EXTEND to 5 mirrors 000633 1741692235.577027 trunc from 0x040000 to 0x025f67 (0x1a098 bytes) 000634 1741692235.594766 MIRROR_EXTEND to 6 mirrors 000635 1741692235.693160 read 0x011ef5 thru 0x0132fb (0x1407 bytes) 000636 1741692235.696246 mapread 0x002ce3 thru 0x00adb3 (0x80d1 bytes) 000637 1741692235.703821 punch from 0x003daf to 0x012256 (0xe4a8 bytes) 000638 1741692235.712463 mapwrite 0x02648d thru 0x0281ed (0x1d61 bytes) 000639 1741692235.728039 mapwrite 0x005540 thru 0x00b764 (0x6225 bytes) 000640 1741692235.736066 mapread 0x00fc9a thru 0x014213 (0x457a bytes) 000641 1741692235.738376 write 0x02be2f thru 0x030a2e (0x4c00 bytes) 000642 1741692235.739163 fallocate 0x039588 thru 0x03ffff (0x6a78 bytes) 000643 1741692235.740819 mapread 0x030b3a thru 0x03c894 (0xbd5b bytes) 000644 1741692235.744344 mapwrite 0x028ca1 thru 0x02a088 (0x13e8 bytes) 000645 1741692235.751741 MIRROR_RESYNC 6 mirrors 000648 1741692236.016981 mapwrite 0x0011e2 thru 0x010d64 (0xfb83 bytes) 000649 1741692236.047270 MIRROR_RESYNC 6 mirrors 000650 1741692236.300460 read 0x0026e2 thru 0x00e265 (0xbb84 bytes) 000651 1741692236.302485 trunc from 0x040000 to 0x03743b (0x8bc4 bytes) 000652 1741692236.311636 mapwrite 0x0216d7 thru 0x02617d (0x4aa7 bytes) 000654 1741692236.321509 read 0x0249fd thru 0x02a2e0 (0x58e4 bytes) 000655 1741692236.321915 fallocate 0x038ce6 thru 0x03c677 (0x3992 bytes) 000656 1741692236.323629 read 0x015b0e thru 0x021534 (0xba27 bytes) 000657 1741692236.326406 mapread 0x031f65 thru 0x03743b (0x54d7 bytes) 000659 1741692236.327154 fallocate 0x03d7ce thru 0x03ffff (0x2832 bytes) 000660 1741692236.329158 mapwrite 0x03d832 thru 0x03ffff (0x27ce bytes) 000661 1741692236.335573 fallocate 0x01b87f thru 0x0225eb (0x6d6d bytes) 000662 1741692236.337337 write 0x012b07 thru 0x01b687 (0x8b81 bytes) 000663 1741692236.338140 read 0x020bfd thru 0x023797 (0x2b9b bytes) 000664 1741692236.338394 trunc from 0x040000 to 0x00ed3a (0x312c5 bytes) 000665 1741692236.345280 MIRROR_EXTEND to 7 mirrors 000666 1741692236.424849 mapread 0x00dfa6 thru 0x00ed3a (0xd95 bytes) 000668 1741692236.429387 fallocate 0x016ae9 thru 0x02681d (0xfd35 bytes) 000669 1741692236.440733 MIRROR_RESYNC 7 mirrors 000670 1741692236.578182 read 0x0016b4 thru 0x0029a4 (0x12f1 bytes) 000671 1741692236.579624 mapwrite 0x03d18d thru 0x03ffff (0x2e73 bytes) 000672 1741692236.600684 write 0x016790 thru 0x0248e3 (0xe154 bytes) 000674 1741692236.604698 mapwrite 0x02156b thru 0x0284dd (0x6f73 bytes) 000675 1741692236.612451 fallocate 0x009285 thru 0x00d75b (0x44d7 bytes) 000676 1741692236.614040 fallocate 0x017e07 thru 0x01ee7f (0x7079 bytes) 000677 1741692236.615397 fallocate 0x02f439 thru 0x037065 (0x7c2d bytes) 000678 1741692236.617011 fallocate 0x0154a6 thru 0x016c78 (0x17d3 bytes) 000679 1741692236.618401 mapread 0x017bd2 thru 0x022783 (0xabb2 bytes) 000680 1741692236.618855 mapread 0x02d0eb thru 0x0367bd (0x96d3 bytes) 000682 1741692236.622822 punch from 0x01dfd4 to 0x02d764 (0xf791 bytes) 000683 1741692236.624636 write 0x0075d4 thru 0x009237 (0x1c64 bytes) 000684 1741692236.627126 write 0x00b240 thru 0x01a6ad (0xf46e bytes) 000685 1741692236.627898 mapread 0x0198b5 thru 0x01ac8b (0x13d7 bytes) 000686 1741692236.628372 mapread 0x02a3ca thru 0x037e49 (0xda80 bytes) 000689 1741692236.631075 write 0x0002a2 thru 0x0057ea (0x5549 bytes) 000690 1741692236.632860 trunc from 0x040000 to 0x012ef3 (0x2d10c bytes) 000691 1741692236.638615 MIRROR_RESYNC 7 mirrors 000692 1741692236.767775 punch from 0x0121a1 to 0x012ef3 (0xd53 bytes) 000695 1741692236.776611 mapread 0x00c22f thru 0x00de5b (0x1c2d bytes) 000696 1741692236.778911 read 0x0037d4 thru 0x00a9ba (0x71e7 bytes) 000697 1741692236.780852 write 0x03a3c0 thru 0x03ffff (0x5c40 bytes) 000699 1741692236.781895 fallocate 0x029505 thru 0x0376f4 (0xe1f0 bytes) 000701 1741692236.783842 write 0x0004e3 thru 0x008121 (0x7c3f bytes) 000702 1741692236.786457 write 0x022945 thru 0x02f3f0 (0xcaac bytes) 000703 1741692236.789734 mapread 0x0138f2 thru 0x015b0e (0x221d bytes) 000704 1741692236.792056 write 0x03fa77 thru 0x03ffff (0x589 bytes) 000705 1741692236.792351 read 0x034155 thru 0x039912 (0x57be bytes) 000707 1741692236.794364 punch from 0x011761 to 0x016fa2 (0x5842 bytes) 000709 1741692236.795995 write 0x036884 thru 0x03ffff (0x977c bytes) 000710 1741692236.796912 mapwrite 0x02d857 thru 0x03ac11 (0xd3bb bytes) 000715 1741692236.804396 write 0x0090c5 thru 0x010d29 (0x7c65 bytes) 000716 1741692236.804987 read 0x035720 thru 0x03a011 (0x48f2 bytes) 000718 1741692236.805436 mapwrite 0x00ee1f thru 0x0169a3 (0x7b85 bytes) 000719 1741692236.812041 mapwrite 0x00d3cf thru 0x0115e5 (0x4217 bytes) 000720 1741692236.816431 punch from 0x024b5f to 0x02a0b0 (0x5552 bytes) 000721 1741692236.818116 write 0x03c6f2 thru 0x03fd9d (0x36ac bytes) 000723 1741692236.820176 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000724 1741692236.866608 punch from 0x02c7fb to 0x034a93 (0x8299 bytes) 000725 1741692236.868042 write 0x037243 thru 0x03ffff (0x8dbd bytes) 000726 1741692236.870679 mapwrite 0x019803 thru 0x02824a (0xea48 bytes) 000727 1741692236.882116 read 0x03be3e thru 0x03ffff (0x41c2 bytes) 000728 1741692236.882594 trunc from 0x040000 to 0x000515 (0x3faea bytes) 000729 1741692236.890380 mapread 0x0003f8 thru 0x000515 (0x11e bytes) 000730 1741692236.892169 mapread 0x00004f thru 0x000515 (0x4c7 bytes) 000731 1741692236.892606 trunc from 0x000516 to 0x01295b (0x12446 bytes) 000732 1741692236.898143 MIRROR_EXTEND to 7 mirrors 000733 1741692236.993808 mapread 0x004450 thru 0x00511e (0xccf bytes) 000734 1741692237.000113 write 0x011fae thru 0x016dcb (0x4e1e bytes) 000735 1741692237.010545 read 0x007e48 thru 0x012536 (0xa6ef bytes) 000736 1741692237.012099 punch from 0x00b291 to 0x016dcb (0xbb3b bytes) 000737 1741692237.015470 write 0x037338 thru 0x038a56 (0x171f bytes) 000738 1741692237.015983 mapwrite 0x03a1c7 thru 0x03e304 (0x413e bytes) 000739 1741692237.030268 write 0x0255a1 thru 0x028668 (0x30c8 bytes) 000741 1741692237.034941 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000742 1741692237.082097 fallocate 0x017848 thru 0x01c478 (0x4c31 bytes) 000743 1741692237.083733 write 0x02c920 thru 0x033a49 (0x712a bytes) 000744 1741692237.087821 read 0x00d07e thru 0x013917 (0x689a bytes) 000745 1741692237.089899 trunc from 0x03e305 to 0x02f87b (0xea89 bytes) 000748 1741692237.094189 mapread 0x029144 thru 0x02f87b (0x6738 bytes) 000749 1741692237.097622 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000750 1741692237.144156 mapread 0x015f8c thru 0x0162d7 (0x34c bytes) 000751 1741692237.146853 trunc from 0x02f87c to 0x038935 (0x90ba bytes) 000753 1741692237.151748 fallocate 0x00b035 thru 0x0193c3 (0xe38f bytes) 000754 1741692237.153128 read 0x0209e1 thru 0x02d5d2 (0xcbf2 bytes) 000755 1741692237.156329 fallocate 0x005aa2 thru 0x00b5d9 (0x5b38 bytes) 000757 1741692237.158087 mapread 0x0313e3 thru 0x033ab5 (0x26d3 bytes) 000759 1741692237.161454 mapread 0x01ea18 thru 0x021341 (0x292a bytes) 000760 1741692237.163725 read 0x008f15 thru 0x0113cb (0x84b7 bytes) 000761 1741692237.166110 mapwrite 0x0287b1 thru 0x0376b4 (0xef04 bytes) 000762 1741692237.175006 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000763 1741692237.223061 trunc from 0x038936 to 0x0349e3 (0x3f52 bytes) 000764 1741692237.229564 punch from 0x016c13 to 0x01e142 (0x7530 bytes) 000765 1741692237.232150 trunc from 0x0349e4 to 0x0129a7 (0x2203c bytes) 000766 1741692237.236994 write 0x01b44d thru 0x01dc75 (0x2829 bytes) 000767 1741692237.237617 mapread 0x008561 thru 0x016035 (0xdad5 bytes) 000768 1741692237.247308 MIRROR_RESYNC 4 mirrors 000770 1741692237.472587 fallocate 0x0217a7 thru 0x021dc0 (0x61a bytes) 000771 1741692237.485205 mapwrite 0x00caf2 thru 0x01180f (0x4d1e bytes) 000773 1741692237.503223 punch from 0x00fe53 to 0x013f18 (0x40c6 bytes) 000774 1741692237.506527 trunc from 0x021dc1 to 0x00e5c7 (0x137f9 bytes) 000775 1741692237.521133 MIRROR_EXTEND to 5 mirrors 000776 1741692237.628752 punch from 0x00e4fa to 0x00e5c7 (0xce bytes) 000778 1741692237.638015 punch from 0x00cda7 to 0x00d16b (0x3c5 bytes) 000779 1741692237.639323 fallocate 0x024063 thru 0x02e01a (0x9fb8 bytes) 000780 1741692237.640965 write 0x00aeb4 thru 0x014a53 (0x9ba0 bytes) 000781 1741692237.644461 mapwrite 0x003d5c thru 0x00f700 (0xb9a5 bytes) 000782 1741692237.652535 fallocate 0x0177dd thru 0x01fbc5 (0x83e9 bytes) 000783 1741692237.654084 punch from 0x018ba6 to 0x027480 (0xe8db bytes) 000784 1741692237.655443 fallocate 0x00f585 thru 0x0197fc (0xa278 bytes) 000786 1741692237.657110 read 0x006fd3 thru 0x00800d (0x103b bytes) 000787 1741692237.657392 mapwrite 0x001796 thru 0x00e4c5 (0xcd30 bytes) 000788 1741692237.663325 trunc from 0x02e01b to 0x02c18c (0x1e8e bytes) 000789 1741692237.672078 mapread 0x01738c thru 0x022178 (0xaded bytes) 000790 1741692237.676330 read 0x013b58 thru 0x01d397 (0x9840 bytes) 000792 1741692237.676836 trunc from 0x02c18d to 0x01ab02 (0x1168a bytes) 000793 1741692237.681473 read 0x005089 thru 0x0104b9 (0xb431 bytes) 000794 1741692237.681818 mapwrite 0x018226 thru 0x021616 (0x93f1 bytes) 000795 1741692237.693251 write 0x025dae thru 0x035197 (0xf3ea bytes) 000796 1741692237.694291 read 0x001a01 thru 0x011402 (0xfa02 bytes) 000797 1741692237.694704 read 0x001e6a thru 0x010705 (0xe89c bytes) 000798 1741692237.695109 mapread 0x02e4bf thru 0x035197 (0x6cd9 bytes) 000800 1741692237.695924 trunc from 0x035198 to 0x00f67a (0x25b1d bytes) 000801 1741692237.701100 punch from 0x00f0f7 to 0x00f67a (0x584 bytes) 000802 1741692237.702619 fallocate 0x007d25 thru 0x00c33d (0x4619 bytes) 000803 1741692237.706142 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000805 1741692237.770113 trunc from 0x00f67b to 0x00b38b (0x42ef bytes) 000806 1741692237.775031 mapread 0x001a23 thru 0x00b38b (0x9969 bytes) 000807 1741692237.781290 trunc from 0x00b38c to 0x016c37 (0xb8ac bytes) 000808 1741692237.785159 fallocate 0x0381a4 thru 0x03ffff (0x7e5c bytes) 000809 1741692237.787257 trunc from 0x040000 to 0x03528c (0xad73 bytes) 000810 1741692237.791446 mapwrite 0x0375b3 thru 0x03cba0 (0x55ee bytes) 000811 1741692237.804541 MIRROR_RESYNC 4 mirrors 000812 1741692237.942453 punch from 0x02e81c to 0x0388bb (0xa0a0 bytes) 000813 1741692237.950684 read 0x01dc6e thru 0x02250f (0x48a2 bytes) 000814 1741692237.952753 read 0x00ecd0 thru 0x01c2c7 (0xd5f8 bytes) 000815 1741692237.955722 trunc from 0x03cba1 to 0x016ffb (0x25ba5 bytes) 000816 1741692237.965069 MIRROR_RESYNC 4 mirrors 000818 1741692238.094758 trunc from 0x016ffc to 0x01e05a (0x705f bytes) 000820 1741692238.103736 fallocate 0x02053a thru 0x02c9ed (0xc4b4 bytes) 000821 1741692238.105498 fallocate 0x0374f2 thru 0x03ffff (0x8b0e bytes) 000822 1741692238.106984 mapread 0x009c5e thru 0x010c09 (0x6fac bytes) 000823 1741692238.112532 punch from 0x00d357 to 0x011be8 (0x4892 bytes) 000824 1741692238.114281 trunc from 0x02c9ee to 0x03823a (0xb84d bytes) 000825 1741692238.120601 punch from 0x01348a to 0x02083c (0xd3b3 bytes) 000826 1741692238.122143 mapread 0x017da6 thru 0x020cd0 (0x8f2b bytes) 000827 1741692238.125933 fallocate 0x007b9f thru 0x00fd19 (0x817b bytes) 000828 1741692238.127280 write 0x009ec8 thru 0x017bc8 (0xdd01 bytes) 000829 1741692238.128109 punch from 0x0176be to 0x01de8d (0x67d0 bytes) 000830 1741692238.131840 trunc from 0x03823b to 0x018be0 (0x1f65a bytes) 000831 1741692238.135581 trunc from 0x018be1 to 0x01dde4 (0x5204 bytes) 000833 1741692238.139375 fallocate 0x02193e thru 0x02cc40 (0xb303 bytes) 000834 1741692238.140905 write 0x033ebd thru 0x03948a (0x55ce bytes) 000835 1741692238.141553 mapwrite 0x011a2d thru 0x01bac7 (0xa09b bytes) 000836 1741692238.150439 mapread 0x02dd21 thru 0x033064 (0x5344 bytes) 000839 1741692238.153236 mapread 0x02b900 thru 0x032163 (0x6864 bytes) 000840 1741692238.153626 mapwrite 0x032348 thru 0x03d45d (0xb116 bytes) 000841 1741692238.164443 fallocate 0x01768b thru 0x022e13 (0xb789 bytes) 000842 1741692238.166334 trunc from 0x03d45e to 0x01c96b (0x20af2 bytes) 000843 1741692238.170374 trunc from 0x01c96c to 0x036589 (0x19c1e bytes) 000844 1741692238.174009 trunc from 0x03658a to 0x03c969 (0x63e0 bytes) 000845 1741692238.177569 mapwrite 0x00f302 thru 0x01515d (0x5e5c bytes) 000846 1741692238.183246 trunc from 0x03c96a to 0x01127c (0x2b6ed bytes) 000847 1741692238.187809 MIRROR_EXTEND to 5 mirrors 000848 1741692238.273645 write 0x01fea6 thru 0x023982 (0x3add bytes) 000850 1741692238.284309 MIRROR_EXTEND to 6 mirrors 000851 1741692238.394209 punch from 0x014986 to 0x01d974 (0x8fef bytes) 000852 1741692238.404397 write 0x02a513 thru 0x02afd2 (0xac0 bytes) 000853 1741692238.406161 punch from 0x016592 to 0x024308 (0xdd77 bytes) 000854 1741692238.407763 write 0x002819 thru 0x009ee9 (0x76d1 bytes) 000855 1741692238.410727 write 0x03afe5 thru 0x03ffff (0x501b bytes) 000856 1741692238.412918 MIRROR_EXTEND to 7 mirrors 000857 1741692238.561501 read 0x00e050 thru 0x00eeb1 (0xe62 bytes) 000858 1741692238.564623 trunc from 0x040000 to 0x007f2d (0x380d2 bytes) 000859 1741692238.578874 fallocate 0x030071 thru 0x038da7 (0x8d37 bytes) 000861 1741692238.580784 fallocate 0x01c99b thru 0x02af87 (0xe5ed bytes) 000862 1741692238.583583 MIRROR_EXTEND to 8 mirrors 000863 1741692238.689946 fallocate 0x016562 thru 0x024db4 (0xe853 bytes) 000864 1741692238.698315 punch from 0x01a79f to 0x029304 (0xeb66 bytes) 000865 1741692238.699602 mapwrite 0x017f4a thru 0x019323 (0x13da bytes) 000866 1741692238.709139 fallocate 0x024f2b thru 0x03257a (0xd650 bytes) 000867 1741692238.710673 write 0x03df5f thru 0x03ffff (0x20a1 bytes) 000869 1741692238.713115 mapwrite 0x0045aa thru 0x01025d (0xbcb4 bytes) 000870 1741692238.721556 MIRROR_EXTEND to 9 mirrors 000871 1741692238.836825 MIRROR_EXTEND to 10 mirrors 000872 1741692238.949314 mapread 0x00b5d9 thru 0x011dd5 (0x67fd bytes) 000873 1741692238.953309 read 0x004759 thru 0x007bf1 (0x3499 bytes) 000874 1741692238.954601 punch from 0x033eb9 to 0x03ffff (0xc147 bytes) 000875 1741692238.963101 fallocate 0x0071f9 thru 0x009bdd (0x29e5 bytes) 000876 1741692238.964498 fallocate 0x02a53b thru 0x038648 (0xe10e bytes) 000879 1741692238.966785 fallocate 0x00221b thru 0x00b85b (0x9641 bytes) 000880 1741692238.968432 trunc from 0x040000 to 0x01597f (0x2a680 bytes) 000881 1741692238.979555 MIRROR_RESYNC 10 mirrors 000883 1741692239.221723 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000884 1741692239.304244 punch from 0x00e646 to 0x00f9b6 (0x1371 bytes) 000885 1741692239.315180 mapread 0x01036a thru 0x01597f (0x5616 bytes) 000886 1741692239.320324 mapwrite 0x0125bd thru 0x01d901 (0xb345 bytes) 000887 1741692239.335076 fallocate 0x009e79 thru 0x0125fc (0x8784 bytes) 000888 1741692239.338050 MIRROR_RESYNC 9 mirrors 000889 1741692239.500343 trunc from 0x01d902 to 0x03605d (0x1875c bytes) 000890 1741692239.513070 trunc from 0x03605e to 0x028373 (0xdcea bytes) 000891 1741692239.517203 read 0x011170 thru 0x020d0a (0xfb9b bytes) 000892 1741692239.519979 trunc from 0x028374 to 0x0010a6 (0x272cd bytes) 000893 1741692239.524048 mapwrite 0x03e342 thru 0x03f634 (0x12f3 bytes) 000894 1741692239.535287 trunc from 0x03f635 to 0x02c48e (0x131a6 bytes) 000895 1741692239.540145 MIRROR_EXTEND to 10 mirrors 000896 1741692239.642052 mapwrite 0x02818c thru 0x02c78a (0x45ff bytes) 000897 1741692239.665767 punch from 0x028764 to 0x02c78a (0x4027 bytes) 000899 1741692239.667749 trunc from 0x02c78b to 0x03127f (0x4af5 bytes) 000900 1741692239.673887 mapread 0x028703 thru 0x028a88 (0x386 bytes) 000901 1741692239.676681 mapwrite 0x03f9a0 thru 0x03ffff (0x660 bytes) 000903 1741692239.686626 fallocate 0x033d4a thru 0x03ffff (0xc2b6 bytes) 000904 1741692239.688160 write 0x00a2bc thru 0x00d37b (0x30c0 bytes) 000905 1741692239.690581 trunc from 0x040000 to 0x03b90a (0x46f5 bytes) 000907 1741692239.694230 mapwrite 0x02a1ce thru 0x02c64a (0x247d bytes) 000908 1741692239.699031 trunc from 0x03b90b to 0x0112f8 (0x2a612 bytes) 000910 1741692239.702618 fallocate 0x0116db thru 0x017ba1 (0x64c7 bytes) 000911 1741692239.704041 fallocate 0x018166 thru 0x018a60 (0x8fb bytes) 000912 1741692239.705301 punch from 0x0122ea to 0x018a60 (0x6777 bytes) 000914 1741692239.708011 mapwrite 0x022c54 thru 0x025744 (0x2af1 bytes) 000915 1741692239.718687 trunc from 0x025745 to 0x0187fc (0xcf48 bytes) 000916 1741692239.722486 fallocate 0x02ccfd thru 0x03b73c (0xea40 bytes) 000917 1741692239.725518 MIRROR_EXTEND to 11 mirrors 000918 1741692239.836279 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000919 1741692239.887809 write 0x02174e thru 0x0263b4 (0x4c67 bytes) 000921 1741692239.897561 read 0x030f78 thru 0x03b73c (0xa7c5 bytes) 000924 1741692239.899965 trunc from 0x03b73d to 0x012d62 (0x289da bytes) 000925 1741692239.906617 write 0x0118b7 thru 0x01f5b6 (0xdd00 bytes) 000926 1741692239.908983 punch from 0x013a68 to 0x01d4f3 (0x9a8c bytes) 000927 1741692239.912191 mapwrite 0x015e0d thru 0x0250ab (0xf29f bytes) 000928 1741692239.926392 mapread 0x00ad33 thru 0x00f474 (0x4742 bytes) 000930 1741692239.927062 write 0x01bf1f thru 0x027f56 (0xc038 bytes) 000931 1741692239.927749 punch from 0x00911d to 0x009922 (0x806 bytes) 000932 1741692239.928959 punch from 0x0110c8 to 0x013673 (0x25ac bytes) 000933 1741692239.930310 punch from 0x0277c2 to 0x027f56 (0x795 bytes) 000934 1741692239.933435 mapwrite 0x03af89 thru 0x03d0d7 (0x214f bytes) 000935 1741692239.942954 mapread 0x00678a thru 0x00fb69 (0x93e0 bytes) 000936 1741692239.945406 read 0x017f0e thru 0x0204c5 (0x85b8 bytes) 000937 1741692239.945800 fallocate 0x006baa thru 0x009f86 (0x33dd bytes) 000939 1741692239.947680 trunc from 0x03d0d8 to 0x0206b3 (0x1ca24 bytes) 000940 1741692239.951057 trunc from 0x0206b4 to 0x03f317 (0x1ec64 bytes) 000941 1741692239.955350 MIRROR_RESYNC 10 mirrors 000942 1741692240.427582 mapwrite 0x011006 thru 0x01a8b2 (0x98ad bytes) 000943 1741692240.454456 punch from 0x021338 to 0x02f319 (0xdfe2 bytes) 000947 1741692240.456341 punch from 0x01231a to 0x01ee4b (0xcb32 bytes) 000948 1741692240.457708 mapwrite 0x034681 thru 0x03ffff (0xb97f bytes) 000949 1741692240.471091 MIRROR_EXTEND to 11 mirrors 000950 1741692240.588335 write 0x00790f thru 0x011aa2 (0xa194 bytes) 000951 1741692240.600794 punch from 0x02952d to 0x0323a0 (0x8e74 bytes) 000952 1741692240.602512 punch from 0x02bade to 0x032773 (0x6c96 bytes) 000953 1741692240.604079 write 0x013a5e thru 0x01a73d (0x6ce0 bytes) 000954 1741692240.608524 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000955 1741692240.661706 mapwrite 0x0045ab thru 0x0060cd (0x1b23 bytes) 000956 1741692240.673279 fallocate 0x03946c thru 0x03b3c8 (0x1f5d bytes) 000957 1741692240.675054 trunc from 0x040000 to 0x0334ed (0xcb12 bytes) 000958 1741692240.683164 mapread 0x009260 thru 0x012ed9 (0x9c7a bytes) 000959 1741692240.685530 punch from 0x01bfc2 to 0x023610 (0x764f bytes) 000960 1741692240.688006 fallocate 0x029b1a thru 0x033c76 (0xa15d bytes) 000961 1741692240.689709 write 0x001cad thru 0x01048e (0xe7e2 bytes) 000962 1741692240.690720 trunc from 0x0334ee to 0x0287a5 (0xad48 bytes) 000964 1741692240.694576 mapwrite 0x02c2a1 thru 0x0341b8 (0x7f18 bytes) 000965 1741692240.705839 punch from 0x01e022 to 0x026763 (0x8742 bytes) 000966 1741692240.707083 read 0x023263 thru 0x02a8a3 (0x7641 bytes) 000968 1741692240.709076 read 0x00cac2 thru 0x00d8e0 (0xe1f bytes) 000969 1741692240.709333 write 0x0110ce thru 0x01c99e (0xb8d1 bytes) 000970 1741692240.711544 mapwrite 0x03e980 thru 0x03ffff (0x1680 bytes) 000971 1741692240.723823 MIRROR_EXTEND to 11 mirrors 000972 1741692240.873064 mapwrite 0x02418b thru 0x02a709 (0x657f bytes) 000973 1741692240.917040 read 0x01a169 thru 0x022435 (0x82cd bytes) 000974 1741692240.918837 fallocate 0x03a762 thru 0x03ffff (0x589e bytes) 000976 1741692240.923399 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000977 1741692240.988776 fallocate 0x002483 thru 0x00a4ab (0x8029 bytes) 000978 1741692240.991045 fallocate 0x0019c2 thru 0x00f9b8 (0xdff7 bytes) 000980 1741692240.992979 mapread 0x038fef thru 0x03ffff (0x7011 bytes) 000981 1741692240.999526 trunc from 0x040000 to 0x0001ce (0x3fe31 bytes) 000983 1741692241.010532 mapwrite 0x03c814 thru 0x03ffff (0x37ec bytes) 000985 1741692241.025734 punch from 0x0346e9 to 0x03b954 (0x726c bytes) 000986 1741692241.028041 mapwrite 0x004f91 thru 0x00d18e (0x81fe bytes) 000987 1741692241.040948 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000988 1741692241.116182 mapwrite 0x02338f thru 0x03167b (0xe2ed bytes) 000990 1741692241.136349 write 0x021198 thru 0x02630f (0x5178 bytes) 000991 1741692241.139417 punch from 0x033144 to 0x03dbc1 (0xaa7e bytes) 000992 1741692241.142193 punch from 0x02d0f3 to 0x033a3d (0x694b bytes) 000993 1741692241.144683 mapwrite 0x00c936 thru 0x016488 (0x9b53 bytes) 000994 1741692241.160609 punch from 0x036410 to 0x03ffff (0x9bf0 bytes) 000995 1741692241.164693 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000996 1741692241.243636 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbf3:0x0]: Invalid argument (22) 000997 1741692241.322723 write 0x030d17 thru 0x037782 (0x6a6c bytes) 000999 1741692241.332423 read 0x02089e thru 0x02e8a7 (0xe00a bytes) 001000 1741692241.337188 mapwrite 0x03245e thru 0x03eb88 (0xc72b bytes) All operations completed A-OK! PASS 70a (16s) == sanity-flr test 71: check mirror extend parent fid ==== 07:24:02 (1741692242) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000404:0x1e:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (3s) == sanity-flr test 200a: stress test ===================== 07:24:05 (1741692245) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre2 Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6048992 .. Extending file size to 7270752 .. done Extending file size to 6193728 .. Extending file size to 8060704 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7872288 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8135040 .. Extending file size to 8404672 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8668064 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8887232 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9126976 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8679328 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8969824 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9417984 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9071808 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9094528 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 4942 4943 4944 4946 4949 failed 192.168.203.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg316-client.virtnet /mnt/lustre2 (opts:) 192.168.203.116@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg316-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 07:25:09 (1741692309) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre2 Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3924224 .. Extending file size to 3759328 .. Extending file size to 7052192 .. done Extending file size to 5350112 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8372544 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 8681152 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8795200 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9381216 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8987008 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9233632 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 9538 9539 9540 9542 9545 192.168.203.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg316-client.virtnet /mnt/lustre2 (opts:) 192.168.203.116@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg316-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 07:26:13 (1741692373) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4312 1283376 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1924 1285764 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18008 3589012 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 20060 3586960 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38068 7175972 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4312 1283376 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1924 1285764 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18008 3589012 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 20060 3586960 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 38068 7175972 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0583021 s, 36.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0782393 s, 26.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg316-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 07:26:18 (1741692378) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0263191 s, 79.7 MB/s PASS 202 (1s) == sanity-flr test 203: mirror file preserve mirror ID === 07:26:20 (1741692380) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xc46:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b0:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b0:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b1:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b1:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 07:26:21 (1741692381) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0195921 s, 107 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 07:26:23 (1741692383) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0244007 s, 85.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b3:0x0] } PASS 204b (1s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 07:26:24 (1741692384) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0208126 s, 101 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b5:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 07:26:48 (1741692408) Waiting for MDT destroys to complete oleg316-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg316-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg316-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0119422 s, 87.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x30:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x30:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2c:0x0] } PASS 204d (31s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 07:27:19 (1741692439) Waiting for MDT destroys to complete oleg316-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg316-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg316-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0180799 s, 116 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (32s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 07:27:51 (1741692471) Waiting for MDT destroys to complete oleg316-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg316-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg316-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg316-server: Pool lustre.test_204f created Adding targets to pool oleg316-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg316-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0312826 s, 67.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg316-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg316-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg316-server: Pool lustre.test_204f destroyed PASS 204f (44s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 07:28:35 (1741692515) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 07:28:37 (1741692517) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 07:28:39 (1741692519) oleg316-server: Pool lustre.test_206 created Adding targets to pool oleg316-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg316-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg316-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg316-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg316-server: Pool lustre.test_206 destroyed PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 07:28:52 (1741692532) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0316565 s, 33.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0130324 s, 80.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x8b8d3e13 Mirror 3: 0x8b8d3e13 PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 07:28:55 (1741692535) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 07:28:56 (1741692536) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 07:28:57 (1741692537) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0128757 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1610000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1b80000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1b80000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1b80000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xe00000 Read this (4 bytes): BBBB 192.168.203.116@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg316-client.virtnet /mnt/lustre2 (opts:) PASS 209a (15s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 07:29:13 (1741692553) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00210989 s, 3.9 MB/s the very first read read with warm cache PASS 209b (1s) == sanity-flr test 210a: handle broken mirrored lovea ==== 07:29:14 (1741692554) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0148993 s, 70.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc67 lmm_fid: [0x200000405:0xc67:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc67 lmm_fid: [0x200000405:0xc67:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c2:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339614311 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-03-11 07:29:15.000000000 -0400 Modify: 2025-03-11 07:29:15.000000000 -0400 Change: 2025-03-11 07:29:15.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 07:29:16 (1741692556) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00873261 s, 120 MB/s oleg316-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg316-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg316-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg316-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 oleg316-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg316-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg316-server: O/280000401/d1/10c1: File not found by ext2_lookup PASS 210b (14s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 07:29:30 (1741692570) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0220306 s, 1.9 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00418102 s, 980 kB/s size after second write -rw-r--r-- 1 root root 4096 Mar 11 07:29 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xc6c:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Mar 11 07:29 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 693 sec ============ 07:29:32 (1741692572) === sanity-flr: start cleanup 07:29:32 (1741692572) === === sanity-flr: finish cleanup 07:29:34 (1741692574) ===