-----============= acceptance-small: sanity-flr ============----- Mon Mar 31 16:51:29 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg134-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 16:51:33 (1743454293) === oleg134-client.virtnet: executing check_config_client /mnt/lustre oleg134-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg134-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b5b9d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b5b9d800.idle_timeout=debug disable quota as required oleg134-server: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 16:51:42 (1743454302) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7642] == sanity-flr test 0a: lfs mirror create with -N option == 16:51:42 (1743454302) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 16:51:45 (1743454305) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 16:51:46 (1743454306) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 16:51:47 (1743454307) PASS 0d (4s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 16:51:51 (1743454311) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (0s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 16:51:52 (1743454312) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (0s) == sanity-flr test 0g: lfs mirror create flags support === 16:51:52 (1743454312) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b5b9d800.stats=clear osc.lustre-OST0001-osc-ffff8800b5b9d800.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 16:51:54 (1743454314) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 16:51:56 (1743454316) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 16:51:58 (1743454318) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (0s) == sanity-flr test 1: create components with setstripe options ========================================================== 16:51:58 (1743454318) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 16:52:00 (1743454320) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 16:52:02 (1743454322) PASS 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 16:52:04 (1743454324) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td lfs extend: cannot get the last init comp error: lfs extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (1s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 16:52:05 (1743454325) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 16:52:08 (1743454328) PASS 7 (1s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 16:52:09 (1743454329) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB) copied, 0.122571 s, 59.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0204428 s, 51.3 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 16:52:11 (1743454331) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0161423 s, 130 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 16:52:13 (1743454333) Stopping /mnt/lustre-ost1 (opts:) on oleg134-server oleg134-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5b9d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5b9d800.ost_server_uuid in DISCONN state after 1 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg134-server: oleg134-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg134-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg134-server oleg134-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5b9d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5b9d800.ost_server_uuid in DISCONN state after 2 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg134-server: oleg134-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg134-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (24s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 16:52:37 (1743454357) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB) copied, 0.109524 s, 57.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg134-server oleg134-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5b9d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5b9d800.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg134-server: oleg134-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg134-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (9s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 16:52:46 (1743454366) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg134-server oleg134-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5b9d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5b9d800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg134-server: oleg134-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg134-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg134-server oleg134-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5b9d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5b9d800.ost_server_uuid in DISCONN state after 4 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg134-server: oleg134-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg134-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (20s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 16:53:07 (1743454387) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.813009 s, 64.5 MB/s -rw-r--r-- 1 root root 52428800 Mar 31 16:53 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.232s user 0m0.000s sys 0m0.112s Stopping /mnt/lustre-ost1 (opts:) on oleg134-server oleg134-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5b9d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5b9d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.216s user 0m0.000s sys 0m0.105s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg134-server: oleg134-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg134-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg134-server oleg134-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5b9d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5b9d800.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost1)... real 0m0.258s user 0m0.000s sys 0m0.114s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg134-server: oleg134-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg134-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg134-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b5b9d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5b9d800.ost_server_uuid in FULL state after 0 sec PASS 33b (21s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 16:53:28 (1743454408) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 16:53:29 (1743454409) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 16:53:30 (1743454410) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (0s) == sanity-flr test 35: allow to write to mirrored files == 16:53:31 (1743454411) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 16:53:32 (1743454412) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.119439 s, 105 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (2s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 16:53:34 (1743454414) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00780571 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00857368 s, 122 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 16:53:36 (1743454416) PASS 36c (2s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 16:53:38 (1743454418) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000445109 s, 2.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000440695 s, 2.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000446554 s, 2.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000502309 s, 2.0 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 16:53:40 (1743454420) create test files with size 15737613 .. 52e82bb4dfb3b75e3965cd496d507144 - 62a8eedd7a298fc638f1cb9fd9fc84b1 - a81ccafb9b80ae257b8084fc339e513c - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg134-client.virtnet /mnt/lustre (opts:) Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre PASS 37 (4s) == sanity-flr test 38: resync ============================ 16:53:44 (1743454424) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (2s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 16:53:46 (1743454426) PASS 39 (1s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 16:53:47 (1743454427) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0177899 s, 58.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00769409 s, 136 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x41:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 16:53:49 (1743454429) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0191789 s, 109 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0756475 s, 55.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0356422 s, 58.8 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) == sanity-flr test 42: lfs mirror verify ================= 16:53:51 (1743454431) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 16:53:52 (1743454432) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 16:53:53 (1743454433) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00877134 s, 120 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (1s) == sanity-flr test 44a: lfs mirror split check =========== 16:53:54 (1743454434) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0721051 s, 43.6 MB/s PASS 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 16:53:56 (1743454436) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4a:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x48:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x48:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 16:53:58 (1743454438) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.064873 s, 162 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg134-client.virtnet /mnt/lustre (opts:) Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (23s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 16:54:21 (1743454461) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800b3b7d800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0718157 s, 146 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg134-client.virtnet /mnt/lustre (opts:) Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff88012e125800.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 16:54:25 (1743454465) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (6s) == sanity-flr test 46: Verify setstripe --copy option ==== 16:54:31 (1743454471) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (7s) == sanity-flr test 47: Verify mirror obj alloc =========== 16:54:38 (1743454478) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 16:54:39 (1743454479) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.079876 s, 39.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.053098 s, 59.2 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 23f0d118f3ded36079d8f5c83b8a5af2 - mirror 1 checksum: 34e6bc15552211e124faf2d9aab2ed1e - mirror 2 checksum: 23f0d118f3ded36079d8f5c83b8a5af2 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Mar 31 16:54:39 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Mar 31 16:54:39 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 23f0d118f3ded36079d8f5c83b8a5af2 - mirror 1 checksum: 34e6bc15552211e124faf2d9aab2ed1e - mirror 2 checksum: 34e6bc15552211e124faf2d9aab2ed1e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Mar 31 16:54:39 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Mar 31 16:54:39 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } PASS 48 (2s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 16:54:41 (1743454481) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109533 s, 95.7 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 529408.. 530431: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 16:54:43 (1743454483) Failing mds1 on oleg134-server Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 16:54:49 (1743454489) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover mds1 to oleg134-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 16:55:02 (1743454502) targets are mounted 16:55:02 (1743454502) facet_failover done oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00849297 s, 123 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } PASS 50A (26s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 16:55:09 (1743454509) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00961227 s, 2.1 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00475034 s, 4.3 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 16:55:12 (1743454512) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0101245 s, 2.0 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00291919 s, 7.0 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 16:55:14 (1743454514) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00197453 s, 8.3 MB/s PASS 50c (1s) == sanity-flr test 50d: mirror rsync keep holes ========== 16:55:15 (1743454515) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00942324 s, 2.2 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 16:55:17 (1743454517) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00163091 s, 2.5 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00636703 s, 643 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 16:55:19 (1743454519) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (4s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 16:55:23 (1743454523) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 16:55:41 (1743454541) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 16:55:58 (1743454558) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 16:56:15 (1743454575) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 16:56:17 (1743454577) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 26297 mirror split pid 26298 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 26297 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 26298 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (61s) == sanity-flr test 70a: flr mode fsx test ================ 16:57:18 (1743454638) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 4639 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported 000002 1743454639.320751 write 0x01c03c thru 0x02788b (0xb850 bytes) fallocating to largest ever: 0x17f71 000003 1743454639.322533 fallocate 0x010af5 thru 0x017f70 (0x747c bytes) truncating to largest ever: 0x37129 000004 1743454639.324001 trunc from 0x02788c to 0x037128 (0xf89d bytes) truncating to largest ever: 0x3ca23 000006 1743454639.328638 trunc from 0x037129 to 0x03ca22 (0x58fa bytes) 000009 1743454639.333286 write 0x016082 thru 0x020a02 (0xa981 bytes) fallocating to largest ever: 0x40000 000010 1743454639.335309 fallocate 0x03ba56 thru 0x03ffff (0x45aa bytes) 000013 1743454639.337396 trunc from 0x03ca23 to 0x016cc6 (0x25d5c bytes) 000015 1743454639.342776 MIRROR_EXTEND to 2 mirrors 000016 1743454639.429019 MIRROR_RESYNC 2 mirrors 000017 1743454639.461426 read 0x006d77 thru 0x00d2a5 (0x652f bytes) 000018 1743454639.463347 mapread 0x00d60f thru 0x016cc6 (0x96b8 bytes) 000019 1743454639.467205 mapread 0x010b7f thru 0x016cc6 (0x6148 bytes) 000020 1743454639.467548 fallocate 0x010c3e thru 0x01e2a8 (0xd66b bytes) 000021 1743454639.476073 mapwrite 0x00c5d6 thru 0x012774 (0x619f bytes) 000022 1743454639.484113 punch from 0x00843a to 0x017297 (0xee5e bytes) 000024 1743454639.486525 mapread 0x0183c2 thru 0x01e2a8 (0x5ee7 bytes) 000025 1743454639.492137 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000026 1743454639.532984 mapwrite 0x01df5f thru 0x01f2f9 (0x139b bytes) 000029 1743454639.548872 fallocate 0x0049c1 thru 0x00cdd2 (0x8412 bytes) 000030 1743454639.550025 mapwrite 0x013ac5 thru 0x01913a (0x5676 bytes) 000036 1743454639.559191 write 0x00cf1e thru 0x00f825 (0x2908 bytes) 000037 1743454639.559835 punch from 0x01caf2 to 0x01e936 (0x1e45 bytes) 000038 1743454639.561487 write 0x02e933 thru 0x038f30 (0xa5fe bytes) 000041 1743454639.564122 write 0x00f94b thru 0x010c25 (0x12db bytes) 000044 1743454639.565244 mapwrite 0x02c145 thru 0x0340b9 (0x7f75 bytes) 000045 1743454639.571920 mapread 0x024034 thru 0x031fcd (0xdf9a bytes) 000046 1743454639.575712 fallocate 0x02ad6f thru 0x02c21a (0x14ac bytes) 000047 1743454639.577139 fallocate 0x024570 thru 0x02f3d9 (0xae6a bytes) 000048 1743454639.580165 fallocate 0x03d4c2 thru 0x03ffff (0x2b3e bytes) 000050 1743454639.582514 mapwrite 0x02cce2 thru 0x0317b3 (0x4ad2 bytes) 000051 1743454639.587384 punch from 0x01013e to 0x01f722 (0xf5e5 bytes) 000052 1743454639.590662 MIRROR_EXTEND to 2 mirrors 000053 1743454639.706120 trunc from 0x040000 to 0x00706b (0x38f94 bytes) 000054 1743454639.722108 fallocate 0x0077fe thru 0x012039 (0xa83c bytes) 000055 1743454639.724576 MIRROR_RESYNC 2 mirrors 000058 1743454639.801755 write 0x028e64 thru 0x038437 (0xf5d4 bytes) 000059 1743454639.808647 mapwrite 0x006687 thru 0x009d0f (0x3689 bytes) 000060 1743454639.818592 mapread 0x02f159 thru 0x034dc0 (0x5c68 bytes) 000061 1743454639.819117 punch from 0x0175e1 to 0x01c631 (0x5051 bytes) 000062 1743454639.820823 mapwrite 0x025cb0 thru 0x032e01 (0xd152 bytes) 000064 1743454639.828703 trunc from 0x038438 to 0x00105c (0x373db bytes) 000065 1743454639.837830 mapread 0x00074f thru 0x00105c (0x90e bytes) 000066 1743454639.839918 trunc from 0x00105d to 0x02598b (0x2492f bytes) 000067 1743454639.844448 MIRROR_RESYNC 2 mirrors 000068 1743454639.932895 mapread 0x01e6fd thru 0x02598b (0x728f bytes) 000069 1743454639.938207 mapwrite 0x028887 thru 0x0318b3 (0x902d bytes) 000071 1743454639.962472 mapwrite 0x0019c9 thru 0x008529 (0x6b61 bytes) 000072 1743454639.973576 mapread 0x01c892 thru 0x01e128 (0x1897 bytes) 000073 1743454639.975769 punch from 0x006dfb to 0x011c29 (0xae2f bytes) 000074 1743454639.977340 punch from 0x0207a2 to 0x025f3c (0x579b bytes) 000076 1743454639.979167 write 0x012345 thru 0x01a65f (0x831b bytes) 000077 1743454639.982831 punch from 0x025e31 to 0x028e1d (0x2fed bytes) 000078 1743454639.984341 read 0x027f57 thru 0x0318b3 (0x995d bytes) 000079 1743454639.986170 mapread 0x01c92d thru 0x02452e (0x7c02 bytes) 000080 1743454639.988332 punch from 0x009629 to 0x013e76 (0xa84e bytes) 000081 1743454639.991559 punch from 0x01f289 to 0x026ead (0x7c25 bytes) 000083 1743454639.993054 trunc from 0x0318b4 to 0x03127e (0x635 bytes) 000084 1743454639.997897 mapread 0x006be8 thru 0x012943 (0xbd5c bytes) 000085 1743454640.002820 write 0x00976e thru 0x00c579 (0x2e0c bytes) 000086 1743454640.003495 punch from 0x02b8b5 to 0x03127e (0x59ca bytes) 000088 1743454640.006249 trunc from 0x03127f to 0x020a14 (0x1086a bytes) 000089 1743454640.010417 punch from 0x008af1 to 0x01448f (0xb99f bytes) 000090 1743454640.014401 mapread 0x01f10f thru 0x020a14 (0x1906 bytes) 000091 1743454640.017532 punch from 0x00691a to 0x00a750 (0x3e37 bytes) 000092 1743454640.019384 write 0x029cbc thru 0x031816 (0x7b5b bytes) 000093 1743454640.020297 mapread 0x01e203 thru 0x025144 (0x6f42 bytes) 000094 1743454640.022944 mapread 0x02789e thru 0x02aef5 (0x3658 bytes) 000095 1743454640.023402 trunc from 0x031817 to 0x00340b (0x2e40b bytes) 000097 1743454640.027606 read 0x00203f thru 0x00340b (0x13cd bytes) 000098 1743454640.027958 read 0x002e87 thru 0x00340b (0x585 bytes) 000099 1743454640.029241 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000101 1743454640.060627 write 0x01cf70 thru 0x026f77 (0xa008 bytes) 000103 1743454640.062769 fallocate 0x022ecf thru 0x02cec2 (0x9ff4 bytes) 000104 1743454640.064629 trunc from 0x02cec3 to 0x008169 (0x24d59 bytes) 000105 1743454640.068746 punch from 0x002e3e to 0x008169 (0x532c bytes) 000107 1743454640.070238 fallocate 0x00b879 thru 0x0106c1 (0x4e49 bytes) 000108 1743454640.071459 write 0x011ee9 thru 0x017094 (0x51ac bytes) 000110 1743454640.072412 trunc from 0x017095 to 0x03725e (0x201ca bytes) 000111 1743454640.076119 fallocate 0x03dc57 thru 0x03ffff (0x23a9 bytes) 000114 1743454640.078628 fallocate 0x022479 thru 0x02b41c (0x8fa4 bytes) 000115 1743454640.080030 punch from 0x023af7 to 0x027489 (0x3993 bytes) 000116 1743454640.082122 fallocate 0x02ef81 thru 0x03b61d (0xc69d bytes) 000117 1743454640.084713 MIRROR_EXTEND to 2 mirrors 000119 1743454640.235736 punch from 0x027d25 to 0x02976a (0x1a46 bytes) 000120 1743454640.245816 read 0x025ffc thru 0x0330bd (0xd0c2 bytes) 000121 1743454640.250695 MIRROR_EXTEND to 3 mirrors 000122 1743454640.381337 read 0x039b56 thru 0x03ffff (0x64aa bytes) 000123 1743454640.385358 mapwrite 0x02fc0f thru 0x039e19 (0xa20b bytes) 000124 1743454640.424316 punch from 0x027f64 to 0x03741f (0xf4bc bytes) 000126 1743454640.426811 mapread 0x032daf thru 0x03f113 (0xc365 bytes) 000127 1743454640.429534 mapread 0x039fe5 thru 0x03ffff (0x601b bytes) 000128 1743454640.430300 punch from 0x011c5b to 0x011e2f (0x1d5 bytes) 000129 1743454640.433125 fallocate 0x00bd70 thru 0x010be1 (0x4e72 bytes) 000130 1743454640.434805 read 0x003e07 thru 0x00a316 (0x6510 bytes) 000131 1743454640.439185 fallocate 0x026864 thru 0x033250 (0xc9ed bytes) 000132 1743454640.441147 punch from 0x027f1c to 0x0301f8 (0x82dd bytes) 000133 1743454640.442624 trunc from 0x040000 to 0x03cfe2 (0x301d bytes) 000134 1743454640.458346 write 0x028c82 thru 0x031b70 (0x8eef bytes) 000135 1743454640.460793 write 0x01989a thru 0x01af0e (0x1675 bytes) 000136 1743454640.461243 read 0x00688a thru 0x01218d (0xb904 bytes) 000137 1743454640.461848 write 0x0032cf thru 0x010f7e (0xdcb0 bytes) 000138 1743454640.462830 trunc from 0x03cfe3 to 0x03085d (0xc785 bytes) 000139 1743454640.467195 read 0x028ed8 thru 0x03085d (0x7986 bytes) 000140 1743454640.467624 fallocate 0x033f4c thru 0x037f45 (0x3ffa bytes) 000141 1743454640.469557 mapwrite 0x006ed4 thru 0x011b7a (0xaca7 bytes) 000143 1743454640.476987 write 0x013f73 thru 0x019262 (0x52f0 bytes) 000145 1743454640.479179 MIRROR_RESYNC 3 mirrors 000146 1743454640.634014 punch from 0x014d26 to 0x01bc64 (0x6f3f bytes) 000147 1743454640.642655 write 0x004af9 thru 0x00c902 (0x7e0a bytes) 000148 1743454640.645349 mapread 0x004477 thru 0x00f96b (0xb4f5 bytes) 000149 1743454640.648778 MIRROR_EXTEND to 4 mirrors 000150 1743454640.758425 MIRROR_RESYNC 4 mirrors 000152 1743454640.942497 read 0x005dd3 thru 0x01061a (0xa848 bytes) 000153 1743454640.947418 punch from 0x02d9f5 to 0x03085d (0x2e69 bytes) 000154 1743454640.957297 mapwrite 0x032236 thru 0x036d95 (0x4b60 bytes) 000155 1743454640.981227 mapread 0x01a74c thru 0x027b8c (0xd441 bytes) 000157 1743454640.988293 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000158 1743454641.042504 mapwrite 0x0239ec thru 0x025124 (0x1739 bytes) 000159 1743454641.053060 read 0x008ce8 thru 0x013769 (0xaa82 bytes) 000160 1743454641.056842 mapread 0x015f4c thru 0x022ccd (0xcd82 bytes) 000161 1743454641.060763 punch from 0x0310fe to 0x034c0d (0x3b10 bytes) 000162 1743454641.062338 mapread 0x0329f1 thru 0x036d95 (0x43a5 bytes) 000164 1743454641.064852 read 0x00487e thru 0x00e09e (0x9821 bytes) 000165 1743454641.068054 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000168 1743454641.117390 read 0x00dd0b thru 0x01d5c4 (0xf8ba bytes) 000169 1743454641.120062 fallocate 0x004cbe thru 0x00c8b2 (0x7bf5 bytes) 000171 1743454641.123385 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000172 1743454641.152350 mapread 0x024900 thru 0x0280f4 (0x37f5 bytes) 000173 1743454641.156445 trunc from 0x036d96 to 0x03b90e (0x4b79 bytes) 000174 1743454641.161610 trunc from 0x03b90f to 0x020e1d (0x1aaf1 bytes) 000175 1743454641.165342 write 0x011152 thru 0x015f91 (0x4e40 bytes) 000176 1743454641.167732 read 0x012dbf thru 0x0164e8 (0x372a bytes) 000177 1743454641.168134 mapread 0x00b137 thru 0x00fecd (0x4d97 bytes) 000180 1743454641.169380 write 0x02cb26 thru 0x0392d7 (0xc7b2 bytes) 000182 1743454641.170974 fallocate 0x013004 thru 0x01400d (0x100a bytes) 000183 1743454641.172453 fallocate 0x032bc9 thru 0x03f915 (0xcd4d bytes) 000186 1743454641.174731 fallocate 0x03e497 thru 0x03ffff (0x1b69 bytes) 000189 1743454641.177092 mapread 0x01b65d thru 0x01eddb (0x377f bytes) 000191 1743454641.180295 punch from 0x02fcd0 to 0x031f10 (0x2241 bytes) 000192 1743454641.184261 mapwrite 0x032228 thru 0x03d771 (0xb54a bytes) 000193 1743454641.194690 punch from 0x038d7a to 0x03d870 (0x4af7 bytes) 000194 1743454641.196661 trunc from 0x03f916 to 0x00f116 (0x307ff bytes) 000196 1743454641.201667 read 0x004145 thru 0x00f116 (0xafd2 bytes) 000197 1743454641.204091 read 0x00978b thru 0x00b799 (0x200f bytes) 000198 1743454641.204505 fallocate 0x01f324 thru 0x0295af (0xa28c bytes) 000199 1743454641.206394 mapread 0x004e59 thru 0x00e59c (0x9744 bytes) 000201 1743454641.207378 mapread 0x00191b thru 0x009701 (0x7de7 bytes) 000202 1743454641.207855 mapread 0x00a65d thru 0x00f116 (0x4aba bytes) 000203 1743454641.208320 mapwrite 0x01cf75 thru 0x01ddea (0xe76 bytes) 000204 1743454641.219467 mapread 0x00f928 thru 0x011dd2 (0x24ab bytes) 000205 1743454641.219985 trunc from 0x01ddeb to 0x03cfbb (0x1f1d1 bytes) 000206 1743454641.224072 fallocate 0x02809e thru 0x03184f (0x97b2 bytes) 000208 1743454641.225701 write 0x00d175 thru 0x015132 (0x7fbe bytes) 000209 1743454641.226405 write 0x001797 thru 0x0106ef (0xef59 bytes) 000210 1743454641.227108 mapwrite 0x01086b thru 0x016a91 (0x6227 bytes) 000213 1743454641.234954 mapwrite 0x007f8d thru 0x013b68 (0xbbdc bytes) 000214 1743454641.244593 punch from 0x004713 to 0x00cf0a (0x87f8 bytes) 000215 1743454641.246839 mapread 0x02b191 thru 0x034359 (0x91c9 bytes) 000216 1743454641.249648 read 0x03b447 thru 0x03cfbb (0x1b75 bytes) 000217 1743454641.249988 punch from 0x024dec to 0x02de86 (0x909b bytes) 000218 1743454641.251525 mapread 0x02bd72 thru 0x03afd7 (0xf266 bytes) 000219 1743454641.255156 write 0x0355f6 thru 0x03ffff (0xaa0a bytes) 000220 1743454641.256020 mapread 0x01edf4 thru 0x0281c2 (0x93cf bytes) 000221 1743454641.256572 mapread 0x034b18 thru 0x03ffff (0xb4e8 bytes) 000222 1743454641.257315 mapread 0x02f2be thru 0x0382f0 (0x9033 bytes) 000223 1743454641.257827 trunc from 0x040000 to 0x004c1f (0x3b3e0 bytes) 000224 1743454641.262646 fallocate 0x0056e0 thru 0x00ac5a (0x557b bytes) 000225 1743454641.265709 MIRROR_EXTEND to 2 mirrors 000227 1743454641.354888 punch from 0x000656 to 0x004742 (0x40ed bytes) 000229 1743454641.365514 write 0x01e566 thru 0x02a14a (0xbbe5 bytes) 000231 1743454641.368355 write 0x00b7a4 thru 0x0125a9 (0x6e06 bytes) 000232 1743454641.370671 mapwrite 0x0276c3 thru 0x032f87 (0xb8c5 bytes) 000233 1743454641.390329 trunc from 0x032f88 to 0x026883 (0xc704 bytes) 000234 1743454641.394786 write 0x030f96 thru 0x03e006 (0xd071 bytes) 000235 1743454641.396193 punch from 0x032c9c to 0x03e006 (0xb36b bytes) 000236 1743454641.401172 MIRROR_RESYNC 2 mirrors 000237 1743454641.546314 fallocate 0x007d94 thru 0x016418 (0xe685 bytes) 000239 1743454641.554744 mapwrite 0x02999f thru 0x02e972 (0x4fd4 bytes) 000241 1743454641.568659 MIRROR_RESYNC 2 mirrors 000242 1743454641.704168 read 0x00a80f thru 0x011fdd (0x77cf bytes) 000243 1743454641.706784 mapwrite 0x033c99 thru 0x03bbba (0x7f22 bytes) 000244 1743454641.737404 write 0x026bcc thru 0x02ad2e (0x4163 bytes) 000245 1743454641.740088 trunc from 0x03e007 to 0x004131 (0x39ed5 bytes) 000246 1743454641.751143 mapread 0x0036d6 thru 0x004131 (0xa5c bytes) 000247 1743454641.754735 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000249 1743454641.793302 read 0x003837 thru 0x004131 (0x8fb bytes) 000251 1743454641.795896 write 0x00093c thru 0x004d8a (0x444f bytes) 000255 1743454641.803239 read 0x00012f thru 0x002864 (0x2736 bytes) 000256 1743454641.803567 trunc from 0x004d8b to 0x0354fa (0x30770 bytes) 000258 1743454641.807534 mapread 0x03121c thru 0x0354fa (0x42df bytes) 000259 1743454641.810946 punch from 0x014fb7 to 0x01fa7a (0xaac4 bytes) 000260 1743454641.813171 fallocate 0x0312fc thru 0x0321d9 (0xede bytes) 000261 1743454641.814718 trunc from 0x0354fb to 0x01bd3b (0x197bf bytes) 000262 1743454641.821648 trunc from 0x01bd3c to 0x037ec8 (0x1c18d bytes) 000263 1743454641.825705 fallocate 0x013d45 thru 0x01f647 (0xb903 bytes) 000264 1743454641.827509 mapwrite 0x003e05 thru 0x00f07b (0xb277 bytes) 000265 1743454641.835127 mapread 0x00e1d8 thru 0x00f2b9 (0x10e2 bytes) 000266 1743454641.835634 fallocate 0x030173 thru 0x03c540 (0xc3ce bytes) 000267 1743454641.837158 trunc from 0x037ec9 to 0x036dbc (0x110c bytes) 000268 1743454641.840575 mapwrite 0x006d2a thru 0x00d97a (0x6c51 bytes) 000270 1743454641.847663 mapread 0x02a937 thru 0x02e6f3 (0x3dbd bytes) 000271 1743454641.854883 fallocate 0x03258d thru 0x03f567 (0xcfdb bytes) 000273 1743454641.858158 punch from 0x0166a5 to 0x01e5e3 (0x7f3f bytes) 000274 1743454641.859708 mapwrite 0x032a04 thru 0x03627c (0x3879 bytes) 000276 1743454641.871600 trunc from 0x03f568 to 0x026f07 (0x18660 bytes) 000277 1743454641.876634 punch from 0x00bc77 to 0x013a58 (0x7de2 bytes) 000278 1743454641.878061 punch from 0x017402 to 0x0195d4 (0x21d3 bytes) 000279 1743454641.879189 read 0x00c64e thru 0x00d706 (0x10b9 bytes) 000281 1743454641.882334 read 0x0177e9 thru 0x0268fa (0xf112 bytes) 000282 1743454641.882962 read 0x003dec thru 0x00c191 (0x83a6 bytes) 000283 1743454641.883352 read 0x00397a thru 0x0071d6 (0x385d bytes) 000284 1743454641.884933 MIRROR_EXTEND to 2 mirrors 000285 1743454642.017514 fallocate 0x012a0b thru 0x013beb (0x11e1 bytes) 000286 1743454642.025777 punch from 0x00fc09 to 0x0171df (0x75d7 bytes) 000287 1743454642.026937 read 0x00a662 thru 0x015832 (0xb1d1 bytes) 000288 1743454642.029284 mapwrite 0x016eaf thru 0x019a7d (0x2bcf bytes) 000289 1743454642.041746 fallocate 0x018bc1 thru 0x0250ea (0xc52a bytes) 000290 1743454642.043181 write 0x00a990 thru 0x0198f2 (0xef63 bytes) 000291 1743454642.046150 MIRROR_EXTEND to 3 mirrors 000292 1743454642.155564 punch from 0x01f5d3 to 0x026aaa (0x74d8 bytes) 000293 1743454642.166697 trunc from 0x026f08 to 0x016fd3 (0xff34 bytes) 000295 1743454642.177949 fallocate 0x01bf14 thru 0x01e74f (0x283c bytes) 000296 1743454642.181276 MIRROR_RESYNC 3 mirrors 000297 1743454642.303335 MIRROR_EXTEND to 4 mirrors 000298 1743454642.420208 mapwrite 0x0345cd thru 0x03ffff (0xba33 bytes) 000299 1743454642.448050 read 0x016448 thru 0x01b11c (0x4cd5 bytes) 000301 1743454642.450875 write 0x02300c thru 0x028fa8 (0x5f9d bytes) 000302 1743454642.453543 write 0x033df9 thru 0x037690 (0x3898 bytes) 000303 1743454642.454173 punch from 0x03b20f to 0x03ffff (0x4df1 bytes) 000304 1743454642.455717 read 0x032e5f thru 0x038d77 (0x5f19 bytes) 000305 1743454642.456052 punch from 0x001365 to 0x008ca1 (0x793d bytes) 000307 1743454642.459318 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000308 1743454642.515853 mapwrite 0x028281 thru 0x0341c4 (0xbf44 bytes) 000310 1743454642.532784 mapread 0x0004df thru 0x00f390 (0xeeb2 bytes) 000311 1743454642.538259 write 0x003652 thru 0x00b763 (0x8112 bytes) 000312 1743454642.541340 write 0x01cc7e thru 0x024450 (0x77d3 bytes) 000313 1743454642.544205 mapread 0x018577 thru 0x0211b6 (0x8c40 bytes) 000314 1743454642.544945 punch from 0x0366bb to 0x03ffff (0x9945 bytes) 000315 1743454642.547120 read 0x013509 thru 0x019783 (0x627b bytes) 000317 1743454642.547791 write 0x0123d6 thru 0x01e15d (0xbd88 bytes) 000318 1743454642.548600 mapwrite 0x01d78d thru 0x01e1b9 (0xa2d bytes) 000320 1743454642.553858 mapwrite 0x009469 thru 0x010781 (0x7319 bytes) 000321 1743454642.561575 read 0x01d2c6 thru 0x027ae0 (0xa81b bytes) 000322 1743454642.562024 read 0x020241 thru 0x023aa1 (0x3861 bytes) 000323 1743454642.562405 trunc from 0x040000 to 0x03a3ab (0x5c54 bytes) 000324 1743454642.569107 trunc from 0x03a3ac to 0x00c5ac (0x2ddff bytes) 000326 1743454642.574826 fallocate 0x018b2c thru 0x022d3b (0xa210 bytes) 000327 1743454642.576958 punch from 0x00931d to 0x00c5ac (0x3290 bytes) 000328 1743454642.578411 mapread 0x00723e thru 0x00c5ac (0x536f bytes) 000329 1743454642.581445 mapread 0x00abe9 thru 0x00c5ac (0x19c4 bytes) 000330 1743454642.582131 read 0x0065f9 thru 0x007c80 (0x1688 bytes) 000331 1743454642.583663 MIRROR_EXTEND to 4 mirrors 000332 1743454642.691413 trunc from 0x00c5ad to 0x025fd7 (0x19a2b bytes) 000333 1743454642.706306 trunc from 0x025fd8 to 0x01e2d4 (0x7d03 bytes) 000334 1743454642.710577 read 0x0133a0 thru 0x01a941 (0x75a2 bytes) 000335 1743454642.712702 mapread 0x00ac9c thru 0x010bf7 (0x5f5c bytes) 000336 1743454642.718585 MIRROR_EXTEND to 5 mirrors 000337 1743454642.834162 mapread 0x013a56 thru 0x01955d (0x5b08 bytes) 000338 1743454642.843305 write 0x0313bc thru 0x03a9bc (0x9601 bytes) 000339 1743454642.855301 write 0x032cbd thru 0x03ef7e (0xc2c2 bytes) 000340 1743454642.856297 write 0x03ea9b thru 0x03ffff (0x1565 bytes) 000341 1743454642.857014 mapwrite 0x00e387 thru 0x00ed86 (0xa00 bytes) 000342 1743454642.865139 punch from 0x02811a to 0x0285a6 (0x48d bytes) 000343 1743454642.867723 MIRROR_RESYNC 5 mirrors 000344 1743454643.184991 mapwrite 0x020418 thru 0x02ecdf (0xe8c8 bytes) 000346 1743454643.220316 mapwrite 0x03dd84 thru 0x03ffff (0x227c bytes) 000347 1743454643.227312 mapread 0x02e188 thru 0x035983 (0x77fc bytes) 000348 1743454643.228984 mapread 0x026cfc thru 0x028493 (0x1798 bytes) 000349 1743454643.231113 MIRROR_EXTEND to 6 mirrors 000350 1743454643.369798 punch from 0x03a8dd to 0x03ffff (0x5723 bytes) 000351 1743454643.378755 mapwrite 0x00b1c4 thru 0x014385 (0x91c2 bytes) 000352 1743454643.389464 mapwrite 0x01cf2e thru 0x0286a8 (0xb77b bytes) 000353 1743454643.398547 read 0x024879 thru 0x02fd14 (0xb49c bytes) 000355 1743454643.400227 trunc from 0x040000 to 0x01bfbe (0x24041 bytes) 000356 1743454643.409634 punch from 0x019aa8 to 0x01bfbe (0x2517 bytes) 000357 1743454643.411088 trunc from 0x01bfbf to 0x0333a8 (0x173ea bytes) 000358 1743454643.414747 mapwrite 0x034093 thru 0x03e984 (0xa8f2 bytes) 000359 1743454643.429487 mapwrite 0x020fea thru 0x02b098 (0xa0af bytes) 000360 1743454643.438332 fallocate 0x03835b thru 0x03ffff (0x7ca5 bytes) 000361 1743454643.440344 fallocate 0x011910 thru 0x013df2 (0x24e3 bytes) 000362 1743454643.442016 write 0x02e5a8 thru 0x02fe11 (0x186a bytes) 000364 1743454643.443190 trunc from 0x03e985 to 0x02972a (0x1525a bytes) 000365 1743454643.448792 mapread 0x01f956 thru 0x027659 (0x7d04 bytes) 000366 1743454643.449593 fallocate 0x014a23 thru 0x01b898 (0x6e76 bytes) 000367 1743454643.451221 write 0x01c88e thru 0x02bb0a (0xf27d bytes) 000369 1743454643.452964 write 0x01beb5 thru 0x0234d7 (0x7623 bytes) 000371 1743454643.454272 mapwrite 0x006079 thru 0x008617 (0x259f bytes) 000372 1743454643.460870 mapwrite 0x006991 thru 0x00b478 (0x4ae8 bytes) 000374 1743454643.467697 read 0x016fa4 thru 0x022c53 (0xbcb0 bytes) 000375 1743454643.468173 mapwrite 0x01c7a5 thru 0x021710 (0x4f6c bytes) 000376 1743454643.473716 write 0x0371f3 thru 0x038059 (0xe67 bytes) 000377 1743454643.476195 MIRROR_EXTEND to 7 mirrors 000378 1743454643.599264 trunc from 0x03805a to 0x0219f3 (0x16666 bytes) 000379 1743454643.614813 write 0x037f9e thru 0x03a7a1 (0x2804 bytes) 000380 1743454643.615473 read 0x012411 thru 0x02021a (0xde0a bytes) 000382 1743454643.618445 trunc from 0x03a7a2 to 0x005fb2 (0x347ef bytes) 000383 1743454643.623873 punch from 0x0016b4 to 0x005fb2 (0x48ff bytes) 000384 1743454643.627728 read 0x002796 thru 0x005fb2 (0x381d bytes) 000385 1743454643.629919 read 0x005b16 thru 0x005fb2 (0x49d bytes) 000386 1743454643.630327 write 0x00a359 thru 0x00df98 (0x3c40 bytes) 000387 1743454643.631096 read 0x00c97b thru 0x00df98 (0x161e bytes) 000388 1743454643.631596 punch from 0x00b774 to 0x00df98 (0x2825 bytes) 000389 1743454643.634553 read 0x0094b4 thru 0x00df98 (0x4ae5 bytes) 000392 1743454643.637327 punch from 0x005957 to 0x00df98 (0x8642 bytes) 000393 1743454643.638841 trunc from 0x00df99 to 0x024d25 (0x16d8d bytes) 000394 1743454643.642849 fallocate 0x00ee3b thru 0x01bd76 (0xcf3c bytes) 000395 1743454643.644705 mapread 0x019233 thru 0x024d25 (0xbaf3 bytes) 000396 1743454643.648223 mapwrite 0x01edc0 thru 0x023cf2 (0x4f33 bytes) 000397 1743454643.653317 read 0x019cf2 thru 0x024d25 (0xb034 bytes) 000398 1743454643.653579 trunc from 0x024d26 to 0x03633d (0x11618 bytes) 000401 1743454643.657269 mapwrite 0x0130a2 thru 0x022b0c (0xfa6b bytes) 000402 1743454643.665342 mapwrite 0x01e6df thru 0x02d95f (0xf281 bytes) 000403 1743454643.674121 punch from 0x0116cd to 0x015ac7 (0x43fb bytes) 000404 1743454643.675603 fallocate 0x00ec1d thru 0x01524a (0x662e bytes) 000406 1743454643.677661 fallocate 0x01bc7c thru 0x022a1f (0x6da4 bytes) 000407 1743454643.679287 write 0x015397 thru 0x01f1b7 (0x9e21 bytes) 000408 1743454643.681256 trunc from 0x03633e to 0x01559d (0x20da0 bytes) 000410 1743454643.686425 punch from 0x013997 to 0x01559d (0x1c07 bytes) 000411 1743454643.690725 MIRROR_EXTEND to 8 mirrors 000412 1743454643.806741 trunc from 0x01559e to 0x012840 (0x2d5d bytes) 000413 1743454643.823298 mapwrite 0x0000fa thru 0x0045c6 (0x44cd bytes) 000415 1743454643.839606 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000416 1743454643.901213 MIRROR_RESYNC 7 mirrors 000417 1743454644.147641 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000418 1743454644.222382 mapread 0x0072c6 thru 0x00c2d6 (0x5011 bytes) 000419 1743454644.227417 fallocate 0x00029c thru 0x005a55 (0x57ba bytes) 000420 1743454644.236682 trunc from 0x012841 to 0x024b1a (0x122da bytes) 000421 1743454644.242584 mapwrite 0x0184c0 thru 0x027e5c (0xf99d bytes) 000422 1743454644.257061 trunc from 0x027e5d to 0x005eca (0x21f92 bytes) 000423 1743454644.261288 punch from 0x0037cc to 0x005eca (0x26ff bytes) 000424 1743454644.262939 write 0x01ed58 thru 0x0200db (0x1384 bytes) 000425 1743454644.263621 write 0x003cbb thru 0x00a8e3 (0x6c29 bytes) 000427 1743454644.268020 MIRROR_RESYNC 6 mirrors 000428 1743454644.451934 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000431 1743454644.517710 fallocate 0x02ea7d thru 0x038ddc (0xa360 bytes) 000432 1743454644.526325 fallocate 0x0368a1 thru 0x03ff81 (0x96e1 bytes) 000433 1743454644.527721 write 0x00b581 thru 0x019fef (0xea6f bytes) 000434 1743454644.531180 fallocate 0x0172d4 thru 0x01fdc3 (0x8af0 bytes) 000435 1743454644.532686 fallocate 0x02bf55 thru 0x0384c3 (0xc56f bytes) 000436 1743454644.533854 mapwrite 0x035c4e thru 0x03ffff (0xa3b2 bytes) 000438 1743454644.549836 mapread 0x00a75b thru 0x00ec84 (0x452a bytes) 000439 1743454644.551930 fallocate 0x039018 thru 0x03ffff (0x6fe8 bytes) 000440 1743454644.553137 punch from 0x006410 to 0x008fb0 (0x2ba1 bytes) 000441 1743454644.554354 read 0x0223ca thru 0x022601 (0x238 bytes) 000442 1743454644.557391 MIRROR_RESYNC 5 mirrors 000443 1743454644.765290 fallocate 0x024e82 thru 0x02e9e7 (0x9b66 bytes) 000445 1743454644.774877 mapread 0x03b60d thru 0x03ef7a (0x396e bytes) 000446 1743454644.779294 punch from 0x038216 to 0x03a6f6 (0x24e1 bytes) 000448 1743454644.782724 MIRROR_EXTEND to 6 mirrors 000449 1743454644.912240 fallocate 0x0009fe thru 0x009067 (0x866a bytes) 000450 1743454644.921261 mapwrite 0x02e141 thru 0x02ed80 (0xc40 bytes) 000452 1743454644.927705 trunc from 0x040000 to 0x03d972 (0x268d bytes) 000455 1743454644.938758 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000457 1743454644.992931 MIRROR_EXTEND to 6 mirrors 000459 1743454645.119436 mapread 0x00d180 thru 0x019ea8 (0xcd29 bytes) 000460 1743454645.129643 mapread 0x017e8b thru 0x01aa26 (0x2b9c bytes) 000461 1743454645.133269 mapread 0x008544 thru 0x014738 (0xc1f5 bytes) 000463 1743454645.139081 trunc from 0x03d973 to 0x011bf0 (0x2bd82 bytes) 000464 1743454645.151003 mapwrite 0x011889 thru 0x01bed7 (0xa64f bytes) 000465 1743454645.166266 mapread 0x014bd3 thru 0x01bed7 (0x7305 bytes) 000466 1743454645.166930 punch from 0x003ddd to 0x00baf6 (0x7d1a bytes) 000467 1743454645.169753 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000468 1743454645.225499 fallocate 0x036aec thru 0x03c320 (0x5835 bytes) 000469 1743454645.228408 read 0x0239bc thru 0x0326a0 (0xece5 bytes) 000471 1743454645.231683 read 0x02eb12 thru 0x03c320 (0xd80f bytes) 000472 1743454645.234728 trunc from 0x03c321 to 0x00a3a3 (0x31f7d bytes) 000473 1743454645.242244 read 0x001e48 thru 0x00a31a (0x84d3 bytes) 000474 1743454645.242647 punch from 0x0080f8 to 0x00a3a3 (0x22ac bytes) 000476 1743454645.245010 mapread 0x0002b9 thru 0x005776 (0x54be bytes) 000477 1743454645.245576 mapread 0x001280 thru 0x006b14 (0x5895 bytes) 000478 1743454645.246079 write 0x0290c7 thru 0x02be5b (0x2d95 bytes) 000479 1743454645.248679 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000480 1743454645.306614 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000481 1743454645.357605 mapwrite 0x0099c4 thru 0x00ff39 (0x6576 bytes) 000485 1743454645.371462 mapread 0x01a980 thru 0x0277c4 (0xce45 bytes) 000486 1743454645.377135 mapwrite 0x031d25 thru 0x03c016 (0xa2f2 bytes) 000487 1743454645.396768 mapwrite 0x00ae73 thru 0x01aa03 (0xfb91 bytes) 000488 1743454645.406389 fallocate 0x0020ba thru 0x00b476 (0x93bd bytes) 000491 1743454645.408357 mapwrite 0x036ed3 thru 0x03ffff (0x912d bytes) 000493 1743454645.421984 trunc from 0x040000 to 0x01de0e (0x221f1 bytes) 000494 1743454645.426699 punch from 0x0131f5 to 0x01b132 (0x7f3e bytes) 000495 1743454645.429006 trunc from 0x01de0f to 0x00f8d1 (0xe53d bytes) 000497 1743454645.434229 trunc from 0x00f8d2 to 0x03b085 (0x2b7b4 bytes) 000499 1743454645.439220 write 0x002410 thru 0x01145d (0xf04e bytes) 000504 1743454645.446720 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000505 1743454645.501850 mapread 0x013bd4 thru 0x018f86 (0x53b3 bytes) 000506 1743454645.507163 punch from 0x023f2b to 0x02800a (0x40e0 bytes) 000508 1743454645.508973 trunc from 0x03b086 to 0x01033f (0x2ad46 bytes) 000509 1743454645.517343 fallocate 0x00f693 thru 0x013d01 (0x466f bytes) 000510 1743454645.519159 write 0x03d327 thru 0x03ffff (0x2cd9 bytes) 000511 1743454645.519876 trunc from 0x040000 to 0x025423 (0x1abdc bytes) 000512 1743454645.525210 mapread 0x001d74 thru 0x00ab1b (0x8da8 bytes) 000513 1743454645.528266 mapread 0x019198 thru 0x0204ec (0x7355 bytes) 000514 1743454645.530815 mapwrite 0x019a16 thru 0x0212af (0x789a bytes) 000515 1743454645.536721 read 0x019cf8 thru 0x0233d2 (0x96db bytes) 000516 1743454645.536981 mapread 0x00ac78 thru 0x018df6 (0xe17f bytes) 000517 1743454645.539894 fallocate 0x00c1f6 thru 0x00c251 (0x5c bytes) 000518 1743454645.541402 mapread 0x01d03d thru 0x025423 (0x83e7 bytes) 000519 1743454645.541796 punch from 0x0199ca to 0x01acf3 (0x132a bytes) 000520 1743454645.543186 mapwrite 0x03ea66 thru 0x03eb47 (0xe2 bytes) 000521 1743454645.552797 punch from 0x02090a to 0x0284f2 (0x7be9 bytes) 000522 1743454645.554337 mapread 0x01f610 thru 0x020042 (0xa33 bytes) 000523 1743454645.557101 write 0x005494 thru 0x00f047 (0x9bb4 bytes) 000524 1743454645.557726 mapwrite 0x030271 thru 0x037a19 (0x77a9 bytes) 000526 1743454645.564557 write 0x00019a thru 0x00d67b (0xd4e2 bytes) 000527 1743454645.566377 mapread 0x035737 thru 0x03eb47 (0x9411 bytes) 000528 1743454645.566872 punch from 0x02c72e to 0x033bcb (0x749e bytes) 000529 1743454645.568706 trunc from 0x03eb48 to 0x01268a (0x2c4bd bytes) 000530 1743454645.572394 trunc from 0x01268b to 0x000617 (0x12073 bytes) 000531 1743454645.577529 MIRROR_EXTEND to 3 mirrors 000532 1743454645.663116 mapread 0x0004e7 thru 0x000617 (0x131 bytes) 000533 1743454645.664903 mapwrite 0x01a405 thru 0x0255b2 (0xb1ae bytes) 000534 1743454645.695918 trunc from 0x0255b3 to 0x009d26 (0x1b88c bytes) 000535 1743454645.700811 read 0x00058d thru 0x009d26 (0x979a bytes) 000536 1743454645.702910 read 0x005faa thru 0x009d26 (0x3d7d bytes) 000537 1743454645.704458 MIRROR_RESYNC 3 mirrors 000538 1743454645.804870 read 0x003dd8 thru 0x008953 (0x4b7c bytes) 000540 1743454645.807274 fallocate 0x035515 thru 0x03ffff (0xaaeb bytes) 000542 1743454645.816121 mapread 0x028c36 thru 0x02dc70 (0x503b bytes) 000543 1743454645.820689 punch from 0x01620b to 0x017810 (0x1606 bytes) 000544 1743454645.821713 trunc from 0x040000 to 0x0002d8 (0x3fd27 bytes) 000545 1743454645.832052 mapwrite 0x0325f3 thru 0x0350c4 (0x2ad2 bytes) 000546 1743454645.843647 write 0x022315 thru 0x0252e6 (0x2fd2 bytes) 000547 1743454645.846896 mapread 0x00538f thru 0x00e515 (0x9187 bytes) 000548 1743454645.851388 trunc from 0x0350c5 to 0x03ea15 (0x9951 bytes) 000551 1743454645.856292 fallocate 0x00b28b thru 0x00df77 (0x2ced bytes) 000553 1743454645.858667 punch from 0x01ecd3 to 0x027b0e (0x8e3c bytes) 000554 1743454645.861449 punch from 0x01dbe0 to 0x02cc31 (0xf052 bytes) 000555 1743454645.862656 mapwrite 0x03e84e thru 0x03ffff (0x17b2 bytes) 000556 1743454645.871667 punch from 0x03a2ed to 0x03ceb2 (0x2bc6 bytes) 000557 1743454645.873056 write 0x01961b thru 0x0231ce (0x9bb4 bytes) 000559 1743454645.879318 trunc from 0x040000 to 0x015350 (0x2acaf bytes) 000560 1743454645.883868 trunc from 0x015351 to 0x03a47b (0x2512b bytes) 000562 1743454645.888173 mapread 0x020aea thru 0x02aa5b (0x9f72 bytes) 000563 1743454645.892841 fallocate 0x019dc5 thru 0x01cc39 (0x2e75 bytes) 000566 1743454645.896501 write 0x017d5b thru 0x0227b6 (0xaa5c bytes) 000567 1743454645.899139 write 0x00fd17 thru 0x017a3b (0x7d25 bytes) 000568 1743454645.899807 fallocate 0x013ab5 thru 0x01eb7f (0xb0cb bytes) 000569 1743454645.901425 trunc from 0x03a47c to 0x014547 (0x25f34 bytes) 000570 1743454645.907557 write 0x02ef2c thru 0x03249a (0x356f bytes) 000571 1743454645.908289 mapread 0x02181a thru 0x0264b5 (0x4c9c bytes) 000572 1743454645.910595 punch from 0x02e5b3 to 0x03249a (0x3ee8 bytes) 000573 1743454645.914390 fallocate 0x00a235 thru 0x01871a (0xe4e6 bytes) 000575 1743454645.916280 write 0x03ba5a thru 0x03ffff (0x45a6 bytes) 000576 1743454645.916795 mapwrite 0x027de4 thru 0x035f89 (0xe1a6 bytes) 000578 1743454645.926400 write 0x033fd0 thru 0x03ffff (0xc030 bytes) 000579 1743454645.927600 punch from 0x0142a5 to 0x021ae1 (0xd83d bytes) 000580 1743454645.929158 mapread 0x016f1e thru 0x025e70 (0xef53 bytes) 000582 1743454645.933591 trunc from 0x040000 to 0x03d93f (0x26c0 bytes) 000583 1743454645.937347 write 0x033511 thru 0x03ffff (0xcaef bytes) 000584 1743454645.937989 mapread 0x016a78 thru 0x023d6f (0xd2f8 bytes) 000585 1743454645.938618 trunc from 0x040000 to 0x034d3d (0xb2c2 bytes) 000587 1743454645.942251 write 0x014110 thru 0x019032 (0x4f23 bytes) 000588 1743454645.942977 mapread 0x0090e8 thru 0x00c37e (0x3297 bytes) 000589 1743454645.943485 fallocate 0x00a20c thru 0x016956 (0xc74b bytes) 000590 1743454645.944846 mapread 0x02f0df thru 0x033afd (0x4a1f bytes) 000591 1743454645.945341 mapread 0x0221a3 thru 0x028343 (0x61a1 bytes) 000592 1743454645.947206 MIRROR_EXTEND to 4 mirrors 000593 1743454646.077549 trunc from 0x034d3e to 0x0085c6 (0x2c777 bytes) 000594 1743454646.088684 write 0x00e3da thru 0x013d72 (0x5999 bytes) 000595 1743454646.089358 trunc from 0x013d73 to 0x019a1b (0x5ca9 bytes) 000596 1743454646.093970 punch from 0x00497e to 0x014844 (0xfec7 bytes) 000598 1743454646.097389 fallocate 0x008a92 thru 0x00ed63 (0x62d2 bytes) 000600 1743454646.098744 punch from 0x00509a to 0x007b7e (0x2ae5 bytes) 000601 1743454646.100716 mapwrite 0x03d9a5 thru 0x03ffff (0x265b bytes) 000602 1743454646.111281 write 0x011db2 thru 0x012b0e (0xd5d bytes) 000603 1743454646.114116 read 0x03fd4d thru 0x03ffff (0x2b3 bytes) 000605 1743454646.114947 trunc from 0x040000 to 0x0389b6 (0x7649 bytes) 000606 1743454646.118697 mapwrite 0x00bb39 thru 0x018be9 (0xd0b1 bytes) 000607 1743454646.125565 MIRROR_RESYNC 4 mirrors 000608 1743454646.273793 trunc from 0x0389b7 to 0x01addb (0x1dbdb bytes) 000609 1743454646.286934 mapwrite 0x01af4d thru 0x0251a2 (0xa256 bytes) 000611 1743454646.304794 mapread 0x019c9b thru 0x0251a2 (0xb508 bytes) 000613 1743454646.305775 fallocate 0x01bb52 thru 0x01ffdc (0x448b bytes) 000614 1743454646.306993 mapread 0x00ee6d thru 0x00f676 (0x80a bytes) 000615 1743454646.310145 mapread 0x009890 thru 0x00a96d (0x10de bytes) 000616 1743454646.310659 trunc from 0x0251a3 to 0x03fb95 (0x1a9f3 bytes) 000617 1743454646.316275 MIRROR_RESYNC 4 mirrors 000619 1743454646.494548 read 0x02e464 thru 0x0316f2 (0x328f bytes) 000620 1743454646.497324 fallocate 0x02b31c thru 0x03a58c (0xf271 bytes) 000621 1743454646.507643 MIRROR_RESYNC 4 mirrors 000623 1743454646.696638 write 0x03ff51 thru 0x03ffff (0xaf bytes) 000624 1743454646.706172 mapwrite 0x0182fb thru 0x02386a (0xb570 bytes) 000625 1743454646.720679 fallocate 0x00ef5c thru 0x0147a4 (0x5849 bytes) 000627 1743454646.722706 fallocate 0x0166fb thru 0x023244 (0xcb4a bytes) 000628 1743454646.724120 trunc from 0x040000 to 0x03f915 (0x6ea bytes) 000629 1743454646.734862 trunc from 0x03f916 to 0x0007cd (0x3f148 bytes) 000630 1743454646.739459 write 0x028cbb thru 0x030b9d (0x7ee3 bytes) 000632 1743454646.740625 punch from 0x0239d3 to 0x02921b (0x5849 bytes) 000633 1743454646.745276 punch from 0x009186 to 0x014397 (0xb212 bytes) 000634 1743454646.746964 write 0x0177dd thru 0x024d1a (0xd53e bytes) 000635 1743454646.752665 mapread 0x02ecc3 thru 0x030b9d (0x1edb bytes) 000636 1743454646.753299 read 0x0115c7 thru 0x011e00 (0x83a bytes) 000637 1743454646.753768 trunc from 0x030b9e to 0x01ec39 (0x11f64 bytes) 000638 1743454646.758901 punch from 0x01684e to 0x01ec39 (0x83ec bytes) 000640 1743454646.762518 write 0x01792a thru 0x0229e5 (0xb0bc bytes) 000641 1743454646.765716 read 0x006274 thru 0x00e488 (0x8215 bytes) 000642 1743454646.768393 mapwrite 0x03f170 thru 0x03ffff (0xe90 bytes) 000643 1743454646.780467 trunc from 0x040000 to 0x02368b (0x1c974 bytes) 000644 1743454646.784335 mapread 0x010c5e thru 0x015ecc (0x526f bytes) 000645 1743454646.785066 trunc from 0x02368c to 0x032c26 (0xf59b bytes) 000648 1743454646.789250 punch from 0x029c5d to 0x03058f (0x6933 bytes) 000649 1743454646.790724 mapread 0x00eca1 thru 0x01d15d (0xe4bd bytes) 000651 1743454646.791846 trunc from 0x032c27 to 0x03b968 (0x8d42 bytes) 000653 1743454646.796377 mapread 0x009d74 thru 0x0195b0 (0xf83d bytes) 000654 1743454646.796906 mapread 0x003ef8 thru 0x008ecd (0x4fd6 bytes) 000655 1743454646.797289 trunc from 0x03b969 to 0x003bf6 (0x37d72 bytes) 000656 1743454646.800980 mapwrite 0x0387b1 thru 0x039edd (0x172d bytes) 000657 1743454646.811439 mapwrite 0x036777 thru 0x03ffff (0x9889 bytes) 000658 1743454646.830284 fallocate 0x0024a4 thru 0x006872 (0x43cf bytes) 000659 1743454646.832266 write 0x03bbeb thru 0x03bc76 (0x8c bytes) 000660 1743454646.834366 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000661 1743454646.886132 fallocate 0x02b28b thru 0x03a4b8 (0xf22e bytes) 000662 1743454646.887939 mapwrite 0x039617 thru 0x03a0c0 (0xaaa bytes) 000663 1743454646.893499 read 0x0008ae thru 0x003428 (0x2b7b bytes) 000665 1743454646.895804 trunc from 0x040000 to 0x02d20a (0x12df5 bytes) 000666 1743454646.902235 fallocate 0x000f39 thru 0x003115 (0x21dd bytes) 000667 1743454646.904571 mapwrite 0x0136b9 thru 0x020157 (0xca9f bytes) 000668 1743454646.919085 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000669 1743454646.968095 punch from 0x00fa52 to 0x014b40 (0x50ef bytes) 000670 1743454646.969655 mapwrite 0x003caa thru 0x00cd64 (0x90bb bytes) 000671 1743454646.978811 read 0x01c68c thru 0x02880a (0xc17f bytes) 000673 1743454646.984122 MIRROR_RESYNC 2 mirrors 000674 1743454647.101250 punch from 0x020d6c to 0x023c49 (0x2ede bytes) 000675 1743454647.109800 write 0x034954 thru 0x036b9a (0x2247 bytes) 000676 1743454647.110761 mapread 0x02bb4e thru 0x02ec32 (0x30e5 bytes) 000677 1743454647.115183 write 0x025d5e thru 0x034f05 (0xf1a8 bytes) 000678 1743454647.116440 mapwrite 0x0350ba thru 0x039116 (0x405d bytes) 000679 1743454647.129122 punch from 0x03264e to 0x036b63 (0x4516 bytes) 000683 1743454647.131840 mapread 0x001263 thru 0x00eb17 (0xd8b5 bytes) 000684 1743454647.135965 read 0x029755 thru 0x038457 (0xed03 bytes) 000685 1743454647.137665 mapread 0x00c553 thru 0x01879f (0xc24d bytes) 000686 1743454647.139887 punch from 0x01e26f to 0x02b1ff (0xcf91 bytes) 000687 1743454647.142653 punch from 0x019198 to 0x022276 (0x90df bytes) 000689 1743454647.146068 write 0x028bff thru 0x02a6c0 (0x1ac2 bytes) 000690 1743454647.148579 mapread 0x033180 thru 0x039116 (0x5f97 bytes) 000692 1743454647.149380 mapread 0x017767 thru 0x027518 (0xfdb2 bytes) 000695 1743454647.152308 fallocate 0x022192 thru 0x026ea8 (0x4d17 bytes) 000696 1743454647.154124 write 0x02333a thru 0x02c97c (0x9643 bytes) 000697 1743454647.155577 fallocate 0x017a15 thru 0x021ace (0xa0ba bytes) 000698 1743454647.159123 punch from 0x017c53 to 0x025f87 (0xe335 bytes) 000699 1743454647.163291 trunc from 0x039117 to 0x03802d (0x10e9 bytes) 000701 1743454647.166943 trunc from 0x03802e to 0x0180b3 (0x1ff7a bytes) 000702 1743454647.171605 fallocate 0x002fd1 thru 0x00c8b1 (0x98e1 bytes) 000703 1743454647.174250 punch from 0x013116 to 0x0180b3 (0x4f9e bytes) 000704 1743454647.175956 trunc from 0x0180b4 to 0x01eeb5 (0x6e02 bytes) 000705 1743454647.179816 write 0x030c37 thru 0x031c86 (0x1050 bytes) 000706 1743454647.180441 write 0x02693b thru 0x029f85 (0x364b bytes) 000707 1743454647.183855 mapwrite 0x018403 thru 0x023839 (0xb437 bytes) 000708 1743454647.196626 MIRROR_EXTEND to 3 mirrors 000709 1743454647.322491 fallocate 0x016136 thru 0x01a800 (0x46cb bytes) 000710 1743454647.332827 write 0x01be8d thru 0x01dfea (0x215e bytes) 000711 1743454647.336530 write 0x03faf4 thru 0x03ffff (0x50c bytes) 000712 1743454647.337083 punch from 0x009a45 to 0x00e4b5 (0x4a71 bytes) 000714 1743454647.338775 fallocate 0x02bfbc thru 0x02c26a (0x2af bytes) 000715 1743454647.340257 mapread 0x01737a thru 0x022b9e (0xb825 bytes) 000716 1743454647.345523 mapread 0x02f1c7 thru 0x02f407 (0x241 bytes) 000717 1743454647.348646 mapwrite 0x02bd7d thru 0x03863c (0xc8c0 bytes) 000719 1743454647.357166 fallocate 0x036196 thru 0x036e88 (0xcf3 bytes) 000720 1743454647.360172 MIRROR_EXTEND to 4 mirrors 000721 1743454647.507870 punch from 0x0295b3 to 0x0298a7 (0x2f5 bytes) 000722 1743454647.519504 write 0x0259a9 thru 0x03524b (0xf8a3 bytes) 000723 1743454647.523684 write 0x007133 thru 0x00e91d (0x77eb bytes) 000725 1743454647.526529 punch from 0x022d55 to 0x0271e4 (0x4490 bytes) 000726 1743454647.532322 mapwrite 0x00d07f thru 0x01239a (0x531c bytes) 000728 1743454647.541639 write 0x0158df thru 0x0216d3 (0xbdf5 bytes) 000729 1743454647.543384 fallocate 0x01b92d thru 0x025c3b (0xa30f bytes) 000730 1743454647.545150 write 0x010a31 thru 0x019dc0 (0x9390 bytes) 000731 1743454647.546191 punch from 0x025011 to 0x02c250 (0x7240 bytes) 000732 1743454647.547607 mapwrite 0x00d9fe thru 0x00ff32 (0x2535 bytes) 000733 1743454647.554988 fallocate 0x01bff7 thru 0x02befe (0xff08 bytes) 000734 1743454647.556879 punch from 0x01eedb to 0x0218c5 (0x29eb bytes) 000736 1743454647.559160 trunc from 0x040000 to 0x026a91 (0x1956e bytes) 000737 1743454647.569680 fallocate 0x0015b8 thru 0x00cfaa (0xb9f3 bytes) 000740 1743454647.572205 mapwrite 0x033fcf thru 0x03ce5b (0x8e8d bytes) 000743 1743454647.588149 fallocate 0x005a23 thru 0x00cb35 (0x7113 bytes) 000744 1743454647.589944 punch from 0x004c68 to 0x013b20 (0xeeb9 bytes) 000745 1743454647.593245 write 0x00cfae thru 0x016f2d (0x9f80 bytes) 000746 1743454647.596982 punch from 0x025aee to 0x03053b (0xaa4e bytes) 000747 1743454647.598581 mapread 0x00be0e thru 0x00f077 (0x326a bytes) 000748 1743454647.600366 trunc from 0x03ce5c to 0x0328d3 (0xa588 bytes) 000749 1743454647.604966 punch from 0x02552e to 0x029fc7 (0x4a9a bytes) 000750 1743454647.606694 fallocate 0x0366cf thru 0x03edaa (0x86dc bytes) 000753 1743454647.610670 mapread 0x02001a thru 0x027d46 (0x7d2d bytes) 000754 1743454647.614838 fallocate 0x013beb thru 0x01f6cd (0xbae3 bytes) 000755 1743454647.616531 mapwrite 0x0246e7 thru 0x02e1d1 (0x9aeb bytes) 000756 1743454647.626408 punch from 0x01138f to 0x01ee17 (0xda89 bytes) 000757 1743454647.627989 punch from 0x0018a5 to 0x0035ea (0x1d46 bytes) 000758 1743454647.629614 trunc from 0x0328d4 to 0x034d90 (0x24bd bytes) 000759 1743454647.635109 punch from 0x0279c0 to 0x0289c9 (0x100a bytes) 000761 1743454647.636982 read 0x007326 thru 0x010fb7 (0x9c92 bytes) 000762 1743454647.639866 MIRROR_RESYNC 4 mirrors 000763 1743454647.911074 fallocate 0x00d82d thru 0x018706 (0xaeda bytes) 000764 1743454647.920708 fallocate 0x03b42d thru 0x03d2a6 (0x1e7a bytes) 000765 1743454647.922175 fallocate 0x0255b4 thru 0x02cad2 (0x751f bytes) 000767 1743454647.924449 read 0x00507b thru 0x011049 (0xbfcf bytes) 000768 1743454647.927407 fallocate 0x012c3a thru 0x01eae3 (0xbeaa bytes) 000770 1743454647.929386 fallocate 0x0288bd thru 0x02bdfe (0x3542 bytes) 000773 1743454647.931320 trunc from 0x034d91 to 0x00bde9 (0x28fa7 bytes) 000774 1743454647.944926 read 0x00274e thru 0x00a59c (0x7e4f bytes) 000775 1743454647.947948 fallocate 0x029e10 thru 0x0381cc (0xe3bd bytes) 000776 1743454647.949745 trunc from 0x00bdea to 0x02c72f (0x20946 bytes) 000777 1743454647.953705 mapread 0x01324a thru 0x01fed5 (0xcc8c bytes) 000778 1743454647.958203 mapread 0x001089 thru 0x01011d (0xf095 bytes) 000779 1743454647.959030 fallocate 0x033ec6 thru 0x03ffff (0xc13a bytes) 000780 1743454647.961090 fallocate 0x015347 thru 0x019936 (0x45f0 bytes) 000781 1743454647.962589 trunc from 0x02c730 to 0x0077b3 (0x24f7c bytes) 000782 1743454647.967664 mapwrite 0x01803c thru 0x0240e0 (0xc0a5 bytes) 000783 1743454647.995489 mapread 0x01f8c4 thru 0x0240e0 (0x481d bytes) 000786 1743454647.996662 punch from 0x01ab26 to 0x01c5b2 (0x1a8d bytes) 000787 1743454647.998580 punch from 0x002620 to 0x00f92c (0xd30d bytes) 000788 1743454647.999853 read 0x0137bf thru 0x021c56 (0xe498 bytes) 000789 1743454648.002499 read 0x004bb3 thru 0x009e5a (0x52a8 bytes) 000790 1743454648.003949 write 0x007d20 thru 0x008266 (0x547 bytes) 000791 1743454648.004589 punch from 0x008960 to 0x011855 (0x8ef6 bytes) 000792 1743454648.008668 fallocate 0x031b73 thru 0x039660 (0x7aee bytes) 000793 1743454648.010563 read 0x002265 thru 0x00b2f3 (0x908f bytes) 000794 1743454648.013022 read 0x0038aa thru 0x00f31c (0xba73 bytes) 000795 1743454648.013407 trunc from 0x0240e1 to 0x009084 (0x1b05c bytes) 000796 1743454648.020530 MIRROR_RESYNC 4 mirrors 000797 1743454648.181192 write 0x00a96f thru 0x016fe5 (0xc677 bytes) 000798 1743454648.188946 mapwrite 0x006855 thru 0x0095da (0x2d86 bytes) 000799 1743454648.199338 read 0x011a15 thru 0x016fe5 (0x55d1 bytes) 000800 1743454648.199795 write 0x004fb0 thru 0x00de01 (0x8e52 bytes) 000801 1743454648.201739 MIRROR_RESYNC 4 mirrors 000802 1743454648.331676 mapread 0x00a3cf thru 0x00f623 (0x5255 bytes) 000803 1743454648.336792 punch from 0x00d3af to 0x00dd7f (0x9d1 bytes) 000804 1743454648.345615 read 0x00aae6 thru 0x016689 (0xbba4 bytes) 000805 1743454648.348513 mapwrite 0x01dfd6 thru 0x024b72 (0x6b9d bytes) 000806 1743454648.368295 fallocate 0x02ac30 thru 0x03459b (0x996c bytes) 000808 1743454648.370424 punch from 0x021efe to 0x023ce5 (0x1de8 bytes) 000809 1743454648.372417 mapread 0x023cdb thru 0x024b72 (0xe98 bytes) 000811 1743454648.374822 read 0x016152 thru 0x01908b (0x2f3a bytes) 000812 1743454648.375178 write 0x0235ed thru 0x025a72 (0x2486 bytes) 000813 1743454648.375898 mapwrite 0x014fb2 thru 0x01806b (0x30ba bytes) 000814 1743454648.381323 read 0x004178 thru 0x00bece (0x7d57 bytes) 000815 1743454648.384195 read 0x022336 thru 0x025a72 (0x373d bytes) 000816 1743454648.384817 mapread 0x0162e9 thru 0x01ec15 (0x892d bytes) 000817 1743454648.385664 fallocate 0x0278be thru 0x028173 (0x8b6 bytes) 000820 1743454648.388610 fallocate 0x00ab29 thru 0x01040a (0x58e2 bytes) 000822 1743454648.392384 MIRROR_RESYNC 4 mirrors 000825 1743454648.580425 read 0x0187fa thru 0x01972f (0xf36 bytes) 000826 1743454648.582356 mapread 0x020447 thru 0x02593d (0x54f7 bytes) 000827 1743454648.586092 mapwrite 0x000c64 thru 0x0012c3 (0x660 bytes) 000829 1743454648.617607 write 0x00a145 thru 0x01655b (0xc417 bytes) 000831 1743454648.620277 mapread 0x0000b4 thru 0x00b944 (0xb891 bytes) 000832 1743454648.622635 MIRROR_EXTEND to 5 mirrors 000833 1743454648.748093 punch from 0x019886 to 0x0248fb (0xb076 bytes) 000834 1743454648.759015 write 0x003d07 thru 0x0095d3 (0x58cd bytes) 000835 1743454648.763178 mapwrite 0x018738 thru 0x01fe83 (0x774c bytes) 000836 1743454648.775433 mapwrite 0x00c1e3 thru 0x01a141 (0xdf5f bytes) 000837 1743454648.784929 trunc from 0x025a73 to 0x036b89 (0x11117 bytes) 000839 1743454648.796682 write 0x00d599 thru 0x00f1c9 (0x1c31 bytes) 000841 1743454648.797651 mapread 0x02ad22 thru 0x03136a (0x6649 bytes) 000842 1743454648.800814 write 0x02e86a thru 0x03ce87 (0xe61e bytes) 000843 1743454648.801837 read 0x000b17 thru 0x00b2ed (0xa7d7 bytes) 000844 1743454648.803442 trunc from 0x03ce88 to 0x016c4b (0x2623c bytes) 000846 1743454648.809877 mapwrite 0x014f60 thru 0x01c325 (0x73c6 bytes) 000847 1743454648.826903 fallocate 0x03f400 thru 0x03ffff (0xc00 bytes) 000848 1743454648.828796 punch from 0x00f47d to 0x0122d5 (0x2e59 bytes) 000849 1743454648.830262 trunc from 0x01c326 to 0x0304b6 (0x14191 bytes) 000850 1743454648.833805 write 0x027cdf thru 0x032ede (0xb200 bytes) 000851 1743454648.836702 mapwrite 0x029b4a thru 0x02be3c (0x22f3 bytes) 000852 1743454648.843711 mapwrite 0x013504 thru 0x01fb6a (0xc667 bytes) 000854 1743454648.852014 read 0x02fef0 thru 0x032ede (0x2fef bytes) 000855 1743454648.852609 trunc from 0x032edf to 0x00b681 (0x2785d bytes) 000857 1743454648.857913 punch from 0x000583 to 0x000b1e (0x59c bytes) 000858 1743454648.859141 read 0x00951a thru 0x00b681 (0x2168 bytes) 000859 1743454648.859617 punch from 0x002574 to 0x00b681 (0x910e bytes) 000862 1743454648.861451 mapread 0x0007a9 thru 0x00203f (0x1897 bytes) 000863 1743454648.863684 write 0x017246 thru 0x024951 (0xd70c bytes) 000864 1743454648.866092 MIRROR_RESYNC 5 mirrors 000865 1743454649.055661 read 0x023123 thru 0x024951 (0x182f bytes) 000867 1743454649.058263 MIRROR_RESYNC 5 mirrors 000868 1743454649.101175 mapread 0x020e47 thru 0x022692 (0x184c bytes) 000869 1743454649.105184 write 0x035773 thru 0x03a6b6 (0x4f44 bytes) 000872 1743454649.113534 trunc from 0x03a6b7 to 0x0302f9 (0xa3bd bytes) 000873 1743454649.117952 write 0x0002f2 thru 0x001b1c (0x182b bytes) 000875 1743454649.121574 mapwrite 0x0197d3 thru 0x02825a (0xea88 bytes) 000876 1743454649.133284 trunc from 0x0302fa to 0x016fcf (0x1932a bytes) 000878 1743454649.137775 mapread 0x01364a thru 0x016fcf (0x3986 bytes) 000879 1743454649.139761 MIRROR_EXTEND to 6 mirrors 000880 1743454649.256243 mapwrite 0x0131b4 thru 0x01e642 (0xb48f bytes) 000881 1743454649.286068 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000882 1743454649.336105 fallocate 0x021e3a thru 0x02e219 (0xc3e0 bytes) 000883 1743454649.337635 write 0x03793e thru 0x03ffff (0x86c2 bytes) 000885 1743454649.340194 fallocate 0x001d04 thru 0x00457d (0x287a bytes) 000886 1743454649.341489 fallocate 0x032bef thru 0x03fef1 (0xd303 bytes) 000887 1743454649.342890 trunc from 0x040000 to 0x001414 (0x3ebeb bytes) 000888 1743454649.347428 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000889 1743454649.399405 mapwrite 0x00faac thru 0x01899f (0x8ef4 bytes) 000890 1743454649.418294 MIRROR_RESYNC 4 mirrors 000891 1743454649.548320 trunc from 0x0189a0 to 0x023bf1 (0xb252 bytes) 000892 1743454649.563185 read 0x011eeb thru 0x01c655 (0xa76b bytes) 000893 1743454649.565291 read 0x022b8f thru 0x023bf1 (0x1063 bytes) 000895 1743454649.569015 fallocate 0x0310a5 thru 0x036459 (0x53b5 bytes) 000897 1743454649.571132 fallocate 0x02660e thru 0x02e060 (0x7a53 bytes) 000898 1743454649.573113 write 0x034ef1 thru 0x037a6f (0x2b7f bytes) 000899 1743454649.577632 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000900 1743454649.634843 fallocate 0x034486 thru 0x03a798 (0x6313 bytes) 000901 1743454649.636463 mapread 0x01d1dc thru 0x026348 (0x916d bytes) 000902 1743454649.641386 read 0x01a0d8 thru 0x0234d1 (0x93fa bytes) 000903 1743454649.642949 fallocate 0x02e9d4 thru 0x02f2cd (0x8fa bytes) 000904 1743454649.644649 read 0x00e9ce thru 0x016f94 (0x85c7 bytes) 000906 1743454649.647676 punch from 0x03635d to 0x03a798 (0x443c bytes) 000908 1743454649.649714 write 0x024aba thru 0x024b6f (0xb6 bytes) 000909 1743454649.651574 fallocate 0x03fed4 thru 0x03ffff (0x12c bytes) 000910 1743454649.654437 MIRROR_EXTEND to 4 mirrors 000911 1743454649.784451 fallocate 0x029ed5 thru 0x031229 (0x7355 bytes) 000912 1743454649.791967 punch from 0x01c53c to 0x020c9b (0x4760 bytes) 000913 1743454649.793394 mapread 0x02fe65 thru 0x03af9c (0xb138 bytes) 000914 1743454649.801250 fallocate 0x0396ae thru 0x03ffff (0x6952 bytes) 000915 1743454649.803053 mapread 0x006466 thru 0x012e42 (0xc9dd bytes) 000916 1743454649.808425 trunc from 0x040000 to 0x024355 (0x1bcaa bytes) 000917 1743454649.823994 trunc from 0x024356 to 0x02fab7 (0xb762 bytes) 000918 1743454649.828645 trunc from 0x02fab8 to 0x03fc06 (0x1014f bytes) 000919 1743454649.835484 mapwrite 0x02f135 thru 0x03c72a (0xd5f6 bytes) 000920 1743454649.851238 trunc from 0x03fc07 to 0x011016 (0x2ebf0 bytes) 000922 1743454649.856774 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000923 1743454649.908063 write 0x03295c thru 0x03ffff (0xd6a4 bytes) 000925 1743454649.910734 write 0x003399 thru 0x00ae0f (0x7a77 bytes) 000927 1743454649.914201 trunc from 0x040000 to 0x038c55 (0x73aa bytes) 000928 1743454649.919265 MIRROR_RESYNC 3 mirrors 000930 1743454650.084602 trunc from 0x038c56 to 0x03be02 (0x31ad bytes) 000931 1743454650.094023 read 0x00ff66 thru 0x016f81 (0x701c bytes) 000932 1743454650.096521 mapread 0x035cb0 thru 0x03be02 (0x6153 bytes) 000933 1743454650.100930 fallocate 0x034261 thru 0x03ffff (0xbd9f bytes) 000934 1743454650.102652 mapread 0x00703d thru 0x00c431 (0x53f5 bytes) 000935 1743454650.105681 fallocate 0x002e58 thru 0x011ade (0xec87 bytes) 000936 1743454650.107375 write 0x034790 thru 0x036121 (0x1992 bytes) 000937 1743454650.111116 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000938 1743454650.163502 trunc from 0x040000 to 0x022abd (0x1d542 bytes) 000939 1743454650.169613 MIRROR_EXTEND to 3 mirrors 000941 1743454650.279987 trunc from 0x022abe to 0x02b7a3 (0x8ce6 bytes) 000942 1743454650.293991 MIRROR_EXTEND to 4 mirrors 000944 1743454650.396108 mapwrite 0x00ae95 thru 0x01a833 (0xf99f bytes) 000945 1743454650.436659 fallocate 0x01153f thru 0x01f135 (0xdbf7 bytes) 000946 1743454650.438559 mapwrite 0x030aac thru 0x03ed33 (0xe288 bytes) 000947 1743454650.463115 mapwrite 0x02deb4 thru 0x03d387 (0xf4d4 bytes) 000948 1743454650.471292 trunc from 0x03ed34 to 0x03f0f6 (0x3c3 bytes) 000949 1743454650.475751 read 0x01c10a thru 0x02184b (0x5742 bytes) 000951 1743454650.478176 mapread 0x017402 thru 0x01eeb9 (0x7ab8 bytes) 000952 1743454650.478745 mapread 0x0032c5 thru 0x008b58 (0x5894 bytes) 000953 1743454650.480548 trunc from 0x03f0f7 to 0x030d0c (0xe3ea bytes) 000954 1743454650.484577 read 0x008ebc thru 0x0178e9 (0xea2e bytes) 000956 1743454650.485429 mapwrite 0x014003 thru 0x019a6d (0x5a6b bytes) 000958 1743454650.493325 write 0x00e1fa thru 0x01b272 (0xd079 bytes) 000959 1743454650.494378 fallocate 0x01a7f5 thru 0x029fe6 (0xf7f2 bytes) 000960 1743454650.495689 punch from 0x02164c to 0x022a0e (0x13c3 bytes) 000961 1743454650.496838 write 0x0155cd thru 0x016413 (0xe47 bytes) 000962 1743454650.497271 trunc from 0x030d0d to 0x0210de (0xfc2e bytes) 000963 1743454650.501681 read 0x014038 thru 0x0153ec (0x13b5 bytes) 000964 1743454650.502143 fallocate 0x0025db thru 0x00ef30 (0xc956 bytes) 000965 1743454650.503498 write 0x031765 thru 0x03334f (0x1beb bytes) 000966 1743454650.505153 MIRROR_EXTEND to 5 mirrors 000968 1743454650.615888 punch from 0x022a9d to 0x02d831 (0xad95 bytes) 000969 1743454650.625501 mapwrite 0x00b567 thru 0x00bffb (0xa95 bytes) 000971 1743454650.631231 fallocate 0x024f13 thru 0x030a6b (0xbb59 bytes) 000972 1743454650.634691 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000973 1743454650.683057 trunc from 0x033350 to 0x003d43 (0x2f60c bytes) 000974 1743454650.690626 mapread 0x001733 thru 0x003d43 (0x2611 bytes) 000976 1743454650.694278 punch from 0x002bcc to 0x003d43 (0x1178 bytes) 000977 1743454650.695356 fallocate 0x03a38c thru 0x03ffff (0x5c74 bytes) 000978 1743454650.696622 fallocate 0x00dbdd thru 0x01c75f (0xeb83 bytes) 000979 1743454650.698237 fallocate 0x00f6fe thru 0x01d8fe (0xe201 bytes) 000980 1743454650.699420 mapwrite 0x034c85 thru 0x03ffff (0xb37b bytes) 000982 1743454650.712899 fallocate 0x00940a thru 0x010a2c (0x7623 bytes) 000983 1743454650.714359 fallocate 0x02dc76 thru 0x037b20 (0x9eab bytes) 000984 1743454650.715753 punch from 0x029e01 to 0x02aaf2 (0xcf2 bytes) 000985 1743454650.717235 fallocate 0x0321ba thru 0x03ffff (0xde46 bytes) 000986 1743454650.718760 write 0x003d31 thru 0x004a5f (0xd2f bytes) 000987 1743454650.722047 read 0x033d09 thru 0x03ffff (0xc2f7 bytes) 000988 1743454650.722479 trunc from 0x040000 to 0x0299db (0x16624 bytes) 000989 1743454650.726506 write 0x014640 thru 0x0148df (0x2a0 bytes) 000991 1743454650.729196 mapread 0x00b7c8 thru 0x017a3d (0xc276 bytes) 000992 1743454650.729626 write 0x020a2c thru 0x0234ca (0x2a9f bytes) 000993 1743454650.732240 fallocate 0x020cab thru 0x025882 (0x4bd8 bytes) 000994 1743454650.733779 fallocate 0x005aa2 thru 0x0127e7 (0xcd46 bytes) 000995 1743454650.735014 fallocate 0x03ca70 thru 0x03ffff (0x3590 bytes) 000996 1743454650.736974 fallocate 0x00ac47 thru 0x01060b (0x59c5 bytes) 000997 1743454650.739081 fallocate 0x01b675 thru 0x0235d4 (0x7f60 bytes) 000999 1743454650.740743 fallocate 0x019d28 thru 0x01d565 (0x383e bytes) 001000 1743454650.741999 mapread 0x023fc3 thru 0x024db9 (0xdf7 bytes) All operations completed A-OK! PASS 70a (13s) == sanity-flr test 71: check mirror extend parent fid ==== 16:57:31 (1743454651) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000404:0x52:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (3s) == sanity-flr test 200a: stress test ===================== 16:57:34 (1743454654) Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre2 Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5619104 .. Extending file size to 7054528 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5819968 .. Extending file size to 7790368 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8455872 .. Extending file size to 6194016 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7436320 .. done Extending file size to 8259936 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8916320 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8404576 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8452768 .. failed Extending file size to 9098368 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9055264 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9112000 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9393536 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 3804 3805 3807 3808 3811 192.168.201.134@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg134-client.virtnet /mnt/lustre2 (opts:) 192.168.201.134@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg134-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (63s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 16:58:37 (1743454717) Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre2 Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7427936 .. Extending file size to 7143904 .. done Extending file size to 8275680 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8923168 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8683840 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9094112 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9094144 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9405504 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 8143 8144 8146 8148 8150 192.168.201.134@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg134-client.virtnet /mnt/lustre2 (opts:) 192.168.201.134@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg134-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 16:59:41 (1743454781) Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4052 1283636 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2040 1285648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 14964 3592056 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 23184 3583836 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38148 7175892 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4052 1283636 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2040 1285648 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 14964 3592056 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 23184 3583836 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 38148 7175892 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0394421 s, 53.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0614459 s, 34.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.134@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg134-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 16:59:45 (1743454785) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0356792 s, 58.8 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 16:59:47 (1743454787) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa8b:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xec8:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xec8:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xec9:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec8:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 16:59:49 (1743454789) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0319724 s, 65.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x46:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x46:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x47:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 16:59:51 (1743454791) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0257313 s, 81.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xeca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xecb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xeca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xecb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xeca:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 16:59:53 (1743454793) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0274999 s, 76.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xecc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xecb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xecc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xecb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xecc:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 17:00:16 (1743454816) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg134-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0214073 s, 49.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xecd:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xece:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xecd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xecd:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xece:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xecd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xece:0x0] } PASS 204d (32s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 17:00:48 (1743454848) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg134-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xecf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xed0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0324592 s, 64.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xecf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xed0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xecf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xecf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xed0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xecf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (34s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 17:01:23 (1743454883) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg134-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg134-server: Pool lustre.test_204f created Adding targets to pool oleg134-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg134-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Updated after 2s: want 'lustre-OST0000_UUID lustre-OST0001_UUID ' got 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0197533 s, 106 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xed1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xed0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xed1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xed0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg134-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg134-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg134-server: Pool lustre.test_204f destroyed PASS 204f (48s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 17:02:11 (1743454931) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 17:02:13 (1743454933) PASS 205b (1s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 17:02:14 (1743454934) oleg134-server: Pool lustre.test_206 created Adding targets to pool oleg134-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg134-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg134-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg134-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg134-server: Pool lustre.test_206 destroyed PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 17:02:27 (1743454947) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0192666 s, 54.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00963481 s, 109 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x5fdea395 Mirror 3: 0x5fdea395 PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 17:02:29 (1743454949) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 17:02:30 (1743454950) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (0s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 17:02:30 (1743454950) Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.00916041 s, 0.4 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x14d0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xf90000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xf90000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xf90000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1580000 Read this (4 bytes): BBBB 192.168.201.134@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg134-client.virtnet /mnt/lustre2 (opts:) PASS 209a (14s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 17:02:44 (1743454964) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00251284 s, 3.3 MB/s the very first read read with warm cache PASS 209b (1s) == sanity-flr test 210a: handle broken mirrored lovea ==== 17:02:45 (1743454965) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122371 s, 85.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xaae lmm_fid: [0x200000405:0xaae:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xaae lmm_fid: [0x200000405:0xaae:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xedb:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339613870 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-03-31 17:02:46.000000000 -0400 Modify: 2025-03-31 17:02:46.000000000 -0400 Change: 2025-03-31 17:02:46.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 17:02:47 (1743454967) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00910333 s, 115 MB/s oleg134-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg134-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg134-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg134-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg134-server: O/280000401/d27/edb: File not found by ext2_lookup PASS 210b (13s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 17:03:00 (1743454980) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.022619 s, 1.8 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00489132 s, 837 kB/s size after second write -rw-r--r-- 1 root root 4096 Mar 31 17:03 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xab3:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Mar 31 17:03 /mnt/lustre/f211.sanity-flr PASS 211 (1s) == sanity-flr test complete, duration 692 sec ============ 17:03:01 (1743454981) === sanity-flr: start cleanup 17:03:02 (1743454982) === === sanity-flr: finish cleanup 17:03:03 (1743454983) ===